Title: Horizon: A Hyper-Edge Observability Engine for Live Streaming Networks
Authors: Ziqian Liu, Daqian Ding (The University of Hong Kong); Rui Han, Shixian Guo, Zhendong Xie, Aifang Xu, Changqian Wang, Kefei Liu (ByteDance); Jialin Li (National University of Singapore); Yunming Xiao (The Chinese University of Hong Kong, Shenzhen); Heming Cui, Yiming Qiu (The University of Hong Kong)
Introduction
Live streaming networks use edge nodes distributed across regions and ISPs to deliver real-time media from broadcasters to large audiences. Server-side probes usually remain within the backbone and edge infrastructure, making it difficult to observe user-facing paths or validate streaming-protocol semantics. Client SDKs can collect playback metrics, but they depend on real user sessions and software rollout and cannot perform continuous monitoring or preflight testing on demand. Horizon uses provider-managed hyper-edge devices, such as set-top boxes and home gateways, to run controllable probes near users. The same device fleet supports both runtime monitoring and scenario-driven preflight testing. The system has been deployed in production for more than three years across a large global live streaming network.
Key idea and contribution:
Horizon uses a controller-agent architecture. A centralized controller in the datacenter maintains device state and generates tasks, while agents on hyper-edge devices execute probes in sandboxes and report their results. The paper divides the system into two capabilities built on this shared infrastructure (Figure 5).
-
Runtime monitoring: The controller continuously estimates each agent’s reliability using reference targets, filters out unstable devices, and assigns the same task to multiple reliable agents for result aggregation. Probing tasks enter a global queue according to region, ISP, and protocol, and online agents pull compatible tasks according to their capabilities. If a device’s state changes, unfinished tasks return to the queue for reassignment. On each device, the agent schedules execution according to a probe-conflict matrix and CPU, memory, and bandwidth budgets, preventing streaming, HTTP, and ICMP probes from interfering with one another.
-
Preflight testing: Horizon uses a unified domain-specific language to describe network environments and user behavior. The environment specification defines topology, scale, link conditions, and fault injection, and the controller selects the required hyper-edge devices and service-side nodes. The behavior specification uses a signal-driven state machine to describe stream fetching, waiting, switching, and quality checks, so subsequent actions are triggered by playback state and observed results. Horizon compiles both parts into a coordinated cross-device execution plan that can repeatedly validate a specific scenario before a release or major live event.
Evaluation
In 2025, Horizon’s runtime monitoring detected more than 1,700 LiveNet incidents with a mean time to discovery of 3.4 minutes, while weekly preflight tests found more than 300 additional bugs. Manual validation showed that runtime monitoring covered more than 98% of eventually confirmed major incidents and achieved 95% alert precision.
The system manages more than 100,000 hyper-edge agents across ICMP, HTTP, FLV, and HLS probes, while the controller uses an average of only 14% CPU and 7% memory. Reliability filtering increases the share of agents with a perfect score from 60.4% to 86.7% and reduces the share below 80% from 26.3% to 1.3%, substantially limiting the effect of unstable devices on monitoring results (Figure 11).
For a mixed workload containing four TS tasks and 40 lightweight probes, serial execution requires 109 seconds, while fully parallel execution introduces resource contention that distorts latency measurements. Horizon executes only non-conflicting tasks concurrently and completes the workload in 21.45 seconds, providing an approximately fivefold speedup while preserving measurement fidelity (Figure 13).
In a pre-event stress test, edge nodes remain stable before the load reaches its peak, but an edge-capacity limit appears at minute 48. Pull success falls to 70%, and playback bitrate drops from the 3.5 Mbps target to 2.4 Mbps. Horizon exposes the insufficient capacity before the actual event, allowing operators to expand resources in advance (Figure 14). These results matter because they show that a volatile fleet of near-user devices can provide accurate continuous monitoring and controlled preflight validation at production scale.
Q&A
Q1: Are you able to place these devices in every user’s home, or only in a fraction of homes? How do you know whether the deployment is sufficient?
A1: Not in every home. These devices are managed and provided by third-party providers, which have agreements with users. We use the probing function in a sandbox.
Q2: Have you correlated the locations of these hyper-edge devices with the locations of your users? In other words, how do you know that these devices provide good coverage?
A2: The coverage generally matches our business distribution. Large cities and regions with more users tend to have more hyper-edge devices, while regions with fewer users have fewer devices. It covers about 90% of our key business regions.
Q3: Do you see coverage gaps? Are there cases where the coverage is not good enough, but you have to accept that limitation?
A3: We have other monitoring engines that complement Horizon, such as server-side monitoring and SDK telemetry.
Personal thoughts
Horizon’s treatment of observation-point placement is more important than simply increasing the number of probes. By incorporating managed household-side devices into the observability plane, the system can inspect streaming protocols and playback quality along real user paths while retaining active scheduling and fault-injection capabilities. Reliability-aware selection, pull-based allocation, and conflict-aware scheduling make these otherwise difficult-to-control resources practical for long-running production monitoring.
Coverage depends on partnerships with device providers and the regional density of available devices. Horizon also still requires operators to correlate detected anomalies with system telemetry, flow logs, and configuration changes. The infrastructure therefore has clear boundaries in geographic coverage and automated diagnosis.