Paper: Horizon: A Hyper-Edge Observability Engine for Live Streaming Networks
Authors: Ziqian Liu, Daqian Ding, Rui Han, Shixian Guo, Zhendong Xie, Aifang Xu, Changqian Wang, Kefei Liu, Jialin Li, Yunming Xiao, Heming Cui, Yiming Qiu
Presenter: Yixiu Zhou, Xiamen University
Guest of Honor: Ziqian Liu, Hong Kong University
Q:When an agent cannot reach several reference targets, what signals are most useful for distinguishing a local device problem from a wider access network problem?
A:The authors assume that the servers in the live-streaming network are relatively stable. They also have a server-side monitoring system, including a ping-mesh mechanism, which can detect server-side problems such as CPU or OS failures. In most cases, the server side is healthy. Therefore, if an agent repeatedly fails to reach multiple reference targets, Horizon considers the agent unreliable and temporarily excludes it from monitoring tasks. After half a day or one day, the system checks the agent again to determine whether it has recovered.
Q:How can Horizon coexist with traditional network monitoring systems?
A:The different monitoring approaches serve different purposes and have evolved over time. Initially, live-streaming networks relied on server-side monitoring and ping-mesh mechanisms. However, these approaches cannot cover the entire path from the edge to the end user. As the system scaled, client-side SDKs were introduced to collect playback information, but the amount of telemetry became too large to process efficiently, and these measurements were not sufficiently controllable for pre-release testing. Horizon introduces controllable, user-proximate agents that can conduct experiments and support both runtime monitoring and pre-release testing. These approaches are complementary rather than mutually exclusive.
Q:What is the network overhead of Horizon, particularly when conducting large-scale stress tests before major streaming events?
A:Lightweight probes, such as ICMP ping and HTTP requests, introduce relatively little overhead. The largest overhead comes from stress testing before major streaming events, such as the World Cup. To minimize the impact on users, the authors schedule these tests during midnight or other periods when devices are less active. The system can then utilize most of the available bandwidth without affecting users’ experience. The authors also rely on bandwidth allocated by the ISP to the users, allowing them to utilize otherwise idle capacity.
Q:Does Horizon introduce additional CPU and memory overhead on the end devices?
A:Yes. The third-party provider rents virtual machines on users’ devices and allocates fixed resources to Horizon. Each agent is typically limited to one CPU core and approximately 200 MB of memory. Different probing tasks have different resource requirements. For example, ICMP and HTTP probes are relatively lightweight, whereas HLS or FLV streaming tests consume more bandwidth, CPU, and memory. Therefore, the authors designed scheduling mechanisms to coordinate different probing tasks and maximize the utilization of the available resources.
Q:Which characteristics of live-streaming networks had the greatest influence on Horizon’s design compared with more general network monitoring systems?
A:The most challenging characteristic is the large number of paths between edge nodes and end users. Traditional Internet monitoring methods, such as sending pings from cloud servers, mainly cover the backbone network and the paths between servers. They cannot adequately observe the many different paths from edge nodes to users. Therefore, Horizon introduces hyper-edge devices to improve observability from the user’s perspective. These devices can monitor actual video delivery and capture problems that traditional server-side monitoring cannot detect.
Q:Does Horizon consider user-level metrics, such as bandwidth and queuing delay, when evaluating network conditions and user experience?
A:Yes. The ultimate goal is to ensure a good user experience. The authors are particularly interested in whether users are affected and how many users experience problems. For runtime monitoring, Horizon uses multiple levels of checks, progressing from ICMP to HTTP and then to actual streaming protocols such as HLS and FLV. However, even a successful streaming request does not necessarily guarantee a good playback experience. Therefore, application-level playback checks provide more meaningful information about user experience than basic connectivity tests alone.
Q:How are applications deployed on hyper-edge devices? Are they deployed as containers or microservices?
A:They are deployed as virtual machines. Third-party providers rent individual VMs on users’ hyper-edge devices and make them available to the Horizon system. The authors can then manage these VMs and deploy their monitoring agents.
Q:Does the hyper-edge device run a full-fledged operating system?
A:Yes. The hyper-edge device provides a full-fledged operating system environment through which the monitoring VM can be deployed and managed.
Q:Can existing observability frameworks designed for virtual machines in traditional data centers be directly applied to hyper-edge devices?
A:Not directly. The main difference is that hyper-edge devices themselves are inherently unreliable. They may change ISPs or geographic locations, lose power, or go offline or enter sleep mode. Therefore, Horizon cannot simply assume that its monitoring agents are always available. Instead, it explicitly designs mechanisms to evaluate agent reliability and temporarily exclude unreliable devices. In traditional data centers, VMs are generally assumed to be much more stable, so this particular problem is less significant.
Q:Are there any similarities between monitoring hyper-edge devices and monitoring servers in traditional data centers?
A:Yes. Both environments can benefit from multi-layer monitoring. For example, when monitoring a high-performance network, ICMP ping can be used at the lowest level to check basic connectivity, but this is insufficient to guarantee that the virtual network is functioning correctly. Higher-level checks, such as RDMA-related tests or VPC-level and virtual-network monitoring, may also be necessary. The general principle of combining multiple monitoring layers is similar, although the reliability assumptions and specific challenges differ between hyper-edge devices and data-center servers.
Q:What were some of the most interesting or unexpected findings from this work?
A:One particularly interesting finding was cross-region ISP throttling. ISPs may throttle traffic when users consistently receive traffic from other regions or provinces because carrying such traffic can be more expensive. As a result, available bandwidth may be degraded even when the service-side monitoring system reports that the infrastructure is healthy. This kind of behavior is difficult to identify through traditional monitoring because ISPs do not necessarily provide explicit guarantees about such cross-region traffic conditions.
Q:Which parts of the original design changed the most as you gained experience from production deployments?
A:Initially, the authors tried to use relatively complicated algorithms to profile the CPU, memory, and bandwidth consumption of different probing tasks offline. However, they found that this approach was not very effective in production environments. Even when different VMs were used, the underlying hardware resources and actual resource availability could vary. Consequently, the authors moved toward simpler algorithms and more straightforward conflict-aware policies to schedule probing tasks and allocate resources more effectively.
Q:How are the VMs used by Horizon funded? Do you rent them directly from users or through third-party providers?
A:The authors work with third-party providers that have contracts with individual users. The research team establishes contracts with these providers and pays for the VM resources. The providers handle the rental arrangements with users, which may involve agreements such as renting a VM for one year. The specific financial details are part of the business arrangement and were not disclosed during the discussion.
Q:For students who do not have access to a live production network, what would be a good experimental starting point for research on user-side observability?
A:There are many open research questions in user-side observability. One direction is to investigate what metrics are actually necessary to guarantee smooth video playback. Another is to establish relationships between different measurement layers, such as the physical network, HTTP, FLV, and HLS, and understand how they relate to user experience.
Researchers can also use hyper-edge devices in their own homes to conduct experiments. For example, they could compare the playback experience when a phone accesses a hyper-edge device over Wi-Fi with the experience when the video is played directly on the hyper-edge device. If the results differ, this may help identify whether the problem originates from the Wi-Fi connection or from the network path between the edge and the user. These experiments can help identify which parts of the network contribute to degraded QoE.
Q:Can this work contribute to traditional WebRTC systems and their monitoring frameworks?
A:There are some important differences between live-streaming networks and WebRTC networks. WebRTC focuses heavily on low latency, so the internal network between servers is particularly important. If the network path between a user and an edge server is poor, WebRTC communication may fail. Therefore, server-side monitoring and ping-mesh mechanisms may be more important in large-scale WebRTC networks. In contrast, live-streaming edge nodes often serve a large number of users, so monitoring the paths between edge nodes and end users becomes particularly important.
Q:How is Horizon related to CDN infrastructure? Is the hyper-edge device essentially part of a CDN?
A:Yes. The edge nodes discussed in the paper are part of the CDN infrastructure. Horizon uses these existing edge nodes and additional hyper-edge devices to improve the observability of the live-streaming service.
Q:Does Horizon change the deployment decisions or architecture of a CDN?
A:Not directly. The deployment of CDN infrastructure is primarily driven by performance requirements, whereas Horizon focuses on observability. Its purpose is to help operators understand the performance of the existing live-streaming network rather than change where CDN nodes should be deployed. However, if hyper-edge devices were used for actual video delivery rather than only monitoring, as explored in other research, they could potentially influence CDN deployment decisions.
Q:For new graduate students who want to work in this area, what would be the best way to get started?
A:The author recommends conducting more experiments, exploring different applications, and identifying new cross-layer problems. Rather than focusing exclusively on infrastructure, students could start by investigating new applications and understanding their requirements. Since many infrastructure designs are already well established, exploring new applications may provide more accessible opportunities to identify interesting research problems.