Title: LITE： Loss-resilient Immersive Telepresence with Multimodal Semantics
Authors: Ruizhi Cheng, Nan Wu, Bo Han (George Mason University); Harshvardhan Takawale, Nirupam Roy, Sennur Ulukus (University of Maryland); Matteo Varvello, Eugene Chai (Nokia Bell Labs)
Introduction
This paper studies how to preserve quality of experience in immersive telepresence systems operating over lossy networks. Immersive telepresence transmits 3D content to enable free interaction in virtual spaces, but packet loss can cause severe freezing, stalling, and visual degradation. Traditional retransmission and forward error correction (FEC) cannot satisfy strict latency requirements, while deep-learning techniques developed for 2D video do not transfer directly to 3D content because 3D processing is expensive, representations are diverse, and temporal dependencies are difficult to model.
Key Idea and Contribution
The authors build LITE, a loss-resilient immersive telepresence system based on multimodal semantics. Instead of recovering raw 3D data, LITE recovers compact and user-relevant semantic representations, such as motion and audio semantics. These semantic representations are more compact, more predictable, and less sensitive to packet loss. LITE compresses semantics by more than 1,000 times and requires only about 3 ms for inference, leaving sufficient time for loss recovery even under tight latency constraints.
LITE has three main components. First, it uses semantic communication to transmit compact motion and audio representations. Second, it uses a lightweight multimodal model that exploits the correlation between audio and facial motion to recover missing motion semantics. Third, it introduces neural redundancy for audio semantics: historical audio information is compressed and transmitted as neural redundancy with current frames, requiring only 0.006 Mb/s of additional bandwidth. The authors implement a complete prototype and evaluate it using real network traces and user studies.
Evaluation
LITE is evaluated using real packet-loss traces from the Akamai cellular network, with an average loss rate of 29.7%, and the Hairpin cloud-gaming network, with an average loss rate of 10.1% and burst losses reaching 90%. Under the Akamai traces, LITE maintains SSIM above 0.90, while nine baseline configurations remain below 0.75. Its lip-sync error remains at or below 0.2, compared with more than 3.5 for the baselines; it sustains over 30 FPS, while the baselines fall below 20 FPS; and users rate it 6.5 out of 7, compared with 4.7 for the best baseline. LITE maintains similar performance under burst losses and controlled loss rates up to 30%. This result is significant because it shows that multimodal semantic recovery can provide near-lossless immersive telepresence quality under severe packet loss while meeting real-time requirements with far less overhead than conventional recovery techniques.
Q1: How did you evaluate the loss of audio semantics? Did you test cases where audio semantics are completely lost, or where frames are lost while audio remains complete? Is there an ablation study showing the contribution of audio semantics?
A1: The evaluation replays real loss traces at an accelerated rate and includes cases where audio is lost while motion is available, and cases with the opposite pattern. The ablation study shows that adding audio semantics improves SSIM, while the full multimodal model provides a further improvement.
Q2: If bandwidth is stable and compression/transmission are perfect, can semantic communication still outperform perfect compression?
A2: LITE focuses on robustness to packet loss rather than the no-loss compression limit. Internet loss can be bursty, including sudden loss rates around 90%. If there is truly no loss, the gain studied here is not the main benefit; prior work such as MagicStream focuses on compression. In practice, current 3D content can still require 60–70 Mb/s after intra-frame compression, whereas semantic transmission can reduce the rate to below 1 Mb/s.
Q3: What is the computation and resource overhead of semantic extraction? Can it run on resource-limited edge devices?
A3: The semantic features are lightweight, including facial keypoints and acoustic features. The authors evaluated running the pipeline on an edge/embedding device and report support for about three concurrent users on a single Jetson-class edge device.
Personal Thoughts
LITE addresses a timely problem as immersive telepresence moves toward practical deployment. I particularly like its decision to recover semantics rather than pixels or raw 3D frames. The more than 1,000-fold semantic compression not only improves robustness to packet loss but also creates a useful computation window for recovery. Using audio semantics as an auxiliary signal for motion reconstruction is another clever cross-modal design choice.
The evaluation is strong because it includes both cellular and bursty cloud-gaming traces, as well as user studies. Future work could examine how well the approach generalizes across users, languages, environments, and different 3D representations. It would also be valuable to understand the energy cost of semantic inference on mobile or head-mounted devices.