Title: Artic: AI-oriented Real-time Communication for MLLM Video Assistant
Authors: Jiangkai Wu, Zhiyuan Ren, Junquan Zhong, Liming Liu, Xinggong Zhang (Peking University)
Introduction
AI Video Assistants powered by Multimodal Large Language Models (MLLMs) represent a new paradigm of real-time communication (RTC), where one participant in the communication process is an AI model deployed in the cloud. Users continuously send video and audio streams to the AI assistant and receive responses, creating a more natural interaction experience similar to communicating with a human assistant. However, current RTC frameworks are mainly designed for human perception and optimize metrics such as visual quality and smooth playback, which do not match the requirements of AI Video Assistants. These systems need high response accuracy and extremely low latency instead. In addition, mobile and wearable devices introduce unstable network conditions and limited uplink bandwidth. Existing RTC systems fail in these scenarios, causing latency spikes and accuracy drops.
Key idea and contribution
The authors propose Artic, an AI-oriented RTC framework designed specifically for MLLM Video Assistants. Instead of optimizing communication for humans watching videos, Artic redesigns RTC around the goal of enabling AI to better understand video content. The framework includes three major contributions: Response Capability-aware Adaptive Bitrate, Zero-overhead Context-aware Streaming, and the Degraded Video Understanding Benchmark.
The first component, Response Capability-aware Adaptive Bitrate (ReCapABR), changes the traditional bitrate adaptation strategy. Conventional congestion control algorithms continuously increase bitrate when bandwidth is available because higher quality benefits human viewers. However, MLLMs often reach an accuracy saturation point where additional bitrate no longer improves their answers. ReCapABR uses feedback from the MLLM’s confidence level to determine whether more bitrate is necessary. If the model already has enough information to answer correctly, the system limits bitrate growth and reserves bandwidth for future fluctuations, reducing latency.
The second component, Zero-overhead Context-aware Streaming (ZeCoStream), addresses the problem of limited bandwidth. Instead of allocating bits uniformly across the whole video frame, ZeCoStream allows the MLLM to identify regions that are important for answering the current question. The system then allocates more bitrate to these regions while compressing less important areas. This approach avoids additional client-side computation and improves accuracy under low bandwidth conditions.
The third contribution is DeViBench (Degraded Video Understanding Benchmark), which evaluates how video quality degradation caused by RTC affects MLLM accuracy. Unlike traditional video benchmarks that focus on human visual quality, DeViBench measures whether degraded videos still allow MLLMs to correctly answer questions. The benchmark contains automatically generated degradation-sensitive samples and supports evaluation and system tuning.
Evaluation
The authors implemented an Artic prototype and evaluated it using real-world mobile uplink traces. Compared with existing RTC approaches, Artic improves MLLM response accuracy by 15.12% and reduces latency by 135.31 ms. The evaluation also shows that Artic introduces minimal computation, bandwidth, and monetary overhead. This result is significant because it demonstrates that future communication systems for AI applications should optimize for machine understanding rather than human perception, enabling more reliable and interactive AI assistants in real-world environments.
Q&A
Q: I have a question about the video queries. Are the queries arriving online, or do you already know the queries in advance?
A: That is a very good question. Actually, there are two possible settings. The first setting is query-driven, where the user asks a question and the AI provides a response. This is a turn-based interaction, meaning the system only processes the video when there is a specific user query. The second setting is proactive video understanding, where the user continuously streams video to the AI. The AI keeps watching the video and provides responses or alerts when necessary. In this case, the user does not need to explicitly ask a question each time. Instead, the user can define certain conditions, such as “notify me when there is a warning” or “tell me when something important happens.”
Q: Which setting are you mainly targeting? Because for tasks like bitrate selection or video compression, the optimal strategy depends heavily on the query. If you do not know what the query will be asking about, especially which part of the video is important, it seems difficult to decide what information to preserve.
A: Yes, that is an excellent point. Our primary target is the second setting, the proactive video understanding scenario. The reason is that the model needs to identify important regions or events in the video and allocate more encoding resources to those areas. If we do not transmit sufficient video information, the AI cannot accurately determine which regions are important. However, the first setting also has potential applications. In that case, we can use the context from previous interactions to identify important regions. Instead of relying only on the latest question, the system can use the conversation history or a summary of previous queries as context. This approach can also help guide video compression, although it may not be as accurate as the proactive setting.
Personal thoughts
This paper provides an important perspective that communication systems should evolve as the receiver changes from humans to AI models. I like that the authors identify the fundamental mismatch between existing RTC objectives and AI assistant requirements. The idea of allowing the MLLM itself to provide feedback for network optimization is especially interesting because it avoids relying on manually designed quality models and allows the system to adapt to different tasks.
However, several questions remain open. First, Artic depends on the reliability of MLLM feedback, but the confidence scores generated by models may not always accurately represent true understanding ability. Future work could explore more robust feedback mechanisms. Second, as MLLMs become more powerful, their memory and reasoning capabilities may change the relationship between video quality and accuracy. It would be interesting to investigate communication systems that jointly optimize networking, compression, and AI reasoning.