Title: DeepSFU: Scalable Deepfake Detection for Video Conferencing
Authors: Tuan Tran, Shirin Ebadi, S. M. H. Hosseini, Woongsub Shin, Evan Ram(University of Colorado Boulder); Youngwook Son(University of Colorado Boulder & Seoul National University); Seyeon Kim (Korea University); Nam Bui (University of Colorado Denver); Kyunghan Lee (Seoul National University); Eric Keller, Sangtae Ha(University of Colorado Boulder)
Introduction
Deepfake technology has become a serious threat to online communication because attackers can impersonate executives, public figures, or trusted contacts during video calls. Although existing deepfake detection models can achieve high accuracy offline, deploying them in real-time video conferencing is difficult. Conventional detection pipelines must decode high-resolution video frames, transfer them to the GPU, and run computationally expensive inference for every stream. These operations consume substantial CPU, GPU, and memory bandwidth, causing high latency, frame drops, and poor scalability. Therefore, existing video conferencing systems cannot provide continuous deepfake detection while maintaining interactive quality and supporting large meetings.
Key idea and contribution:
The authors propose DeepSFU, a video conferencing architecture that integrates deepfake detection directly into the Selective Forwarding Unit (SFU). Instead of decoding every frame, DeepSFU analyzes information from encoded video bitstreams to detect possible facial changes and deepfake onsets. It only requests high-resolution decoding and heavyweight model inference when suspicious changes are detected. This selective strategy avoids the high cost of continuous frame decoding while preserving detection accuracy.
DeepSFU also introduces a more efficient media-processing pipeline. It uses SmartNICs to offload packet forwarding, encryption, and data movement, and uses GPUDirect RDMA to transfer frame data directly from the network interface to GPU memory. The system includes a lightweight two-stage detector based on residual energy and adaptive cluster scores. Together, these techniques reduce CPU contention, shorten the critical path, and allow deepfake detection to operate as a native capability of the SFU.
Evaluation
The authors implement DeepSFU and evaluate it on a large-scale testbed against a Jitsi-based deepfake detection system. DeepSFU reduces per-frame processing latency by 143.7× and supports 26.2× more concurrent conferences at 720p, while maintaining continuous detection and video quality. Its onset detector achieves 95.8% precision, 97.6% recall, and a 96.7% F1-score. DeepSFU also provides an average end-to-end alert latency of approximately 153 ms and achieves 18× higher throughput with only about 0.74% bandwidth overhead. This result is significant because it demonstrates that real-time deepfake detection can be integrated into large-scale video conferencing without sacrificing the scalability and low latency expected from an SFU.
Q: How does the system ensure that the Superblock itself is not the Deepfake artifact?
A: The system does not directly classify a Superblock as fake or real. Instead, it monitors sudden changes in residual energy across the facial region. If the entire face changes, the residual energy of the corresponding Superblocks will increase significantly. The system then treats this as suspicious, requests a keyframe, and performs further verification using a computationally expensive CNN-based detector.
Q: Why is a hardware cryptographic pipeline needed on the DPU?
A: Video conferencing requires a large amount of hop-by-hop encryption. To reduce this overhead, the system offloads most cryptographic operations to the hardware crypto pipeline on the BlueField-3 DPU. This reduces CPU utilization and improves the efficiency of media forwarding.
Q: Has the system been evaluated using real-world applications such as Zoom or Google Meet?
A: The authors did not directly evaluate the system on Zoom or Google Meet because they could not access the production code of these commercial platforms. Instead, they implemented the system on Jitsi Videobridge, a popular open-source SFU, and integrated the Deepfake detection pipeline into it. Since commercial platforms and Jitsi use similar SFU architectures, Jitsi provides a reasonable representation of the core system architecture.
Q: How would the system handle end-to-end encrypted video?
A: One possible approach is to run the detector on the client or endpoint. However, endpoints are usually laptops or smartphones with limited computing resources. Although the video frames are decoded on the receiver side, high-accuracy detection typically requires a heavy CNN model, which can impose substantial GPU and computational overhead. Therefore, running high-accuracy Deepfake detection directly on end-user devices remains challenging.
Personal thoughts
The main strength of this paper is that it identifies the real bottleneck in real-time deepfake detection: not necessarily the neural network itself, but the repeated decoding and movement of video frames. The idea of analyzing encoded-domain information first and invoking expensive inference only when necessary is both practical and elegant. The combination of selective detection, SmartNIC offloading, hardware cryptography, and zero-copy GPU access also shows strong systems-level thinking.
One limitation is that the approach depends on the encoded-domain signals being sufficiently reliable across different codecs, cameras, lighting conditions, and deepfake generation methods. It would be interesting to evaluate DeepSFU in more diverse real-world environments, especially with encrypted or end-to-end encrypted conferencing, mobile clients, and rapidly evolving generative models. Another open question is how the system should balance detection sensitivity, false alarms, privacy, and user consent in practical deployments.