Thursday, 29 October 2026 · 7 min read
Falcon enables efficient, real-time audio-visual target speaker extraction in complex scenes
Falcon, a lightweight causal AVTSE model with a novel multi-scale separator and encoder-space speaker contrastive loss, achieves state-of-the-art extraction quality on realistic multi-speaker benchmarks while drastically reducing computation and latency for edge deployment.
Based on the paper Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes by Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang and 1 other
Challenges in real-time audio-visual target speaker extraction
Audio-visual target speaker extraction (AVTSE) aims to isolate a target speaker's voice from a mixture using their lip movements as visual cues. This task is crucial for latency-sensitive applications like video conferencing and hearing aids, where causal (real-time) processing is required.
Existing benchmarks for AVTSE typically fix the number of speakers at two and lack realistic acoustic conditions such as varying speaker counts, near- and far-field interference, reverberation, and sudden environmental noise. This limits the evaluation of models in real-world scenarios.
On the modeling side, high-performance causal AVTSE architectures are computationally heavy, making them unsuitable for edge devices. Lightweight models often reuse small separators iteratively, which increases computation and latency, offsetting efficiency gains. Additionally, standard training losses do not explicitly discourage residual leakage from near-field interfering speakers, which worsens as the number of interferers grows.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Two key formulas from the paper, explained symbol by symbol
- Two figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short