بوصلة الكلام: تسهيل المحادثات الجماعية من خلال تحديد موقع الصوت

Imagine a world where following group conversations is effortless, even with hearing impairments or in noisy environments. Google Research and DeepMind are bringing that vision closer to reality with SpeechCompass, a novel approach to enhancing mobile captioning using multi-microphone sound localization. This technology, recently awarded “Best Paper” at CHI 2025, addresses a significant limitation of existing speech-to-text applications: the difficulty in distinguishing speakers in group settings. SpeechCompass aims to eliminate cognitive overload and enhance accessibility by providing real-time speaker diarization and directional guidance.

Overcoming Limitations of Existing Mobile Captioning

Current mobile automatic speech recognition (ASR) apps often concatenate all transcribed speech, making it challenging to follow who is speaking. This presents a considerable hurdle for users requiring accessibility features, language translation, note-taking, or meeting transcripts. Existing solutions, such as audio-visual speech separation and speaker embedding, are often impractical for mobile use due to camera requirements or the need to pre-register voiceprints. SpeechCompass offers a more practical and privacy-conscious alternative.

Introducing SpeechCompass: Diarization and Directional Guidance

SpeechCompass enhances mobile captioning with two key features:

• Speaker Diarization: Separating speakers in the ASR transcript with color-coded visual cues.
• Real-time Localization: Directional indicators, such as arrows, guide the user to the source of the speech.

This multi-microphone approach offers several advantages:

• Lower Computational Costs: The algorithm runs on small microcontrollers with limited memory and compute, unlike ML-based approaches.
• Reduced Latency: Directional information is extracted from basic sound properties, enabling real-time operation with minimal lag.
• Greater Privacy Preservation: The system assumes that speakers are physically separated and doesn’t require video or unique personally identifying information.
• Language-Agnostic Operation: SpeechCompass analyzes differences between audio waveforms without prior assumptions about the content.
• Instant Reconfiguration: Moving the phone instantly reconfigures SpeechCompass.

Implementation and Technical Details

SpeechCompass is implemented in two forms:

• Phone Case Prototype: A custom phone case with four microphones connected to a low-power microcontroller allows for 360-degree sound localization.
• Software Implementation: A software version for existing phones with two or more microphones (e.g., Pixel phones) provides 180-degree localization.

The system addresses the challenge of sound reverberation in indoor environments by using a localization algorithm based on time-difference of arrival (TDOA). The algorithm estimates the TDOA between microphone pairs using Generalized Cross Correlation with Phase Transform (GCC-PHAT) and applies statistical estimations to improve precision. The use of 3 or more microphones solves “front-back” confusion issues present with 2 microphone solutions.

User Interface and Visualization Styles

The Android application displays the enhanced transcripts with multiple visualization styles to indicate speaker direction:

• Colored Text: Speakers are distinguished by different colored text.
• Directional Glyphs: Arrows, dials, or color highlights point to the speaker’s location.
• Minimap: A radar-like display shows the speaker’s position.
• Edge Indicators: Visual cues around the screen edges highlight speaker direction.
• Unwanted Speech Suppression: Users can click on screen edges to suppress speech from specific directions, removing their own speech or irrelevant conversations, enhancing privacy.

Performance and User Evaluation

Technical evaluations demonstrated that SpeechCompass can accurately localize sound direction, with an average error of 11°-22° at normal conversational loudness. This accuracy is comparable to human sound localization abilities. The four-microphone configuration consistently outperformed the three-microphone setup in diarization error rate (DER).

User surveys revealed that current mobile captioning technology struggles with distinguishing speakers in group conversations. User feedback on the SpeechCompass prototype highlighted the value of directional guidance, with colored text and directional arrows being the most preferred visualization methods.

Future Directions

The potential applications of SpeechCompass are vast, including:

• Classroom settings for following discussions.
• Business meetings, interviews, and social gatherings for tracking speaker changes.

Future development may include:

• Integration with wearable form factors like smart glasses and smartwatches.
• Enhanced noise robustness through machine learning.
• Further customization of visualization preferences.
• Longitudinal studies to understand adoption and behavior in everyday scenarios.

SpeechCompass represents a significant step towards making communication more accessible and inclusive, and it inspires further innovation in this critical area.

Source: Google Research, Google DeepMind