Vision augmented hearing

The most natural form of human interaction is face-to-face, integrating auditory information from the voice of the talker with visual information from the face of the talker. A dramatic illustration of the influence of visual information on speech perception is provided by the McGurk effect. In this illusion, pairing an auditory "ba" with an incongruent visual "ga" sometimes results in the percept of a different syllable. We show that an artificial neural network known as AVHuBERT also perceives the McGurk effect. Both humans and AVHuBERT report a mixture of percepts to McGurk stimuli, including the auditory component of the stimulus, the fusion percept of "da" reported in the original description of the illusion, and other syllables, including "fa" and "ah". The similar responses of humans and AVHuBERT to McGurk stimuli suggest that artificial neural networks may provide a useful model for human audiovisual speech perception. The neural basis of audiovisual speech perception was examined using stereoencephalographic (SEEG) recordings from neurosurgical patients. These recording demonstrate an anatomical boundary between the superior temporal gyrus, which responds similarly to auditory-only and audiovisual speech, and the superior temporal sulcus, which shows larger and faster responses to audiovisual compared with auditory-only speech. Consistent with behavioural studies, audiovisual enhancement was more pronounced in the presence of auditory noise.

Dr Michael Beauchamp

University of Pennsylvania, US

Dr Michael Beauchamp

University of Pennsylvania, US

Michael Beauchamp grew up in Waterloo, Ontario, Canada before attending Harvard University, where he received an undergraduate degree in Biology in 1992. He received his PhD from the Neuroscience Graduate Program at University of California, San Diego in 1997. As a postdoctoral fellow in the NIH Intramural Research Program in Bethesda, he worked with Jim Haxby (1997 - 2000) and Alex Martin (2000 - 2005). Michael's first faculty position was at the University of Texas at Houston Medical School, followed by a transition to Baylor College of Medicine in 2015. In 2021, Michael moved to Philadelphia to become a Professor and the Vice-Chair for Research of the Neurosurgery Department in the Perelman School of Medicine at the University of Pennsylvania.

Understanding speech in noisy, multisensory environments is a fundamental challenge for both listeners and researchers. While most studies focus on speech intelligibility, less is known about how the perceptual construct of loudness is influenced by the structure of the audiovisual (AV) scene. This work presents how systematically manipulating the temporal synchrony between auditory and visual signals, as well as the linguistic content of the speech modulates perceived loudness. Our results show that increased AV asynchrony leads to a significant drop in perceived loudness, with effects emerging beyond natural synchrony ranges. Surprisingly, linguistic complexity also alters loudness ratings, even when physical intensity is held constant. By transforming subjective ratings into an objective measure of perceived target-to-masker ratio (TMR), we demonstrate that extreme AV asynchrony results in a 2–3 dB drop in perceived TMR, independent of linguistic content.

These findings highlight the value of loudness perception as a sensitive index of AV scene analysis. This approach offers new avenues for studying multisensory processing in both typical and neurodiverse populations, and for developing accessible protocols that decouple scene analysis from linguistic ability.

Dr Liesbeth Gijbels

Meta Reality Labs Research, US

Dr Liesbeth Gijbels

Meta Reality Labs Research, US

Liesbeth Gijbels is a research scientist with a rich academic background in Speech and Hearing Sciences (PhD, Master, Bachelor), Psychology, and Education. The focus of her work is to bridge clinical practice and academic research in Speech and Hearing Sciences. Liesbeth has 10+ years of hands-on clinical experience supporting individuals with communication, learning, and hearing challenges. After moving to the US in 2018, she completed her PhD at the University of Washington, focusing on audiovisual speech perception and the cognitive processes underlying human communication. Building on her clinical and academic foundations, Liesbeth’s current work as a research scientist at Meta Reality Labs centers on developing AI-driven tools and technologies to enhance hearing and communication.

The mistakes that listeners make in determining the direction of a sound source are not solely the consequence of sampling error from a distribution centered around the source’s true position. Burgeoning evidence suggests that the distributions themselves have characteristic shifts or biases that depend on adaptive processes of different time courses, the spatial statistics and recent history of the auditory scene itself, the relative angle of the head and of the torso, and on eye gaze angle, to name a few contributing factors. These effects indicate that perceived acoustic space is not fixed but undergoes continual bias remapping in response to behavioural and sensory context. It is the intent of this talk to review the history of what we know about such systematic changes in spatial acoustic perception and in particular the interaction with gaze. We will then present a neurophysiologically inspired model of how head angle and eye gaze influence sound localization, as well as related phenomena such as sound source segregation and spatial release from masking. We conclude by comparing model predictions with key results from the literature, demonstrating that the model captures a common structure underpinning the diverse ways in which gaze alters spatial auditory perception.

Dr Owen Brimijoin

Reality Labs Research (Meta), US

Dr Owen Brimijoin

Reality Labs Research (Meta), US

Dr Brimijoin earned his bachelor's degree in Neuroscience and Behavior at Wesleyan University before completing his PhD in Brain and Cognitive Sciences at the University of Rochester. His doctoral research focused on nonlinearity in spectrotemporal auditory receptive fields—exploring how neurons develop preferences for the way sounds change in spectral content over time. During his postdoc, he used multi-unit physiology in mouse models to examine how aging and hearing loss affect this nonlinearity, contributing to the difficulty many people experience understanding speech in noisy environments.

He then transitioned to human psychophysics, spending nearly a decade as a scientist at the MRC Institute of Hearing Research in Glasgow. There, he studied speech intelligibility, listener behavior, sound localization, and auditory motion perception, with a particular focus on how both normal-hearing and hearing-impaired listeners use their own movement – intentional and unintentional – to understand speech and build a spatial understanding of the acoustic world.

Now at Reality Labs Research, he continues to work at the intersection of speech perception, listening intent, and spatial hearing with the goal of building augmented reality devices with the potential to significantly improve hearing and understanding in challenging environments.

The organisation of neuronal networks in the brain is highly structured; however, the principles underlying how structure relates to function are only beginning to be understood. What are the rules governing communication between sensory areas involved in processing different streams of information? I will present how the structure of connections in cortical and subcortical areas supports cross-modal spatiotemporal representations. I will also share our developments in biologically inspired AI models for the analysis of large-scale neuronal recordings, and how these models capture cortical cross-modal plasticity in the representation of natural scenes across days. Ultimately, by combining experimental and theoretical approaches, we aim to establish how circuit motifs support multisensory integration.

Dr Florencia Iacaruso

The Francis Crick Institute, UK

Dr Florencia Iacaruso

The Francis Crick Institute, UK

Florencia Iacaruso is a group leader at the Francis Crick Institute. At the interface between systems neuroscience, computational modelling, and sensory physiology, her lab investigates how the brain integrates information from different sensory modalities to guide behavior.

Speech enhancement technologies have rapidly developed in the last decade. Multi-modal speech perception has inspired the next generation of speech enhancement technologies to explore multi-modal approaches that leverage visual information to overcome scenarios that can be challenging for audio-only speech enhancement models (e.g., overlapping speakers).

The Audio-Visual Speech Enhancement Challenge (AVSEC) provided the first benchmark to assess algorithms that use lip-reading information to augment their speech enhancement capabilities. Throughout its four editions, we provided carefully designed datasets that enabled us to explore AV-SE performance in a range of listening conditions different from those commonly used in laboratory settings. In this talk, I will present our proposed approach to scalable stimuli design from in-the-wild data and introduce the AVSEC protocol for human listening evaluation of AV-SE systems. I will present an overview of the listening test results throughout different editions of the challenge and discuss them in relation to characteristics of the designed stimuli.

Dr Lorena Aldana

University of Edinburgh, UK

Dr Lorena Aldana

University of Edinburgh, UK

Lorena Aldana is a Research Associate at the University of Edinburgh. She has a background in sound engineering and computer science. She was a DAAD scholar at The University of Bielefeld and finished her PhD in 2021.

Her research interests lie at the intersection of multi-modal speech and hearing technologies, audio signal processing and machine learning.

Lorena has been a technical lead of the four successful editions of the International Audio-Visual Speech Enhancement Challenge (AVSEC). Her current research focuses on advancing evaluation methods for speech and hearing technologies addressing ecological validity and integrating individual differences in hearing beyond current standard clinical methods.

Spatial hearing is most commonly assessed using sound localisation tasks that are widely regarded as a gold standard for evaluating auditory spatial function. These paradigms typically rely on highly controlled experimental conditions, in which listeners are visually deprived, required to keep their head stationary, and exposed to brief noise bursts presented in different locations within anechoic environments. While such methods provide precise and reproducible measurements, they represent listening situations that are largely detached from everyday auditory experience. In natural environments, sound sources are multiple and rather complex from a spectral envelope point of view, listeners actively move their head and body, and spatial judgments are frequently resolved through dynamic interactions between proprioception, audition and vision.

This mismatch raises fundamental questions about the ecological validity of traditional spatial hearing assessments and about the role of vision in both assessing and training spatial auditory functions. Even alternative approaches, such as speech-based measures of spatial release from masking, implicitly depend on visual information, which supports spatial expectation, source identification, and audio-visual speech integration. Consequently, vision is not merely an auxiliary modality but an integral component of spatial hearing in real-world contexts.

I will present recent work exploring assessment paradigms that explicitly incorporate vision while preserving the functional contribution of auditory spatial cues. A first study focuses on a visual search task in which spatialised sound guides attention toward a visually defined target will be introduced. Although the task can ultimately be completed using vision alone, auditory information significantly affects reaction times, movement trajectories, and search strategies, providing sensitive performance metrics beyond localisation accuracy. For example, this paradigm has revealed subtle but systematic differences between individualised and non-individualised head-related transfer functions. In a second one, a game-based localisation tasks employing audio-visual anchors and auditory distractors shows that visible and head-tracked sound sources can facilitate faster learning than non-tracked and visually absent ones.

Finally implications of these findings for spatial hearing training. Across laboratory and clinical applications, including training for bilateral cochlear implant users, our results highlight the importance of multimodal feedback for perceptual remapping. Coupling auditory and visual information supports adaptation to altered spatial cues, while progressively reducing visual guidance enables the transfer of learning to audition alone. Together, these findings argue for a systematic integration of vision into spatial hearing assessment and rehabilitation, moving beyond purely auditory paradigms toward more ecologically valid and functionally informative approaches.

Professor Lorenzo Picinali

Imperial College London, UK

Professor Lorenzo Picinali

Imperial College London, UK

I am a Professor in Spatial Acoustics and Immersive Audio, and I lead the Audio Experience Design team (www.axdesign.co.uk). Our research explores both the perceptual and computational aspects of acoustics, alongside their practical applications. We aim to understand how humans perceive spatial sound features, such as source locations and reverberation, and use this knowledge to design algorithms and tools for creating virtual acoustic simulations. Our goal is to seamlessly blend real and virtual acoustic environments, ultimately enhancing hearing. I also work on eco-acoustic monitoring, designing autonomous recorders, and using audio to study the impact of human activity on remote ecosystems.

Speech comprehension is often boosted by watching a talker’s face and has been reported vary between individuals and across the materials used for testing. However, the benefit is dependent on exact stimulus conditions and highly non-linear, making it challenging to compare across individuals and groups, experimental manipulations and treatments. I will describe a model which is simple enough to use for quantitative analysis of data and appears to capture relationship between speech perception performance in audio-only, visual-only and audio-visual situations. The Multi-Stage Noise model considers speech perception as a process limited first by modality specific low-level processing followed by later supra-modal limitations. In tests so far (~400 people) audio-visual speech perception is well predicted from individual differences in unisensory processing across a wide range of people, implying that the integration process itself is relatively stable. Here, I will use the model as source of simulated data, which presumably captures the non-linear properties of real data, and compare various other methods of quantifying the relationships in performance across modalities. This reveals that many conventional ways of quantifying integration and the “benefit of seeing the talker” result in high-false positive rates – which could be interpreted as changes in audio-visual integration. The Multi-Stage Noise model, which can be used within a Bayesian statistical framework, may offer a more robust “null” model to quantify audio-visual integration.

Professor Chris Sumner

Nottingham Trent University, UK

Professor Chris Sumner

Nottingham Trent University, UK

Chris’ research focuses on understanding the neural computation underlying how we hear. He does this with a variety of methods, but often by building computer models (simulations) of neurons, neural systems, and behaviour, with the aim of relating processing by single neurons to our perception of sound. He believes that this understanding is critical in order to tackle the problems associated with hearing loss. Recent research interests include: How sensory processing influences the coding and recognition of complex acoustic signals such as (but not limited to) speech and how this is affected by hearing loss; the mechanisms underlying resolution of sound frequency in the auditory system; audio-visual integration of speech.