Could virtual reality gaming help hearing impaired children improve their performance in the classroom?

04 July 2023

Training children and young people with bilateral cochlear implants to locate sounds in virtual reality (VR) games could help them learn how to filter out distractions and perform better in the classroom and in everyday life, according to scientists.

A clinical trial, led by UK scientists and clinicians from the Both Ears (BEARS) project, is testing whether a suite of virtual reality games could improve participants’ sound localisation, speech-in-noise performance and music listening skills.

Up to 384 participants, aged between eight and 16, with two cochlear implants are being recruited to take part in the randomised clinical trial where they will practise with a VR headset and headphones for at least an hour a week for three months.

Visitors to the Royal Society Summer Science Exhibition, a free science festival in central London running 4 – 9 July, will be able to try out the VR games and explore the science behind the technology.

“We know children with cochlear implants can find it difficult to differentiate sounds in background noise or find where a sound is coming from,” said Professor Helen Cullington, University of Southampton, Principal Clinical Scientist on the NIHR-funded trial.

“Using both ears, or implants, can help these functions. Locating sounds is part of daily life and important for keeping us safe: if you are out and about and a car beeps you need to know where it is coming from. Picking out the sounds you want to hear and ignoring the background noise can help us navigate noisy environments like classrooms.

“So, we hope the BEARS trial will show improvements in these kinds of tasks, which in future could bring a real ‘spend to save’ benefit. We hope that by improving children’s hearing their educational outcomes will also improve and that in turn will help them throughout their lives.”

The games take advantage of state-of-the-art simulations which account for the specific way sounds arrive at the two ears from the surrounding environment, engaging participants in tasks such as locating a customer in a noisy café and understanding their order; identifying a musical rhythm, or detecting a target based on a sound. The games begin with audio-visual interactions, to facilitate familiarisation and learning, but as the player advances, the visual cues are reduced, up to the point that the tasks need to be completed only relying on auditory cues.

Another factor in locating sound sources and distinguishing speech from noise is our individual head-related transfer functions (HRTFs), or audio ‘filters’. These are the cues that allow our hearing system to determine where a sound is coming from, and to separate a target speech source from several maskers, for example when in a noisy restaurant. To perform accurate and realistic simulations, individual features of these HRTFs need to be accounted for.

Audio engineers at Imperial College London are the key designers and developers of the BEARS applications and are working on the same technology that underpins the virtual reality experience as part of the EU-funded SONICOM project.

The Imperial team has built a state-of-the art sound-proofed laboratory to collect data on the different sizes of people’s heads and ears to ascertain how they perceive sound through their own individual audio filter.

The data will be used to create personalised, realistic virtual audio using artificial intelligence.

Dr Lorenzo Picinali, head of Imperial’s Audio Experience Design (AXD) research group, which is running the SONICOM project, said: “While from the visual point of view we are still rather far from being able to blend real and virtual domains, with one always being clearly distinguishable from the other, from the auditory perspective we are very close to being able to trick someone into believing that a sound source is in the surrounding space, for example someone else talking to them, even if there is no sound in the real environment.

“Hearing impairment is a big societal challenge and the technology, particularly cochlear implants, has changed so much. We have gone from a unilateral implant to bilateral implants, opening up possibilities which were not available just a few years ago.

“So, this is the right moment to do something on bilateral hearing for cochlear implant users. The technology and lessons from the SONICOM project are feeding into this trial, helping to recreate these real-world environments and experiences in virtual reality, and training the cochlear implant users to use the altered cues more effectively, ultimately improving these young people’s lives, and outcomes as well.”

There are more than 5,000 children in the UK with bilateral cochlear implants, and if successful, the team hopes the virtual reality training and low-cost headsets could be rolled out more widely on the NHS, and to other people with hearing impairments.