Face2face: advancing the science of social interaction

Enabling human-like communication has been a driving force as well as a linchpin for AI systems ever since. Recent data-driven approaches in fields like conversational AI, computational linguistics, or social behaviour processing have yielded models that can process and generate impressive instances of communicative surface behaviour. However, today's systems still suffer from clear shortcomings when it comes to being able to engage in a fluent, dynamic face to face interaction. Professor Kopp will discuss the current state of the art and point out principles and approaches that need to (and can) be used to build embodied conversational agents that are able to create and maintain a 'socially interactive and intelligent presence' in face to face interaction. This will include establishing multiple interaction loops, adapting dynamically to the evolving interaction context, and combining communicative behavior processing with mental state attribution and dialogue models. 

Experiencing social interaction and social communication difficulties is core to a diagnosis on the autism spectrum. Understanding how and why social interaction differences occur between autistic and non-autistic people will facilitate understanding of how difficulties can be reduced. In order to create good models of social interaction and communication processes it is important to study behaviour in real-world contexts. Conducting well designed, rigorous face-to-face studies is practically challenging in many ways but if we fail to do so, our understanding will be incomplete. In this talk, Dr Freeth will consider evidence from a range of different paradigms that aim to assess key processes and constructs involved in face-to-face interactions including consideration of how these can be isolated and investigated. These include 1. the perception of social presence; 2. the perception of agency as social or non-social; 3. the experience of a live vs pre-recorded social encounter. Evidence will be drawn from paradigms that involve structured interactions measuring behavioural responses and tracking of eye movements, using both desk-mounted and mobile eye-trackers. Overall, the evidence presented will demonstrate the potential that face-to-face interaction studies afford in furthering our understanding of the processes underlying social interaction and social communication.

During early life, healthy neurodevelopment depends on warm, responsive and closely-coordinated social interactions between infants and caregivers. These rich multidimensional sensory experiences act through multiple pathways to orchestrate healthy maturation of the neonatal brain, mind and body. Conversely, adverse early life experiences (including abuse or neglect) seed vulnerabilities for poor cognition and emotional instability throughout the lifespan. 

Despite the pivotal role played by social interaction in early development, we still lack medical models and precision tools that can accurately assess a child’s social interactive capacities (their ability to engage in and respond to social input) and measure the health of their social interactions. Similarly, although psychological tools exist to measure children’s cognitive capacities – including intelligence, attention, working memory, and inhibition skills – these tests cannot be used in the first year of life with young infants. Therefore, a needs gap exists in the early assessment and detection of abnormalities in infant psychosocial health and consequent impact on neurocognitive development. We currently miss many, if not most, early warning signs of suboptimal psychosocial development, along with valuable opportunities for prophylactic intervention, before a child’s first birthday.

Here, Associate Professor Leong will discuss neural sociometrics – real-time multi-sensor high-dimensional imaging of adult-infant social interactive behaviour and neurophysiology – as a possible tool and framework for precision measurement of early mental health and neurodevelopment. Early risk identification and mitigation, paired with precision therapeutics, could fundamentally alter a child’s development trajectory toward lifelong mental wellbeing and productivity.

Human music and speech are peculiar behaviours from a biological perspective: although extremely common in humans, at first sight they do not seem to confer any direct evolutionary advantage. In particular, many hypotheses have been proposed for the origins of rhythm, often in connection with vocal learning, a precursor to speech. Because music and speech do not fossilize, and lacking a time machine, the comparative approach provides a powerful tool to tap into human cognitive history. Notably, behaviours that are homologous or analogous to human rhythm and speech can be found across a few animal species and developmental stages. Hence, investigating rhythm across species is not only interesting in itself, but it is crucial to unveil protomusical and protolinguistic behaviours present in early hominids. Here Dr Ravignani suggests how three strands of research – partly neglected until now – can be particularly fruitful in shedding light on the evolution of rhythm and vocal learning. He will present rhythm experiments in marine mammals, primates, and other species, suggesting that rhythm research in non-human animals can also benefit from ecologically-relevant setups, combining strengths and knowledge from cognitive neuroscience and behavioural ecology. Second, he will discuss the interplay between vocal anatomy, (social) learning, and vocal development in harbor seal pups, arguing for their importance as model species for human speech. Finally, he will present computational modeling work on rhythmic and interactive signaling across species. These results suggest that, while many species may share one or more building blocks of speech and music, the ‘full package’ may be uniquely human.

Although emotional contagion is a primitive and foundational feature of social interactions, the underlying neural mechanisms are not well understood. Here we apply a dyadic approach to investigate face-based emotional contagion using hyperscanning and functional near infrared spectroscopy. One participant, 'Video-Watcher', watched emotionally provocative silent videos and the other partner, 'Face-Watcher', watched the face of the Video-Watcher. The library of videos included 'neutrals' featuring landscapes, 'adorables', featuring animal antics, and 'creepy crawlies' featuring spiders, worms and states of decay. Ratings of the positive and negative emotional experience were acquired simultaneously for both participants. The correlation of emotional ratings between the 'Video and Face-Watchers' (r = 0.60) confirmed that the emotion was transmitted by facial expressions and taken as confirmation of emotional contagion.  Facial Action Units of the 'Video Watcher’s' face were taken as the stimuli for the 'Face- Watcher’s' brain. Neural responses included a complex of right TPJ regions consisting of supramarginal gyrus, angular gyrus, and superior temporal gyrus (p < 0.05, FDR corrected peak voxel) consistent with a face-emotion-sensing function. A dyadic model of emotion contagion and face sensing is suggested where human brains and faces are tuned to send and receive emotional states.

In WEIRD (Western, educated, industrialized, rich, democratic) societies, face-to-face interactions are an important component of mother-infant interactions. While facing each other, they engage in a variety of non-verbal means of communication, including facial expressions, such as smiles, and eye contact. It has been suggested that face-to-face interactions are unique to humans, as they are a form of emotional communication indicative of the long-lasting bond between mothers and their infants.

However, other great apes, including the closest relatives of humans, the chimpanzees, also form close and long-lasting relationships with their offspring, but little is known about the role of face-to-face contact in mother-infant interactions. In terms of quantitative measures, research has shown that gazing bouts in chimpanzees are shorter than in human mother-infant dyads. However, less is known about the quality of such face-to-face contact, as it remains an open question what chimpanzee mothers or their infants do while engaging in such interactions. 

Therefore, the group studied face-to-face contact between mothers and their infants in captive chimpanzees and humans from a Western society, to investigate both qualitative and quantitative aspects of their face-to-face interactions. They predicted that 1) human mothers spend more time engaging in face-to-face contact with their infants, and 2) both species vary with regard to the types of behaviours used during such gazing bouts. They used focal observations to video-record spontaneous interactions of ten human and 8 chimpanzee mothers and their 6-months-old infants, and coded facial expressions, facial touches, and the duration of face-to-face contact. If feasible, eye contact was also coded. As predicted, human dyads spend significantly more time in face-to-face interactions than chimpanzees. Species also differed with regard to the behaviours shown during face-to-face contact: human mothers frequently engaged in direct eye contact and to some extent, also mirrored facial expressions of their infant, while chimpanzee mothers often touched the face of their infants, eg, during grooming bouts. Thus, these findings indicate that during face-to-face contact, human mothers use different behaviors to communicate with their infants, while in chimpanzees face-to-face contact seems to be rather a by-product of the mother’s bodily actions on the infant, rather than a communicative situation. 

The game-changer that afforded human culture is widely considered to be language. But turning thoughts into vocalizations was only part of the story. The real innovation was the development of a common language – a sounds-to-meanings map shared across people that enabled one idea to leap from one mind to the next. Conversation provided a platform for minds to meet, allowing information to leap from one brain to the next. But how does conversation work?

We know very little about conversation because the fields of psychology and neuroscience have focused near-exclusively on the individual. In part this was because interacting minds create rich, complex data that were challenging to capture and analyse. However, recent technological advances have made the study of conversation newly tractable.

Here, Professor Wheatley will present research demonstrating how face to face conversation aligns minds by increasing neural synchrony and how neural synchrony, in turn, changes conversation. She will also discuss evidence for why breaking interpersonal synchrony, from time to time, may be important to striking the right balance between shared and independent modes of thought.

Conversation is a collaborative process where speakers and listeners produce information together, continuously coordinating to incrementally co-construct the evolving content. Smooth turn exchange is achieved through tight coordination of interlocutors’ verbal and non-verbal behaviour and becomes problematic when this deviates from expectations. 

Patients with a diagnosis of schizophrenia are one of the most socially excluded in society. It is well documented that they have problems with language and social cognitive skills, including with self-monitoring and turn-taking, yet little research has investigated how these impact interaction. Interactions involving patients offer an opportunity to observe the strategies that people employ when interaction is problematic, and shed light on how ‘normal’ interactions are managed.

Using data from a corpus of triadic conversations containing 20 dialogues involving one patient with a diagnosis of schizophrenia and two healthy controls and 20 dialogues involving three healthy participants, Dr Howes will show that dialogues involving a patient differ from controls in terms of turn-taking, disfluencies, gesture and repair. Furthermore, the presence of the patient influences the behaviour of the healthy controls they interact with. The data supports the idea that disfluencies are communicative solutions, not problems.

This unique data demonstrates that not only are there communication difficulties in schizophrenia but they also impact on social interactions more broadly, thus providing new insights into the social deficits of this complex disorder.