Face2face: advancing the science of social interaction

Enabling human-like communication has been a driving force as well as a linchpin for AI systems ever since. Recent data-driven approaches in fields like conversational AI, computational linguistics, or social behaviour processing have yielded models that can process and generate impressive instances of communicative surface behaviour. However, today's systems still suffer from clear shortcomings when it comes to being able to engage in a fluent, dynamic face to face interaction. Professor Kopp will discuss the current state of the art and point out principles and approaches that need to (and can) be used to build embodied conversational agents that are able to create and maintain a 'socially interactive and intelligent presence' in face to face interaction. This will include establishing multiple interaction loops, adapting dynamically to the evolving interaction context, and combining communicative behavior processing with mental state attribution and dialogue models. 

Professor Stefan Kopp, Bielefeld University, Germany

Professor Stefan Kopp, Bielefeld University, Germany

Stefan Kopp is Professor of Computer Science and head of the Social Cognitive Systems Group at Bielefeld University, Germany. He obtained his PhD in AI for work on generating fluent multimodal behaviour of artificial agents. After a postdoc stay at Northwestern University (IL) and a research fellowship at the Center for Interdisciplinary Research ZiF (Bielefeld), he has been deputy coordinator of CRC 673 'Alignment in Communication', principal investigator at the Center of Excellence 'Cognitive Interaction Technology' (CITEC), and chairman of the German Cognitive Science Society (GK). Stefan is internationally renowned and awarded for his interdisciplinary research at the intersection of human communication, embodied-cognitive models of social intelligence, and bootstrapping it with conversational agents or social robots. He has been involved as principal investigator in leading research projects on human/child-robot interaction, multimodal communication, cognitive assistants, or explainable systems. His current research interests centre around the interplay and coordinative dynamics of social cognition and communication in human-agent/robot interaction.

Experiencing social interaction and social communication difficulties is core to a diagnosis on the autism spectrum. Understanding how and why social interaction differences occur between autistic and non-autistic people will facilitate understanding of how difficulties can be reduced. In order to create good models of social interaction and communication processes it is important to study behaviour in real-world contexts. Conducting well designed, rigorous face-to-face studies is practically challenging in many ways but if we fail to do so, our understanding will be incomplete. In this talk, Dr Freeth will consider evidence from a range of different paradigms that aim to assess key processes and constructs involved in face-to-face interactions including consideration of how these can be isolated and investigated. These include 1. the perception of social presence; 2. the perception of agency as social or non-social; 3. the experience of a live vs pre-recorded social encounter. Evidence will be drawn from paradigms that involve structured interactions measuring behavioural responses and tracking of eye movements, using both desk-mounted and mobile eye-trackers. Overall, the evidence presented will demonstrate the potential that face-to-face interaction studies afford in furthering our understanding of the processes underlying social interaction and social communication.

Dr Megan Freeth, The University of Sheffield, UK

Dr Megan Freeth, The University of Sheffield, UK

Dr Megan Freeth is Senior Lecturer in Psychology and Departmental Director of Research and Innovation at The University of Sheffield. She is director of the Sheffield Autism Research Lab. The scope of Dr Freeth's research is broad and includes using eye-tracking to understand mechanisms of social attention, neuroimaging and behavioural studies aimed at gaining insight into neural and cognitive divergence and applied studies aimed at improving understanding of the lived experience of autism and genetic syndromes. Her work has an overarching goal of having a positive impact on the lives of autistic individuals and those with genetic syndromes via advancements in research.

During early life, healthy neurodevelopment depends on warm, responsive and closely-coordinated social interactions between infants and caregivers. These rich multidimensional sensory experiences act through multiple pathways to orchestrate healthy maturation of the neonatal brain, mind and body. Conversely, adverse early life experiences (including abuse or neglect) seed vulnerabilities for poor cognition and emotional instability throughout the lifespan. 

Despite the pivotal role played by social interaction in early development, we still lack medical models and precision tools that can accurately assess a child’s social interactive capacities (their ability to engage in and respond to social input) and measure the health of their social interactions. Similarly, although psychological tools exist to measure children’s cognitive capacities – including intelligence, attention, working memory, and inhibition skills – these tests cannot be used in the first year of life with young infants. Therefore, a needs gap exists in the early assessment and detection of abnormalities in infant psychosocial health and consequent impact on neurocognitive development. We currently miss many, if not most, early warning signs of suboptimal psychosocial development, along with valuable opportunities for prophylactic intervention, before a child’s first birthday.

Here, Associate Professor Leong will discuss neural sociometrics – real-time multi-sensor high-dimensional imaging of adult-infant social interactive behaviour and neurophysiology – as a possible tool and framework for precision measurement of early mental health and neurodevelopment. Early risk identification and mitigation, paired with precision therapeutics, could fundamentally alter a child’s development trajectory toward lifelong mental wellbeing and productivity.

Associate Professor Victoria Leong, Nanyang Technological University, Singapore and University of Cambridge, UK

Associate Professor Victoria Leong, Nanyang Technological University, Singapore and University of Cambridge, UK

Victoria Leong is a developmental cognitive neuroscientist who has pioneered the use of dyadic-EEG to study parent-infant neural dynamics during social interaction, particularly in the context of play, communication and social learning. She is Associate Professor of Psychology and Medicine at Nanyang Technological University, Singapore and Honorary Senior Fellow with the Department of Pediatrics, Cambridge University, UK. She is also Deputy Director of the Cambridge-NTU Centre for Lifelong Individualised Learning which aims to develop neuropersonalised training programmes for flexible lifespan learning. Vicky is a recipient of the Federation of Associations in Behavioral and Brain Sciences Early Career Impact Award (2022), MOE Social Sciences and Humanities Research Fellowship (Singapore, 2021), Parke Davis Exchange Fellowship (Harvard, 2015), Sutasoma Junior Research Fellowship (Cambridge, 2013-15), Cognitive Science Society Glushko Dissertation Prize (2014) and holds awards from the Wellcome Trust, British Academy, UK Economic & Social Research Council, and Rosetrees Medical Trust.

Human music and speech are peculiar behaviours from a biological perspective: although extremely common in humans, at first sight they do not seem to confer any direct evolutionary advantage. In particular, many hypotheses have been proposed for the origins of rhythm, often in connection with vocal learning, a precursor to speech. Because music and speech do not fossilize, and lacking a time machine, the comparative approach provides a powerful tool to tap into human cognitive history. Notably, behaviours that are homologous or analogous to human rhythm and speech can be found across a few animal species and developmental stages. Hence, investigating rhythm across species is not only interesting in itself, but it is crucial to unveil protomusical and protolinguistic behaviours present in early hominids. Here Dr Ravignani suggests how three strands of research – partly neglected until now – can be particularly fruitful in shedding light on the evolution of rhythm and vocal learning. He will present rhythm experiments in marine mammals, primates, and other species, suggesting that rhythm research in non-human animals can also benefit from ecologically-relevant setups, combining strengths and knowledge from cognitive neuroscience and behavioural ecology. Second, he will discuss the interplay between vocal anatomy, (social) learning, and vocal development in harbor seal pups, arguing for their importance as model species for human speech. Finally, he will present computational modeling work on rhythmic and interactive signaling across species. These results suggest that, while many species may share one or more building blocks of speech and music, the ‘full package’ may be uniquely human.

Dr Andrea Ravignani, Max Planck Institute for Psycholinguistics, The Netherlands

Dr Andrea Ravignani, Max Planck Institute for Psycholinguistics, The Netherlands

Andrea is an Independent Group Leader at the Max Planck Institute for Psycholinguistics, where he leads the Comparative Bioacoustics Research Group. He is also Associate Professor at the Center for Music in the Brain, Aarhus University, Denmark. Andrea has studied, researched and worked in several areas, including mathematics, biology, speech sciences, musicology, computer science and cognitive psychology – this multidisciplinarity is mirrored in his research team. Andrea’s current research group at the MPI is highly interdisciplinary, featuring 10 scientists from many disciplines, including cognitive neuroscience, ethology, experimental psychology, linguistics, communication sciences, computer science, AI, bioacoustics, primatology and marine mammalogy.

Andrea’s research, and that of his group, tackles some fundamental questions: Why do we speak? And why are humans such musical animals? Could our abilities to speak and to make music be connected, sharing a common evolutionary history? Andrea investigates the evolutionary and biological bases of music cognition and flexible vocal sound production, and the role they played in the origins of music and speech in our species. His team performs sound recordings and behavioural non-invasive experiments in non-human animals (mostly seals), as a comparative effort to understand the evolutionary history of human capacities. The team complements animal research with human testing, neurobiological evidence, mathematical models, and agent-based simulations.

Andrea has recently been awarded an ERC Starting Grant to investigate the origins of human rhythm using a multi-species and multi-method approach. He firmly believes in and supports kindness in science.

Although emotional contagion is a primitive and foundational feature of social interactions, the underlying neural mechanisms are not well understood. Here we apply a dyadic approach to investigate face-based emotional contagion using hyperscanning and functional near infrared spectroscopy. One participant, 'Video-Watcher', watched emotionally provocative silent videos and the other partner, 'Face-Watcher', watched the face of the Video-Watcher. The library of videos included 'neutrals' featuring landscapes, 'adorables', featuring animal antics, and 'creepy crawlies' featuring spiders, worms and states of decay. Ratings of the positive and negative emotional experience were acquired simultaneously for both participants. The correlation of emotional ratings between the 'Video and Face-Watchers' (r = 0.60) confirmed that the emotion was transmitted by facial expressions and taken as confirmation of emotional contagion.  Facial Action Units of the 'Video Watcher’s' face were taken as the stimuli for the 'Face- Watcher’s' brain. Neural responses included a complex of right TPJ regions consisting of supramarginal gyrus, angular gyrus, and superior temporal gyrus (p < 0.05, FDR corrected peak voxel) consistent with a face-emotion-sensing function. A dyadic model of emotion contagion and face sensing is suggested where human brains and faces are tuned to send and receive emotional states.

Professor Joy Hirsch, Yale University, USA and UCL, UK

Professor Joy Hirsch, Yale University, USA and UCL, UK

Humans, by nature, are irresistibly social. The overarching goal of Professor Hirsch's research is to understand the fundamental neural mechanisms that underlie these social behaviours. Live faces serve as the primary social stimuli for this research. Two-person neuroimaging based on near infrared spectroscopy is configured for real-time live face-to-face hyperscanning including simultaneous measures of facial classifications, eye-tracking, pupillometry, EEG, and behavioural reports of subjective effects. Emerging theoretical frameworks lead to high-level models of multi-function dyadic face and social processes including cross-brain neural synchrony suggesting that brain-to-brain coupled mechanisms are integrated components of dynamic face and social processing. 

In WEIRD (Western, educated, industrialized, rich, democratic) societies, face-to-face interactions are an important component of mother-infant interactions. While facing each other, they engage in a variety of non-verbal means of communication, including facial expressions, such as smiles, and eye contact. It has been suggested that face-to-face interactions are unique to humans, as they are a form of emotional communication indicative of the long-lasting bond between mothers and their infants.

However, other great apes, including the closest relatives of humans, the chimpanzees, also form close and long-lasting relationships with their offspring, but little is known about the role of face-to-face contact in mother-infant interactions. In terms of quantitative measures, research has shown that gazing bouts in chimpanzees are shorter than in human mother-infant dyads. However, less is known about the quality of such face-to-face contact, as it remains an open question what chimpanzee mothers or their infants do while engaging in such interactions. 

Therefore, the group studied face-to-face contact between mothers and their infants in captive chimpanzees and humans from a Western society, to investigate both qualitative and quantitative aspects of their face-to-face interactions. They predicted that 1) human mothers spend more time engaging in face-to-face contact with their infants, and 2) both species vary with regard to the types of behaviours used during such gazing bouts. They used focal observations to video-record spontaneous interactions of ten human and 8 chimpanzee mothers and their 6-months-old infants, and coded facial expressions, facial touches, and the duration of face-to-face contact. If feasible, eye contact was also coded. As predicted, human dyads spend significantly more time in face-to-face interactions than chimpanzees. Species also differed with regard to the behaviours shown during face-to-face contact: human mothers frequently engaged in direct eye contact and to some extent, also mirrored facial expressions of their infant, while chimpanzee mothers often touched the face of their infants, eg, during grooming bouts. Thus, these findings indicate that during face-to-face contact, human mothers use different behaviors to communicate with their infants, while in chimpanzees face-to-face contact seems to be rather a by-product of the mother’s bodily actions on the infant, rather than a communicative situation. 

Professor Katja Liebal, Leipzig University, Germany

Professor Katja Liebal, Leipzig University, Germany

Katja Liebal is a biologist and comparative psychologist, interested the evolution of human language. She uses a comparative approach to investigate if and which building blocks of human language might be already present in our closest relatives, the nonhuman apes. She studies the variability of individual repertoires, the intentional and flexible usage of signals across social contexts to achieve different goals, and how young apes acquire their communicative repertoires. In this context, she also investigates the facial communication in interactions between mothers and their infants, over the infants’ first year of life. She works at Leipzig University, where she is currently heading the LeipzigLab, an interdisciplinary initiative to encourage collaborations between scholars from the humanities and natural sciences.

The game-changer that afforded human culture is widely considered to be language. But turning thoughts into vocalizations was only part of the story. The real innovation was the development of a common language – a sounds-to-meanings map shared across people that enabled one idea to leap from one mind to the next. Conversation provided a platform for minds to meet, allowing information to leap from one brain to the next. But how does conversation work?

We know very little about conversation because the fields of psychology and neuroscience have focused near-exclusively on the individual. In part this was because interacting minds create rich, complex data that were challenging to capture and analyse. However, recent technological advances have made the study of conversation newly tractable.

Here, Professor Wheatley will present research demonstrating how face to face conversation aligns minds by increasing neural synchrony and how neural synchrony, in turn, changes conversation. She will also discuss evidence for why breaking interpersonal synchrony, from time to time, may be important to striking the right balance between shared and independent modes of thought.

Professor Thalia Wheatley, Dartmouth College, USA

Professor Thalia Wheatley, Dartmouth College, USA

Thalia Wheatley is the Lincoln Filene Professor in Human Relations at Dartmouth College, external professor at the Santa Fe Institute, and inaugural Director of the Consortium for Interacting Minds. Her research programme investigates how people communicate with each other, how social networks form, and why social connection is so important for mental health.

Conversation is a collaborative process where speakers and listeners produce information together, continuously coordinating to incrementally co-construct the evolving content. Smooth turn exchange is achieved through tight coordination of interlocutors’ verbal and non-verbal behaviour and becomes problematic when this deviates from expectations. 

Patients with a diagnosis of schizophrenia are one of the most socially excluded in society. It is well documented that they have problems with language and social cognitive skills, including with self-monitoring and turn-taking, yet little research has investigated how these impact interaction. Interactions involving patients offer an opportunity to observe the strategies that people employ when interaction is problematic, and shed light on how ‘normal’ interactions are managed.

Using data from a corpus of triadic conversations containing 20 dialogues involving one patient with a diagnosis of schizophrenia and two healthy controls and 20 dialogues involving three healthy participants, Dr Howes will show that dialogues involving a patient differ from controls in terms of turn-taking, disfluencies, gesture and repair. Furthermore, the presence of the patient influences the behaviour of the healthy controls they interact with. The data supports the idea that disfluencies are communicative solutions, not problems.

This unique data demonstrates that not only are there communication difficulties in schizophrenia but they also impact on social interactions more broadly, thus providing new insights into the social deficits of this complex disorder. 

Dr Christine Howes, University of Gothenburg, Sweden

Dr Christine Howes, University of Gothenburg, Sweden

Christine Howes is a senior lecturer in the Centre for Linguistic Theory and Studies in Probability (CLASP) at the Department of Philosophy, Linguistics and Theory of Science, University of Gothenburg, Sweden. Her main interest is linguistic interaction and how people use the resources provided by language in conversation to coordinate their understanding, with a focus on phenomena that are often considered to be outside the remit of linguistics, such as repair, split utterances and laughter.