Bridging senses: new developments in synaesthesia

09:00-09:10 Welcome by the Royal Society and Professor Simon E Fisher

09:10-09:40 Synaesthesia genetics: past, present and future perspectives

Professor Simon E Fisher, Max Planck Institute for Psycholinguistics, The Netherlands

Abstract

Writing in the 1880s, Francis Galton was intrigued by synaesthesia, observing that it tended to cluster in families. This talk will describe how views of the molecular underpinnings of this fascinating trait have evolved over the years that followed these early descriptions. It will introduce the diverse methods that have been used to gain insights into synaesthesia genetics, ranging from twin studies and linkage analysis, to case-control screens and next-generation DNA sequencing, while highlighting the promise and pitfalls of each approach. Initial work in this area hoped for explanation in terms of a single gene (or perhaps a few genes) of large effect, but it has become clear that the genomic architecture underlying synaesthesia is complex and multifactorial. Nonetheless, while there is significant heterogeneity between families/cases in terms of the specific genes involved, the first clues from studies of rare gene variants appear to highlight shared pathways. Ongoing investigations in extended families, as well as large numbers of unrelated cases, should help clarify this point. Crucially, the identification of genetic factors robustly associated with synaesthesia can open up new avenues of research, giving novel entry points into neurobiological mechanisms that are important for the way we experience the world.

• Listen to the audio (mp3)

09:40-10:10 Children with synaesthesia

Professor Julia Simner, University of Sussex, UK

Abstract

People with synaesthesia experience their senses in unusual ways. For example, they might see colours when reading numbers, or experience tastes in the mouth when they hear sounds. These experiences start early in childhood - so what is it like to be a child with synaesthesia? Simner will describe findings from her ERC-MULTISENSE research project on childhood synaesthesia, which tested over 7 thousand children between the ages of 5 and 12 years of age and gave a battery of tests to the children, to their parents and to their teachers. She’ll describe a number of her novel tools which can diagnose a range of synaesthesias including those that trigger colours (eg the 7 is red), tastes (eg 7 tastes bitter), and even personality traits (eg 7 is a bossy child). She’ll describe some of her findings related to the prevalence of synaesthesia in schools, and the impact of childhood synaesthesia on literacy, numeracy, learning, creativity, personality, happiness and health-related well-being. She’ll also evaluate how schools might better equip themselves to deal with the needs of synaesthetic children and how the identification of synaesthesia early in childhood might bring about beneficial changes in later-life well-being.

• Listen to the audio (mp3)

10:10-10:40 Discussion

10:40-11:00 Coffee

11:00-11:25 Associations in Japanese grapheme-colour synaesthesia: from the perspective of grapheme learning

Associate Professor Michiko Asano, Rikkyo University, Japan

Abstract

Grapheme-colour synaesthetic association is consistent over time. Although graphemes do not elicit the same colour across different synaesthetes, some regularities in synaesthetic experience have been reported. In this talk, the authors present their recent work on factors that affect synaesthetic colours for Japanese Kanji characters (logographic characters). In the study, they first explored the influence meaning has on synaesthetic colours for Kanji characters representing abstract meanings by examining synaesthetic colours for antonym pairs (i.e. characters with meanings opposed to each other) in Japanese synaesthetes. Results show that antonym character pairs that were learned in lower school grades elicited synaesthetic colours that were dissimilar from each other. This suggests that semantic relations are reflected in synaesthetic colours, at least in the early stages of learning abstract Kanji characters. Second, the effect that learning new sounds or meanings of graphemes has on synaesthetic colours for those graphemes was examined. Japanese synaesthetes were taught new sounds or new meanings for familiar Kanji characters. Results indicate that the pre- and post-learning consistency of synaesthetic colour choice was slightly, but significantly, lower for Kanji characters for which new sounds/meanings were taught than for those for which no novel information was taught. This suggests that synaesthetic colours can be modulated to reflect the synaesthete’s latest knowledge about graphemes. Implications of these findings will be discussed in relation to the model of synaesthetic grapheme-colour association by Asano and Yokosawa (2013), which takes into account the developmental process of grapheme learning.

11:25-11:50 Rereading rainbows: The role of meaning and morphology in grapheme-colour synaesthesia

Dr Jennifer Mankin, University of Sussex, UK

Abstract

In the years since synaesthesia was proposed to be an essentially psycholinguistic phenomenon (Simner, 2007), a variety of research has begun to investigate the complex relationship between language and grapheme-colour synaesthesia. By identifying the elements of language that are reflected in synaesthetes’ colour experiences, researchers gain valuable insight into the underlying structure and processing of words. This talk explores how grapheme-synaesthesia can be used as a tool to test and investigate psycholinguistic theories.

In particular, this talk focuses on two aspects of words: structure and meaning. The former investigates how synaesthetic colours reflect the morphological structure of words – for example, how the synaesthetic colours of rain and bow result in the colour(s) of rainbow. These correspondences between colour and morphology may also provide insight into the processing of other morphological processes, such as affixing (eg re- + read = reread). However, words also convey meaning, and in some cases these meanings are already intrinsically coloured, such as the green of leaf or the blue of sky. Contrasting such canonically coloured words with similarly spelled control words (eg fire vs fine) shows that the synaesthetic colours of words are also influenced by the conceptual, canonical colours inherent in those words. The conclusion addresses some substantial gaps remaining in the psycholinguistic study of synaesthesia, particularly fundamental questions about the everyday experience of synaesthetes using language that will provide essential context for future research.

• Listen to the audio (mp3)

11:50-12:15 Discussion

12:15-13:15 Lunch

13:15-13:40 Cognitive differences in synaesthesia as a window into neurodevelopmental Mechanisms

Professor Jamie Ward, University of Sussex, UK

Abstract

This talk will give an overview of differences in cognitive ability between synaesthetes and non-synaesthetes; it will discuss whether these differences can potentially serve as a unifying behavioural marker for different varieties of synaesthesia; and whether they speak to the issue of whether synaesthesia is categorical in nature or continuous with neurotypical cognition. Specifically, evidence will be presented that people with synaesthesia show enhanced episodic memory, more vivid mental imagery, and show certain enhancements in perception. The latter resemble those found in autism. These traits are often shared across different kinds of synaesthesia and can even show dose-like effects: the more kinds of synaesthesia a person has, the more extreme the scores. It will be argued that these differences may be part of the neurodevelopment ‘start-up kit’ for synaesthesia, but that possession of these traits alone does not make someone a synaesthete (not even partially). Instead, it is proposed that synaesthesia is a discrete category that forces us to consider that there are multiple ways of being ‘normal’ – a lumpy rather than continuous space of neurodiversity.

• Listen to the audio (mp3)

13:40-14:05 Synaesthesia and autism spectrum disorder: shared characteristics of visual perception

Dr Tessa van Leeuwen, Radboud University, The Netherlands

Abstract

Synaesthesia is a mixing of the senses, eg letters elicit colour. Approximately 20% of people with autism spectrum disorder (ASD) have synaesthesia: the reason for this high co-occurrence is unclear, but it is likely that underlying neural mechanisms are at least partly shared. In several experiments it was investigated whether perceptual alterations in synaesthetes resemble those reported for ASD. Visual perception tests were conducted and autistic traits assessed in >60 synaesthetes and >40 non-synaesthetes. Synaesthetes revealed stronger autistic traits on subscales of the Autism-Spectrum Quotient and reported increased sensory sensitivity compared to controls (Glasgow Sensory Questionnaire). Interestingly, in non-synaesthetes, the degree of synaesthesia as expressed in a consistency score correlated positively with AQ scores, suggesting a relationship between these traits in non-synaesthetes. Similar to reports in the ASD literature, motion coherence thresholds of synaesthetes were elevated (p<.001) suggesting that synaesthetes are less sensitive to global motion patterns. Motion coherence thresholds correlated with the sensory sensitivity score in synaesthetes (p<.001). Similar to reports of enhanced detail processing in ASD, synaesthetes made less errors than controls on an Embedded Figures Task (p<.05). Altogether the results point to a bias towards detail processing in synaesthetes, which is also typical of visual perception in ASD. Atypical sensory processing may be a shared feature of synaesthesia and ASD. Implications of the findings will be discussed.

14:05-14:30 Discussion

14:30-14:50 Tea

14:50-15:15 The colours of our mind: exploring mental imagery and synaesthesia

Dr Mary Jane Spiller, University of East London, UK

Abstract

Our ability to imagine a visual scene or to hear music in our minds is a common everyday experience. For some people these mental images are extremely clear and vivid, whereas others report no image-like experience and merely the awareness that they are thinking about the image content. Research suggests that people who have synaesthesia have particularly vivid mental imagery, and that this may be linked to the senses involved in the synaesthesia.  For instance, people who experience colours with smells may have more vivid imagery for smells. Other research has looked at whether synaesthesia can be experienced from a visual image formed in the mind. For example, whether imagining a letter can induce the colour experience usually induced upon seeing such a letter. This talk will provide an overview of the research to date concerning mental imagery and synaesthesia. It will highlight key areas for future research including a focus on individual differences and the determining factors of the synaesthetic experience. Overall, the research discussed will raise important questions about the potential impact of our perceptual experiences on the development and capabilities of our visual spatial cognition.

• Listen to the audio (mp3)

15:15-15:40 The impact of synesthesia and multisensory stimulation on memory

Professor Edward Hubbard, University of Wisconsin-Madison, USA

Abstract

Dr Hubbard will review a series of studies that his group has recently completed examining the impact of synesthesia on memory.  Although there is now a substantial literature, spanning from Luria to modern investigations, demonstrating the impact of synesthesia on long-term memory (LTM), studies of earlier stages of memory remain sparse.  They propose that LTM advantages may arise from advantages during encoding arising at earlier stages of memory.  To test this hypothesis, they investigated the impact of grapheme-color synesthesia on three different stage of memory, from early visual sensory memory (referred to as “iconic memory”), visual working memory (WM), and long-term memory.  In all three studies, they found a synesthetic memory advantage.  Critically this advantage was most prominent when memory was stressed by asking participants to remember large arrays.  These findings suggest that memory load may be important for understanding some of the discrepancies in the literature, as some of the failures to find differences may not have stressed memory enough to reveal a synesthetic memory advantage.  They then frame these findings in terms of classic dual-coding theories (Pavio, 1971; Pavio, 1991) and the “sensory memory” account of working memory (see, eg Postle and D’Esposito, 2015 and a recent debate in TICS Xu, 2017; Gayet, Paffen & Van der Stigchel, 2018; Scimeca, Kiyonaga & D’Esposito, 2018; reply by Xu, 2018) and extend these investigations to multisensory stimulation in non-synesthetes.

• Listen to the audio (mp3)

15:40-16:00 Discussion

16:00-18:00 Poster session and drinks reception

09:30-09:40 Welcome by Professor Simon E Fisher

09:40-10:05 Investigating the relationship between grapheme-colour synaesthesia and genetic risk for neuropsychiatric disorders

Dr Amanda Tilot, Max Planck Institute for Psycholinguistics, The Netherlands

Abstract

Synaesthesia is a neurological phenomenon that is likely influenced by genetic factors. Reports of families with multiple generations of synaesthetes span over 130 years, with recent data pointing toward rare variation in genes related to axonogenesis. A study on the broader neurological profile of synaesthesia found evidence for phenotypic overlap with the sensory abnormalities present in individuals with autism, a condition with a complex genetic basis including both rare and common variation. Geneticists at the MPI for Psycholinguistics began recruiting grapheme-colour synaesthetes in 2013, and today the collection includes over 750 DNA samples, including contributions from a worldwide network of collaborators. A majority of the sample is from the UK and the Netherlands, and all subjects passed standard consistency tests for grapheme-colour synaesthesia. Where well-powered genome wide association studies exist for a condition, a person’s aggregate genetic risk for that condition can be estimated from their balance of risk increasing and decreasing variants and the relative size of their effects (a ‘polygenic risk score’). At the meeting, Dr Tilot will share the results of an analysis comparing polygenic risk for multiple neuropsychiatric disorders across unrelated individuals with and without grapheme-colour synaesthesia. These results will provide a first look at the role of common genetic variation in synaesthesia and may enhance our understanding of the mechanisms behind shared sensory experiences in synaesthesia and autism.

10:05-10:30 A unifying model for the emergence of synaesthetic associations

Associate Professor Kevin Mitchell, Trinity College Dublin, Republic of Ireland

Abstract

There is strong evidence that synaesthesia is a genetic condition affecting some aspects of cortical development. Resultant differences in cortical connectivity can explain the emergence of mostly arbitrary cross-sensory percepts and associations. However, there also seems to be some role for learning of such associations from the environment or from semantic cues. A model is presented that unifies the factors of altered development and associative learning. This model can explain how synaesthetic associations arise during early critical periods, as well as the observed trends in specific pairings. Once formed, such associations become incorporated into the schemas of the inducing objects and remain extremely stable over long time frames. This is observed even in cases where the conscious synaesthetic experience has been suppressed due to injury or medication.

• Listen to the audio (mp3)

10:30-11:00 Discussion

11:00-11:20 Coffee

11:20-11:45 Commonalities and differences between synesthetic and non-synesthetic multimodal associations

Dr Romke Rouw, University of Amsterdam, The Netherlands

Abstract

A person with grapheme-color synesthesia experiences colors with alphanumerical symbols. While this is an extraordinary trait, the degree and manner in which synesthetes are ‘different’ is topic of debate. There are many examples of multisensory associations that are present in all of us (even though we might not be aware of them). I will present evidence of shared associations between synesthetes and non-synesthetes, and show how certain associations may be universal. Yet, there are also neurobiological and behavioral characteristics that set synesthetes apart from non-synesthetes. Moreover, only synesthetes have the remarkable sensory experiences resembling ‘real’ sensations. To understand these seemingly conflicting results, we need a better understanding of what a ‘synesthetic predisposition’ entails. First, having this predisposition may have neurological consequences; we obtained (in non-synesthetes) neural markers related to having synesthesia in the family. Second, a synesthetic predisposition may cause a differential response to a common environmental factor; we found that a shared environmental influence in early childhood can cause different associations in adulthood, in synesthetes as compared with non-synesthetes. Overall, differences between synesthetes and non-synesthetes may not be present in having cross-modal associations per se, but instead in how these associations are ‘locked in’, in brain and behavioural responses.

11:45-12:00 Discussion

12:00-13:00 Lunch

13:00-13:25 Synaesthesia, multisensory integration and imagery: associated or independent processes to bridge the senses?

Professor Fiona Newell, Trinity College Dublin, Ireland

Abstract

Although cross-modal interactions in the brain are often assumed to mediate synaesthesia, relatively few investigations have attempted to elucidate the nature of these interactions. Synaesthesia can be driven by integrated, multisensory rather than unisensory inputs, which is consistent with growing evidence that multisensory interactions are the norm, both within the brain and on behaviour. Moreover, evidence for cross-modal, synaesthetic-like associations in non-synaesthetes as well as enhanced multisensory processes in synaesthetes suggests that synaesthesia may be scaffolded onto multisensory processes which are common to all. However, this line of research has primarily focussed on cross-modal interactions that lie outside of synaesthetic experiences and has yet to address whether synaesthesia is associated with differences in how related sensory information is processed. In particular, the phenomenology and prevalence of synaesthesia suggests that cross-modal interactions associated with synaesthesia may be discrete, and not cognitively penetrable. However, evidence that synaesthesia can also be triggered by a mental image of the inducing stimulus challenges the idea that synaesthesia relies on discrete processes. Mental imagery itself has also been shown to induce multisensory illusions, but its role in broader synaesthetic experiences is mainly unknown. This can largely be attributed to our relatively poor understanding of mental imagery in sensory domains beyond vision. Current investigations are aimed at exploring individual differences in the ability to imagine stimulation across other sensory domains. Such abilities may, in turn, be linked with specific types of synaesthesia thus providing further insight into the cross-modal mechanisms underpinning synaesthesia.

13:25-13:50 Synesthesia evoked through mild sensory deprivation

Assistant Professor David Brang, University of Michigan, USA

Abstract

The natural environment contains auditory and visual cues, conveying both redundant and unique information about multisensory objects and events. To facilitate the processing of this information, the brain integrates auditory and visual signals, leading to better detection and response. While these multisensory benefits can result from one sensory system influencing another (eg, the spatial position of a tone modulates neural activity in areas of visual cortex that process similar regions of space), tones presented in isolation do not typically evoke conscious visual experiences. In individuals with synesthesia, however, these multisensory interactions do lead to qualitatively different experiences such as tones evoking flashes of light. Why, if multisensory interactions are present in all individuals, do only synesthetes experience multisensory hallucinations? Models of synesthesia propose that this difference is due to either reduced inhibition or increased connectivity between associated modalities in synesthetes. Case reports suggest that non-synesthetes can experience these sensations through drug ingestion, raising the possibility that synesthesia exists as a latent feature in all individuals, manifesting only when the balance of activity across the senses has been altered. This talk with presents data from a series of studies showing that aspects of generic multisensory interactions present in all individuals operate at early sensory levels, and that perturbation of these networks can lead to auditory-visual hallucinations (synesthesia) in the general population, providing tentative links between these conventionally distinct processes.

• Listen to the audio (mp3)

13:50-14:15 Discussion

14:15-14:35 Tea

14:35-15:00 Using mirror-sensory synaesthesia to examine how we perceive and understand others

Professor Michael Banissy, Goldsmiths, University of London, UK

Abstract

Our capacity to share the experiences of others is a critical part of human behaviour. One process thought to be important for this is vicarious perception - the ability to co-represent the experiences of other people by matching the observed state onto representations of our own first-hand experience. For example, observing pain in other people activates some of the same network of brain regions as the first-hand experience of pain. The degree of vicarious perception is contextually and socially embedded. It has thus been used as a model system for exploring the broader mechanisms that underpin inter-personal representations and complex phenomena such as empathy. For most of us vicarious perception is implicit (i.e. unconscious), but for some individuals viewing another person’s state results in them literally experiencing a conscious sensation of the observed event (known as mirror-sensory synaesthesia). This talk will discuss what factors contribute to mirror-sensory synaesthesia and ways in which mirror-sensory synaesthesia can be used to enhance our knowledge of how we understand the experiences of others.

• Listen to the audio (mp3)

15:00-15:25 Neurophenomenological investigations of natural and trained synaesthesia

Professor Anil Seth, University of Sussex, UK

Abstract

Synaesthesia offers a unique window into the neurocognitive mechanisms underlying conscious perception. Intriguingly, recent evidence suggests that extensive perceptual and cognitive training can induce synesthesia-like experiences in non-synaesthetic adult volunteers (Bor et al, Scientific Reports, 2014, 4:7089). On the other hand, neuroimaging findings of colour-selective responses during natural (grapheme-colour) synaesthesia have been inconsistently reported. Focusing on grapheme-colour synaesthesia, I will describe a series of studies linking neural responses to phenomenology in both natural and ‘trained’ synaesthesia. For natural synaesthetes, our results show that colour-specific specific brain responses can be predicted by individual differences in synaesthetic phenomenology captured by ‘localisation’ and ‘automaticity’. For trained non-synaesthetes, we find coordinated phenomenological, behavioural and neurophysiological changes following a battery of adaptive cognitive training, revealing an unexpectedly powerful capability for perceptual plasticity even in adults. Finally, I will highlight an overlooked property of synaesthesia, which is that synaesthetic concurrents usually lack perceptual ‘presence’; that is, they are not experienced as being part of the external world. A theory based on counterfactual predictive processing suggests why this might be so.

• Listen to the audio (mp3)

15:25-15:50 Discussion

15:50-17:00 Synthesis discussion

Professor Simon E Fisher, Max Planck Institute for Psycholinguistics, The Netherlands