Symposium: Multisensory Integration

Organizer: Charles Spence

List of confirmed speakers in alphabetical order:

David Burr, "Multi-sensory perception of space and time, in adults and in children"

Many studies have shown that our perceptual systems integrate information from different senses optimally. Here I will overview some recent work from our laboratory studying the how visual, haptic and auditory signals about size, orientation, motion and time are integrated perceptually. I will also review recent work showing that the capacity to integrate visuo-haptic information develops late, at about 8 years of age. Evidence from blind and low-vision subjects suggests that the late development could reflect cross-sensory calibration in the developing brain.
Michael Kubovy, "Auditory Objects, Visual Objects, and Audiovisual objects"

I will discuss two kinds of linkages between vision and audition. The first is a duality.The the visual system detects and identifies surfaces; the auditory system detects and identifies sources. Surfaces are illuminated by sources of light; sound is reflected off surfaces. However, the visual system discounts sources, and the auditory system discounts surfaces. These and similar considerations lead to the Theory of Indispensable Attributes that states the conditions for the formation of gestalts in the two modalities. The second linkage involves the formation of audiovisual objects, integrated cross-modal experiences. I will describe research that reveals the role of cross-modal causality in the formation of such objects. These experiments use the canonical example of a causal link between vision and audition: a visible impact that causes a percussive sound.
Ladan Shams, "Crossmodal interactions in visual perception and learning"

Humans are generally considered as visual animals. Visual perception, however, can be strongly and qualitatively altered by other modalities. We have found that sound can radically change visual perception, and ERP, MEG, and fMRI studies show that this alteration can occur at short latencies and as early as primary visual cortex. These findings together with a wealth of other recent findings have established that crossmodal interactions are ubiquitous in human perception and can occur at various levels of processing, including very early stages of sensory processing. Therefore, we asked whether crossmodal interactions play a role in perceptual learning. We compared visual training with auditory-visual training for learning of a low-level visual task, and discovered that auditory-visual training accelerates and enhances performance in a visual task even in the absence of sound. An fMRI study of this multisensory learning suggests that a vast network of brain areas are affected by this type of learning. Altogether these findings implicate that understanding multisensory mechanisms is essential in understanding visual perception and learning.
Salvador Soto-Faraco, "Multisensory contributions to the perception of motion"

There is increasing recognition of the importance of multisensory integration in order to achieve accurate and coherent perceptual representations. Some of the classic demonstrations are based on the perceptual consequences of presenting conflicting information to different senses, such as the ventriloquist illusion or the McGurk effect. However, these traditional lines of research have often focused on spatially static stimulation, in contrast with the highly dynamic nature of everyday life environments. Here I discuss some of our recent work addressing the contribution of multisensory integration processes to the perception of direction and speed of motion. Our initial studies showed that motion direction judgments in one sensory modality (i.e, audition) can be strongly influenced by the direction of motion in other modalities (i.e., vision). As in other multisensory phenomena, these influences are not always bi-directional and extend to other modality combinations. Furthermore, I will present several lines of evidence to support the idea that the integration of motion information has a perceptual basis, and that it cannot be accounted for solely by the local interactions occurring at the level of static information (i.e., the ventriloquist illusion). I will try to draw some conclusions regarding the neural underpinnings of such interactions and, finally, will discuss the relevance of this research in the context of audiovisual media applications and virtual reality environments.
Charles Spence, Jordi Navarra, Argiro Vatakis, Jess Hartcher-O’Brien and Cesare Parise, "The multisensory perception of synchrony"

The last few years have seen a rapid growth of interest in issues related to the temporal aspects of multisensory perception. We will highlight recent research that has investigated people’s sensitivity to temporal asynchrony for both simple (e.g., beeps, flashes, punctuate touch, and laser pain) and more complex stimuli (e.g., speech, music, and object action video clips) using both simultaneity and temporal order judgment tasks. We will review some of the latest findings to have emerged from our laboratory looking at how the brain responds (i.e., adapts) to various kinds of on-going asynchronous stimulation (again using both simple and complex stimuli). Recent findings demonstrating the effect of the “unity effect” on multisensory temporal perception will be outlined, as will research showing that synesthetic correspondences can modulate multisensory integration (both temporal and spatial) in normal participants.