Symposium: Population encoding for high-dimensional visual representations

Organizer: J. Haxby

List of confirmed speakers in alphabetical order:

John-Dylan Haynes, "Decoding conscious and unconscious visual representations from patterns of human brain activity"

It has recently emerged that the sensitivity of fMRI can be dramatically increased if the full information present in large ensembles of voxels is appropriately taken into account. For example, supervised learning can be used to train a pattern classifier to distinguish between several orientation stimuli viewed by a subject based on the characteristic distributed brain responses they evoke in visual cortex. This holds even though the relevant features are represented at a finer spatial scale than the nominal resolution of single voxels. Here several studies will be presented that apply such supervised learning to the study of conscious and unconscious visual processing in humans. In one study the information about a stimulus that is available to a subject for a perceptual decision is compared to the information that can be decoded from early visual areas. This reveals that V1 has information about stimulus features even when they are rendered completely invisible due to masking, suggesting that V1 can have information about visual stimuli that is not available for conscious access. A second study demonstrates that pattern classification can be used to accurately predict on a second-by-second basis participants' conscious perception while it undergoes many spontaneous changes during binocular rivalry. Importantly, this reveals that the source of predictive information differs between visual areas, being more eye-based in V1 and more percept-based in V3. A third study demonstrates that a subject's behavioral choices can be predicted from different brain regions depending on whether a stimulus is either clearly visible or almost invisible, thus suggesting that the brain adopts different modes of decision making under high and low visibility conditions. Taken together this provides valuable information about the nature of perceptual coding in visual cortex and how such encoding is affected by changes in awareness.
Yuki Kamitani, "Visual image reconstruction from human brain activity: A modular decoding approach"

Perceptual experience consists of an enormous number of possible states. Previous fMRI studies have predicted a perceptual state by classifying brain activity into pre-specified categories. Constraint-free visual image reconstruction is more challenging, as it is impractical to specify brain activity for all possible images. Here, we present a modular decoding approach in which a visual image was assumed to consist of multiscale local image bases (modules). The contrasts of the local image bases were independently decoded from fMRI activity, and then combined to create a reconstructed image. We show that arbitrary binary-contrast images were accurately reconstructed by the decoding model trained with fMRI activity patterns only for several hundred random images. Reconstruction was also used to identify the presented image among millions of candidates. The results suggest that our approach provides an effective means to read out complex perceptual states from brain activity.
Roozbeh Kiani, "Representation of object category structure by the neuronal population of inferior temporal cortex"

Our mental representation of categories has a hierarchical organization and facilitates rapid and effortless categorization of visual objects. I studied the neural underpinning of this representation by recording responses of 674 neurons in inferior temporal cortex (IT) of two macaque monkeys. During the recordings, the monkeys passively viewed several presentations of more than 1000 natural and artificial object images. I used unsupervised clustering methods to investigate the grouping of the images according to the similarity of the responses they elicited in the population of IT neurons. The results revealed a hierarchical representation that approximates our intuitive category structure. Animate and inanimate objects created distinguishable clusters. Within the animate cluster bodies, hands and faces created separate clusters. Each of these was further divided into subordinate categories. The representation of this categorical structure was distributed across the neural population and was highly preserved even when categorically-selective cells were excluded from the population. This distributed representation allows easy and fast retrieval of categorical membership of images because the response patterns were linearly separable for different categories. Finally, the categorical representation seems to be specific to IT; simulation of LGN and V1 responses failed to replicate the hierarchical structure. The experimental results suggest that IT population represents the hierarchy of natural categories through selective development of tunings for complex visual attributes that define these categories.
Gabriel Kreiman, "Fast and robust decoding of visual information from intracranial field potentials in the human visual cortex"

The remarkable pattern recognition abilities of humans and other primates surpass the most sophisticated computational algorithms available today. The difficulty of the recognition problem stems from the need to achieve high selectivity in a fraction of a second while maintaining robustness to object transformations. We quantified, at high temporal resolution, the amount of information conveyed about objects and their transformations by intracranial field potentials from 1494 electrodes in 18 human subjects. Subjects were presented with images containing one or more objects or with movies. Using a statistical classifier, we could accurately decode object category information in single trials as early as 100 ms after stimulus onset. Decoding performance was robust to changes in rotation, scale and clutter. Furthermore, visual information could also be decoded under dynamic viewing conditions and in the presence of background clutter. The results revealed that physiological activity in the human temporal lobe can account for some of the key properties of visual recognition, and they provide strong constraints for computational models of human vision.
Nikolaus Kriegeskorte, "Characterizing categorical and continuous visual-object codes in man, monkey, and computational models with representational similarity analysis"

In man and monkey, inferior temporal (IT) cortex contains visual object representations at the interface between perception and cognition. Although IT has been intensively studied in both species, IT representations of the same objects have never been compared between the species and IT's computational mechanism and role in categorization is not well understood. Here we presented monkeys and humans with the same images of real-world objects and measured the IT response pattern elicited by each image. In order to relate the representations between the species and to computational models, we use representational similarity analysis (RSA). RSA characterizes a neuronal population code by means of a representational dissimilarity matrix. For each pair of experimental conditions (e.g. each pair of stimuli), the representational dissimilarity matrix contains an entry reflecting the dissimilarity of the activity patterns associated with the two conditions. We find that IT response patterns form category clusters, which match between the species. The clusters correspond to animate and inanimate objects; within the animate objects, faces and bodies form subclusters. Within each category, IT distinguishes individual exemplars and the within-category exemplar similarities also match between monkey and human. In the human, categoricality appears to arise early on along the ventral stream, at the level of the lateral occipital complex. A range of computational models we tested did not account for the categorical nature of the object representations in IT, while human behavioral similarity judgments showed even greater categoricality than IT and a similar yet distinct categorical structure, in which the human-nonhuman distinction was more prominent than in IT. Our findings suggest that primate IT across species may host a common code, which combines a categorical and a continuous representation of objects. At the methodological level, RSA allows us to bridge fundamental divides of our science so as to relate human to monkey, cell-recordings to fMRI, and brain-activity data to behavioral measures and computational theory in an information-rich, quantitative framework that is well-motivated by cognitive theory.