Outcome: Researchers from Carnegie Mellon University have shown that it is possible to reconstruct a novel face image based on the observer's behavioral or neural response to a very large set of homogeneous faces (Nestor, A., Plaut, D. C. and Behrmann, M.). From a practical perspective, these findings make possible a broad range of image-reconstruction applications via a straightforward methodological approach and, from a theoretical perspective, the current results provide key insights into the nature of high-level visual representations.
Impact/benefits: The present work establishes a novel approach to the study of visual representations. This approach allows researchers to estimate the structure of human face space as encoded by high-level visual cortex, to extract image-based facial features from this structure and to use such features for the purpose of facial image reconstruction. The derivation of visual features from empirical data provides an important step in elucidating the nature and the specific content of face representations. Further, the integrative character of this work sheds new light on the existing concept of face space by rendering it instrumental in image reconstruction. Last, the robustness and generality of the reconstruction approach is established by its ability to handle both neuroimaging and psychophysical data.
Explanation: The reconstruction of images from neural data can provide a unique window into the content of human perceptual representations. While recent efforts have established the viability of this enterprise using functional magnetic resonance imaging (fMRI) patterns, these efforts have relied on a variety of prespecified image features. Here, we take on the twofold task of deriving features directly from empirical data and of using these features for facial image reconstruction. First, we use a version of reverse correlation to derive visual features from fMRI patterns elicited by a large set of homogeneous face exemplars. Then, we combine these features to reconstruct novel face images from the corresponding neural patterns. This novel approach allows us to estimate collections of facial features associated with different cortical areas as well as to achieve significant levels of reconstruction accuracy. Furthermore, we establish the robustness and the utility of this approach by reconstructing face images from patterns of behavioral data.
Participants viewed a set of 120 face images (60 identities x 2 expressions), carefully controlled with respect to both high-level and low-level image properties. Each image was presented at least 10 times per participant across 5 fMRI sessions using a slow event-related design and participants performed a one-back identity task across variation in expression. Cortical areas that exhibited separable patterns of activation to different facial identities were first demarcated. Then confusability matrices from behavioral and neural data in these areas to determine the general organization of face space were constructed and visual features accounting for this structure were extracted by means of a procedure akin to reverse correlation. Last, the very same features were used for the purpose of face reconstruction.
The results revealed that: (i) a range of facial properties such as eyebrow salience and skin tone govern face encoding; (ii) the broad organization of behavioral face space reflects that of its neural homologue, and (iii) high-level face representations retain sufficient detail to support reconstructing the visual appearance of different facial identities from either neural or behavioral data.
Legend: Examples of face stimuli and their reconstructions from behavioral and fMRI data for neutral facial expressions. Numbers in the top corners of each reconstruction show its average experimentally-based accuracy (green, left corner) along with its image-based accuracy (red, right corner).
Nestor, A., Plaut, D. C. and Behrmann, M. Feature-based face representations and image reconstruction from behavioral and neural data. Proc Nat Acad Sci, in press.
(NSF Highlight 2015-2016)