Multivariate pattern analysis (MVPA) of fMRI data has allowed the investigation of neural representations of stimuli on the basis of distributed patterns of activity within a brain region, independently from overall brain activity. For instance, several studies on early visual cortex have reported reliable MVPA decoding of the identity of a stimulus representation that was kept in working memory or internally generated, despite the fact that the overall BOLD response was low or even at baseline levels. Here we ask how it is possible that reliable stimulus information can be decoded from early visual cortex even when the overall BOLD signal remains low. We reanalyzed a data set in which human participants (N = 24) imagined or kept in working memory an oriented visual grating. We divided voxels from V1, V2, and V3 into groups based on orientation preference, and compared the time course of mean BOLD responses to preferred and non-preferred orientations with the time course of the multivariate decoding performance. Decoding accuracy related to a numerically small, but reliable univariate difference in the mean BOLD response to preferred and non-preferred stimuli. The time course of the difference in BOLD responses to preferred and non-preferred orientations was highly similar to the time course of the multivariate pattern classification accuracy. The reliability of the classification strongly correlated with the magnitude of differences in BOLD signal between preferred and non-preferred stimuli. These activity differences were small compared to the large overall BOLD modulations. This suggests that a substantial part of the task-related BOLD response to visual stimulation might not be stimulus-specific. Rather, stimulus-evoked BOLD signals in early visual cortex during a task context may be an amalgam of small stimulus-specific responses and large task-related but non-stimulus-specific responses. The latter are not evident during the maintenance or internal generation of stimulus representations, but provide an explanation of how reliable stimulus information can be decoded from early visual cortex even though its overall BOLD signal remains low.
