The theory of core affect posits that the neural system processes affective aspects of stimuli encountered by the organism quickly and automatically, resulting in a unified affective state described along the dimensions of valence and arousal. Core affect theory posits two functional subsystems that guide affective processing: a sensory integration and a visceramotor network. The proposed study investigates how the representation of affective dimensions depends on sensory modality, features of the task, and brain regions. A series of behavioral studies was run to develop an experimental stimulus set of silent videos and musical clips that met requirements of equating valence across stimulus types while holding arousal constant across valence categories. Valence manipulation was successful, with valence categories were equated on arousal ratings. The stimulus sets in the current study matched many of low level features between valence categories so that any difference between experimental conditions can most likely be attributed to the valence of the stimuli and not to the arousal levels or low level features of the stimuli. The fMRI study applied multiple multivariate analysis tools to analyze the fMRI data. General valence was successfully decoded from patterns of whole brain activation within participants. The successful cross-modal classification demonstrated that there is modality-general processing of valence at the whole brain level. The multidimensional scaling (MDS) results supported these conclusions by showing that a common valence dimension for visual and auditory trials as well as visual- and auditory-specific valence dimensions. The same analyses were applied to the predefined anatomical ROIs (mPFC, OFC, and STS) and revealed modality-general valence processing, evidenced by cross-modal classification and the MDS solution. Successful within-participant cross-modal classifications and unsuccessful cross-participant cross-modal classifications implies that modality-general representation of valence could be individual-specific, whereas, successful within-participant within-modal classifications and successful cross-participant within-modal classifications implies that modality-specific representations of valence might be individual-general. A first searchlight analysis was performed to localize the brain regions that were involved in modality-general valence and it identified three significant clusters: right transverse temporal gyrus, left superior temporal gyrus, and right middle temporal gyrus. These searchlight results were validated with cross-modal classification and MDS. The modality-specific regions found by a second searchlight analysis were in the occipital region for visual stimuli and the temporal region for auditory stimuli, as expected. Within-modality classification confirmed that those modality-congruent areas are involved in valence processing of the corresponding modality. Interestingly, each modality’s valence was also decoded from the modality-incongruent regions. These results imply modality-specific valence valuation for both modalities in each region, because cross-modal classification was not successful in these regions and MDS did not reveal a general valence dimension in either region. In sum, the neural representation of both modality-general and modality-specific valence were found at a whole brain level as well as frontal and temporal regions, consistent with the two system approach to core affect posited by Barrett and Bliss-Moreau (2009). This conclusion was bolstered by converging methodologies.