Emotion plays a significant role in goal-directed behavior, yet its neural basis is yet poorly understood. In several psychological models the cardinal dimensions that characterize the emotion space are considered to be valence and arousal. Here 3T functional magnetic resonance imaging (fMRI) was used to reveal brain areas that show valence- and arousal-dependent blood oxygen level dependent (BOLD) signal responses. Seventeen healthy adults viewed pictures from the International Affective Picture System (IAPS) for brief 100 ms periods in a block design paradigm. In many brain regions BOLD signals correlated significantly positively with valence ratings of unpleasant pictures. Interestingly, partly in the same regions but also in several other regions BOLD signals correlated negatively with valence ratings of pleasant pictures. Therefore, there were several areas where the correlation across all pictures was of inverted U-shape. Such correlations were found bilaterally in the dorsolateral prefrontal cortex (DLPFC), dorsomedial prefrontal cortex (DMPFC) extending to anterior cingulate cortex (ACC), and insula. Self-rated arousal of those pictures which were evaluated to be unpleasant correlated with BOLD signal in the ACC, whereas for pleasant pictures arousal correlated positively with the BOLD signal strength in the right substantia innominata. We interpret our results to suggest a major division of brain mechanisms underlying affective behavior to those evaluating stimuli to be pleasant or unpleasant. This is consistent with the basic division of behavior to approach and withdrawal, where differentiation of hostile and hospitable stimuli is crucial.