Encoding and retrieval of emotionally arousing stimuli depend on the activation of multiple interconnected brain regions, with people showing differences in their individual strength of emotional perception and recollection. Understanding the association between these brain regions and the behavioral outcome might therefore have important clinical implications as dysfunctional emotional memory processes are characteristic of many psychiatric disorders. Based on behavioral and fMRI data collected from healthy young adults (N = 1'385), we investigated brain activation patterns, arousal ratings and memory performance during encoding and retrieval of negative and neutral pictures. We performed multi-voxel pattern analysis (MVPA) and voxel-wise association analyses. Subjects' individual strength of perceived arousal at encoding and subjects' memory performance at recognition could be predicted from the fMRI data of the respective tasks by using a topographically identical network of brain regions. This network was mainly left lateralized including dense clusters of voxels in the occipital and parietal lobe and including the amygdala. Voxel-wise association analyses confirmed the close link between the brain activation of both tasks and their relation to the respective behavioral outcome. These results point to the importance of the here identified brain network for emotional memory processes in health and, possibly, disease.