Facial expressions provide critical details about social partners' inner states. We investigated whether event-related potentials (ERP) related to the visual processing of facial expressions are modulated by participants' perceived arousal and valence at the stimulus level. ERPs were recorded while participants (N = 80) categorized the gender of faces expressing fear, anger, happiness, and no emotion. Participants then viewed each face again and rated them on arousal and valence using 1-9 Likert scales. For each participant, ratings of each unique face were linked back to corresponding ERP trials. ERPs were analyzed at all time points and electrodes using hierarchical mass univariate statistics. Three different ANOVA models were employed: the original emotion model, and models with valence or arousal ratings as trial-level regressors. Results from models with ratings highly overlapped with the original model, although they were more temporally restricted. The N170 component was the most impacted by arousal and valence ratings, with four out of six emotion contrasts revealing significant valence or arousal interactions. Emotion effects on the P2 component were mostly unrelated to ratings. On the EPN component, only two contrasts related to both arousal and valence ratings. Thus, ERP emotion effects are related to participants' perceived arousal and valence of the stimuli, although this association depends on the contrast analyzed. These findings, their limitations, and generalizability are discussed in reference to existing theories and literature.