Building on evidence for experience-specific grounding of word meaning and assumed interindividual differences therein, this study investigates how specific aspects of empathy influence the processing and representation of abstract emotional words. We investigated single-trial N400 amplitudes as a measure of semantic retrieval in 78 healthy adults during a lexical decision task with emotion-label, emotion-laden, and neutral abstract words. We further measured the participants’ levels of empathic concern, fantasy, personal distress, and perspective taking. Additionally, ratings on valence, arousal, and emotional experience quantified the words’ emotional representational content. While direct comparison yielded no evidence for N400 differences between word types, N400 amplitudes in response to emotion-label words decreased with increasing fantasy scores, with this modulation being stronger than for emotion-laden and neutral words. Additionally, participants with higher fantasy rated emotional words higher in absolute valence. The observed N400 reductions thus seem to reflect fantasy-driven processing facilitation graded by the words’ emotionality level. In contrast, we found no evidence for N400 modulations by empathic concern, personal distress, or perspective taking while affective ratings on all scales increased with increasing empathic concern scores. Our findings suggest that fantasy facilitates especially emotion-label word processing and empathic concern enriches emotional word meaning representations, respectively, demonstrating interindividual differences in the experiential grounding of emotional abstract concepts.