Colors serve as an important embodied source of perceptual experiences for conceptualizing emotional concepts. This study offers an image-based visual corpus analysis on perceptual (dis)similarities among six basic color terms (black, white, red, green, yellow, blue) and 63 emotion-label words in English. The investigation focuses on two primary questions: (i) whether the six colors are significantly different regarding their contributions in conceptualizing emotional concepts, and (ii) how color-emotion association patterns are shaped by valence and arousal. Empirical analyses were conducted and discussed using the degree of perceptual (dis)similarities between the target color terms and emotion words as an indicator of the strength of color-emotion associations, based on a dataset of 69*100 images collected from Google Image, comprising 63 emotion words and 6 color terms, with 100 images retrieved for each term. The results reveal that (i) significant differences exist among the six colors in their effectiveness at capturing affective (dis)similarities among various emotional concepts, as well as in their association strength with the affective domain, and (ii) the valence and arousal ratings of emotion words are significantly correlated to the strength of their associations with five of the six color terms, with the exception of blue. This study provides an image-based empirical exploration of the cognitive mechanisms responsible for color-emotion mappings, highlighting the essential and distinctive roles of colors in understanding emotional concepts.