Abstract Understanding the somatovisceral responses to auditory affective imagery has important implications in disorders like misophonia. The current study compared physiological responses to aversive and nonaversive states across three modalities: audiovisual, auditory, and auditory imagery. Electromyographic activity over corrugator supercilii (EMGc) and zygomaticus major (EMGz), electrodermal activity (EDA), heart rate (HR), and finger skin temperature (SKT) were measured. There was significant differentiation in EMGc, EDA, and HR deceleration between aversive and nonaversive audiovisual stimuli. EMGc potentiation was the only physiological measure showing consistent differentiation across the three modalities. Cross-modal aversiveness classification results revealed a similar physiological response pattern between audiovisual and auditory modalities. The physiological response pattern during auditory affective imagery was useful for predicting the aversiveness in audiovisual modality but not the other way around. Vividness in auditory imagery correlated with subjective hedonic valence ratings, but not physiological responses. Taken together, the current data suggest that the aversiveness of auditory imagery is differentiable in subjective affective experience and facial muscle potentiation. These results of the physiological responses to imagined aversive sounds in nonclinical population would serve as a comparison baseline for the study of misophonia.