Emotional responses to sounds are often described along two dimensions: arousal (exciting–calming) and valence (pleasant–unpleasant). Whereas sound level is a robust cue for arousal, the acoustic cues supporting valence are less well defined, particularly across listeners with different hearing abilities. Valence ratings were obtained from 236 adults (with and without hearing loss) for 75 sounds from the International Affective Digitized Sounds corpus (IADS-2). These data were pooled from several studies. For each sound, acoustic characteristics reflecting spectral shape and its time-varying changes were extracted. For each listener, age, gender, pure-tone average hearing threshold (0.5–2 kHz), and anxiety/depression symptom scores were collected. Valence ratings for previously unseen listeners were predicted using an interpretable machine-learning model (gradient-boosted trees with a mixed-effects structure). Out-of-fold performance was moderate (R² = 0.458; Spearman ρ = 0.679). The strongest predictors included variability in spectral “peakiness” (spectral crest), mid-frequency spectral contrast near 1–2 kHz (mean and variability), and loudness variability. Hearing loss also contributed, suggesting audibility and/or altered sound representation affects pleasantness judgments. Overall, listeners appear to rely on time-varying spectral structure, particularly in mid frequencies, when forming pleasant–unpleasant judgments, providing candidate acoustic cues for affective sound processing in listeners with and without hearing loss.