Emotion perception in speech supports speech comprehension and depends on listeners’ integration of acoustic and lexico-semantic cues. The computational principles underlying this integration remain unclear. Here, we model the perceived speaker emotion as an adaptive internal state that updates continuously during speech perception, revealing temporally evolving weights on acoustic and lexico-semantic features. We presented 90 natural conversational speech excerpts (30–90 s each) from the Buckeye Corpus (Pitt et al. 2007) to 173 listeners, who rated speaker emotional valence continuously over time from most positive to most negative. Participants also rated excerpts without lexico-semantic cues (via low-pass filtering) and without acoustic cues (via reading time-locked transcripts). We find that time-resolved emotional valence ratings for some excerpts are more affected by removing lexico-semantic cues, and others are more affected by removing acoustic cues, implying that listeners flexibly utilize different features to infer speaker emotion at any given moment. We then explicitly map the temporal relationship between acoustic/lexico-semantic features and emotional valence ratings by fitting time-lagged random forest regressors across a range of lag values, identifying the relative weighting between features that drive perceived speaker emotion and resolving the timing of their integration during speech perception.