We carry out two studies on affective state modeling for communication settings that involve unilateral intent on the part of one participant (the evoker) to shift the affective state of another participant (the experiencer). The first investigates viewer response in a narrative setting using a corpus of docu-mentaries annotated with viewer-reported narrative peaks. The second investigates affective triggers in a conversational set-ting using a corpus of recorded interactions, annotated with continuous affective ratings, between a human interlocutor and an emotionally colored agent. In each case, we build a “one-sided ” model using indicators derived from the speech of one participant. Our classification experiments confirm the viabil-ity of our models and provide insight into useful features. Index Terms: affect, speech recognition, audio analysis, natural language communication