Humans often make summarized visual judgments about previously experienced affective situations to inform future decisions. However, these summarized judgments are subject to an overestimation bias: Negative events are recalled as more negative and positive events as more positive than they truly were. It is currently unknown whether the strength of overestimation bias in affective judgments varies across observers. If this overestimation bias represents an observer-specific cognitive trait, it should display idiosyncratic and stable individual differences. Here, we investigated whether the overestimation bias in perceived affect is idiosyncratic and stable within observers across days and different stimuli. Using a novel continuous psychophysics measure of perceived affect, observers continuously tracked, in real-time, the affect of people in videos using a two-dimensional valence-arousal rating grid. At the end of each video, participants then reported what they believed to be the average affect of the previously tracked person. By comparing observers' continuous ratings with the average affect reported at the end of the video, we found that observers often overestimated the affect in their summarized judgments. Importantly, the strength of the overestimation bias was unique to each observer and stable across days and across different sets of videos. Our findings also highlight the value of the continuous psychophysical affect tracking paradigm: Continuous affect tracking was reliable and accurate, with high between-observer agreement, and it can be collected both online and in the lab. Together, our results suggest that continuous affect tracking is a powerful approach to isolate and identify idiosyncratic perceptual and cognitive mechanisms of affect understanding.