The intricate interplay between visual perception and emotion determines how waking experience influences mentation through a 'day residue' at once conspicuous yet hard to predict. Here we set out to map the neural sources associated with the visuo-affective processing of the 'day residue' during hypnagogic sleep. To this end, we assessed 28 healthy participants on a combined nap protocol with serial awakenings, pre-sleep stimulation with affective visual images, yoked measures of the semantic similarity between image and imagery reports, affect ratings, estimation of 64-channel EEG sources, and functional connectivity analysis. Overall, low-frequency EEG power was associated with weaker residues, and high-frequency EEG power was associated with stronger residues. The source networks most significantly correlated with imagetic and affective residues were markedly different across wake-sleep states, partially overlapping with the default mode network during N1 for up to 50% and 61%, respectively. The results allowed us to identify neural correlates of the visuo-affective processing of the day's residue, showing that the hypnagogic processing of the waking experience involves complex, dynamic and sequential bi-hemispheric interactions among multiple cortical, subcortical, and cerebellar structures with visual, limbic, optokinetic, and cognitive functions.