Buildings shape how people feel, yet the mechanisms through which specific facade properties drive affective states remain empirically underspecified. Here we introduce the Cambridge Facade Affect Dataset (CFAD), 86 orthogonally rectified facade images annotated with continuous arousal and valence ratings from 85 participants, and establish a validated pipeline linking machine-vision-derived surface metrics to human affective responses. Focusing on three quantifiable attributes, complexity, transparency (window-to-wall ratio), and materiality (proportion of natural versus artificial surface composition), we show that perceived complexity is the dominant affective predictor, with significant positive associations for both arousal (beta = 0.507, p < 0.001) and valence (beta = 0.376, p < 0.001) and a curvilinear amplification at higher complexity levels. Transparency exhibits an inverted-U relationship with valence, while increasing surface artificiality suppresses arousal and reduces pleasantness consistent with biophilic response theory. Critically, machine-derived metrics show limited direct predictive power over affective outcomes; mediation analyses reveal that human perceptual evaluation functions as a necessary intermediate layer, with perceived materiality significantly mediating the machine-valence relationship (indirect effect = -0.205, p = 0.003). Cross-context validation demonstrates moderate stability of complexity and materiality ratings across image-based and in-situ conditions, while affective responses, particularly valence, exhibit significant context-dependence (ICC = 0.332). These findings advance facade research from descriptive morphological analysis toward predictive, perception-grounded modelling, and provide an empirically validated basis for affect-informed design of the urban environment.