This study presents a new computer vision approach to perform affective analysis of a scene or object. The approach utilises a simulation of the phenomenon of face pareidolia that can be described as the perception of non-existent faces, for example, in random textures, clouds or rock formations. The emergence of face pareidolia in product designs and natural scenes can modulate affective perception of our everyday experiences without our conscious awareness. We propose a new deep learning method to simulate the face pareidolia ability of humans and predict associated emotional responses in two-dimensional valence and arousal space. Starting from a face detector that was trained on images of human faces, we propose a novel cross-domain weakly supervised three-step progressive domain adaptation approach to simulate face pareidolia by fine-tuning the human face detector on three types of synthetically generated sample images. Our approach fuses two deep network models, one model for predicting valence and arousal from abstract and minimal face-like patterns producing face pareidolia, and a second for recognising overall moods and context associated with the scene. To evaluate our approach, we constructed a new dataset containing instance-level annotations of face pareidolia occurrences as well as valence and arousal emotional values associated with the overall scene. The quantitative and qualitative experimental results of the present study demonstrate that our approach can outperform other state-of-the-art methods in face pareidolia detection as well as in predicting associated emotions.