Unraveling emotional signatures: comparing physiological methods and algorithm-based recognition of spontaneous emotional facial expressions

Abstract

Recognizing others emotions is central to social interaction. Traditional biological psychology infers emotional responding via laboratory measures, whereas contemporary computer vision algorithms claim to identify emotions unobtrusively from facial video. However, the validity of such algorithms for classifying spontaneous emotional responses occurring without explicit communicative intent remains debated. We compared established psychophysiological measures (EEG, facial EMG, EDA activity) with the open-source facial behavior toolkit OpenFace for classifying participants spontaneous responses during free viewing of happiness-inducing, disgust-inducing, and neutral pictures. Participants provided valence and arousal ratings and later selected the basic emotion that best matched their reaction which served as the classification criterion. Using within-participants single-trial support vector machine (SVM) classification, EEG achieved the highest accuracy (40%), followed by facial EMG (37%); OpenFace reached 36%. All methods except EDA exceeded chance performance (33.3%) and were lower compared to human raters (48%). Predictions declined slightly for across-participants SVMs, being at chance for OpenFace and EDA. The results indicate that in principle both, psychophysiological measures and video-derived facial action units, can capture diagnostically relevant aspects of emotional responding during picture viewing, but that their performance is limited when expressions are spontaneous and not produced for communicative purposes. Inter-individual variability in expressivity and physiological responding likely contributes to these limitations and should be considered when deploying automatic emotion recognition in research or applied settings.