Scoring facial attractiveness with deep convolutional neural networks: How training on standardized images reduces the bias of facial expressions

Abstract

Objective: In many medical disciplines, facial attractiveness is part of the diagnosis, yet its scoring might be confounded by facial expressions. The intent was to apply deep convolutional neural networks (CNN) to identify how facial expressions affect facial attractiveness and to explore whether a dedicated training of the CNN is able to reduce the bias of facial expressions. Materials and Methods: Frontal facial images (n = 840) of 40 female participants (mean age 24.5 years) were taken adapting a neutral facial expression and the six universal facial expressions. Facial attractiveness was computed by means of a face detector, deep convolutional neural networks, standard support vector regression for facial beauty, visual regularized collaborative filtering and a regression technique for handling visual queries without rating history. CNN was first trained on random facial photographs from a dating website and then further trained on the Chicago Face Database (CFD) to increase its suitability to medical conditions. Both algorithms scored every image for attractiveness. Results: Facial expressions affect facial attractiveness scores significantly. Scores from CNN additionally trained on CFD had less variability between the expressions (range 54.3–60.9 compared to range: 32.6–49.5) and less variance within the scores (P ≤.05), but also caused a shift in the ranking of the expressions' facial attractiveness. Conclusion: Facial expressions confound attractiveness scores. Training on norming images generated scores less susceptible to distortion, but more difficult to interpret. Scoring facial attractiveness based on CNN seems promising, but AI solutions must be developed on CNN trained to recognize facial expressions as distractors.