Audiovisual Information Processing in Emotion Recognition: An Eye Tracking Study

Abstract

In audiovisual information processing, auditory information may interfere with eye movement planning in visual processing due to competition for attentional resources. Here we hypothesize that this interference may be mitigated in the recognition of emotions involving strong audiovisual coupling. Participants judged the emotion of a talking head video under audiovisual, video-only, and audio-only conditions. While participants generally performed the best in the audiovisual condition, their eye movement pattern did not change significantly across the three conditions except for the recognition of disgust. In disgust recognition, eye movements in the audiovisual condition were less eyes-focused than the video-only condition, and the larger the difference, the less the audiovisual advantage in performance. Disgust recognition develops later in life and may involve weaker audiovisual coupling. Accordingly, our results suggest that whether emotional voice information facilitates emotion recognition without interfering with eye movement planning depends on the strength of audiovisual coupling in emotion processing.