The role of eye movements in multimodal information processing : the case of documentary comprehension and emotion recognition

Abstract

In daily life, people receive extensive information from visual and auditory modalities. Understanding how people process these audiovisual inputs can help better understand how people perceive the world. Online eye movement behavior has been shown to reflect one’s perceptual style and cognitive strategy. Thus, the role of eye movements can be of vital importance in investigating multimodal information processing under various contexts. This MPhil thesis aimed to examine how people adopting differential eye movement patterns differ in their performance in various audiovisual tasks and how the automaticity of audiovisual integration in these tasks interferes with people’s online eye movement planning under the context of multimedia learning and emotion recognition. The Eye Movement analysis with Hidden Markov Models (EMHMM) method was adopted to better capture the temporal and spatial features of eye movement behaviors. Study 1 focused on audiovisual integration during documentary comprehension in multimedia learning. Whether subtitles and video content facilitate comprehension of documentaries consisting of statements of facts and whether the comprehension depends on participants’ cognitive abilities and eye movement strategy during video viewing were investigated. 60 native Mandarin speakers completed documentary comprehension tasks with their eye tracked and several cognitive tasks. It was found that subtitles, which have more automatic audiovisual association with auditory narratives than video content, facilitated comprehension. In contrast, with video content but no subtitles, comprehension depended on participants’ eye movement strategy, auditory working memory capacity, and task switching ability. Specifically, a more centralized (looking mainly at the center) eye movement pattern predicted better comprehension as opposed to a distributed pattern (with distributed regions of interest) after variance due to cognitive abilities was controlled. Study 2 examined the hypothesis that different emotions may differ in their automaticity of audiovisual integration, which results in differential audiovisual information processing and thus the eye movement pattern. 65 participants judged the emotion of a talking head video under audiovisual, video-only (with no sound), and audio-only (with a static neutral face image) conditions. Among the six basic emotions, disgust had the largest audiovisual advantage over unimodal conditions in accuracy. In the recognition of all emotions except for disgust, participants’ eye movement patterns did not change significantly across conditions, suggesting mandatory audiovisual integration. In contrast, in disgust recognition, participants looked less at the eyes (the most diagnostic features) in the audiovisual than the video-only condition, suggesting that audio information interfered with eye movement planning for diagnostic features. Disgust recognition is developed later in life and thus involves less audiovisual associative learning, leading to decreased automaticity and higher attention demands in audiovisual integration. In conclusion, how audiovisual information influences eye movements and performance in cognitive tasks depends on the automaticity of audiovisual integration. With high automaticity of audiovisual integration, online eye movement behavior and task performance are affected less by audio information. With low automaticity of audiovisual integration, eye movements are influenced more and in turn modulate the performance. These findings have important implications for real-life multimedia learning and emotion recognition.