Multi-Modal and Multi-Level Analysis of Oral Discourse

Abstract

This chapter focuses on discussing the approach of multi-modal and multi-level analysis of oral discourse. Specifically, the consideration of linguistic characteristics, non-verbal behaviors, and prosodic features in unimpaired and disordered discourse production are presented. First, methodologies to identify, differentiate, and quantify non-verbal behaviors (including a wide range of activities, such as gestures, body postures and movements, touching behaviors, facial expressions, eye gaze, as well as vocal cues that accompany spoken words) are summarized. Examples of applying different frameworks to code facial expressions, bodily actions accompanying oral discourse, and different types and functions of gestures are also provided. This is followed by illustrations of computing simple measures to reflect the relationship between oral discourse performance and co-verbal gestures. Technical procedures in conducting prosodic analysis and coding speech prosody are also described, with explanations of the corresponding prosodic deficits demonstrated by speakers with aphasia and other related cognitive communicative disorders. Finally, use of computational tools for annotation of non-linguistic behaviors in spoken discourse are discussed. The chapter ends with a conclusion to justify the need and to explain the clinical value of performing a multi-modal analysis of oral discourse.