Chinese multimodal reading comprehension in a hypertext environment : measurement and its antecedents

Abstract

With the growing prevalence and accessibility of the Internet and new communication technologies, multimodal reading literacy has become essential for students navigating multimodal texts in the 21st century. In recent years, multimodal reading has garnered considerable attention in large-scale international assessments projects, such as PISA and PIRLS. As part of Hong Kong’s continual effort to maintain its high international standards, the Chinese language education should seize the opportunity to implement multimodal reading assessment. However, several issues still require further advancement. First, there is a paucity of instruments designed to evaluate students’ multimodal reading comprehension. Additionally, the few that exist tend to inadequately address explicit image-text relationships and the assessment of higher-order reading skills. Second, while there has been a surge in research probing into text and reader factors influencing traditional reading comprehension, the predictability of these factors with respect to multimodal texts remains underexplored, leaving strategies for improving multimodal reading unclear. To bridge these gaps, this study was designed to achieve three objectives: (1) develop a multimodal reading comprehension test; (2) examine the influence of basic and higher-order cognitive skills on students’ multimodal reading from the reader’s perspective; and (3) explore how different modalities and image-text relationships affect students’ multimodal reading comprehension from a textual perspective. These objectives inform the design of three interrelated studies. Mixed-methods approach was utilized to address the first two research objectives. First, this study developed and validated the multimodal reading comprehension test with a sample of 251 fourth graders, examining the characteristics of students’ performance in multimodal reading. The results showed that the test demonstrated good psychometric properties. Moreover, it was observed that students’ proficiency in integrating and evaluating information, responding to short-answer itmes, and understanding the extension image-text relationship require further enhancement. Consequently, the findings provide further evidence that basic (word decoding, linguistic comprehension, and image comprehension) and higher-order skills (prior knowledge and inference) positively influenced multimodal reading, and the interaction between basic and higher-order skills was confirmed by using structural equation modelling. To address the third research objective, a qualitative research design was implemented. Semi-structured interviews with 40 students revealed the multifaceted role of different modalities. Text was found to facilitate comprehension at the cognitive level, whereas the main function of pictures was reflected in attracting attention, enhancing motivation, facilitating comprehension, and increasing the speed of information processing. Furthermore, the study identified potential hindrances that students encounter when interpreting multimodal materials. For the image-text relationships, it was noted that illustration and elaboration relationships were relatively easy and the extension relationship was relatively difficult. Nevertheless, the complexity of image-text relationships is not absolute; each type presents its own accessibility issues and unique challenges. The findings of this study provide a theoretical framework for understanding the role of various modalities and their synergy in comprehension, enhancing our insight into multimodal reading. Practically, this study provides a standardised instrument that can be harnessed for educational purposes. Furthermore, it offers valuable insights into the design of multimodal reading resources and the improvement of multimodal reading skills. (500 words)