A proposed multidimensional information system framework for journal evaluations : a case study in the field of education and educational research

Abstract

As bibliometrics is a budding field, this doctoral dissertation aims to enrich the practice of journal evaluations with a proposed design, development and application of a multidimensional journal evaluation information system (JEIS). Based on three overarching caveats established by leading bibliometricians, I first examined the purposes, philosophies and practices of journal evaluations in an attempt to understand why the bibliometrics community has not been able to come to a universally accepted measure. Through an extensive literature review, it appears that the multifaceted characteristics of journals are not being amply captured since extant journal evaluation studies are mostly rank-based, onedimensional, and lacking in qualitative depth and scope. Consequently, I investigated the extent to which journal evaluation practices can be technically rigorous yet easy enough to use, and at the same time, relevant and versatile enough to accommodate varying stakeholder purposes and philosophies. In the process, I learned that the multifaceted characteristics of journals could not be succinctly captured by a single measure. I then identified several existing approaches that could capture some unique characteristics of journals while evincing technical rigor. Based on these findings, I then proposed a research framework involving the design and development of a JEIS. In applying the JEIS, I used Education and Educational Research (EER) journals as the case study, and employed quantitative methods such as PageRank to produce multifaceted journal evaluations. As such, I was able to qualitatively uncover some interesting relationships among the 95 EER journals, produce cross-sectional and longitudinal journal impact evaluations, and identified the roles that the 95 journals play within and outside the EER domain. Based on the outputs, I also illustrated how portions of the results can be qualitatively applied to aid purpose-based decision-making. And in concluding this dissertation, I have deliberated on the research contributions and limitations, drawn research implications, and derived future research tracks.