File Download

There are no files associated with this item.

  Links for fulltext
     (May Require Subscription)
Supplementary

Conference Paper: New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness

TitleNew measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness
Authors
KeywordsExperimentation
Human factors
Measurement
Issue Date2010
Citation
ASIS&T 2010 Annual Meeting, Pittsburgh, PA, 22-27 October 2010. In Proceedings of the ASIST Annual Meeting, 2010, v. 47 How to Cite?
AbstractUser satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.
Persistent Identifierhttp://hdl.handle.net/10722/180707
ISSN
2015 SCImago Journal Rankings: 0.196

 

DC FieldValueLanguage
dc.contributor.authorCheng, Jen_US
dc.contributor.authorHu, Xen_US
dc.contributor.authorHeidorn, PBen_US
dc.date.accessioned2013-01-28T01:41:31Z-
dc.date.available2013-01-28T01:41:31Z-
dc.date.issued2010en_US
dc.identifier.citationASIS&T 2010 Annual Meeting, Pittsburgh, PA, 22-27 October 2010. In Proceedings of the ASIST Annual Meeting, 2010, v. 47en_US
dc.identifier.issn1550-8390en_US
dc.identifier.urihttp://hdl.handle.net/10722/180707-
dc.description.abstractUser satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.en_US
dc.languageengen_US
dc.relation.ispartofProceedings of the ASIST Annual Meetingen_US
dc.subjectExperimentationen_US
dc.subjectHuman factorsen_US
dc.subjectMeasurementen_US
dc.titleNew measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectivenessen_US
dc.typeConference_Paperen_US
dc.identifier.emailHu, X: xiaoxhu@hku.hken_US
dc.identifier.authorityHu, X=rp01711en_US
dc.description.naturelink_to_subscribed_fulltexten_US
dc.identifier.doi10.1002/meet.14504701144en_US
dc.identifier.scopuseid_2-s2.0-84861422067-
dc.identifier.volume47en_US

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats