File Download

There are no files associated with this item.

  Links for fulltext
     (May Require Subscription)
Supplementary

Conference Paper: Generating ground truth for music mood classification using mechanical turk

TitleGenerating ground truth for music mood classification using mechanical turk
Authors
KeywordsCrowdsourcing
Evaluation
Gold Standard
Ground Truth
Mechanical Turk
Mood
Music Information Retrieval
Issue Date2012
PublisherI E E E. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000206
Citation
Proceedings Of The Acm/Ieee Joint Conference On Digital Libraries, 2012, p. 129-138 How to Cite?
AbstractMood is an important access point in music digital libraries and online music repositories, but generating ground truth for evaluating various music mood classification algorithms is a challenging problem. This is because collecting enough human judgments is time-consuming and costly due to the subjectivity of music mood. In this study, we explore the viability of crowdsourcing music mood classification judgments using Amazon Mechanical Turk (MTurk). Specifically, we compare the mood classification judgments collected for the annual Music Information Retrieval Evaluation eXchange (MIREX) with judgments collected using MTurk. Our data show that the overall distribution of mood clusters and agreement rates from MIREX and MTurk were comparable. However, Turkers tended to agree less with the pre-labeled mood clusters than MIREX evaluators. The system evaluation results generated using both sets of data were mostly the same except for detecting one statistically significant pair using Friedman's test. We conclude that MTurk can potentially serve as a viable alternative for ground truth collection, with some reservation with regards to particular mood clusters. © 2012 ACM.
Persistent Identifierhttp://hdl.handle.net/10722/180714
ISSN
References

 

DC FieldValueLanguage
dc.contributor.authorLee, JHen_US
dc.contributor.authorHu, Xen_US
dc.date.accessioned2013-01-28T01:41:33Z-
dc.date.available2013-01-28T01:41:33Z-
dc.date.issued2012en_US
dc.identifier.citationProceedings Of The Acm/Ieee Joint Conference On Digital Libraries, 2012, p. 129-138en_US
dc.identifier.issn1552-5996en_US
dc.identifier.urihttp://hdl.handle.net/10722/180714-
dc.description.abstractMood is an important access point in music digital libraries and online music repositories, but generating ground truth for evaluating various music mood classification algorithms is a challenging problem. This is because collecting enough human judgments is time-consuming and costly due to the subjectivity of music mood. In this study, we explore the viability of crowdsourcing music mood classification judgments using Amazon Mechanical Turk (MTurk). Specifically, we compare the mood classification judgments collected for the annual Music Information Retrieval Evaluation eXchange (MIREX) with judgments collected using MTurk. Our data show that the overall distribution of mood clusters and agreement rates from MIREX and MTurk were comparable. However, Turkers tended to agree less with the pre-labeled mood clusters than MIREX evaluators. The system evaluation results generated using both sets of data were mostly the same except for detecting one statistically significant pair using Friedman's test. We conclude that MTurk can potentially serve as a viable alternative for ground truth collection, with some reservation with regards to particular mood clusters. © 2012 ACM.en_US
dc.languageengen_US
dc.publisherI E E E. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000206en_US
dc.relation.ispartofProceedings of the ACM/IEEE Joint Conference on Digital Librariesen_US
dc.subjectCrowdsourcingen_US
dc.subjectEvaluationen_US
dc.subjectGold Standarden_US
dc.subjectGround Truthen_US
dc.subjectMechanical Turken_US
dc.subjectMooden_US
dc.subjectMusic Information Retrievalen_US
dc.titleGenerating ground truth for music mood classification using mechanical turken_US
dc.typeConference_Paperen_US
dc.identifier.emailHu, X: xiaoxhu@hku.hken_US
dc.identifier.authorityHu, X=rp01711en_US
dc.description.naturelink_to_subscribed_fulltexten_US
dc.identifier.doi10.1145/2232817.2232842en_US
dc.identifier.scopuseid_2-s2.0-84863549107en_US
dc.relation.referenceshttp://www.scopus.com/mlt/select.url?eid=2-s2.0-84863549107&selection=ref&src=s&origin=recordpageen_US
dc.identifier.spage129en_US
dc.identifier.epage138en_US
dc.publisher.placeUnited Statesen_US
dc.identifier.scopusauthoridLee, JH=36064170000en_US
dc.identifier.scopusauthoridHu, X=55496358400en_US

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats