File Download
  Links for fulltext
     (May Require Subscription)
Supplementary

Conference Paper: On optimality of jury selection in crowdsourcing

TitleOn optimality of jury selection in crowdsourcing
Authors
Issue Date2015
PublisherOpenProceedings.org.
Citation
The 18th International Conference on Extending Database Technology (EDBT 2015), Brussels, Belgium, 23-27 March 2015. In Conference Proceedings, 2015, p. 193-204 How to Cite?
AbstractRecent advances in crowdsourcing technologies enable computationally challenging tasks (e.g., sentiment analysis and entity resolution) to be performed by Internet workers, driven mainly by monetary incentives. A fundamental question is: how should workers be selected, so that the tasks in hand can be accomplished successfully and economically? In this paper, we study the Jury Selection Problem (JSP): Given a monetary budget, and a set of decision-making tasks (e.g., “Is Bill Gates still the CEO of Microsoft now?”), return the set of workers (called jury), such that their answers yield the highest “Jury Quality” (or JQ). Existing JSP solutions make use of the Majority Voting (MV) strategy, which uses the answer chosen by the largest number of workers. We show that MV does not yield the best solution for JSP. We further prove that among all voting strategies (including deterministic and randomized strategies), Bayesian Voting (BV) can optimally solve JSP. We then examine how to solve JSP based on BV. This is technically challenging, since computing the JQ with BV is NP-hard. We solve this problem by proposing an approximate algorithm that is computationally efficient. Our approximate JQ computation algorithm is also highly accurate, and its error is proved to be bounded within 1%. We extend our solution by considering the task owner’s “belief” (or prior) on the answers of the tasks. Experiments on synthetic and real datasets show that our new approach is consistently better than the best JSP solution known.
Persistent Identifierhttp://hdl.handle.net/10722/208740
ISBN

 

DC FieldValueLanguage
dc.contributor.authorZheng, Y-
dc.contributor.authorCheng, R-
dc.contributor.authorManiu, S-
dc.contributor.authorMo, L-
dc.date.accessioned2015-03-18T09:08:04Z-
dc.date.available2015-03-18T09:08:04Z-
dc.date.issued2015-
dc.identifier.citationThe 18th International Conference on Extending Database Technology (EDBT 2015), Brussels, Belgium, 23-27 March 2015. In Conference Proceedings, 2015, p. 193-204-
dc.identifier.isbn978-3-89318-067-7-
dc.identifier.urihttp://hdl.handle.net/10722/208740-
dc.description.abstractRecent advances in crowdsourcing technologies enable computationally challenging tasks (e.g., sentiment analysis and entity resolution) to be performed by Internet workers, driven mainly by monetary incentives. A fundamental question is: how should workers be selected, so that the tasks in hand can be accomplished successfully and economically? In this paper, we study the Jury Selection Problem (JSP): Given a monetary budget, and a set of decision-making tasks (e.g., “Is Bill Gates still the CEO of Microsoft now?”), return the set of workers (called jury), such that their answers yield the highest “Jury Quality” (or JQ). Existing JSP solutions make use of the Majority Voting (MV) strategy, which uses the answer chosen by the largest number of workers. We show that MV does not yield the best solution for JSP. We further prove that among all voting strategies (including deterministic and randomized strategies), Bayesian Voting (BV) can optimally solve JSP. We then examine how to solve JSP based on BV. This is technically challenging, since computing the JQ with BV is NP-hard. We solve this problem by proposing an approximate algorithm that is computationally efficient. Our approximate JQ computation algorithm is also highly accurate, and its error is proved to be bounded within 1%. We extend our solution by considering the task owner’s “belief” (or prior) on the answers of the tasks. Experiments on synthetic and real datasets show that our new approach is consistently better than the best JSP solution known.-
dc.languageeng-
dc.publisherOpenProceedings.org.-
dc.relation.ispartofProceedings of the 18th International Conference on Extending Database Technology, EDBT 2015-
dc.rightsCreative Commons: Attribution 3.0 Hong Kong License-
dc.rightsAuthor owns CopyRight-
dc.titleOn optimality of jury selection in crowdsourcing-
dc.typeConference_Paper-
dc.identifier.emailZheng, Y: ydzheng2@cs.hku.hk-
dc.identifier.emailCheng, R: ckcheng@cs.hku.hk-
dc.identifier.emailManiu, S: smaniu@cs.hku.hk-
dc.identifier.emailMo, L: lymo@cs.hku.hk-
dc.identifier.authorityCheng, R=rp00074-
dc.description.naturepublished_or_final_version-
dc.identifier.doi10.5441/002/edbt.2015.18-
dc.identifier.hkuros242618-
dc.identifier.spage193-
dc.identifier.epage204-
dc.customcontrol.immutablesml 150511-

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats