Bringing reputation-awareness into crowdsourcing

In crowdsourcing systems, worker selection is a challenging issue. Researchers have advocated the incorporation of reputation management into crowdsourcing systems to address this issue. However, current reputation-based decisions often result in concentrating the delegation of human intelligence tasks (HITs) to a small number of highly reputable workers to reduce risk. It conflicts with the main objective of crowdsourcing systems which is to promote mass collaboration. In this paper, we propose a situation-aware approach - SWORD - to enable existing reputation models to work in crowdsourcing systems. We holistically consider the objectives of all stake-holders in a crowdsourcing system (including requesters, workers, and system operators) to formulate the HIT allocation problem as a trade-off between quality and timeliness, and propose an efficient approach via constraint optimization to produce solutions to this problem with low computational complexity. Extensive simulations designed based on the actual conditions from Amazon's Mechanical Turk system demonstrates significant advantage of SWORD compared to existing approaches in improving social welfare.

Resource

Access the Research publication

Bringing reputation-awareness into crowdsourcing
Yu, H., Shen, Z. and Leung, C. (2013), 'Bringing reputation-awareness into crowdsourcing', ICICS 2013 - Conference Guide of the 9th International Conference on Information, Communications and Signal Processing, 10-13 December 2013, Tainan, Taiwan.

Metadata

  • no specific sector focus
  • Other
  • 2013
  • Research publication
  • work organisation, rating
  • English
  • Conference on Information, Communications and Signal Processing (Publisher)
  • Qualitative research
  • Subscription
Disclaimer  —  Eurofound aims to keep the information in this database up to date and accurate. If errors are brought to our attention, we will try to correct them. However, Eurofound bears no responsibility or liability whatsoever with regard to the accuracy or content of this database or of external links over which Eurofound services have no control and for which Eurofound assumes no responsibility.