Computational Linguistics

Recently heard at a tutorial in our field: 'It cost me less than one hundred bucks to annotate this using Amazon Mechanical Turk!' Assertions like this are increasingly common, but we believe they should not be stated so proudly; they ignore the ethical consequences of using MTurk (Amazon Mechanical Turk) as a source of labor. Manually annotating corpora or manually developing any other linguistic resource, such as a set of judgments about system outputs, represents such a high cost that many researchers are looking for alternative solutions to the standard approach. MTurk is becoming a popular one. However, as in any scientific endeavor involving humans, there is an unspoken ethical dimension involved in resource construction and system evaluation, and this is especially true of MTurk. We would like here to raise some questions about the use of MTurk. To do so, we will define precisely what MTurk is and what it is not, highlighting the issues raised by the system. We hope that this will point out opportunities for our community to deliberately value ethics above cost savings.

Resource

Access the Research publication

Computational Linguistics
Fort, K., Adda, G. and Cohen, K. (2011), 'Amazon Mechanical Turk: Gold mine or coal mine?' Computational Linguistics, 37(2), pp. 413-420.

Metadata

  • Research publication
  • Other
  • Yes
  • professional services
  • Online moderately skilled click work
  • Amazon Mechanical Turk
  • algorithm, platform characteristics and business model
  • English
  • Computational Linguistics (Publisher)
  • 2011
  • Open access
Disclaimer  —  Eurofound aims to keep the information in this database up to date and accurate. If errors are brought to our attention, we will try to correct them. However, Eurofound bears no responsibility or liability whatsoever with regard to the accuracy or content of this database or of external links over which Eurofound services have no control and for which Eurofound assumes no responsibility.