Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool.

Publisher: Public Library of Science Country of Publication: United States NLM ID: 101285081 Publication Model: eCollection Cited Medium: Internet ISSN: 1932-6203 (Electronic) Linking ISSN: 19326203 NLM ISO Abbreviation: PLoS One Subsets: MEDLINE

Original Publication: San Francisco, CA : Public Library of Science

Crowdsourcing*/methods ; Crowdsourcing*/standards ; Patient Selection* ; Practice Guidelines as Topic*; Behavioral Research/*standards; Adult ; Behavioral Research/methods ; Bias ; Data Accuracy ; Data Collection/methods ; Data Collection/standards ; Datasets as Topic/standards ; Female ; Humans ; Male ; Middle Aged ; Sample Size ; Sampling Studies ; Selection Bias ; Work ; Young Adult

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool. Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.
Competing Interests: We have read the journal’s policy and the authors of this manuscript have the following potential competing interests: All of the authors are employed at Prime Research Solutions. This is the company that owns TurkPrime, the platform whose TurkPrime ToolKit was used to source Mechanical Turk participants, and the database from which some data were queried. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

Science. 2016 Jun 10;352(6291):1263-4. (PMID: 27284175)
Behav Res Methods. 2019 Oct;51(5):2022-2038. (PMID: 31512174)
J Pers Soc Psychol. 2016 Oct;111(4):493-504. (PMID: 27295328)
Behav Res Methods. 2014 Dec;46(4):1023-31. (PMID: 24356996)
PLoS One. 2013;8(3):e57410. (PMID: 23516406)
Behav Res Methods. 2017 Apr;49(2):433-442. (PMID: 27071389)
Psychol Sci. 2015 Jul;26(7):1131-9. (PMID: 26063440)
Monist. 1976 Apr;59(2):204-17. (PMID: 11662247)
Trends Cogn Sci. 2017 Oct;21(10):736-748. (PMID: 28803699)
Science. 2011 Oct 21;334(6054):307. (PMID: 22021834)
Behav Res Methods. 2016 Mar;48(1):400-7. (PMID: 25761395)
Annu Rev Clin Psychol. 2016;12:53-81. (PMID: 26772208)
Perspect Psychol Sci. 2011 Jan;6(1):3-5. (PMID: 26162106)
Behav Res Methods. 2015 Jun;47(2):519-28. (PMID: 24907001)
Science. 1981 Jan 30;211(4481):453-8. (PMID: 7455683)

Date Created: 20191217 Date Completed: 20200416 Latest Revision: 20200416

20240105

PMC6913990

10.1371/journal.pone.0226394

31841534