Understanding Amazon Mechanical Turk
Amazon Mechanical Turk is a crowdsourcing Internet marketplace that allows individuals to use human intelligence to perform tasks that computers cannot. On Amazon Mechanical Turk, Requesters (employers) create HITs – Human Intelligence Tasks – that are completed by Workers (employees). HITs are essentially small, one-time jobs that can take anywhere from a few seconds to several minutes to complete. HITs include categorizing the tone of an article, translating a paragraph from one language to another, selecting the correct spelling for search terms, tagging pictures, market research, and even verifying the fit of Amazon product recommendations. The output of these tasks is used to both build up databases of information, as well as to aide in machine learning.
MTurk is an unregulated marketplace where supply and demand rule. When a Requester posts an HIT, they specify how much they will pay for the completed HIT, and what the job entails. Workers can choose to accept or pass the HIT after reading a brief description. Upon accepting the HIT, they complete the specified work and submit it to the Requester for approval. The Worker is financially compensated, but only if the Requester approves of the job they did. Mechanical Turk touts this system as a way to ensure accurate results, but Workers often complain about Requesters unfairly rejecting their work. HIT compensation varies greatly, but most users earn $5 or less an hour. Workers are paid as contractors, so there are no extra benefits that come from being an MTurk worker.
The platform is online, which allows employers to eliminate overhead and fixed costs associated with managing a temporary or contracted worker. Because of the scale, flexibility, and cost advantages of Mechanical Turk, new ideas and models are being generated daily. This system provides an amazingly efficient way to get work done and collect data, albeit being ethically ambiguous. Regardless of the larger issues at hand, we focus here on workers’ satisfaction with this system, and possible areas for improvement.
We conducted a survey to better understand the users of Amazon Mechanical Turk and their motives. Our first step was to look at who uses MTurk. According to Amazon, there are five hundred thousand workers from nineteen countries currently using MTurk. Of the user population, 80% are from the United States.
-Our data indicates that MTurkers are younger than the general population; their mean age is 33, whilst the estimated average age for Americans (excluding minors, who are not permitted on MTurk) is 41.
-They are evenly split between men and women; among younger users there is a higher percentage of men, but among older users there is a higher percentage of women.
-They have a lower median income than the general US population; their median is $42,478 but the population’s is $51,939.
-They are less likely to currently have children (age 0-18) than the general population; 70% of Turkers do not have children, compared to 58% of the general population.
-They are employed and educated; 50% work full time jobs and 70% have at least some college education.
While MTurkers are a little younger and less well off than the general population, they seem to be like many other Americans. Given this relative similarity, why do they use MTurk, a platform that pays less than minimum wage? This question may be explained by the following findings.
Most users’ primary reason for using MTurk is to make money, but yet they report only making about $3 an hour. While workers report their primary motivation for using MTurk being to make money, they only use MTurk for on average 3 hours a week, suggesting it is not a major source of income. Importantly, there seems to be a second motivation; a significant portion of users also cited “it is entertaining or interesting” (39.6% of users) and “to avoid boredom” (35.4% of users) as reasons for use. Workers who cite entertainment or avoidance of boredom as a reason for using MTurk report significantly greater median income ($48,816) than those that only cite making money as their reason for using MTurk ($38,462).
Interestingly, more than half (52.6%) of the user population has been active on MTurk for 3 months or less. This suggests a high turnover which, in turn, suggests a high dissatisfaction with the platform. The average hourly wage of $3 seems to be one reason for the dissatisfaction. This is lower than minimum wage in the United States, forcing users to realize that the benefits are not worth their time, and discontinuing their use. Despite the low pay our research finds relatively high overall satisfaction, with a mean rating of 75 (out of 100). However, there was a negative correlation (r = – 0.94), between overall satisfaction with MTurk and duration of use. The longer users have used MTurk, the less satisfied they are with it. Mean overall satisfaction was 79 for those who have used MTurk a month, but dropped to 73 for those who have used MTurk for more than a year.
To understand the drop in satisfaction over time, Turkers were asked about specific aspects of MTurk – Amazon Support, Requestors, Worker Pay, and Convenience. The results showed that there was a negative correlation (r = - 0.22) between satisfaction with Amazon Support and duration of use. The longer users have used MTurk, the less satisfied they are with Amazon Support. There was no correlation with the latter three elements of MTurk. One explanation for the negative correlation between satisfaction and duration of use is that newer customers may not have run into as many problems with the website as longer term users. As customers use MTurk longer, they are bound to explore more aspects of the website, and therefore, have the potential to run into more problems and more of an incentive to contact Amazon Support. Amazon should be motivated by these findings to focus more attention on their longer-term users and ensure positive user interactions with their support systems. Significant feedback from workers already exists in the form of discussion forums such as “turkopticon,” “turker nation,” “turk alert,” and on sub-forums at reddit.
Amazon has created an amazingly efficient resource for those looking to get work done and those looking for freelance work. Moving forward, they should work to maintain a high level of user satisfaction. Finally, Amazon has the opportunity to be a leader in the freelance marketplace by establishing policies which allow for free exchange without harming those involved.