Why Mechanical Turk will fail and how to save it
Author: admin
Mechanical Turk has been a topic that a lot of people are talking about, however having been in the Beta status for five years it has almost no improvement in the interface and business model. Having been working on it as a worker and also as a requester (alright, I go ahead and admit that I also tried to submit garbage to the requesters just to test if they can catch me, you can also call me a former scammer), I focused on observing the “society” of MTurk and in this article I am going to talk about why this (I mean, Yes, Mechanical Turk) will fail and how it can be saved. The take home message of this the article is: Crowdsourcing is an economical issue, not a technical issue, the team should focus on a better interacting schema between the Turkers, the Requesters and, finally, keep themselves out of the game.
1. The current ecosystem on Mechanical Turk
OK, first let’s talk about the current ecosystem on Mechanical Turk. I will tell you this is absolutely true, because I have played all the roles in the ecosystem (except the role of Mechanical Turk team) and played against every role.
Roughly speaking there are six entities in the game, MTurk, Amazon, Requesters, Workers and Scammers and in addition, Consumers. Let me explain what are they doing.
- MTurk: They play the roles of the judge, the government, the police, the executer, the landlord. They receive money from the requesters, take 10% from it and allow the requesters to post tasks. When requesters notice scammers they may block them or report to MTurk, and MTurk will perform the role of the judge and suspend the account of the scammer.
- Amazon: The MTurk does not allow non-US workers to get their money, instead the pay was transferred to Amazon GC, what does this mean? It means you can never get real cash if you are not in the US. So foreign workers has to sell their GC or goods to US consumers and (hopefully) get pay from them. And Amazon, earn more revenue from selling the goods.
- Requesters: The requesters pay MTurk money for a slot of posting tasks, and hopefully it can be done. Amazon will not push them to the workers, the workers has to navigate through a huge number of “HIT”s so as to find one to work. You get lucky if your HITs are seen by some workers in the first 10 minutes, then nobody will notice it and it will sit there until it is expired. So, you have to advertise your HITs, on forums or whatever place. After you get your results, you have to decide whether or not it is from (very likely) the scammers, write programs to pick the bad guys out and say “thank you but I cannot pay you”, and be ready to receive hate mails from the non-scammers if you did it wrong.
- Workers: I mean serious workers, aka Turkers. They (if they do exist) go to the site to earn some money for a new Ipod or a new book, or just kill some time. They do not care about the compensate, because it is nothing, believe me, nothing. So in fact it is voluntary for a US-based worker, and you cannot blame a voluntary worker for submitting garbage to the task. But it is not so true for foreigners, working 8 hours on MTurk can earn decent salary in the third world. But they cannot get the money, what they can get is the GCs. And if they know nobody in the US, the GCs are as valuable as (the s-word). So, believe me, if a serious worker who cares about the payment does exist, he/she is a foreigner who knows somebody in the States. But, you can hardly be sure about their English…
- The Scammers: OK, a scammer that only wants money will hack every HIT if they can or just click on “submit” without looking at the real question. Why? because if you have 50% rate to get caught when you can submit 10 HITs in a minute and get 1 dollar from them, or you can spend one minute working seriously on it and get 10 cents, which one has higher earning expectation? Of course it is the first. The scammers are the motivated ones, they are not voluntary, instead they work hard to submit garbage to your HITs and… Now some researchers (Yes, researchers who submit 100 HITs to MTurk and report some results and finish their Ph.D), will say: no, I hardly saw any scammers. Yes, but that is because your number of HITs are not large enough. It is not even worth for a scammer to write a script for your HITs.
- The Consumers: Well, for US-based workers they can transfer the money out to their pocket and go for a dinner, but other workers cannot, they try to sell the goods through relatives or friends to some US consumers and hopefully get some money back. And these consumers get some discount too.
2. The reason why it will fail
Alright, now we know there are six roles. And we also mentioned how they interact and why this interaction will lead to a total failure of the market. We will analyze this by some economics theories but I am not an economist but a geeky computational linguist. Although I do have a degree on economy but it was in China and I can hardly choose correct economics terms in English. So forgive me if this is not professional.
First of all, what is the incentive for a worker or scammer to work on MTurk? Well, many MTurk supporters says it is “fun”. OK, let’s count it. But is it fun enough for any body to do 10000 similar HITs? I suppose the answer is no. Capitalism says: Money. Yes, they want to get money. They will try to do whatever can bring them higher income. And, there is a limitation: Time. Nobody has more than 24 hours a day, so they want to have higher hourly rate. If whatever can save their time, they will do so, as long as the punishment cost less.
Again, researchers, please don’t fight back so soon saying that “the quality is good for my HITs”. I told you it is because your HITs is not worth hacking. I.e. the scammers are not interested in your tasks.
The existence of scammers will cause a very bad consequence: Information asymmetry. As a requester, you believe there must be scammers, and they will definitely try to attack you if you do not do quality control. So you do quality control, hire a programmer paying them 40 dollars per hour to write automatic quality control programs or hire a full-time employee to control the quality, or, pay extra money to third-party companies such as CrowdFlower and use their services. In any way, there is extra cost. Also, how can you do quality control? You put some traps or pre-labeled answers to your questions, or post more then one HITs, say, 3, and compare the answers. Let’s assume you had a budge of 10000 dollars, and you want to complete 1000 HITs. You can pay 10 dollars for one HIT, which takes a serious worker 45 minutes to complete. It is a decent salary, indeed. But now you know there are scammers, you have to ask 5 people to do the same task. Then you cannot pay 10 dollars, instead, 2 dollars. Now, the whole game changed, a decent people will not work on it because it is nothing. Then who will? Yes, somebody that can do it in 9 minutes, or somebody cannot even earn 2 dollars in one hour. You can imagine what you can get from them. So you keep upgrading your quality control program, paying 5000 dollars to the programmers, and now you can only pay 1 dollar to the Turkers…
It is a negative feedback! The market will soon be filled with cheap labor and scammers. You cannot get anything serious done on it. What is the source of the problem? It is not the scammers, nor the requesters or anybody else, the problem is the Mechanical Turk Team. All the actions they are making now really look like the evil capitalists in 1850s. They put all the focus on getting money from the requesters, and try to exploit the workers as hard as possible. If they have ever studied microeconomics they should know the willingness to work depends on the salary. If they pay one cent and ask for serious results, the only way is to restore slavery. Unfortunately, it is 21st century now so the decent workers will simply get away from it. In this free market, information asymmetry comes into play. The very famous “The Market for Lemons” stories can be found in may articles, for example, this
Akerlof, George A. (1970). “The Market for ‘Lemons’: Quality Uncertainty and the Market Mechanism”. Quarterly Journal of Economics (The MIT Press) 84 (3): 488–500. doi:10.2307/1879431. http://jstor.org/stable/1879431.
The existence of scammers brings down the salary. And the lowered salary kicks out the serious workers. The small number of serious workers make it even harder to get good results, and the salary will go further down. Finally, it will become a market for lemons.
One way of solving the problem, is for MTurk team to eliminate scammers. And it is not enough, people can be a scammer whenever they want to, Joseph Stiglitz said:
Unlike other forms of capital, humans can choose their level of effort.
There are indeed no incentive for a worker to be serious, no rewards, no real punishment. Because the requester has enough people (cheap) to do the HITs, they also do not have no incentive to raise the pay, not to mention higher pay won’t give them any better result. It is definitely not enough just for MTurk team to block scammers. So, as I promised, I will suggest a way to save the market, it may or may not work, but I will try to present it.
3. How to save the market
We earn decent salary because of what? Believe me it is not because of the government of the Wall Street. It is because of the Education System and the Union. What does the Education System do? They not only educates you, they also give you certifications about your ability. When a employer saw your diploma, they know roughly what you can do and what you cannot do. Also, you join a Union so that when the employer does not treat you fairly you can strike and negotiate. The Education System is really protection of the employer, they issue certification to qualified graduates, and the punishment of qualifying “lemons” for a school will be a bad fame, and devalued diploma. The Union is the shield for the employee, under which they won’t get unfair treatment or “enslaved” by the evil capitalists. Now, the status in the MTurk market is that: 1. The employer must build their own education system if they want some qualifications. 2. The workers have no protection. If this can be changed, the market can be saved. I will try to suggest a new structure that may work. And the key is: A virtual society for the workers and the requesters.
In the society, we need to have an organization that play both the roles of Education System and the Union. We are in the world of crowdsourcing, of course, the organization should not be organized by MTurk Team, instead, we also crowdsource it. The organization will be based on a forum where the requesters and workers can discuss on it. Every organization issues a single qualification, such as “English and Chinese translation qualification”. Anybody joins the organization must take the qualification and be peer-reviewed. The requester can then require anybody who wants to work for them, take such qualification. Base on their overall experience on the results, they will give the organization a score, whether the organization has done well, or not. There can be 100 organizations issuing “English and Chinese translation qualification”, but soon we will discover who will get highest scores.
The organization, also has the right to choose which requester’s task they take. I.e. if the payment of the requester is too low, the organization can vote to strike. Anybody in the organization is not allow to work on the requester’s task, if they do so, their qualification will be suspended. The requester has a lot of choices, they can negotiate with a famous organization, a smaller organization who will ask for a lower price, or, just risk themselves on the general public. The workers of the organizations, get higher payment because they have the qualification, will value it and work seriously. The organization, to maintain their reputations, need to peer-review their members thoroughly. The system will lead to a positive feedback loop. The competition between organizations will keep the price from being too high, and their reputation will be an endorsement for high-quality results. The serious workers, under the protection of the organizations, can ask for decent price for their work.
The organization will be self-organized, fully crowd-sourcing, the Mechanical Turk Team does not need to intervene the management. The organization itself will take care of kicking out scammers, and enroll good turkers. The change to the system is also minimal. First of all, Mechanical Turk need to build a forum, the user ID should be connected to the worker ID on MTurk. Secondly, MTurk already has the “qualification” interface on MTurk API, simply link the qualification id to an organization, and provide APIs for the chair of the organization to delete or add users. The organization has all privilege to determine how they qualify a worker, such as peer-review, voting, or a dictatorship. The requester can ask to use the qualification, but it must be approved by the organization. Again, the approval procedure can be democratic or by dictatorship. The Mturk should report what HITs worker has done to its organizations, and the leaders of the organizations can determine the punishment for traitors. Also, the requesters can still reject HITs, but the rejection will be reported to the organizations too, and the organizations can determine whether or not to punish the possible scammer, or to strike, against the bad requesters. Finally, organization gets information on what HITs to do, and requesters have a channel of pushing HITs to the workers. Also, a possible way of awarding the organizers of the organization is to impose “tax” on the workers in the organization, which the organization can determine who to award, so people working hard on designing tests get their share.
In the whole game, MTurk team can be hidden, they do not need to do much, just set up the mechanism, and let the invisible hand adjust themselves. I don’t think there is any reason not doing so. Having such virtual society is wonderful and I am really dreaming about it. I would cheer if anybody in the world can build this and well defeat MTurk, but actually if MTurk does this, it could be perfect. Indeed, I do think if MTurk crew see this, they should do this, this can really be done within a month. And if someone does it, it can be a catastrophe for MTurk, especially if CrowdFlower does it. I know they are looking forward to “new channels” other than MTurk, which means if they have this “virtual society” but MTurk does not, they can kick out MTurk in any time, build their own system and let MTurk die miserably.
The MTurk team should switch them focus from attracting good requesters to attracting and protecting good workers. Remember, good workers ARE your RESOURCE, but requesters are NOT, they are your CUSTOMERS. And you are selling your RESOURCE, if they have bad quality, no CUSTOMERS will come to you. Until now you have not put enough effort on helping the Turkers, now it is time to do so.
Well, that’s all, sorry for my poor English and typos, I am not going to edit them:)
