I've been saying this for years as a critique to psychology studies. Do you really expect desperate people getting paid peanuts to yield quality results?
I mentioned this above, but at least in the short term my experience has been that paying more doesn’t yield better results on most tasks, which I naively attribute to those skills not being available at any price from the current population of MTers.
If higher prices were paid across the board, doubtlessly more workers with the sought after skills would sign up though. So maybe that was the thrust of your comment.
> If higher prices were paid across the board, doubtlessly more workers with the sought after skills would sign up though. So maybe that was the thrust of your commen
Maybe a couple? But you're also going to get millions more unskilled workers. Without good filtering, no pay level is going to work.
I mentioned this above, but at least in the short term my experience has been that paying more doesn’t yield better results on most tasks, which I naively attribute to those skills not being available at any price from the current population of MTers.
This is plausible but doesn't change the basic problem. Of course, workers on AMT have years of experience trying to game it. Just more pay isn't going to create a sudden sense of solidarity with the employer.
I would guess that quality results at scale would require a system built from the ground-up to cultivate some degree of loyalty or something.
It's kind of a dilema of capitalism itself.
At scale, you can pay people to accomplish a task for peanuts but they lack loyalty and will cut-corners whenever it's possible and convenient. And that still gets a whole lot of things done for cheap in this world.
But if you want to get loyalty, people you don't have to watch, etc., you have to organize an entire enterprise for that, maybe show loyalty yourself (Turkers are inherently throw-away, why should they be loyal for just today's wages). And the cost of that arrangement tends to multiply.
I'm sorry, but are we talking about the post docs ~running the experiment~ managing the routine tasks beneath the dignity of the department/lab head or the people participating in the experiment?
I am not privy to the state of psychology research, but I'm pretty sure the OP is referring to the reliance of Mechanical Turk in the field of psychology [1].
