How professional survey-takers are shaping scientific research

JUDY WOODRUFF: Finally tonight: a look at enterprise reporting we have done online that we thought would be of interest to you.

It's about the proliferation of scientific and academic research being done online and whether those methods may be leading to flawed or unreliable data. Much of the work was done once by students, but, these days, there is an informal work force of people who participate in studies through an online job forum known as Mechanical Turk.

The name was inspired which an 18th century fake chess-playing robot decorated in Turkish robes. It defeated almost every opponent it faced for years, but it turned out there was a hidden human chess master behind the machine.

Well, the NewsHour's Jenny Marder reported our story. And she fills us in now.

Jenny, it's great to have you here to talk about it.

So, first of all, tell us more about who these people are who are answering these surveys and what exactly do they do.

JENNY MARDER: Yes.

Well, this is a portion of the 500,000 workers on Mechanical Turk that we were looking at. And the workers do all sorts of jobs. The jobs have been — work has been called microlabor because the pay is often very, very low. You see a lot of jobs for 25 cents, 5 cents, even a penny.

So they're really working for pennies on Mechanical Turk. The reason we got interested is it's been, over the past five years, increasingly used by academic researchers as a way of getting data and finding study subjects for their research.

JUDY WOODRUFF: So what kind of research is being done?  What are these surveys about?  I know it's a broad array of subjects. What kind of questions are they trying to answer?

JENNY MARDER: Yes. Yes.

And the research spans all sorts of disciplines. There's — you have a lot of psychology, social science research, but also political scientists are using it. You even see it in medical research.

And we looked at a lot of the studies that are using Mechanical Turk, and a lot of them are asking really big questions. There's a lot of research on human behavior, but also on teen alcohol abuse, a lot of research on decision-making, how people perceive scientists and climate scientists. So these aren't obscure studies. And they're asking some pretty big questions.

JUDY WOODRUFF: That go on to maybe make a difference in terms of how policy is made.

JENNY MARDER: Sure.

JUDY WOODRUFF: Now, we have a short clip of one of the people you profile. This is a young mother.

Let's take a look at that.

SARAH MARSHALL: Done a lot of academic research just put out by different universities. I have done a lot of surveys in general, but, for just colleges, probably about 20,000.

Ah, consumer attitudes towards advertising.

Surveys usually pay better, at least in my experience, because they pay a lump sum of money. It's, like, oh, for your five minutes, you can have a dollar.

Mommy just earned a dollar.

It becomes like kind of robotic. It's, like, I am filling out a survey, so I'm going to do it the same way I did it yesterday.

How much formal schooling have you had?

And it's like, if I see the same block of questions twice on the same day, I even know the pattern for my answers.

JUDY WOODRUFF: So, that is a woman named Sarah Marshall. And she does how many of these a week?

JENNY MARDER: Yes.

So, Sarah Marshall does maybe hundreds of studies a week. She has done 20,000 studies altogether in the past five years.

JUDY WOODRUFF: And you were telling me there are a lot of other people like her. So, that — that's what raises some questions about whether these surveys — the validity of these surveys, these projects.

JENNY MARDER: Right. Right.

There's a question of environmental control. You don't know — these people could be at home and distracted while they're doing these surveys. But they're also seeing the same questions repeated again and again.

Researchers — it's common for researchers to test intuition, to test a person's gut instinct. And you can see how, if somebody's answered a question a hundred times or even three times, they're no longer getting the intuitive response. They're getting a much more trained response.

JUDY WOODRUFF: So, doing it over and over again, you're not getting that — you're not — also not getting that real cross-section that you're looking for of the public.

JENNY MARDER: That's right.

JUDY WOODRUFF: All right, Jenny Marder, again, this is just a glimpse of the reporting you have done.

You can see the entire story, find out a whole lot more by going online to our Web site. That's PBS.org/NewsHour.

Jenny, thank you.

JENNY MARDER: Thank you.

Recently in Science