Do You Trust Polls? Should You?
Paul Solman interviews professor Doug Schwartz at Quinnipiac University’s Polling Institute.
Politics professor Doug Schwartz runs the well-respected CBS/New York Times poll at Quinnipiac University. We interviewed him at the university’s Polling Institute for a story on presidential prediction last week. Why should we believe his numbers?
PAUL SOLMAN: Of 100 phone calls that people are making when collecting polling data, how many times are callers getting through to an eligible respondent?
DOUG SCHWARTZ: I would say out of the numbers that they dial, maybe one out of 10. Typically, when we reach our eligible respondents, about half of them cooperate and do the survey.
PAUL SOLMAN: What’s the reassurance that the people you’ve got are a representative sample of the total population of voters?
DOUG SCHWARTZ: Well the key is, you have to have to have a random sample. If you have a random sample, everyone in the population has an equal chance of being selected and therefore, your sample should be representative.
But in addition to that, we make checks. We look at the census data. We compare our sample to what the census says. And in addition to that, we ask people if they’re registered to vote and then we also ask them how likely they are to vote, to help define who is a likely voter.
PAUL SOLMAN: But as the skeptical journalist, I think, well, wait a second, if you’re only getting at most, 10 percent of people responding to the initial calls when somebody’s there and when you finally get through to somebody only 50 percent or so of them actually respond. And, of course, we can’t possibly really know whether they’re telling you the truth or not. How can you reassure me that this an actual scientific sample of the voting public?
DOUG SCHWARTZ: I guess I would point to our track record. If you look at how we’ve done in terms of predicting elections, we have a very good track record. We’ve done a very good job of predicting the actual election results. Not just the winner, but the margins.
In 2008, we polled on a bunch of swing states; we got within three points of President Obama’s actual margin in Ohio, Pennsylvania, Florida, Colorado and Wisconsin.
PAUL SOLMAN: And how do you reassure yourself that you’ve got an actual random sample?
DOUG SCHWARTZ: Well, first thing that we do, is we use something called random digit dialing, or R.D.D. That’s the industry standard. Phone numbers are randomly generated by a computer and that way we ensure that we’re getting not just listed phone numbers but unlisted as well.
When we call a household we don’t automatically speak to whoever picks up the phone because they tend to be women, so we randomly select the person in the household to speak. The technique we use called ‘The Next Birthday Method.’ We speak to whoever has the next birthday in the household.
We also call over several days because we don’t want to just reach the easy to reach people. For example, seniors and people with young children are easier to reach. So we’ll call a phone number at least four times, between 6 and 9 o’clock during the week, but then we’ll also try on weekends. A lot of folks are working at night during the week. So, we also call on weekends as well.
We call cell phones. Increasingly, Americans use only a cell phone. We also conduct interviewing in Spanish for those who prefer to do the survey in Spanish. So, it all gets back to this principle of making sure that everyone has the equal chance of being selected.
PAUL SOLMAN: Out of 100 attempts, let’s say, the interviewer gets through to three or four people in a given night. What happens to the other 96 or 97?
DOUG SCHWARTZ: We’ll just keep calling. If no one answers, somebody else will get that number and call. So the [original interviewer] calls at 6 o’clock tonight, gets no answer, that number will pop up for an interviewer tomorrow night at 7 o’clock or 8 o’clock. And if again, no answer, then the number will come up again over the weekend. We try hard to reach the hard-to-reach people.
A lot of the 100, you have to keep it in mind, that are bad numbers, they could be disconnects, they could be businesses, so it’s not just people who weren’t home that we didn’t reach or people that refused. A lot of them are simply bad numbers.
PAUL SOLMAN: I read today that the typical response rate on polls, not yours, but overall, was down to nine percent.
DOUG SCHWARTZ: Yeah, that is the Pew study. That’s probably representative of where it is in the field. Again, looking at all phone numbers dialed, we’re not talking about cooperation rate, once we reached an eligible respondent, people are 50-50 likely do it. But we’re talking about all the numbers that — a lot of the bad numbers included — 10 percent response rate, is about right.
I think as long as it just keeps working, I think you know, people will continue to trust the polls. And what I mean by working is we continue to get the elections right. People trust us. But if all a sudden, the polls are way off, then we’d lose our credibility.
This entry is cross-posted on the Making Sen$e page, where correspondent Paul Solman answers your economic and business questions