By the People: Election 2004 . Savvy Voter | PBS

Savvy Voter - Analyze a Poll

They may be controversial, but there is no question that political polls are a central fact of American politics - ubiquitous and influential. What do polls really tell us? Can the results be trusted?

Learn what it takes to distinguish an informative poll from a misleading one.

Analyze a Poll

During the election season, Americans will be inundated with data from polls--polls that tell us who's ahead of whom, by how much, and why. With election night will come exit polls to tell us who won, by how much, and why. These polls are commissioned by candidates, political parties, interest groups and media organizations and they can enlighten but they can also distort or even mislead, sometimes intentionally.

Questions to Ask
Citizens should be astute consumers of poll data, aware of the methods and dangers involved in conducting and interpreting a survey. A smart consumer of polls need not become an expert at drawing samples, constructing questionnaires, or analyzing data, but he or she should ask some of the following questions to better understand the implications of the poll results:

  • Who sponsored or paid for the poll? Who conducted the poll? Who thought the topics in the poll were important enough to spend money finding out what the public thinks? A poll is conducted for a reason, either to gain information or to advance a particular cause. The sponsor's motive for doing a poll should raise questions about the validity of the poll. Polls by campaigns and special interest groups often contain slanted questions and employ inappropriate sampling methodologies.

    Who actually conducted the poll? A reputable firm will avoid problems arising from the wording and order of the question, sampling design and execution, interviewer reliability, and the analysis and interpretation of the results.

  • What population is being analyzed? (The population is the group being targeted by the poll.) Three populations most often targeted for political polls are:
    • persons aged 18 or older
    • registered voters
    • persons who say they will vote in the relevant election

    The sample is the part of the population selected for analysis. The sample must be representative of the broader population or the poll results will be worthless or misleading.

  • How were respondents chosen? Respondents in a sample should be chosen randomly or with a known probability. In a simple random sample, for example, each unit of the population has exactly the same chance of being questioned as any other.

    The first rule of survey sampling is that respondents cannot select themselves. Self-selection introduces socioeconomic and demographic biases; "man-on-the-street" polls, straw polls, call-ins, and mail-ins all violate this rule. Most online polls are also not scientific in nature and should be taken as entertainment.

    One example of the impact of poor methodology on poll results is the 1936 Literary Digest poll. 10 million ballots were mailed to households listed in telephone directories or state auto registrations. This introduced significant biases since, in 1936, Republicans were more likely than working class Democrats or Southern farmers to have phones or cars. 2.4 million ballots were returned, but this type of mail-in return allows people to self-select as poll respondents. The problem with self-selection is that people who respond to mail surveys tend to be better educated, have higher incomes, and feel more strongly about the matters dealt with in the questionnaire than the general population. As a result of so many errors, the poll incorrectly predicted that Republican Gov. Alf Landon of Kansas would defeat Democrat Franklin Delano Roosevelt for President.

    Sampling arouses the greatest public skepticism about polls, while, in fact, it is often the least troublesome. How can a sample of 1,000 or 1,500 accurately measure the views of 195 million adult Americans? Pollsters have a joke: If you don't believe in random sampling, then the next time you go to the doctor for a blood test, have him take it all. It only takes one drop of blood randomly drawn from the body to test for cholesterol, just as it only takes a sample of the population, properly drawn, to know the views of millions of Americans.

  • How were respondents contacted and interviewed? Poll respondents are interviewed either in person or by telephone. Pre-election polls are usually telephone polls. The most common telephone polling methodology is some form of random digit dialing in which random numbers are generated to produce the telephone numbers to be called - both listed and unlisted.

    Even the most rigorous design rarely, if ever, produces a purely random sample. There is always "non-response" - people don't have phones, they refuse to be interviewed, or they aren't home. Although the Census reports that 95 percent of American households has a telephone, those without phones are disproportionately poor, black, Hispanic, older, and rural - thus introducing some upper class bias.

  • What is the size and error of the poll? You pay a price for using samples rather than interviewing everyone in a population. The discrepancy between the sample results and true population values is called sampling error or the margin of error. The more people interviewed in a scientific poll, the smaller the error.

    What error margin is considered acceptable? The error margin for most national polls is +/- 4 percent or less for the entire sample.

  • What questions did the poll ask? In what order were the questions asked? An accurate and informative poll questionnaire must be properly worded and ordered. Bad wording is often the deliberate result of interested parties whose aim is to generate specific responses. One of the best tests of a poll question is your reaction to it. Does it seem fair and unbiased? This "smell test" is not foolproof, however. Seemingly innocent variations in phrasings such as "aid to needy" vs. "public welfare programs," can produce very different poll results.

    One of the most serious criticisms of polling is that it produces "non-attitudes." This is reflected in respondents answering questions on which they have no genuine opinions, but don't want to admit that they are uninformed. In Cincinnati, in the late 1970s, respondents were presented with the following about a fictitious Public Affairs Act: "Some people say that the 1975 Public Affairs Act should be repealed. Do you agree or disagree with this idea?" One third of the respondents offered an opinion.

  • What are the actual percentages upon which conclusions are based? What are the actual numbers? Sometimes poll results are reported using terms that have no universally agreed upon meaning - most, more, slight, sizable, substantial, overwhelming - or terms that have ranges of meanings - majority, plurality, minority. These terms can mislead. Even exact numbers can become imprecise by pairing them with imprecise modifiers: "fully 20%" sounds quite different from "only 20%."

  • When were the interviews conducted? Polls measure sentiment at the time they are conducted -- not next week, not on election day. Pre-election trial heat or horserace polls can have exceedingly short shelf lives; events can have dramatic (and subtle) impact on results.

The 1948 election polls by Gallup, Roper, and Crossley all forecasted that Republican Thomas Dewey would beat Democrat Harry Truman. The Chicago Daily Tribune even printed a headline stating that Dewey had won. The bad call was based in large part on the fact that polling had stopped too far in advance of the election and, thus, did not reflect many Democratic defectors deciding at the last minute to vote for Truman.

Polling organizations now survey right up to the day of the election - often through tracking polls which use rolling samples. For example, a poll might interview 200 people each night for three nights for a total sample of 600; on the fourth night another 200 are interviewed, but the first 200 are dropped from the analysis; on the fifth night another 200 are interviewed and the second 200 dropped, and so forth.

  • What other polls were done on this topic? Do they say the same thing? Results of other polls should be used to check and contrast poll results. First check the timing. The difference may represent a movement in public opinion. Then check the population being studied. Do the polls have the same populations (registered voters, all adults, etc.)? The question wording and order, as well as error margins, also can explain discrepancies between poll results.

The Hard Part
Even if the answers to these questions are available and satisfactory, there is still the matter of interpreting the data. Did the analyst read the data appropriately and draw correct conclusions? Analysts have tremendous leeway in deciding which items to analyze, which subsets to present and how to interpret the statistical data. Interpretation of poll numbers is not an automatic and objective enterprise. There is a high degree of subjectivity, judgment, and potential for distortion - intentional and otherwise.



Learn More

The Online NewsHour: Poll Mania (http://www.pbs.org/newshour/bb/media/july-dec03/polls_10-15.html)
The Online NewsHour speaks with experts about the increasing reliance on polls this election season.

NOW with Bill Moyers: Margin of Error? ("http://www.pbs.org/now/politics/nowpolls.html")
NOW offers tips and advice on understanding poll results.
The NewsHour with Jim Lehrer: Deliberative Polling.(http://www.pbs.org/newshour/btp/dop_background.html)
Find out more about the new technique of deliberative polling and why some people feel it gives a better sense of the mind of the voter.

....................................................................................................................................................

Doug Muzzio, Professor of Political Science at Baruch College, The City University of New York, has authored many studies on American public opinion and voting behavior.

This essay draws heavily from Sheldon Gawiser and G. Evans Witt, "Twenty Questions a Journalist Should Ask About Poll Results," as well as from Herbert Asher, Polling and the Public, 3rd ed., Robert Erikson and Kent Tedin, American Public Opinion, 5th ed., and Norman Bradburn and Seymour Sudman, Polls and Surveys: Understanding What They Tell Us.

....................................................................................................................................................