Steven S. Ross
 |
 The poll is the election.
Campaign polling by the media now borders on the unethical. It's not
time to stop polling. But it is time for a new code of ethical
practices. There are simply too many bad polls and too much uncritical
reporting on bad polls conducted by others -- including academic
institutions.
Don't get me wrong. I'm all for media polling. But many polls, if not
most, are badly designed, badly executed, and badly reported. It takes
money and talent to poll well. Many media organizations are short of
both. Their polls may provide "newsworthy" results only because they are
done poorly enough to be "surprising."
That's a shame, because polling drives almost all of politics today:
- Candidates do early, confidential polls to show to their potential
funders.
- Candidates poll to find out what their positions are or should be on
issues -- the same way toothpaste companies poll to find what features
customers might want to pay for. This is not quite as obnoxious as it
sounds. Various candidates occupy various places in the political
spectrum, and use polls more to fine-tune than to make wholesale image
transplants. Likewise, toothpaste companies know they are in the
toothpaste business -- and use market research to refine the features of
their products and to target their competitors.
- During the campaign, candidates "track" their progress with weekly or
daily polls, to see how well the public is responding to their actions
and to the actions of their opponents.
Thus, news organizations must do their own polls, in part to mimic the
candidates' internal polling and thus put candidates' actions into
context.
Unfortunately, the media usually chooses to use polls for "horserace"
reporting to distract readers from the shallowness of their reporting on
issues. (link to poll types; see sidebar at bottom of this file)
Thus, polling, in a sense, is not simply part of a political campaign.
Polling IS the campaign -- the hidden campaign that few actually see.
Long before Election Day, usually, the candidates' carefully planted
impressions, guided by polling, have decided the outcome.
The media marvels at how the candidate who is "behind" almost always
closes the gap as Election Day approaches -- not enough to win, but
enough to make the race seem close. What's really happening is that the
candidate who is ahead will try to play it safe. Better to let the
opponent get close than to announce some new initiative or commit a
campaign gaffe that could cost the election! How does the candidate now
how far he or she is ahead of the pack? Polling.
That's why "horserace" polling is both important and misleading. It is
important so newspapers understand when a candidate moves into this kind
of "prevent defense." It is misleading because it describes what would
happen "if the election were held today." The election isn't being held
"today," however. It is being held on Election Day.
News organizations also tend not to follow their own polling data. As of
May 2000, for example, Pat Buchanan was getting far more media attention
than Ralph Nader -- although polling suggested that more American voters
are for Nader than for Buchanan. (For instance, Nader gets 5.7 percent
to 3.6 for Buchanan in an early April Zogby poll. At these low levels
and with sample sizes for the overall poll at about 1,000, the two are
barely distinguishable statistically, but Nader was certainly not behind
Buchanan.)
The Minneapolis Star-Tribune treated Jesse Ventura's candidacy as a joke
until he polled about 25 percent of the electorate. He ended up winning
the Minnesota governorship with 37 percent of the vote in a three-way
race.
Typical polls are also much less accurate than readers -- and editors --
believe. Voters are getting harder to sample:
- Most (some pollsters say 80 percent or more) refuse to talk to
pollsters who call. Thus, pollsters have to make five calls to get one
valid response. A decade ago it was two or three calls.
- The strong economy allows people to move more frequently, changing
their voting address. This makes it more difficult for pollsters to
confirm that the people being polled are indeed likely voters.
- Movement of people, and a new, strong trend toward quasi-marriage and
racial intermarriage have made it more difficult to "profile" a voting
district. That is, it is harder to determine voters' age and race to
draw a representative sample for polling. Let's say that 25 black
females from a certain district are asked their opinions on an issue. Do
they stand for 3,000 likely voters in the district's population, or
4,000? If 4,000 then the answers of the 25 might be magnified in the
poll 33% more than if they stand for 3,000. Is a family counted as
Hispanic if only the husband is Hispanic?
- Voters' opinions are often quite fluid and changeable. Times are good
for most likely voters these days. Thus, they tend to worry about a lot
of issues, but don't worry too much about any of them. Small changes in
wording can generate different impressions. A Washington Post/ABC poll
in early April had one question that suggested the public trusts Bush to
do a better job than Gore handling the national economy (47 to 41
percent). But two-thirds of the respondents also said Gore would do a
good job "keeping the economy strong." In late March, a Pew survey had
Gore and Bush even (44 and 43 percent) on the general question, but Bush
ahead 41-25 percent on who would deal better with rising gasoline
prices.
- As politicians know too well, voters forget. So do editors. The
Vanishing Voter Project at Harvard's Kennedy School of Government says
40 percent of the electorate said it knew (in February just after the
New Hampshire primary) whether Bush "favors or opposes requiring people
to register all the guns they own," and 24 percent (or 60% of the 40%)
actually got it right -- that Bush opposes gun registration. By April,
only 24 percent said they knew, and only 13 percent actually did.
Most importantly, even if a perfect sample of voters can be polled, the
results will still not mirror the real world with exactitude. It's the
way God made numbers.
At first this sounds bizarre. But if we think about it, we see
commonplace examples in everyday life. For instance, let's think about
what happens when we "poll" a penny about whether it will vote heads or
tails. We understand intuitively that if a coin is perfectly balanced
and it is tossed many thousands of times the poll will end up even. Half
the time, the penny will vote "heads" and half the time it will vote
"tails."
We also understand that if we toss the penny only 10 times it may vote
seven heads to three tails. Or nine tails to one head. We call this sort
of thing a "winning streak." With thousands of tosses, there are many
streaks and they tend to cancel each other out.
Choosing a person to be polled is like tossing a penny. One person may
say Bush (Heads!). A second person, demographically identical, might
choose Gore (Tails!). The pollster might talk to one and not the other
... and thus must talk to hundreds of voters to make sure the streaks
cancel out.
Not so intuitively, it turns out that such "luck" is predictable.
Mathematician Jacques Bernoulli lived between 1654 and 1705, so he never
met a Senator. But he understood the process. He said that if we poll
the coin 1,000 times we will not always get a 500-500 tie.
In fact, he said, we will RARELY get a 500-500 tie. But 19 times out of
20 we will get a result between 465 and 535 if the coin is perfectly
balanced.
The "19 times out of 20" is what the statisticians today call the
confidence level. If you divide 19 by 20 you get 0.95 or 95 percent. The
difference between 465 and 500 (or 50 and 535) is the margin of error.
In this case, it is 35. The 35 divided by 1000 (the size of the sample)
is 0.035 or 3.5 percent.
When polls are done by the print media, or reported for the first time,
they usually carry a disclaimer based loosely on Mr. Bernoulli's math.
For a poll of 1000 They say that 95 percent of the time, the reported
results are within 3 percent of the results that could be expected if
the entire electorate were polled.
That statement is roughly in line with one approved more than a decade
ago by the American Statistical Association. It actually overstates the
perfection (Bernoulli's formula says that for the confidence level to be
95 percent in a 1000-person poll the error limits must be widened to
plus or minus 3.5 percent). Also in line with the ASA, some news
organizations note that the error limits for subsamples are larger. They
never say how much larger, however. Thus, if a newspaper polls 1000
people and 500 are women, the error limit for women's opinions is plus
or minus 5 percent at a 95 percent confidence level. What about the 100
black females in the sample? The error limit is more than 10 percent!
Remember, this assumes that the sample is perfectly drawn -- and it can
never be. Furthermore, Bernoulli is fairly generous. Other
mathematicians note that the error margin gets larger -- the answer gets
less precise -- as the sample gets closer to a 50-50 split. Also, the
errors increase as the number of choices increase -- in a three-way or
four-way race, for instance.
The "gold standard" for polling is a confidence level of 95 percent.
That is, again, that 19 times out of 20 the poll will be within a
certain percent of reality. There are so many polls, however, that some
are bound to be, well, wrong. And that's so, even if they are well done.
No poll I've read over the past decade points this out. But even a large
poll by a reputable firm can be wrong.
The Wall Street Journal and NBC News, for instance, conducted a joint
poll throughout the Dukakis-Bush presidential race of 1988. On October
18 they reported a poll of more than 1,300 likely voters showing Bush
ahead by 55 to 38 percent. No other large poll showed such a huge gap
between the two candidates. Part of the reason may have had to do with
the timing -- the poll was taken just after a debate that Bush was
deemed to have won. My review of all polls taken during the last month
of that campaign suggests the poll was simply among the 1 in 20 that
fell outside the error limit (of about 3 percent in this case). The real
gap was probably 50-43 at the time. Of course, the 55-38 percent gap was
widely reported... drying up the last of Dukakis's campaign donation
stream.
It is bad enough that news organizations don't include all the relevant
statistical information in their own stories. They tend not to include
any such information when they report on polls by others!
This fools the reader. The reader assumes that if one news organization
reports on the work of another, the organization is applying its own
good news judgment that the original report is worthy of repetition. In
fact, just the opposite is true. A poorly done poll is more prone to
error, and thus more likely to produce "surprising" results. Among
editors and reporters, another word for "surprising" is "newsworthy."
The New York Times runs a regular "campaign briefing" column, usually
compiled by B. Drummond Ayres, Jr. These compilations are particularly
dangerous when reporting polls because they simplify results in one or
two paragraphs. On April 28, for instance, the Times reported that Bush
was leading Gore by 8 points in Ohio, 47 to 39 percent. The poll, taken
by the University of Cincinnati, covered only 531 likely voters and was
taken over a long period -- April 5 to 22. The Times reported that, but
should not have reported the poll at all.
The New York Times also said on April 28 an Elway Research poll taken
April 11 and 12 in Washington State showed Bush ahead 39 to 38 -- a dead
heat statistically but surprising because the state had been considered
Gore territory. The poll was of only 400 voters and was said to have a
margin of error of plus or minus 5 percentage points. (Actually 6 under
Bernoulli's Law.) Gore could actually be comfortably ahead -- and was in
some polls.
The public does like horse races, and tends to lose interest when the
race is won. The Joan Shorenstein Center on the Press, Politics and
Public Policy at Harvard has been polling voters weekly since last fall,
asking what (if any) political issues they have been following and
discussing among themselves. The Center finds that public interest was
very low throughout the fall of 1999, rose during the spring primaries
and then fell sharply by early March when the major party nominees
emerged.
It is up to the press to go beyond the horserace, however. Perhaps we
can start by actually engaging the public in the issues in a meaningful
way, and debunking the cheap shots and soundbites that have become
commonplace in political ads.
Steven S. Ross is Associate Professor of Professional Practice at the
Columbia University Graduate School of Journalism.
|
|
