secrets of the sat
photo of bob schaeffer
Bob Schaeffer: He is Director of Public Education of FairTest,  a standardized test watchdog group based in Cambridge, Massachusetts.
navigation, see text below
Why don't you read that doughnut question test and tell me what it illustrates.

This is one of the most famous SAT questions. It's called the doughnut question. And it shows you a doughnut. In the figure above, what is the greatest number of non-overlapping regions into which the shaded region can be divided with exactly two straight lines? And we used this at the kick-off news conference that launched Fair Test way back in 1985. And David Owen, who wrote the book None of the Above, showed this to the assembled 40 or 50 press people in the room--who ooed, and awed at it. Fifteen years ago, people hadn't seen this. And how simply you solve it if you have been well-coached.

You read it, what is the greatest number? And you see that, well, this is the last math item in this section. Therefore, it's a very difficult question. You know from looking at the SAT that on every SAT section, exception for reading comprehension, items go from the easiest to the hardest. So at the beginning of the section, an obvious answer is right. At the end of a section, an obvious answer is a distracter. It's wrong. You never go for it.

the doughnut problem And what coaching courses like the Princeton Review and others teach you, is that you can analyze--without knowing how to solve the problem, you can have a very good chance of getting the right answer. You look at it and say, what is the greatest number? Well the greatest number in the answer's six. That's wrong. That's the distracter answer. It's designed to take somebody who doesn't understand it and make him guess wrong. But you look at it and say, well, the simplest thing I can do is I can divide it into four pieces by cutting vertically and cutting horizontally. But that's too easy too. That's got to be wrong. But I could get four. So, I must be able to get three or two because they're smaller than four. And six is wrong because it's the easy answer. The answer's five. That is the right answer.

And, I now know how to draw the lines so you can do it. You draw them so they intersect in the middle, inside of the doughnut. So you get two very small triangles and some large ones. But you don't have to know how to solve that problem. And it has nothing to do with math. It has nothing to do with aptitude. And it most certainly has nothing to do with merit. Unless you define merit as being coached.

Wayne Camara says it measures reasoning ability, math ability, logic.

Well, it does measure a sort of reasoning if you can reason like the test maker, and understand how they do it. It's, one of the things the coaching courses teach you. In which you understand your opponent's strengths and weakness. And you use it to tip them on their but. It's the reason a small woman can knock over a big guy--by using their attempt to subdue you--to beat them. That's the same thing that you're doing on the SAT.

Well, maybe that's merit. It's kind of useful skill to have in life. But no where in their technical material do they ever claim that that's what the SAT is trying to measure--test-wisedness.

You want to read this item?

If we're going to do test items. The next item is another classic. And I'm not sure I remember the answer but I can probably puzzle it out. It shows a digital clock with, the time reading 5:05. The twelve hour digital clock above shows one example of the time at which the number representing the hour is equal to the number representing the minutes. What is the least possible number of minutes from the instant one such double reading appears, the instant the next appears?

We've never said that the SAT is not measuring something meaningful. There's a little part of meaning, a little part of background, a little part of schooling. But there's a lot of test-wiseness--how shrewdly you can play the game. There's a lot that can be taught in coaching courses that has nothing to do with any of the skills you need to succeed in college or in life. And the way you're supposed to puzzle this out is--again, it's number 25 on the map.

Well, I mean, you know what you eliminate? You eliminate 11 because it's the least. You, the, the thing you think of immediately--5:05, well, the next time it does it is 6:06. That's the obvious answer. You can quickly go from 12:11 to 1:01 is the right answer. That's the shortest amount--to, so from 12, from 12:12, 12 minutes after 12 to, 11 minutes after 11. That's less than an hour. So 61 can't be right. Sixty can't be right. We know 11 can't be right because it's the smallest. Even if you can't get any further, you're down to two choices. 30 and 49. And you can guess between the two and you'll increase your score on the average if you guess when you've eliminated more than one item.

Again, I have no clue to how to solve that problem anymore. But I've boosted my score on the average by five points. Each item you get right instead of wrong is worth 10 points. My odds of getting that right are one half by guessing between two items.

Are you saying the SAT is a guessing game?

Part of the SAT is a guessing game. Fair Test's position is reasonably new on it. You know, we've never said that the SAT is not measuring something meaningful. There's a little part of meaning. There's a little part of background. There's a little part of schooling. But there's a lot of test-wiseness. There's a lot of how shrewdly you can play the game. There's a lot that can taught in coaching courses that has nothing to do with any of the skills you need to succeed in college or in life.

Wayne Camara says it's a measure of verbal and mathematical reasoning ability.

Well, Wayne Camara, who I know is a decent person, maybe, he believes that. But if you talk to representatives of tobacco companies, they tell you their products are good and healthy and don't lead to diseases. And auto manufacturers have told you that their cars don't burst into flames. And back at the beginning of the century, you know, stockyard owners claimed that they were slaughtering the animals in a clean way. I think we've learned to trust not to--or not to trust those who profit from manufacturing products, as the sole source of information.

Yes, one of the things the SAT is measuring in small part, are those skills. But it's measuring a whole lot of other things. And when you use the SAT as the major factor--or worse, the sole factor--to make high stakes decisions, to define what is merit, you're relying not on what most people think of is merit. But on these very trainable skills that coaching courses and others help people learn.

Does coaching work?

There's no question that coaching works. The only people in this country who don't believe coaching works are the employees of the test manufacturers. Because they've learned from the past that someone who works there and says coaching works, needs to find another job.

Coaching works anecdotally. I've seen my own son take a coaching course. And his score went up 150 points. Not because he became smarter or had more aptitude, was better prepared for college work, but because the coaching course taught him how and when to guess. And he's a very careful thoughtful kid. Probably the sort of kid that you want in college or in life. But that doesn't help you on the SAT. You get points only when you get items right. And that means you need to complete them all. And the coaching courses teach you how to pace yourself and teach you how to guess. That's not aptitude. That's not merit. But it's high scores on the SAT.

John Katzman says, pull the plug. Kill it. It's a bad test. It's a scam.

Well, we don't disagree with John Katzman. Fair Test's position is that the SAT should be optional in the admissions process. Right now, kids do all kinds of wild things to boost their admissions portfolio profiles. They bake cakes. They submit video tapes and projects. If kids want to take tests on Saturday morning and fill in bubbles and think that that adds something to their profile--let them. But a system right now, where a couple million kids a year are forced to go to the SAT--and another million and some through the ACT, it's competitor, is a tragic waste of educational resources and money. And ends up dis-forming the educational process by emphasizing the very narrow range of things an exam like the SAT or the ACT can measure.

ETS research and reviews the strong evidence that ETS can manipulate test content to change score averages for males and females.

Well, that's absolutely correct, and probably the strongest provable bias of the SAT. What Fair Test has proved it a number of times in federal court and elsewhere, is that it's biased against females. The sole scientific claim made by the SAT--when you get down to the bottom line and strip away all the rhetoric and nonsense--is its capacity to predict first year grades. Well, young women get higher grades than young males across the country in colleges despite the fact that they earn lower SAT scores by about 40 points on the average. There's only two ways to square that circle. Either all the colleges in the country are wrong, they're biased towards girls and give them higher grades than they deserve. Or there's something fundamentally flawed about the test.

Well, a number of researchers asked the Educational Testing Service and College Board--have thought through that problem. And they have shown that you can--by adding different types of questions to the exam--skew the answers very differently so that girls get the same or slightly higher score than males. And in fact, ETS researchers have been able to manipulate other tests to substantially reduce or reverse the score gap between blacks and whites.

Talk about that.

Well in the example of white and black score gap--which, by the way, is a red herring that Fair Test is always slapped over the head of with, that it, some claim that we believe somehow that the whole gap is caused by bias. We never said it. We don't believe it. Anybody who thinks that the differences between blacks and whites and test scores in our country, are only the result of test score bias, is not someone who belongs in our organization.

But to deny that there is some bias, is also ignoring a piece of the problem. Yes, some of that gap is a result of differences in preparation, opportunity--going back to prenatal nutrition. But if tests reinforce that by exaggerating the nature of the gap by the way the tests are constructed, so that the only items that can added to the SAT are ones in which those who have already scored high, perform well, so that items that minority groups might do well on and whites don't do on, they can't be added to the SAT or other standardized tests based on the definition of how they're put together. So it becomes a reinforcing cycle.

Is an item performing correctly only if high scorers get the question right more often than low scorers.

One of the great joys of working at Fair Test is we get to read all the publications that ETS and the College Board and other testing industry advocates put forth, because and they have hang themselves with their own words.

Yes, that is exactly how the test is put together. That means if you tried out a question on say, you know--what's the major ingredient of a tortilla? And Mexican American kids said corn quite regularly. And white kids didn't know or said wheat--but corn is the right answer. That item wouldn't appear on the SAT because the wrong group got it right. Mexican Americans, on average--because of language--score lower than whites on the SAT. Such an item doesn't fit the formula and it's thrown away in the pre-testing. So again, self reinforcing. It's a vicious cycle.

Test takers were able to answer nearly 40 percent of reading comprehension questions correctly without reading the passages themselves...

I read that report when it first came out. I said, well that's interesting. And then, I did it myself. I actually did it when they launched the new SAT, before a bunch of reporters at the College Board's annual conference. They handed out this sample of the new, more complicated reading problems. And I said, well let's look at this item without any knowledge and see how it works.

The reading problem compared a funeral speech by Pericles and Lincoln's Gettysburg Address, which most well educated people have some knowledge with. I didn't read the passages. I read the questions. I got 11 out of 13 correct without ever reading the passage.

Why is that important through, other than to show how smart you can be, in a way that the test supposedly isn't scoring? It's because kids who've been coached know that. So if you're running out of time and you have this lengthy reading passage in front of you, and then you get to the questions. Instead of reading the passage--times up. You get no points. You read the questions and answer them without regard for the passage. And you'll get some points. Every item you get right--another ten points. You get five of those 13 items right-- you're up 50 points on the SAT. Bang, like that. Does it mean you have more knowledge or ability or merit? Probably not.

There's a suit that was just filed about ten days ago, on behalf of eight kids who didn't get into UC Berkeley, by five legal defense groups. What they're claiming is two things. One, they don't have access to AP courses state wide in proportional numbers. Two, that UC relies too heavily on SAT's. Therefore, because of the gap, it's discriminating against them. Can you talk about that?

Fair Test worked very closely with the civil rights groups that put that together. And I did a lot of research personally, for that suit. What they're saying there is that the SAT is actually a very good measure of cumulative disadvantage. And if that's what we're trying to reward in our systems and define it as merit--yes, the SAT performs a pretty decent service. But if you want to define merit as something much more meaningful, the SAT falls way short.

What they say, and one of the plaintiffs --you, take, a kid in California who's a child of migrant farm workers, who despite the background, and despite moving around, and despite possibly having to drink poisoned water in the formula--still excelled in high school. Got a 4.0. Was a leader in groups. And probably got quite a decent score on the SAT. That person was not admitted to UC Berkeley. And in his or her place was someone who grew up in Beverly Hills, had all the advantages of a nanny who read to them and parents who were college educated in the best of private schools, and an SAT coaching course. And that person had similar grades. More AP courses because they're in Beverly Hills and not in the Valley. And a higher SAT score. Possibly in part because of coaching. How can you possibly say that the kid from Beverly Hills has more of this mystical thing called merit? You got to at least say that the migrant farm worker has at least the same, or maybe more, because he's over come his background. Yet a system that relies heavily on test scores, ignores all those other components. And says it's only how well you do on this very narrow measure that matters.

How can they possibly take a person who scored lower than someone else?

Well, first of all, the test makers themselves in their technical manuals--which are far better than their advertising--say a 100 points difference between two candidates is absolutely meaningless. It's within the margin of error. Called the standard error of the difference. So that you can't even really be sure that those two scores are that different. That if they took the test again, they might possibly reverse. And they warn explicitly about making decisions based on those kinds of narrow differences.

Now that's one of those rules that is ignored very widely. And of course, the test makers--for their profit reasons--do absolutely nothing to stop institutions which over rely on the SAT or use them as a sole criterion for scholarships. But, why would you argue that that means that you have more merit rather than the kid who's done nearly as well on the test, equally well in the classroom and overcome so much? Which one's going to be a better contributor at UC Berkeley? Which one's going to be a better contributor in life--in California or this country?

The average difference between whites and Asians and blacks and Latinos is--for Latinos it's about 150 points...

Well, even a gap in the 150, 200 point range may not be that meaningful if the kids with the higher score have taken coaching courses which claim to raise scores in that range. The independent research suggests it's a slightly lower--more like 100, 120 points. How do you know that the scores that you're seeing-- whether they're the result of some kid walking in and taking the test cold on a Saturday morning? And the results of some other kid who's been tutored for $700 at the Princeton Review or Kaplan, or $1,500 for some tutor who comes to your house and drills you on the test? Those scores don't mean the same thing.

The colleges are making judgment based on a host of factors. Which include grades, and class ranks, and community service, and all the other kinds of things that particularly a public university system wants to see in its undergraduates and in its future alumni. And that seems perfectly reasonable to balance all those factors.

Derek Bok actually feels the SAT is a common yardstick.

Well, as much as I respect Dr. Bok and the excellent book that he wrote, he's wrong about that. The test makers themselves tell you that high school performance, either as grades or as test scores, is a better predictor than the SAT. Now you think about that. And that's a real important indictment of their own product. Because everybody knows grades vary all across the lot. There are tough schools and there are really very easy schools, even within the same school. Despite all that variation, the test makers admit that grades are a better predictor than their test is. What does it say about the test?

The second answer to Dr. Bok is the experience in Texas and the other several hundred schools in the country that don't require test scores as part of their admissions process for some or all of their applicants. You end up with a better entering class, in terms of other factors--including general academic performance--if you ignore test scores. You look at the very competitive elite end of the university spectrum. Places like Bates and Bowdoin in Maine. Which are not quite at the Harvard level of selectivity but reject two thirds to three quarters of all their applicants. Both of those schools dropped test scores back in the 1980's and both are very, very happy with SAT optional as a way to admit students. And they say they get more and better applicants from across the board.

The other thing--and I've offered this many times at college admissions conferences--is that, okay admissions officers, your administration believes that the SAT is so useful. Let's say the rules were changed. Instead of parents paying for the test to the tune of over a 100 million dollars a year, colleges paid for this useful advice. My hypothesis is, Fair Test's hypothesis is this: If the SAT is gone tomorrow morning. How many of you disagree? I'm waiting for that first hand to go up.

Another reason, the hidden reason, colleges like the SAT--is they get, along with the test scores, a ton of other very useful information from the student demographic questionnaire. Where the kids fill out all kinds of information about their history, their college preferences. And colleges use it for recruiting. You got to college admissions conferences here in the late 1990's, the major focus is what's called yield management. Not, how do we be selective? It's, how do we put buts with tuition checks in the seats? In the vast majority of the colleges the reality is, they're looking for three real factors. High school diploma. You're alive. And there's a tuition check. That's what gets you in at the vast majority of colleges. There's tremendous amount of recruiting that goes on. Even from name brand little Ivy colleges, for kids who have decent--not super--academic records. They are desperate for bodies. And the SAT is a sideline to that. But they get the recruiting information along with the test scores.

Professor John Yoo says, oh, great, now because we've passed prop 209--which he supported because of views of fairness-- the standards are going down in the sewer.

Well, if he really said, limited the damage--that sounds like a fundamentally racist position. Because the data on what affirmative action did do, in California when it could legally be practiced, was that it diversified the jewels but it did not at all lower the quality. It's--the notion that you're playing off equity on the one hand versus excellence on another, is a false dichotomy. You can have them both.

That there is research done I know on the medical schools in California and they looked at the career paths of those who had been admitted under affirmative action in California. I think one could argue that what those young people did with their lives was more valuable, more meritorious to society, than those higher scores perhaps who they replaced. These are people who are in things like community medicine. And in fact, their profiles did not look significantly different, than people who were not admitted according to affirmative action.

There's this notion that somehow among the elite you can very delicately rank order based on small differences in test scores. When in fact, probably everybody in the top third of test scores for something like Boalt as a law school or one of the University of California med schools or UC Berkeley, could do perfectly well. I mean, at MIT where I went to undergraduate school, the admissions office has admitted that probably half their applicants could do adequate level MIT work. And yet they're rejecting three out of every four. So they're throwing away other people who are meritorious based on some other factors.

What is your feeling about the significance of the black/white, test score gap?

Well, first of all, that's not--it's another one of those non either/or. Indeed attention needs to be paid, more attention, by society to some of the controllable factors that go into the black, white test score gap, everything from infant nutrition through lousy schools, etc. But when kids through that system perform well--even though they might not do quite as well no the test scores for blacks who graduated from high school, as white, their white counter parts maybe--you maybe do want to take into account the obstacles they have overcome and give them at shot at the next level, as a way you define merit. But defining merit solely as this test score, particularly a test score that does not predict very well at all, ends up reinforcing the biases that led to that lower test score.

What does the SAT predict?

The sole scientific claim of the SAT is its capacity to predict first year grades. According to the technical studies done by the Educational Testing Service and College Board, the SAT predicts about one factor in six--one sixth of the difference between two kids first year grades. The predictive value declines after that--looking at four year grades or graduation rates. So even the test makers agree that five out of six parts of whatever it takes to predict how well you're going to do in your freshman year, is not their test.

It does correlate extremely highly with an IQ test. It was developed from the army IQ test...

That's part of the seedy under side of the SAT. The SAT was originally developed by straight out racists, eugenicists, people who thought my forbearers--not just people of color--were imbeciles and shouldn't be allowed in their country because they didn't know the language and couldn't score high on their test. I wouldn't suggest the current people who run those companies share those kinds of ugly views. But it's a self reinforcing notion of defining intelligence as that which whatever the dominant group in society has. Ends up giving that group higher scores and lower scores. The fact that test scores correlate with test scores is rather meaningless. The tests are measuring the same set of factors. What's more important is whether the test accurately predicts how well you're going to do.

Would you say that we are the only country in the world that administers a national IQ test?

Well, despite the efforts of the Educational Testing Service-- which is a global corporation with nearly about half a billion dollars in total revenues--the US still is the major country that administers a test like this across the board to college bound seniors. If you take the SAT and its competitor, the ACT--which about 80 percent as many kids take-- the vast majority of college bound kids take those tests. And yes, the SAT in particular has its roots in IQ testing. Which is, are at best controversial and, at worst, quite, quite poor predictors of anything of value.

So is it an IQ test?

It's a variant of an IQ like test. It is set up somewhat differently. It begs the question of, what is an IQ test measuring? What is intelligence? And you talk to test makers. And intelligence is what their test makes. And that's a circular definition. So to the extent that it's measuring the same that an intelligence test is measuring--then, yes it is. But there's three fallacies there: That there is such a thing as intelligence. That it can be measured. And that you can put the measurements on a linear scale. And other, even people who believe that there is such construct as intelligence believes that this thing is not one thing but seven or eight or possibly nine different things. And Robert Sternberg at Yale says it's three different things.

At best, the SAT is badly measuring one of those parts of what goes into intelligence.

Should we get rid of it?

You don't need a test at all. People like Katzman--and he's in the test coaching business. He'd have to retreat to his summer home in the Hamptons if he didn't have it. He says you don't need tests at all. It's like a hypochondriac walking around with a crutch even though they're perfectly able. You say, we're taking your crutch away, and they say, can I have a wheel chair? But they can walk.

Two hundred and eighty one colleges have shown that they can do admissions well. Everything from very competitive to not so competitive, without looking at test scores at all. All the colleges in the country could in fact do that.

Wayne Camara said, our analysis show that it's 30 points--maybe 43 points this time around...

Maybe. He's talking about a study released early this year. I looked at the methodology. Here's how they did it. They did take a group of kids. Give them a pre-test to see their base line. Coach them. And then see the results. Which other people have done and found 130, 140 points. What they did is, after kids had taken the SAT and before they got their score back, kids received a letter on ETS letter head saying, were you coached? How much were you coached? Fill in the bubbles and tell us. And they expect that they got honest answers from kids in that kind of setting. When the coaching companies tell them explicitly, if you're coached, it's up to you to say. they have no idea whether these kids were coached or not. It's, ah, it's quite despicable research to be making the claims they did based on their methodology, which is fundamentally flawed.

We've asked Don Stuart, the president of the College Board, to sponsor that--directly to his face. He's taken it under advisement for a decade. They wouldn't do it because if the coaching works, they're dead. And they can't take the risk because coaching works.

I'm hitting hot buttons here. They are just such liars about coaching. It's, I mean, they are the tobacco companies about coaching.

Because they won't do the research?

They are unwilling to risk endorsing research that could show that their product doesn't work the way they say it does. Back historically, ETS has fired researchers who concluded that coaching works. A guy named Lewis Pike was fired and wrote about it, back in the 70's. It's their soft under belly. And they can't risk exposing it.

But the difference between them and the tobacco companies is, they're supposed to be non-profits operating in the public interest.

Let's go back to the History...

Standardized testing, since it came to this country, in the early part of, of the 20th century, has been used primarily to sort and to reinforce existing social structures. Everything from IQ test used for immigration, to the SAT used for college admissions. Some of the earliest proponents of the SAT were virulent racists and anti-Semites. And it goes without saying at that stage, misogynists.

Carl Brigham was, at the time he was appointed to be the first secretary of the College Board, very heavily involved and I believe he became the president of the American Eugenics Association. Other early testers, Yerkes and other were, were absolute racists. Brigham recanted on his death bed apparently. And I believe that's legitimate.

That's how the tool has been used historically. Testing is a tool that is used to, to funnel black kids, in the elementary schools into dummy classes. It's used to make sure that even in integrated school districts, that the white suburban elite has the advanced classes, so it serves that purpose. Is that the reason why test makers support the product? No, it's not the sole reason, but it meets a need for others in society.

home | discussion | who got in? | interviews | the race issue | sat & test prep | history of the sat
the screening process | test score gap | getting in to berkeley | bibliography | links | tapes & transcripts | press | synopsis

FRONTLINE | pbs online | wgbh

web site copyright 1995-2014 WGBH educational foundation



../test/ ../race/ ../interviews/ ../who/ ../talk/ ../