Beware of economics: The perils of cost-benefit analysis

Counting everything may be about as useful as counting nothing, says psychologist Barry Schwartz. Photo by Flickr user Terry Johnston.

Counting everything may be about as useful as counting nothing, says psychologist Barry Schwartz. Photo by Flickr user Terry Johnston.

Paul Solman: Swarthmore’s Barry Schwartz is one of the country’s best-known psychology professors, a prolific author, a TED talker, and a sometime contributor to this page. We profiled Schwartz and his argument about the paradox of choice — too much of it can paralyze us — on the NewsHour in 2003. Watch that segment below. The thesis has come under attack recently and Schwartz brought our readers up-to-date on the paradox here last month.

In this post, Schwartz questions the bedrock assumption of economics — cost-benefit analysis, beginning with the application of that analysis to early childhood education, which Nobel laureate James Heckman lauded on this page in an especially popular post, and eminent Harvard child psychology professor Jerome Kagan then responded to, skeptically, in a post that drew even more attention.

Where does Schwartz wind up? “Counting everything may be about as useful as counting nothing.” Here’s his argument.

Barry Schwartz: Pretty much everyone seems to support universal early education, which is a good thing. But more cautious observers point out that the outcome data for early education are a mixed bag. The cases in which it seemed to have lasting effects (Perry and Abcedarian programs) were extremely high-quality, with low student-teacher ratios.

The story on much less expensive and uneven-quality Head Start programs is decidedly mixed. There is much less evidence of support for massive expenditures in practice than for the idea of pre-K education in principle.

How does one justify the billions that high-quality, universal pre-kindergarten will cost? Supporters point to downstream benefits in educational attainments, college degrees and good jobs. Similarly, in defending the billions spent creating systems of digital medical records, supporters point to the downstream benefits in reducing serious medical events resulting from overlooked bits of patient history and missed perilous drug interactions that digitized records will prevent.

In both cases — indeed in countless other such cases — such downstream benefits are denominated in dollars, so that apples can be compared to apples. This makes it possible for people to say things like, “for every dollar we invest on pre-K education, we will reap X dollars in return.” No one thinks that such dollar estimates are the only benefit to pre-K education or digital medical records, but they are an important benefit.

Over the years, policy makers and legislators have become enamored of this sort of cost-benefit analysis of potential public policies. Indeed, it is a requirement that all policy initiatives be accompanied by such an analysis to make sure that the initiatives are actually “investments,” with substantial positive expected returns.

There is lots of room for disagreement about such estimates of future benefits, as there is about almost any effort to predict the future. As skeptics like to say, “economists have predicted nine of the last three recessions.” But at least with cost-benefit analysis as the agenda, experts will be disagreeing about the right things, and from such disagreements, more accurate estimates of the future and wiser public policies will emerge.

There is much to admire about the effort to do cost-benefit analysis prior to adopting and implementing a policy. It seems to substitute facts for opinions, data for intuitions and rational analysis for naked political power. It gives people with real expertise a chance to contribute to the building of a rational public policy. Every policy has trade-offs, and there are always more good ideas than money to pay for them. Cost-benefit analysis is a way to assess the trade-offs and choose the most beneficial policies, and thus replace the power brokers with data miners.

The problem, I think, is that there is much less “objectivity” to cost-benefit analysis than meets the eye. There is no algorithm to tell us what should count as a “cost” and what should count as a “benefit.”

Consider an article published by Michael Pollan in the New York Times Magazine more than a decade ago. Pollan bought a baby steer and watched it develop as it was readied for slaughter. He knew what he paid for it, he knew what the rancher paid in feed and other expenses to nurture it, and he knew what it brought in returns at slaughter. You buy it for $X, you rear it for $Y, and you sell it for $Z. If $Z is greater than $X plus $Y, you’ve made a profit.

Pollan was focused on the cost-benefit differential between corn-fed beef and grass-fed beef. It was more expensive to raise steers on grass, and thus more expensive to buy grass-fed beef at the market.

But what Pollan wondered about was the true cost differential between corn-fed and grass-fed beef. That is, what about the federal subsidies for raising the corn that feeds the steers? Should that cost be added to the supermarket price of beef?

What about the medical consequences of creating disease resistant bacteria, owing in part to the prophylactic use of antibiotics in ranching? What about the fact that corn-fed beef is fattier than grass-fed beef, leading to more obesity and cardiovascular disease in the people who consume it?

What about the fact that when steers eat corn, the acidity of their digestive system matches the acidity of the human digestive system, allowing bacteria to survive the trip from steer to person? With grass-fed beef, incompatibility of stomach acidity would assure that virtually all bacteria that can live in the steer will die in the person.

Finally, what about the need for petroleum to make the fertilizer used to grow corn? How much of U.S. foreign policy is shaped by the need for oil? And how much of that would be altered if we used less fertilizer?

Suddenly, the seemingly straightforward cost-benefit analysis had grown very complicated. Should all the costs Pollan identified have been included? If not, why not? Chances are quite good that if the cost-benefit analysis were comprehensive, grass-fed beef would start to look like a bargain, though many of the costs that Pollan identified would not be borne by the rancher, and thus are what economists call “externalities” – costs and benefits — not reflected in the market price of beef.

What this example shows us is that long before the folks with spreadsheets start entering the data, using the best techniques of economic analysis and forecasting, decisions must be made about which costs and which benefits should count.

Decisions about what to include will always influence the results of the analysis, and may frequently determine it. These decisions are political, philosophical and moral — resolvable, if at all, by public discourse and disagreement, not technical expertise. The big question — how do we do the accounting — is not decidable with rigorous empirical analysis. The smaller question — what numbers go in the cells of our spreadsheet — may be.

Americans now face exactly this issue when it comes to assessing the costs and benefits of the Dodd-Frank legislation to regulate the activities of financial institutions. Dodd-Frank is the law, and regulators are now trying to turn that law into a set of specific rules. The bankers, as a group, oppose such regulation, and are trying to water it down at every turn. According to a recent piece in the Times by Eduardo Porter, bankers argue that such regulations will cost more than 3 percent of economic output and up to 7.5 million jobs. That’s quite a hefty price.

But Porter wants to know just what the cost of the economic meltdown is. Surely, we can’t assess whether losing 7.5 million jobs is “worth it” without an estimate of what it will cost if we fail to toughen regulations. There are several attempts to estimate the cost of the meltdown, some of them as high as $150,000 per person in the U.S.

But my guess, following Pollan, is that the true costs are a good deal higher — in houses lost, families shattered, lives transformed, self-worth crushed and communities decimated. Philip Oreopoulus, Marianne Page and Ann Stevens provide an example of some of these effects by showing how job loss by parents has long-term effects on the economic lives of their kids — even into adulthood.

There is certainly room for technocratic expertise in coming up with a good estimate of these true costs of the meltdown. And there is room for dispute among experts, both about the real costs of regulation and the real costs of the financial collapse. But deciding what should go into that analysis is the task of all of us, as citizens.Yes, we can turn over the details of the accounting to the accountants. But to let them determine what is to be counted would be an abdication of our collective responsibility.

We are collectively dazzled by numbers. Numbers are matters of fact, not opinion. Numbers are objective, not subject to the bias and myopia of fallible humans. Numbers are black and white; opinions are grey. And in our current era of big data, we can now attach numbers to pretty much everything.

But what numbers themselves don’t tell us is which numbers are worth computing. And just as the Internet has shown us that having all of the world’s information at our fingertips can be about as useful as having none of it (I just got 7.8 million hits for “grass-fed beef”; now what do I do?), counting everything may be about as useful as counting nothing. It will take the wise judgment of citizens to tell us what is worth counting, and why.

Watch our 2003 report with Barry Schwartz about the “paradox of choice”: