This week I spent some time with the folks from the SAS Institute, which is an institute in name only and is more properly a software company — the biggest software company you've never heard of. With more than $1 billion in sales, 8,000+ employees and a research and development budget in excess of $250 million, the SAS Institute is both the economic powerhouse of Cary, North Carolina, and one of the most aggressive players in the business of corporate data mining. This is using statistical tools to sift through large volumes of data to learn more about customers, suppliers, and your own organization, all with the goal of making things run smoother and more profitably. At the SAS Institute, they call this Business Intelligence.

Business intelligence turns out to be something rather rare, since many companies are only dimly aware of the basis of their own success. Some of us prefer business denial.

When I started using SAS software in 1977, it was a general purpose statistical package, and business intelligence didn't really exist. Most improvements in organizational efficiency back then came from things like the time and motion studies that the United Parcel Service did, that resulted in your UPS driver ringing the bell and knocking at the same time to get you to the door just that little bit faster. That behavior came as a result of data analysis. But efficiency obtained in this way came from external analysis, producing standalone data that was interesting, but bore little relation to the internal workings of your enterprise. It simply wasn't useful for other things.

Back then, business intelligence meant being savvy. I once worked for a company run by a charismatic billionaire who was business savvy. He had an uncanny ability to single out just that line in a financial statement that the business unit head really didn't want to discuss. In one meeting, he pushed and pushed for a particular line to be explained. Finally the business unit head responded, "That's where we pay for your gardener."

But business savvy isn't business intelligence. Business intelligence is using internal data to make informed decisions. Twenty years ago, folks from Apple Computer went on a sales call at the Westinghouse Electric Company in Pittsburgh. They were trying to sell Apple II's and the VisiCalc spreadsheet, but the company's head computer guy wasn't interested. He said Westinghouse would never buy Apple computers, that they were wasting his time. Then they showed him a survey of product registrations. At that very moment there were more than 1000 Apple II's running IN THAT BUILDING, but because they were paid for from petty cash and other non-computing budgets, he had no idea they even existed. Clearly the guy lacked business intelligence.

Business intelligence is knowing where you are, where you want to go, and the best way to get there. Unfortunately, business intelligence is too often isolated within an organization. Fiefdoms of knowledge are based on the term "need to know." Look at Enron, a company that was leaking billions, yet hardly anyone in the organization even knew it. The data was there — it had to be — even the crude tools they used (Microsoft Excel) would have shown the company was in trouble, but there was no mandate to use that information. Nobody was looking.

There will be an aftermath to Enron, an aftermath that goes far beyond ruined pensions and tarnished accountants. Regulators will set new standards for corporate behavior and reporting, requiring data that most outfits don't value and some outfits don't even possess.

Here is a simple fact of corporate life. Organizations tend to value systems over data, but users value data. In this case the users are right. Without good data, systems are meaningless.

But with good data and good analytic tools, and with a top-to-bottom mandate to use that information — literally, the right to know — it is possible to change the world. That's what they did at Wal-Mart, a company that uses information technology and corporate data as a strategic asset. Wal-Mart's success is based entirely on knowing their data and deriving actionable information from that data. It is the same at Home Depot. These companies are world-beaters entirely because of the quality of their data and their willingness to make decisions based on that data.

We live in a time when productivity is increasingly important and ever harder to achieve. The easy gains have all been made, and now we have to find little pockets of inefficiency and little wells of opportunity, both of which can only be discovered through data analysis. It isn't enough any more just being smart, now you have to be smart AND informed.

In many organizations, these facts are not at all obvious and business intelligence can be a hard sell. After all, we've come from 20 years of corporate decision-making by spreadsheet. And while a spreadsheet is a very fine thing, it is typically built with ad hoc data, with hunches and guesses and numbers copied out of the Wall Street Journal. A spreadsheet enables an informed GUESS, while analytic tools like those from SAS (when used correctly!!!!) enable an informed DECISION.

This can be an especially difficult problem in organizations like government and education that think of themselves as not having clear competitors, yet these are the very outfits that need business intelligence most, simply because there are few traditionally accepted ways of measuring productivity and efficiency.

When they were getting ready to introduce the Macintosh computer, Apple had to get independent software developers to write programs for the new platform. The people Apple sent to convince the developers were called evangelists. And it was a hard sell. The first question the software developers would ask was "What's your installed base — how big is the market for my product?" Well, the installed base was zero, since no Macintoshes had yet been sold. "Then how much are you going to pay me to do this?" Apple had no budget to bribe software developers. What they had was a compelling argument. It was already clear that graphical user interfaces were the future of computing. Even if the Macintosh was a failure, the evangelists argued, having early experience developing graphical programs would prepare the developer for later success on some other platform. Remember, the Mac was the only graphical computer really available at the time. So deciding against the Macintosh could very well have meant a software developer deciding to kill his company. The argument worked.

We face the same situation today with business intelligence and the use of analytical tools. To not do it is to risk your company's future. But to do it right requires conviction and rigor on the part of user organizations, and to gain full benefit, they must be embraced at every level. No Enrons allowed. But if your organization won't embrace business intelligence and all that it entails, some competitor just might. And I'll guarantee that competitor will soon be your biggest competitor.

The bigger the organization, the more important it is to use tools of this type, since they are almost the only way to make sense out of a mountain of otherwise meaningless data. Yet many large corporations — some of the ones you would think would have had this stuff in place for years and years — are the worst offenders. Next week I'll write in great detail about one of these companies, a suddenly very troubled IBM.