Wall $treet Week with FORTUNE co-anchor Karen Gibbs recently sat down with Mark Hulbert of the Hulbert Financial Digest and FORTUNE writer Nicholas Stein to talk about mutual fund and stock research firm Morningstar, which is preparing for an initial public offering. A small part of their discussion will appear on our Dec. 10, 2004 program. Here is most of the interview that won't be aired:

KAREN GIBBS: Who are these guys? What's the culture behind Morningstar?

NICHOLAS STEIN: Well, Morningstar, really, culture is a good way to describe it, because they really changed the culture of investing for small investors. It's been awhile, so it's probably a lot of people don't remember the dark days before Morningstar came along, when mutual funds were really veiled in secrecy. And what Morningstar did was really open it up and allow small investors, people who usually at that time didn't have a lot of information about the market, to really learn a lot about the funds that they were going to invest in. And as a result of that, you know a lot of people took the money out of their passbook savings accounts and poured it into mutual funds.

GIBBS: Mark, what about Morningstar's star rating system? It caused a lot of people to chase performance.

MARK HULBERT: There are a variety of reasons. I want to start out by saying I don't want to be overly critical of Morningstar in the sense that they're not the only ones that are having trouble figuring out a rating system that can pick funds that outperform the market. There are plenty of people in academic graduate schools and business schools across the country as well as people on Wall Street who are spending an enormous amount of energy and have been for decades trying to figure out a system that can look at past performance and provide a statistical guarantee of what will beat the market in the future. And most everyone has had some degree of failure at coming up with that system. So when I point out that Morningstar has not figured out a system of picking funds to outperform the market, they're in good company.

Having said all that, I think there's perhaps a couple of things that at least in retrospect we can say that the rating system has done that leaves something to be desired. One of them, in fact the core I would suggest is that they give inordinate weight to performance over the last three years. Even a fund that's been around for five or 10 years, still the majority of their rating will be based on the performance of the last three years.

GIBBS: Do you find the same thing, Nicholas?

STEIN: I do, and if I could just add one thing to what Mark was saying, I think the biggest issue with the star system, which Morningstar themselves will acknowledge, is that it is primarily based on past performance. And, you know, investors, it would be great if we had some stargazing way of looking into what would happen in the future. Unfortunately we don't, so we have to base it on the past. And of course it doesn't really indicate what's going to happen in the future.

Morningstar actually recalibrated the way that they come up with their star system a few years ago, and the reason for that is that a lot of people were complaining that all of the funds were being compared together. So what would happen is, you know, right now energy's hot, so if you happen to have a fund in the energy sector, you would be right up at the top. Similarly, during the Internet bubble, if you were in tech, your fund would be performing really well. So what Morningstar did was they changed their rating system, the categories within that system so that a small-cap growth fund would be compared only to another small-cap growth fund. And for some people that was a good change.

Other people were less impressed. I talk to people in the fund industry who complain that what it really did was give financial advisors a way to get off the hook with their customers, because they could turn to their customer and say, you know, you did great, your mid-cap growth fund outperformed everyone else, everyone else's. In the meantime, not only did it outperform everyone else's, but the S&P 500 outperformed yours by 10 percent. So unfortunately what this change has done is that it's now meant that you might be the equivalent of a five-star restaurant in Cleveland, but that doesn't mean that your five-star restaurant in Cleveland matches up to the five-star restaurant in New York or Paris.

GIBBS: Mark, we're looking at a universe of say 16,000 funds. It's very difficult then to compare apples to apples and oranges to oranges, as Nicholas just said. So, how do we do it?

HULBERT: Well, as I was suggesting earlier, I think one should either focus on shorter term performance, I'll explain why, or on much longer term performance.

If they were to focus on shorter term performance, I think they would have performance that's closer to what some of these other newsletters that I track that do focus on short term performance, they have much better rankings, higher ratings in my system than do Morningstar's or the other newsletters that focus on three years, or if they'd focus on longer term performance. Then they would have separated out those who have ability. And so that's where I think they're in between the two major strands of where they should have gone with the ratings.

STEIN: And if could just add one more thing to that, often what happens, because so many investors follow Morningstar's ratings, and really if you look at the flow of money in and out of funds, when a fund gets four or five stars, billions of dollars flow into it and vice versa. And often what happens is you really start to see a fund that may have done well with a relatively small portfolio, all of a sudden they have to invest a much larger portfolio because they've got all this extra money to deal with, and that causes their performance to decline. So, you know, I agree with Mark completely that over a much longer period of time you can really see a manager who's been able to kind of ride out some of the peaks and valleys and the inflows and outflows of money from the fund.

GIBBS: Well, I hear very clearly, loud and clear, what both of you are saying. But, Nicholas, then why are so many people following the Morningstar rating system?

STEIN: Well, I think once again it's the Morningstar rating system is a very convenient way to look at this very large, very complex universe. And I think what it does is it allows you to avoid the total flops. So, you know, if you talk to people in the industry, even people who aren't big fans of Morningstar will say, you know, granted if you look at the four and five star funds, you're probably not going to end up with something that's a disaster. But at the same time, will it help you find the hits? You know, that's a more open question, and that's something which clearly they've not demonstrated over time that they've been able to do.

GIBBS: Well, Nicholas, what do you know about the Securities and Exchange civil investigation into Morningstar?

STEIN: Well, the civil investigation focuses on just one fund, the Topflight, Rock Canyon Top Flight Growth fund out in Colorado. And the investigation is based on incorrect information on that fund that Morningstar posted on its web site. The larger question that it raises is, you know, number one, was this a deliberate mistake or was it inadvertent? And number two, if it was inadvertent, as I think it was, was it a one time error which you could sort of imagine happening given the millions of pieces of data that are flowing in and out of Morningstar every day? Or is this something that's more of an epidemic at the company, a data error that is happening repeatedly and that this just happened to be one of the first times that it was caught?

GIBBS: Mark, what do think about that?

HULBERT: I have no particular insight into that investigation alone, but I think Nicholas is right, that with lots of data going in and out, it's perfectly reasonable to assume that an inadvertent error caused it. I have nothing more to think about that.

But you asked why would Morningstar be so popular. I think Nicholas picked up on a lot of it, and the other thing I think that's worth pointing out is they're the first. I mean, for example, I focused a lot in my research on the Value Line investment survey, which is a much, in one sense used to be a much better source of independent stock research than Morningstar, and Morningstar I think almost self consciously modeled the pages of their service on what Value Line's service looked like for the presentation of stock data. And if Value Line, for example, had decided to get into the mutual fund rating system as early as Morningstar, I can easily imagine a scenario in which Value Line would be the subject of our conversation tonight rather than Morningstar. And for a variety of reasons that I have no insight into either, Value Line did not get into it until the early '90s. They were seven, eight years behind what Morningstar did.

Morningstar was the first to get into the business of rating mutual funds for the retail individual investors. They deserve a lot of credit for getting into it early. And I think the reason that so many people look at it is that precisely they were one of the first. The same way that you hear a lot more about Amazon.com than you hear about Barnesandnoble.com, even though in some senses you might claim that Barnes & Noble has more books or whatever. You hear about the competition between the two all the time. But because Amazon.com was the first, they end up enjoying a huge benefit for it.

STEIN: Well, I agree to a certain extent, but I also feel like one of Morningstar's real strengths was its ability to market itself. When Joe Mansueto started the company back in 1984 from his Chicago apartment, you know, what he did was he went out and hired a lot of liberal arts grads, people with English backgrounds, because he wanted to be able to not just provide this service to investors, but provide it in a way that was easy for investors to understand.

» W$WWF, Dec. 6, 2002: Don Phillips interview

And really Morningstar had a big competitor in the beginning, which was a company called Lipper. Lipper was there before Morningstar. Lipper offered many of the same services. In many ways Morningstar patterned its offerings after Lipper. The big difference was Morningstar decided to go after the individual investor, and they did that by really marketing themselves, by making themselves available to the media. You probably won't find anyone in American financial journalism today who is quoted more frequently than Don Phillips, the managing director of Morningstar. And that was a very deliberate decision, I would argue, on their part, to never turn down an interview, never turn down an opportunity to give a sound bite. And that is really what I guess cemented Morningstar in the public's imagination.

Morningstar is really many different things. To individual investors, it's very much the way that Mark described it. It's a research company that allows people to really dig deep into the mutual fund universe, and more recently into the stock universe as well, and get the information that people hopefully feel they need before plunging in and buying stocks or mutual funds. But increasingly over the years, Morningstar has also become a giant database where institutions, large institutions, and also financial planners can go to for the information that they need to evaluate the fund universe as well.

GIBBS: Nicholas, if not Morningstar, where else can investors turn to for research?

STEIN: Well, there are a lot of places that investors can turn to for research. The problem is that they seem to be dying off by the day. I mean I think that there was a huge landscape, the landscape of investing for individual investors was hit pretty hard with the scandal a couple of years ago when a lot of the big investment banks that used to provide free research to small investors, it was found that that research wasn't as independent as investors may have hoped. As a result of that, a lot of what became known as the Spitzer settlement, more than $400 million was set aside by the large investment banks for independent research.

Morningstar is one of the companies that's going after that money, and in fact they believe that that's a major growth area for them. But I think that what time has demonstrated is that individual investors aren't really willing to spend a lot of money for research, and you know these days with so many newsletters, so many web sites on the Internet devoted to investor research, so many places investors can go for free, you know, the question remains are people going to be willing to pay for the research that Morningstar provides? They clearly think that people will. A lot of people probably don't know this, but Morningstar now has a lot more equity analysts on their staff than they have fund analysts, and they're pouring a lot of investment into this area in the hopes that people will follow.

But I'm a little bit skeptical as to whether they will, especially because when people do, are willing to pay for research, it's usually research that's based on some sort of performance. You know, you'll have a guru who has shown over a number of years that they can outperform the market, and people will be willing to pay for that advice. But will they be willing to pay just for the Morningstar brand without any type of performance attached to that? That remains to be seen.

I think that the flaw in their business model is that they were a really good small company, and the IPO is sort of their attempt to become a big company. But the question is will they have to kill the part of them that allowed them to become this great little company in the first place in order to graduate and become as big a company as they are a brand? That's sort of the real issue that they're struggling with right now is that, you know, the institutions that pay the most of their money are the same ones that they're covering, and the only way that they can continue to grow and develop as a company is if they sort of move this desire that they have to be the watchdog of the fund industry somewhere further into the background.

GIBBS: Mark, you're nodding your head. Have they become part of the problem instead of part of the solution?

HULBERT: Well, I was nodding my head because I was thinking about the tensions that Nicholas was mentioning. Another tension that's perhaps related is the tension between focusing on the longer term performance that I think would help their returns to the rating system, and on the other hand, the other part of the tension is that if you only focus, let's say that you were to decide that you were only going to rate mutual funds, give them a star rating, if they have the same manager for the last 10 years and have beaten the market on a risk adjusted basis over that period of time, both very reasonable assumptions. A lot of statisticians throw out from the analysis if they've changed a manager over that time, changed their particular investment focus, and have not beaten the market over a particular period of time.

The problem if you do that is that there are very few mutual funds that would satisfy that particular, there are very few that can jump over those hurdles. And it turns out from a marketing point of view, you need to have ratings for most mutual funds, and if most mutual funds aren't even going to be eligible because of either a change in management or haven't been around for 10 years, or most likely haven't beaten the market over a 10-year period, then what, that means that you're not going to be able to sell yourself as having a rating system for the majority of mutual funds. And I've often focused on that particular tension, because it seems to me that on the one hand there's statistical integrity; you're going to go for what you think has the greatest chance of identifying mutual funds that will outperform the market, versus a desire to be in business. And obviously if you're not in business you can't do any help for investors. On the other hand, by focusing on funds that have beaten the market over the last several years, I'd say three years, I'm not sure you've helped the investors that much either.

But nevertheless, they have done, I think made many efforts over the years, especially in the last several, to encourage investors to not rely solely on the rating system, that the rating system is the beginning of research rather than not the end of the research, and I would wholeheartedly endorse that. It's really not the end point at all.

GIBBS: Well, this brings me to another subject, and kind of broaden the picture out now between index funds and managed funds. All the experts tell us that what we should do as investors is just put the money in the index fund, and that very few managed accounts will outperform the index, but that's not necessarily true. We found that six funds, in both five- and 10-year time frames, outperformed the biggest index fund, which is that Vanguard 500 index. What gives?

HULBERT: Well, I think the problem is that the debate has become very stylized over the years. On the one hand, you have people claiming that the market's impossible to be beat. On the other hand, you have people who claim the market is very easy to be beat, and there's this polarization of the debate. It's become very stylized.

And I think in fact if you were to get most people, all the way from in academic circles who are thought to be at least stereotypically as to be believing that the market can't be beaten, all the way to the people in the industry who think the market is easily beaten, if you got them off the public eye and were willing to speak soberly off the record, most of them would admit that the market is hard to be beaten but it's not impossible. So it's really a matter of odds, and the question is whether one wants to take those odds.

Without a doubt the market is difficult to be beaten. The question is is it impossible? I don't know of anyone who says it's impossible to be beaten. So then the question is do you want to take the chance? And I think one of the ways in which the statistical way in which that debate is stylized ends up falling short is that there's a whole psychological dimension as well. It turns out that you'll see, and in fact I've seen this over the years with newsletters, is that the degree to which they are passionate about buy and hold is itself cyclical, is cyclical.

At the top of a bull market, you'll find most people genuflecting at the altar of buy-and-hold, and at the bottom of a bear market everyone is extolling the virtues of market timing. So in late '99 and 2000, there were newsletters who announced that they were buy-and-hold investors at that point, and I knew that they weren't buy-and-hold investors. They hadn't been in the past. And when I would see them at conferences, I'd say what are the chances of you being fully invested at the bottom of the next bear market? And of course it was an inconvenient question that they avoided. I can tell you that none of them was fully invested at the bottom of the bear market in October of 2002. So there's a way in which there's this whole psychological dimension.

So what I suggest to people is as they try to decide whether an index fund is appropriate for them or an actively managed fund is, is this a fund or an approach that they are likely to stick with through thick and thin? And if they are not willing to stick with it through thick and thin, and there's no shame in admitting that you don't have what it takes, then don't even consider it. Because if you invest in an index fund at the top of a bull market for the statistical reasons that are unassailable, but don't psychologically have the patience and discipline to stick with it, you're going to get rid of it near the bottom of a bear market, give or take, and thus be worse off than had you gone with an actively managed fund, let's say, that might be statistically inferior, but is one that you're willing to stick with through thick and thin.

And it turns out that that psychological dimension is absolutely crucial, and people ignore it when they look at it statistically alone.

GIBBS: But newsletters don't necessarily ignore that psychological dimension. In fact, some of them start giving sentiment. What are you finding now? Where is the sentiment toward the market? What's the market outlook?

HULBERT: Well, I take a contrarian approach to that sentiment, as you know. In fact, I'd say that the newsletter editors on balance right now are pretty chipper. They think that the market is more likely to go up than down at this point. It's not at an extreme, and extremes are of course very worrisome from a contrarian point of view. But nonetheless, we're finding that we're much closer to that bullish extreme among investment newsletter editors than we are too a bearish extreme. That is to suggest then, from my point of view at least, if I were to be so bold as to hazard a guess, that we're, that at least the sentiment basis or the foundation in sentiment for this rally is on shaky ground.