February 2007
How we tested for dependable funds
To qualify for this analysis, a mutual fund had be at least 10 years old at the end of September 2006. We didn't include institutional
funds, index funds, or those with less than $50 million in assets. More than 1,300 funds made that first cut. Conveniently,
the last 10 years gave us a boom, a bubble, a bust, and a recovery in the stock market. Stomach-churning as they might have
seemed at the time, those ups and downs allow us to see how the fund managers performed in a variety of situations.
We ran each fund through four different sets of tests. The results from each test were scored and added together to compute the overall consistency score shown in the Ratings. The numbers in parentheses below indicate each test's contribution to the final score.
Frequency test (40%). Here we compared the quarterly results from each fund with an appropriate alternative index investment: Vanguard's 500 Index fund, Mid Capitalization Index fund, Small Capitalization Index fund, Total Stock Market Index fund, or Total International Stock Index fund. We wanted to see how often a fund surpassed its designated index. The funds were ranked by how many quarters each posted index-beating results. We then broke the returns up into rolling 12-month periods and went through the counting and ranking again. The two rank scores were then combined.
Worst-case test (15%). For most of us, avoiding big losses and recovering quickly from the inevitable downturns are probably of greater importance than the occasional windfall. Economists call that avoiding downside risk. To measure it, we looked at each fund's worst 12- and 24-month periods in the last 10 years and scored them in relationship to the rest of the funds.
Business-cycle test (10%). Some stocks do well in a bear market. Others do well during economic booms, and still others shine during the early stages of economic recovery. The same is true for stock funds. We broke the most-recent business cycle down into its three phases and scored the relative returns of each fund during each phase. Then we rated each fund again for the best combination of scores from all three periods.
Volatility test (30%). An investment's volatility is usually measured by its standard deviation, which is basically a measure of the most common range within which you will find the data. The smaller the standard deviation, the more consistent the investment. Typically a mutual fund's volatility is measured against daily, weekly, or monthly returns. That reflects the interests of institutional investors, who are apt to be more frequent traders than individuals who are following a long-term, buy-and-hold strategy. So instead, we took the unorthodox step of comparing the standard deviation of quarterly and 12-month results, time periods that should be more meaningful to most readers.
The remaining 5 percent is a weighting for management tenure. We include that because one of our preliminary tests showed that funds that changed lead managers had noticeably lower average returns over 10 years than those that kept the same manager. Funds whose managers had been on the job less than six months were automatically dropped from our lists.
Once we had our consistency scores, we made one final cut before compiling our list of top funds: We dropped any fund with an expense ratio in the top third of all funds (more than 1.5 percent a year). Morningstar studies suggest that there is a slight but measurable correlation between higher expenses and lower future performance.
We ran each fund through four different sets of tests. The results from each test were scored and added together to compute the overall consistency score shown in the Ratings. The numbers in parentheses below indicate each test's contribution to the final score.
Frequency test (40%). Here we compared the quarterly results from each fund with an appropriate alternative index investment: Vanguard's 500 Index fund, Mid Capitalization Index fund, Small Capitalization Index fund, Total Stock Market Index fund, or Total International Stock Index fund. We wanted to see how often a fund surpassed its designated index. The funds were ranked by how many quarters each posted index-beating results. We then broke the returns up into rolling 12-month periods and went through the counting and ranking again. The two rank scores were then combined.
Worst-case test (15%). For most of us, avoiding big losses and recovering quickly from the inevitable downturns are probably of greater importance than the occasional windfall. Economists call that avoiding downside risk. To measure it, we looked at each fund's worst 12- and 24-month periods in the last 10 years and scored them in relationship to the rest of the funds.
Business-cycle test (10%). Some stocks do well in a bear market. Others do well during economic booms, and still others shine during the early stages of economic recovery. The same is true for stock funds. We broke the most-recent business cycle down into its three phases and scored the relative returns of each fund during each phase. Then we rated each fund again for the best combination of scores from all three periods.
Volatility test (30%). An investment's volatility is usually measured by its standard deviation, which is basically a measure of the most common range within which you will find the data. The smaller the standard deviation, the more consistent the investment. Typically a mutual fund's volatility is measured against daily, weekly, or monthly returns. That reflects the interests of institutional investors, who are apt to be more frequent traders than individuals who are following a long-term, buy-and-hold strategy. So instead, we took the unorthodox step of comparing the standard deviation of quarterly and 12-month results, time periods that should be more meaningful to most readers.
The remaining 5 percent is a weighting for management tenure. We include that because one of our preliminary tests showed that funds that changed lead managers had noticeably lower average returns over 10 years than those that kept the same manager. Funds whose managers had been on the job less than six months were automatically dropped from our lists.
Once we had our consistency scores, we made one final cut before compiling our list of top funds: We dropped any fund with an expense ratio in the top third of all funds (more than 1.5 percent a year). Morningstar studies suggest that there is a slight but measurable correlation between higher expenses and lower future performance.
