Should you be a passive, index investor or an active stock picker? There are competing schools of thought. One is that a passive index fund is just as good as an actively-managed fund, if not better. The other is that managers have a positive effect and it's okay to pay for that person's expertise.
Not Black and White
The truth, as it happens, is not so black and white. It's possible to pick either individual stocks and bonds, or broad classes of assets, and do well. Most investors put their money in mutual funds, and aside from index funds the management is usually active. That means the manager is picking individual securities or groups of securities that are broadly similar, and putting the money where it looks like returns will beat the market. (For more, see: Passively Managed vs. Actively Managed Mutual Funds: Which is Better?)
The key thing to remember is that there's a difference between the effect asset allocation, for example, has on fund variability, the absolute returns, and the volatility of the fund itself. The short version is that the asset allocation – what kind of securities are in the fund and how much weighting they get – accounts for the lion's share of variability. Variability is the amount that one's returns differ between funds. An oft-cited paper by Gary P. Brinson, CFA, Randolph Hood, and Gilbert L. Beebower was the first place to quantify the effect, in 1986. They found that about 93.6% of the variability was due to the allocation, and that active management actually hurt returns, costing the average pension plan 1.10% per year.
To put that in perspective, the mean average return over the 10-year period Brinson, Hood and Beebower used (1974-1983) was 9.01%, whereas the benchmark portfolio returned 10.11%. So active management seems to cost. But that isn't always the case. Sometimes active management could add 3.69% per year. (For more, see: Active Management: Is it Working for You?)
The difference, it appears, was the kind of active management. Selecting broad asset classes and weighting them a certain way seemed to beat picking individual securities. That is, the various kinds of securities one might have (stocks and bonds for most investors, with a smattering of cash) all move differently in the same market. While the details can be pretty complicated, rules of thumb like "stocks go up when bonds go down and vice versa," are generally true. That means if you want returns you want to allocate your assets to minimize the down side and maximize the upside.
This doesn't mean, however, that one can ignore the management of the fund and just plug in a certain percentage of stocks and bonds. The investing world is more complex, and later studies showed that stock picking can work, notably by Craig French and Daman Ko in 2006. In their paper they looked at hedge fund returns and compared it to the Standard & Poor's 500. (For more, see: Hedge Funds: Higher Returns or Just High Fees?)
French and Ko found that hedge fund managers can and do show skill in picking stocks or allocations, and sometimes the returns reflect that. But while a few managers excel at those two skills, market timing (which means accurately choosing when to invest and divest) was something nobody did particularly well. That means that if you're looking at whether a manager is a good one, and provides returns in addition to the market, it's probably better to ask about how they build strategies based on picking stocks or picking weights of asset class, rather than how well they timed the investment. So in that regard, passive management was outperformed by active management most of the time.
In fact, an index fund won't even match the performance of an index, because of transaction costs. So if you're worried about matching the market an index fund is not the way to do it, though you're guaranteed to get at least close to the market return. Besides transaction costs, the reason index funds "underperform" is because no fund can accurately represent the entire market.
Past performance also isn't the only way to pick a manager. "I think it is certainly safe to say that passive investing works, but active management can outperform," French, a portfolio manager at WBI Investments, said in an email. "Past performance is not the only way to find a skilled manager: Jones and Wermers (in a 2011 study) suggest four potential criteria for identification of superior asset managers, including evaluation of past performance, macroeconomic correlations, fund/manager characteristics, and fund holdings." That can be a complicated comparison for most investors to do, however. (For more, see: Misconceptions About Past Performance and Future Returns.)
French noted that even as far back as the 1970s there have been attempts to separate "luck" from "skill" in managing funds. Nobody has come up with a definitive answer yet. That's because a lot of the time investors have to work with probability. There's no set formula that will guarantee your retirement investments will always go up. The best one can do is maximize the probability that your money will grow and mitigate the inevitable losses.
The Bottom Line
Active management can add value, and active funds are perfectly good as part of a portfolio, but be aware that that performance can deviate from an index and vary a lot between managers. And depending on timing the market it is probably a bad idea. (For more, see: A Statistical Look at Passive vs. Active Management.)