Why doesn't an index fund that tracks an index consistently outperform the index?
Why doesn't an index fund that tracks an index (like the S&P) consistently outperform the index? I understand the fee structure to pay for turnover as well as for minimal management. However, shouldn't the fund profit enough off of stock leasing to a short seller to generate enough returns to outperform the actual index?
The premise of your questions is really spot on. Yes, an index fund can earn a significant amount of money lending their securities to short sellers. But, since securities lending is an activity of the fund company, and not the fund, this income is not put back into the fund. This income is added to the other income the fund company makes. As an example, you can look at page 11 of BlackRock's 2017 10-K filing for an explanation of how they handle securities lending.
Additionally, since index fund expense ratios have been driven so low, the fund management likely needs this income to help pay the fund expenses. While index funds don't have the larger (some would call bloated) budgets that active funds have for fund management and analysts, the index funds still pay significant sums for SEC filings and other regulatory costs. Considering many index funds can be found with expense ratios below 0.1%, the fund companies are looking for all the money they can get.
Other than the fees you mentioned, the other cause of underperformance comes from over diversification. The whole idea is that you can only diversify a portfolio to a point where risk can't be reduced any further by diversification. Refer to the Efficient Frontier for these levels which defines the maximum expected return for a given amount of risk. Once you are at the optimal level of diversification each additional stock will essentially cause your portfolio (index fund in this case) to retract to the mean of the market (index) minus the fees you pay. Because these funds own a couple hundred stocks, they are well past the optimal level of diversification causing the portfolio to retract to the mean of the market (index).
Index funds are not designed to outperform indexes. They're designed solely to track or replicate a specific index. Keyword here is 'track', not 'outperform'. The other keyword here is 'index fund', not 'mutual fund.' Unlike mutual funds, index funds are passively managed, meaning there's no team of portfolio managers actively buying and selling in attempt to beat the market or picking specific securities over others in the underlying portfolio. Most index funds will either own every asset in that specific index it's seeking to track or seek to achieve the same end result by holding similar securities. As a result, there's no picking winners or losers, you're simply buying up an entire index and then participating in, not beating, it's performance. Here's a good Investopedia article on how index funds work.
Great question! These are some of the complexities of what many people tout as the simplest way to invest. Strictly speaking, the goal of these funds is to minimize tracking error, not to perform well. I certainly wouldn't expect them to outperform a pure index, since indices incur none of the operational costs of a business. You should expect an index fund to underperform it's benchmark by it's expenses, at least.
There are many ways to build an index fund; some are holding the actual companies of the index; some are synthetic funds not holding any of the actual stocks. There are still humans running these funds, and in many cases, making decisions about what and how much to hold. All of this can easily account for performance differences.
To your specific question, stock leasing revenue is not required to be brought back into the fund as profit. It's not price growth and it's not dividends, so if it doesn't come back into the fund, where does it go? Ferraris? Yachts? Seriously though, it doesn't typically go into investor's pockets, unless the company explicitly states that it does!
Don't get me wrong - an appropriate allocation of several index funds will perform better and be simpler than most of what DIY investors are doing (stock pickers, market timers, and S&P 500 index holders, I'm looking at ALL of you)! But, simpler for most people and best for you are two different animals.
By the nature of index funds they should always underperform the index by their fee structure. There are variances due to timing of stock purchases which can also dictate peformance differences. Albeit small in relation to the old days, there are spreads in buying and selling securities so there are more costs than just the management fee. Some institutions lend securities but they typically do that as a profit center for themselves.