How to Make Better Forecasts
If you’ve ever had a kitchen renovation that took longer and cost more than you anticipated, you are familiar with what psychologists call the planning fallacy: the human tendency to underestimate task-completion times and budgets.
The planning fallacy affects many parts of our lives, from home renovations and auto repairs to large capital projects and M&A. CEOs and investors suffer from the same biases as architects and contractors, which explains why corporate M&A is prone to failure, why capital investment is a negative price return signal for quantitative investors, and why most earnings forecasts are far too optimistic.
Fortunately, there is a solution to the planning fallacy you can implement today: base your forecast on the performance of other, similar ventures.
If you’re planning a kitchen renovation, base your time and cost estimates on a data set of other similar kitchen renovations. If you’re forecasting an investment’s returns, base your estimate on how other similar investments performed. Daniel Kahneman calls this “the single most important piece of advice regarding how to increase accuracy in forecasting through improved methods.”
This methodology—known as “reference class forecasting”—has been proven to benefit fields as diverse as project management for infrastructure and private equity investing. To see this in action, let’s explore two case studies: one from the United Kingdom’s efforts to better forecast infrastructure costs and one from an Australian private equity firm.
UK Infrastructure
In 2003, the UK Department of Transport and Her Majesty’s Treasury identified a significant problem: "There is a demonstrated, systematic tendency for project appraisers to be overly optimistic.” They decided to combat this by commissioning a major study of historical projects. For each category of project, a reference class of completed, comparable transportation infrastructure projects was used to establish probability distributions for cost overruns for new projects similar in scope and risks to the projects in the reference class. For example, they looked at 172 completed and comparable road projects and found the following distribution of cost overruns.
Figure 1: Distribution of Cost Overruns
Source: Bjorn Flyvbjerg, “From Nobel Prize to Project Management: Getting Risks Right”
This study found that the median budget estimate should be revised up by roughly 20% —a methodology now implemented for major infrastructure projects in the UK. By implementing a statistical reference class forecast, the government was able to combat the natural biases of the system and come up with far more accurate forecasts.
Australian Private Equity Firm
In 2012, management researchers performed an experiment at an Australian private equity firm. Thirty-three investors at the firm completed a three-part questionnaire about the investments they were working on. They were asked to:
Forecast the returns of an investment they were currently working on.
Place the investment into a broader category of historical investments (e.g., take-privates, carve outs, etc.) and list five comparable investments from their portfolio or another private equity firm’s portfolio that fell into that category.
Provide the returns of the comparable investments, compare their forecast for their investment to the reference class they had come up with, and decide if they wanted to revise their forecast from part 1 based on their answers in part 2.
The results were fascinating. In part 1, the investors estimated a 29% IRR for the projects they were working on. In part 2, they described comparable investments with an average return of 19%. And in part 3, 82% of investors who described their current investment as having a higher expected return than comparable investments revised their forecasts down closer to the reference class.
However, the researchers sounded a note of caution. The participants generated their own reference classes and generally chose very small reference classes that were disproportionately comprised of successful outcomes. “People do not seem naturally inclined to form a broad reference class of projects even when encouraged to do so,” wrote the researchers. “Participants generated their own, very small sample or reference group in the experiment, yet these practitioners routinely make large investment decisions based on a small set of casual analogies and the case at hand.” Only 42% of the investors could come up with the requested five comparable projects, instead choosing to list a handful of very successful historic investments.
Conclusion
Benjamin Franklin once reasoned that “If you know how to spend less than you get, you have the philosopher’s stone,” the legendary substance which turns base metals into gold. Investing is a relatively straightforward two-part equation: invest less, get back more. The future being uncertain, we lose control of the second half of the equation once we commit to the investment decision. The difficulty many investors, CEOs, government offices, and couples remodeling their homes often encounter is not in devising a plan to get something of value, but in making too optimistic of a forecast. Reference class forecasting provides a powerful tool to help us overcome our natural inclination to plan based on expectations that are outside the realm of statistical probability. The UK example shows the power of reference class forecasting, while the private equity study shows the difficulty of executing such a strategy and the discipline required.