REFERENCE CLASS FORECASTING

(This document is the result of my independent work. All sources of information used in the document have been duly acknowledged.)

Reference class forecasting is a forecasting tool, used to predict the outcome of a planned action, based on actual outcomes of similar actions taken in the past.  The theories of reference class forecasting were built by Daniel Kahneman and Amos Tversky, and were based on theories of decision-making under uncertainty (that won Princeton psychologist Daniel Kahneman the Nobel prize in Economics in 2002).

People usually hold either of the following views while forecasting the outcome of an action or decision. These are:
a) The Inside View
b) The Outside View

In an Inside view, the focus is on the components of the planned action rather than on the actual outcomes of similar ventures that have already occurred in the past. Kahneman and Tversky (1979a, b) found that human judgment is generally optimistic due to overconfidence and insufficient consideration of distributional information about outcomes of decisions/actions in the past.  Therefore, people tend to underestimate the costs, completion times, and risks of planned actions, whereas they tend to overestimate the benefits of those same actions. This leads to an error in forecasting.

In the Outside view, focus is on knowledge about actual performance in a reference class of comparable projects taken up in the past. Reference class forecasting is a method for taking an outside view on planned actions.

The contrast between inside and outside views has been confirmed by systematic research (Gilovich, Griffin, and Kahneman, 2002). The research shows that when people are asked simple questions requiring them to take an outside view, their forecasts become significantly more accurate.

According to Kahneman and Tversky, error occurs when past outcome and distributional information is not taken into consideration. They recommended that forecasters "should therefore make every effort to frame the forecasting problem so as to facilitate utilizing all the distributional information that is available.” 

According to Bent Flyvbjerg, three main types of explanations exist that claim to account for inaccuracy in forecasts of costs and benefits: technical, psychological, and political-economic explanations.

Technical Explanations: This is the most common type of explanation of inaccuracy in forecasts. Technical explanations account for cost overruns and benefit shortfalls in terms of imperfect forecasting techniques, inadequate data, honest mistakes, inherent problems in predicting the future, lack of experience on the part of forecasters, etc. 

Psychological Explanations: Psychological explanations account for cost overruns and benefit shortfalls in terms of planning fallacy and optimism bias, developed by Kahneman and Tversky (1979), Kahneman and Lovallo (1993), and Lovallo and Kahneman (2003). Due to planning fallacy, planners make decisions based on “delusional optimism rather than on a rational weighting of gains, losses, and probabilities.” This is the “inside view” that we described above.

Overoptimism can be traced to cognitive biases, that is, errors in the way the mind processes information. Optimism bias is the demonstrated systematic tendency for people to be over-optimistic about the outcome of planned actions. This includes over-estimating the likelihood of positive events and under-estimating the likelihood of negative events. It is one of several kinds of positive illusion to which people are generally susceptible. It is unconscious and thus not reflected by forecasters. Although these biases are thought to be ubiquitous, their effects can be tempered by simple reality checks, thus preventing organizations and people from making erroneous decisions. 

Political-Economic Explanations: Political-economic explanations see planners and promoters as deliberately and strategically overestimating benefits and underestimating costs when forecasting the outcomes of projects. They do this in order to increase the likelihood that it is their projects, and not the competition's, that gain approval and funding. 

According to Bent Flyvbjerg, “Strategic misrepresentation can be traced to political and organizational pressures, for instance competition for scarce funds or jockeying for position and it is rational in this sense.”

For making a forecast using the outside view, planners need to identify a reference class of analogous past initiatives, determine the distribution of outcomes for those initiatives, and place the project at hand at an appropriate point along that distribution. These can be classified under five steps as given below:

1.       Select a reference class:  This entails identifying the right reference class based on similarities and differences on many variables.

2.       Assess the distribution of outcomes:  Once the reference class is chosen, this involves documenting the outcomes of the prior projects and arranging them as a distribution, showing the extremes, the median, and any clusters. This requires access to credible, empirical data for a sufficient number of projects within the reference class to make statistically meaningful conclusions.

3.      Make an intuitive prediction of your project's position in the distribution: Based on personal understanding of the project at hand and how it compares with the projects in the reference class, predicting where it would fall along the distribution.

4.      Assess the reliability of your prediction: This step is intended to gauge the reliability of the forecast made in Step 3. The goal is to estimate the correlation between the forecast and the actual outcome, expressed as a coefficient between 0 and 1, where 0 indicates no correlation and 1 indicates complete correlation. In the best case, information will be available on how well the past predictions matched the actual outcomes. One can then estimate the correlation based on historical precedent. In the absence of such information, assessments of predictability become more subjective.

5.      Correct the intuitive estimate: Due to bias, the intuitive estimate made in Step 3 will likely be optimistic—deviating too far from the average outcome of the reference class. In this final step, the estimate is adjusted toward the average based on the analysis of predictability in Step 4. The less reliable the prediction, the more the estimate needs to be regressed toward the mean. 

Thus reference class forecasting does not try to forecast the specific uncertain events that will affect the particular project, but instead places the project in a statistical distribution of outcomes from the class of reference projects.

REFERENCE CLASS FORECASTING IN PRACTICE

In 2001 (updated in 2006), AACE International (the Association for the Advancement of Cost Engineering) included Estimate Validation as a distinct step in the recommended practice of Cost Estimating (Estimate Validation is equivalent to Reference class forecasting in that it calls for separate empirical-based evaluations to benchmark the base estimate)

The first instance of reference class forecasting in practice may be found in Flyvbjerg and Cowi (2004): Procedures for Dealing with Optimism Bias in Transport Planning. Based on this study, in the summer of 2004 the UK Department for Transport and HM Treasury decided to employ the method as part of project appraisal for large transportation projects under their jurisdiction. The immediate background to this decision was the revision to "The Green Book" by HM Treasury in 2003 that identified for large public procurement a demonstrated, systematic tendency for project appraisers to be overly optimistic. 

Such optimism was seen as an impediment to prudent fiscal planning, for the government as a whole and for individual departments within government. To redress this tendency HM Treasury recommended that appraisers involved in large public procurement should make explicit, empirically based adjustments to the estimates of a project’s costs, benefits, and duration.

For each category of projects, a reference class of completed, comparable transportation infrastructure projects was used to establish probability distributions for cost overruns for new projects similar in scope and risks to the projects in the reference class, as required by step 2 in reference class forecasting.

In October 2004, the first instance of practical use of the uplifts was recorded, in the planning of the Edinburgh Tram Line 2. Ove Arup and Partners Scotland (2004) had been appointed by the Scottish Parliament's Edinburgh Tram Bill Committee to provide a review of the Edinburgh Tram Line 2 business case developed on behalf of Transport Initiatives Edinburgh.

By framing the forecasting problem to allow the use of the empirical distributional information made available by the UK Department for Transport, Ove Arup was able to take an outside view on the Edinburgh Tram Line 2 capital cost forecast and thus de-bias what appeared to be a biased forecast. As a result Ove Arup's client, The Scottish Parliament, was provided with a more reliable estimate of what the true cost of Line 2 was likely to be. 

The American Planning Association Endorses Reference Class Forecasting

In April 2005, based on a study of inaccuracy in demand forecasts for public works projects by Flyvbjerg, Holm, and Buhl (2005), the American Planning Association (APA) officially endorsed a promising new forecasting method called “reference class forecasting” and made the strong recommendation that planners should never rely solely on conventional forecasting techniques when making forecasts. The emphasis was on transportation project management, because this is where the first instance of reference class forecasting occurred. 

CONCLUSION

Reference class forecasting bypasses human bias--including optimism bias and strategic misrepresentation--by cutting directly to outcomes. Reference class forecasting is extremely useful when it comes to planning for large infrastructure projects, namely, on transportation infrastructure projects. However, according to Bent Flyvbjerg comparative research shows that the problems, causes, and cures we identify for transportation apply to a wide range of other project types, including power plants, dams, water projects, concert halls, museums, sports arenas, convention centers, IT systems, oil and gas extraction projects, aerospace projects, and weapons systems. Thus reference class forecasting has a much wider scope of application.

Reference:

1. Bent Flyvbjerg, Policy and Planning for Large Infrastructure Projects: Problems, Causes, Cures, World Bank Policy Research Working Paper 3781, December 2005, Retrieved on 6th September, 2008 from 

http://flyvbjerg.plan.aau.dk/0512DRWBPUBL.pdf

2. Bent Flyvbjerg, Eliminating Bias through Reference Class Forecasting and Good Governance, Concept Report No 17 Chapter 6, NTNU, Retrieved on 6th September, 2008 from

http://www.concept.ntnu.no/Publikasjoner/Rapportserie/Rapport%2017%20kapittelvis/Concept%2017-6%20Reference%20Class%20Forecasting%20and%20Good%20Governance.pdf

The table has been taken from here.

3. Bent Flyvbjerg, FROM NOBEL PRIZE TO PROJECT MANAGEMENT: GETTING RISKS RIGHT, project management Journal, August 2006, Retrieved on 6th September, 2008 from

http://flyvbjerg.plan.aau.dk/Publications2006/Nobel-PMJ2006.pdf

The graphs have been taken from here.

4. Dan Lovallo and Daniel Kahneman, Delusions of Success: How Optimism Undermines Executives' Decisions, Working Knowledge for Business Leadesr, Harvard Business School, Archive, 18th August, 2003; Retrieved on 6th September, 2008 from

http://hbswk.hbs.edu/archive/3630.html

5. Retrieved on 7th September, 2008 from http://en.wikipedia.org/wiki/Reference_class_forecasting

6. Retrieved on 7th September, 2008 from http://en.wikipedia.org/wiki/Optimism_bias