We all want evidence based policy and decisions. However, deciding what constitutes evidence and what standard of proof is required is another matter. And of course this is especially challenging when we need to explore what will happen, rather than what has happened. Will this policy work? Will this investment pay back? It’s not clear that evidence is of any use in making these decisions at all.
Notoriously, the Jubilee Line Extension failed to pass the value for money tests at the time, which were overruled by the Prime Minister at the time. However, its use in practice, even after overruns of cost, show that it would now pass the test. In other words, forecasts of passenger use and demand were too low. At the same time, studies have shown that projects of all kinds suffer cost over-runs which undermine benefit cost ratios.
More generally, a recent study of the results of 100 studies published in top psychology journals have shown that only one third stood up to the challenge of being replicated. Unconscious bias, poor statistical analysis and pushing for strong results can all result in reaching conclusions which do not later stand up. None involve deliberate falsification but rather that to err is human.
The National Infrastructure Commission has been asked to provide an assessment of infrastructure needs across all classes of infrastructure looking out 30 years. Thinking about such needs means thinking about a future in which very little is held constant – everything can change. Even on a nearer time frame, thinking about the development of the corridor from Oxford to Milton Keynes and to Cambridge, technologies, housing, employment patterns and institutions can all show potentially dramatic shifts. In such circumstances, how should we think about the nature of evidence if everything in the future can be different from the patterns of the past?
A good approach in my view is to assess objectives and plausibility, with a strong dose of transparency.
The vision for a project or programme sets out what it is intended to achieve. Evaluation can only sensibly occur in relation to objectives. HS2 was first presented as having the objective of saving time – in the first instance to Birmingham. It was then castigated as being a very expensive way of saving 20 minutes on such a trip. This accusation still surfaces and if indeed this had been the objective the detractors would be right. However, a more accurate objective, later clarified, is to improve connectivity to northern cities and the capacity of the rail network more generally. Thus any evaluation should be set against these objectives and how they might be achieved. The NAO pointed this out in their initial review of the project, pointing out that no evaluation of value for money could be concluded until the objectives were more clearly set out.
If the objective of an investment is to sell more widgets, this might be easier to describe than if the objective is to improve access to markets for northern cities. Objectives, however, remain objectives. Moreover, setting them out clearly also allows the context and assumptions to be set out clearly too.
To sell more widgets, I need to assume something about the demand for widgets and that they are not likely to become technologically obsolete. To improve market access I will be making assumptions about the size and scale of such markets and whether it will be possible to take advantage of such markets. This brings us to plausibility.
What does it take to reach the objective? Are such changes plausible in either the current state of the world or the possible futures I am assuming? How different is this future and how do its changes compare with past changes?
If I want to double my sales of widgets, it would be helpful to know if this has been done before, or what market share this might imply for my new factory. These questions would be standard in a self-respecting market study and investment appraisal. Somehow they don’t always get asked in a policy study.
If the aim of an investment is to create jobs, what scale of job creation is implied and is this scale unprecedented? In producing long term forecasts for the London economy for the GLA, we looked back the longest possible period to give the most context possible for future growth. Such an analysis showed that the fall in London’s employment over the period from the 1970s had been concentrated in manufacturing but these losses had now ended and in the meantime growth in services had continued at a fairly constant rate. The question became one of asking what it would take to change a trend which had been in place for nearly thirty years. In the absence of such a shift, London’s employment could be expected to grow and this long term view has indeed been justified over the past fifteen years in spite of cyclical variation.
Of course it remains debatable whether a long term trend will continue. Once it is understood what a particular view of the future depends on, it becomes possible to have such a debate in an informed way.
For example, changes in employment patterns might result from even more improved communications technologies and rising self-employment, spelling the end of large offices full of large number of people working for one firm. I would argue strongly that such technological changes have so far resulted in people wanting to work more closely together and not scattering out over the countryside, even though a higher proportion of time is spent working from home. Such a debate can be informed by looking at the trend to self-employment, time spent at home and office, and average sizes of firms. Such things change slowly but they do change and there might come a point where cities start to shrink. This is a debate worth having, and there may be different opinions. We might explore the consequences of different scenarios for these uncertain futures.
Indeed, looking at plausibility definitely requires the analysis of scenarios to consider the range of contexts in which an investment succeeds and those in which it could fail.
For example, statistical analysis shows that skills are important to economic success. However, without market access skills by themselves will be insufficient. If we are investing in market access, what skill scenarios support the investment and which would render it nugatory?
Setting out assumptions about population, technologies and such matters as house prices can also help clarify the degree of plausibility that an investment will meet its objectives and what other supporting changes might be needed or already in place. Understanding what it is necessary to believe about the future for a policy, programme or investment to work is very important in judging it and brings me to transparency.
The models beloved of technocrats can be very powerful. But the adage of ‘garbage in, garbage out’ remains. Strong assumptions and weak data are not a good basis for decision making. Being able to set out clearly the assumptions so that they can be understood by non-specialists is crucial to a good and informed debate about conclusions.
A narrative about the objectives of a policy or investment, how these are expected to be achieved and the plausibility of these mechanisms is essential for a transparent understanding.
In making an assessment, we are taking bets on the future. Futures are uncertain, so setting out a range of possible futures is one step. Then analysing which ones are supported by the investment in question, which ones would be undermined, and which would invalidate the project can then be described.
Facts matter. Reliable data about the past matter. These are what we have to inform us about possible futures. However, they don’t tell us what that future will actually be since this rests on our decisions now. The consideration of possible futures, their degree of likelihood and on what assumptions they depend, rests on fact and data. But it is not controlled by them. That also requires judgement.
Senior Adviser for Volterra Partners
Commissioner, National Infrastructure Commission