We all want evidence based policy and decisions. However, deciding what constitutes evidence and what standard of proof is required is another matter. And of course this is especially challenging when we need to explore what will happen, rather than what has happened. Will this policy work? Will this investment pay back? It’s not clear that evidence is of any use in making these decisions at all.
Notoriously, the Jubilee Line Extension failed to pass the value for money tests at the time, which were overruled by the Prime Minister at the time. However, its use in practice, even after overruns of cost, show that it would now pass the test. In other words, forecasts of passenger use and demand were too low. At the same time, studies have shown that projects of all kinds suffer cost over-runs which undermine benefit cost ratios.
More generally, a recent study of the results of 100 studies published in top psychology journals have shown that only one third stood up to the challenge of being replicated. Unconscious bias, poor statistical analysis and pushing for strong results can all result in reaching conclusions which do not later stand up. None involve deliberate falsification but rather that to err is human.
The National Infrastructure Commission has been asked to provide an assessment of infrastructure needs across all classes of infrastructure looking out 30 years. Thinking about such needs means thinking about a future in which very little is held constant – everything can change. Even on a nearer time frame, thinking about the development of the corridor from Oxford to Milton Keynes and to Cambridge, technologies, housing, employment patterns and institutions can all show potentially dramatic shifts. In such circumstances, how should we think about the nature of evidence if everything in the future can be different from the patterns of the past?
A good approach in my view is to assess objectives and plausibility, with a strong dose of transparency.
The vision for a project or programme sets out what it is intended to achieve. Evaluation can only sensibly occur in relation to objectives. HS2 was first presented as having the objective of saving time – in the first instance to Birmingham. It was then castigated as being a very expensive way of saving 20 minutes on such a trip. This accusation still surfaces and if indeed this had been the objective the detractors would be right. However, a more accurate objective, later clarified, is to improve connectivity to northern cities and the capacity of the rail network more generally. Thus any evaluation should be set against these objectives and how they might be achieved. The NAO pointed this out in their initial review of the project, pointing out that no evaluation of value for money could be concluded until the objectives were more clearly set out.
If the objective of an investment is to sell more widgets, this might be easier to describe than if the objective is to improve access to markets for northern cities. Objectives, however, remain objectives. Moreover, setting them out clearly also allows the context and assumptions to be set out clearly too.
To sell more widgets, I need to assume something about the demand for widgets and that they are not likely to become technologically obsolete. To improve market access I will be making assumptions about the size and scale of such markets and whether it will be possible to take advantage of such markets. This brings us to plausibility.
What does it take to reach the objective? Are such changes plausible in either the current state of the world or the possible futures I am assuming? How different is this future and how do its changes compare with past changes?
If I want to double my sales of widgets, it would be helpful to know if this has been done before, or what market share this might imply for my new factory. These questions would be standard in a self-respecting market study and investment appraisal. Somehow they don’t always get asked in a policy study.
If the aim of an investment is to create jobs, what scale of job creation is implied and is this scale unprecedented? In producing long term forecasts for the London economy for the GLA, we looked back the longest possible period to give the most context possible for future growth. Such an analysis showed that the fall in London’s employment over the period from the 1970s had been concentrated in manufacturing but these losses had now ended and in the meantime growth in services had continued at a fairly constant rate. The question became one of asking what it would take to change a trend which had been in place for nearly thirty years. In the absence of such a shift, London’s employment could be expected to grow and this long term view has indeed been justified over the past fifteen years in spite of cyclical variation.
Of course it remains debatable whether a long term trend will continue. Once it is understood what a particular view of the future depends on, it becomes possible to have such a debate in an informed way.
For example, changes in employment patterns might result from even more improved communications technologies and rising self-employment, spelling the end of large offices full of large number of people working for one firm. I would argue strongly that such technological changes have so far resulted in people wanting to work more closely together and not scattering out over the countryside, even though a higher proportion of time is spent working from home. Such a debate can be informed by looking at the trend to self-employment, time spent at home and office, and average sizes of firms. Such things change slowly but they do change and there might come a point where cities start to shrink. This is a debate worth having, and there may be different opinions. We might explore the consequences of different scenarios for these uncertain futures.
Indeed, looking at plausibility definitely requires the analysis of scenarios to consider the range of contexts in which an investment succeeds and those in which it could fail.
For example, statistical analysis shows that skills are important to economic success. However, without market access skills by themselves will be insufficient. If we are investing in market access, what skill scenarios support the investment and which would render it nugatory?
Setting out assumptions about population, technologies and such matters as house prices can also help clarify the degree of plausibility that an investment will meet its objectives and what other supporting changes might be needed or already in place. Understanding what it is necessary to believe about the future for a policy, programme or investment to work is very important in judging it and brings me to transparency.
The models beloved of technocrats can be very powerful. But the adage of ‘garbage in, garbage out’ remains. Strong assumptions and weak data are not a good basis for decision making. Being able to set out clearly the assumptions so that they can be understood by non-specialists is crucial to a good and informed debate about conclusions.
A narrative about the objectives of a policy or investment, how these are expected to be achieved and the plausibility of these mechanisms is essential for a transparent understanding.
In making an assessment, we are taking bets on the future. Futures are uncertain, so setting out a range of possible futures is one step. Then analysing which ones are supported by the investment in question, which ones would be undermined, and which would invalidate the project can then be described.
Facts matter. Reliable data about the past matter. These are what we have to inform us about possible futures. However, they don’t tell us what that future will actually be since this rests on our decisions now. The consideration of possible futures, their degree of likelihood and on what assumptions they depend, rests on fact and data. But it is not controlled by them. That also requires judgement.
Senior Adviser for Volterra Partners
Commissioner, National Infrastructure Commission
London is changing. But then, it always is. The theme of my new book, ‘Reinventing London’ is that change is the lifeblood of a great city. Over the past thirty years it has replaced around 1 million jobs in manufacturing, largely around the edge of the city and along the radial routes, with more than a million jobs in the centre of the city. Fortunately, and some of it was luck, we had enough infrastructure to make this possible. The Victorian railway and underground legacy has creakily heaved us into the twenty first century as one of the world’s great cities, overcoming the loss of confidence and built infrastructure which was the legacy of war.
Nobody really forecast this shift. In the 1990s it was fashionable to talk about a future of market towns and the continued loss of urban density. Even those of us who were dubious about this were not at the time foreseeing that half the world’s population would now be living in cities. In spite of the trend of continued services growth and an end to manufacturing decline was obvious it still took a leap to see that this meant that London’s employment would now grow.
If identifying existing trends is not easy, spotting the emerging trend is still harder, being a dynamic and emergent process. The developing digital economy on the northern fringe of the City was not planned, though planning can certainly stifle such potential. Spare capacity, whether of buildings, transport or bandwidth is essential to enabling growth. Economists who model maximising outcomes where everything is ‘just right’ forget this at the economy’s peril.
Joseph Bazalgette, when asked to design a sewer system in the 1860s, calculated the effluent of the densest part of London and applied this rate to the whole of the city. Then he doubled it, to allow for growth. Then he built the Embankment, with enough room for a railway – the District line – beside the sewers. None of our current evaluation mechanisms would have given this project a positive ratio of benefits to costs. But the payback has been enormous, as the sewers have only recently needed new capacity and the District line is an essential part of our infrastructure.
Taking a long term view is of course taking a bet. No one can be sure that people will continue to want to come to cities. But the trend seems pretty strong and well established, but we need to provide the backdrop which makes it possible. Transport, water, drains, power distribution, and housing are all essential and require both public and private investment. Both the scale and the regulatory backdrop mean that the public sector has to be bold – it was municipal bonds that financed the sewers after all. A similar funding mechanism is proposed for the extension of the Northern Line into Battersea, to open up new high density areas and enable the refurbishment of the Battersea Power Station after years in the doldrums. The GLA borrows the funds, and they are repaid out of the business rates generated by the development.
Large scale finance is equally necessary to solve the housing problem and provide for the growth in population and demand that is driving up prices in the capital. A wholesale funding mechanism for housing development which makes possible a range of tenures and the rapid buildout which housing for sale makes harder is essential to solve this pressing problem.
Better transport and housing make possible the continued development of high density economic activity, the agglomeration which produces the high productivity which drives growth and performance. Effective labour markets encourage investment in skills and the ability to find the right job. High density makes it easier to find clients for new firms, and to make new ideas effective. Just as in eighteenth century coffee shops, people still congregate to talk and to communicate, and not just with their laptops. Extending the areas of high density has been a theme of the last decades, in Broadgate and Docklands, Kings Cross and St Pancras, Paddington and London Bridge. It is no accident that all of these have invested in the quality of their transport connections and that Crossrail will continue to improve them.
All investment is finally about creating a city in which people can develop and use their skills and abilities. The funding mechanisms for investment, the planning regime, and our institutions are all necessary for this, but not sufficient without the diversity of people which we should celebrate and enhance.
My new book ‘Re-inventing London’ is available now from the London Publishing Partnership. Order your copy here.
More on the book soon, but in the meantime, here is a recording of the talk on cities that I gave at the festival.
At the Treasury Select Committee, in relation to HS2, Professor Graham said that it was important that we get these numbers right and in a scientifically rigorous way. Both he and Professor Overman attacked the work published by KPMG as not only insufficiently rigorous, but also on the grounds that the results are too optimistic.
I want to examine both these propositions. I’d like to start by saying that in general I think that we tend to underestimate the benefits of transport investments, and this will include HS2. One of the major reasons for this is that we have developed an analytical mechanism which is suited to ranking projects in a known economy and used it to evaluate projects in an uncertain future one. In an uncertain world, well into the future, it is essential that we are careful about what assumptions we are making about how far the future is different from the past. Statistical rigour can obscure this.
The KPMG work has been attacked for being creative and taking a non-standard approach. The Crossrail work on Wider Economic Benefits which I pioneered was also attacked for being non-standard and creative at the time – now I notice that it is considered reasonable that there is a productivity response to transport investment and Professors Graham and Overman support the idea.
The idea of ‘wider’ benefits is of course that they are an add on to ‘standard’ benefits. Standard benefits accrue to people and businesses in a known world as far as the project is concerned. The forecasts of activity are independent of the transport system. Wider benefits reflect the potential for relieving constraints on higher productivity activities arising from the transport system which mean that people do less productive work than they might otherwise be able to do. The officially sanctioned estimates of the parameters to be used have been made by Professor Graham, who is understandably protective of the work that he has produced. Standard benefits, which are largely measured in time savings, are equal to real output benefits IF AND ONLY IF firms and people undertake all projects which are even marginally profitable, AND that the values of time have been correctly measured.
While these are elegant theoretical propositions, it is clear that there are some fundamental uncertainties about this edifice. What if the forecasts are wrong, or the values of time inappropriate, or not all investments can be made? More importantly, what if transport makes more difference to activity and productivity than we have so far identified?
Professors Overman and Graham stress the problem of ‘confounding’, which sounds like something out of Harry Potter and refers to mistakenly crediting (say) transport with (say) creating jobs, when actually it was down to (say) investment in skills. A competent researcher looking at the historical data on jobs would want to attribute the change among these potential causes. This historical accounting may however be only of partial help when considering the future, especially the long term future. It will not tell us is whether the skills investment would have been useless without the transport, or indeed vice versa. Moreover, most of the analysis in this area is done on data for a particular year, so does not ask about changes over time, although this is what it wants to capture.
In a similar vein, much is made of the problem of causation. Does the economy generate the transport system or the other way round? I must admit it seems hard to envisage exactly how the economy generates transport investments which are under the control of central government and slow and meticulous planning systems, but there will certainly be feedbacks as transport generates market access, economic success, congestion, and demands for further investment.
Both joint effects and feedbacks mean that using the historical data to estimate the right answers will be fraught with danger.
Of course the data itself is messy and imprecise. Even knowing the number of jobs in an area is subject to errors and revisions, while other data on level of skills, or output is pretty woolly. What is the output of the kinds of firms which operate in city centres, advertising agencies, or accountants, or computer programmers? How do we value it? This is a difficult problem and should not be underestimated.
A scientifically rigorous process cannot be based on inadequate data and on estimating impacts without enough alternative cases to see, for example, what would happen if we made the skills investment without transport and vice versa. Statistical rigour is not a substitute for careful consideration of what the issues might be and what the statistics might miss, and which might be as important as that which is measured. It would be better to be broadly right than precisely wrong.
Hence the Treasury Committee’s probing of the KPMG work for its robustness, reliability, and whether it is a forecast is understandable but misses the point. No single piece of analysis will create a robust and reliable view. But a number of different analyses will present ranges of potential outcomes which must then be compared and judged.
The KPMG work, as I understand it, took the DfT forecasts for growth in jobs in the UK economy and asked how they might be redistributed under a different transport pattern of availability. They generated a relationship between levels of accessibility and output across the UK and used this to develop an average reaction, so that jobs became more productive as they move into more accessible locations. (This is actually pretty similar to the view of wider economic benefits, but pulls together both these and the benefits to ‘standard’ users too.) KPMG’s work effectively takes the transport investment as necessary for the benefits of other investments (such as in skills) to be effective. In other words, people’s skills will not earn the return the statistics suggest they should be able to unless there is additional transport. This could well be correct, though a statistical decomposition ex post of the various impacts will not necessarily show this.
Their estimates do, however, miss out on any effect on the total number of jobs in the economy and any impact that increased freight might have; and this is a non-trivial shortcoming. It takes a total growth forecast which stops growth only a few years after the opening of the line.
The DfT guidance on the evaluation of transport projects instructs that the key scenario is the standard one, based on time saved independently of the economy which is taken to be known. This cannot be sensible for a project of this scale and game changing nature. The guidance is not relevant here. And statistical analysis is only relevant if it illuminates what kind and scale of game changing might be possible or relevant. Professor Overman states there is a good case for rail, and also that we take far too long to make decisions. In my view, this is partly the result of the attempt to find statistical rigour where none exists. The long term effects of long term investment can be much more usefully informed by historical analysis and thinking about the kinds of opportunities that can be generated by such investment.
For example, regarding Crossrail, I judge that the final benefits of Crossrail will generate total output returns of around £80 billion on an investment of £15bn. An investment in new railways of around £40bn should, based on analogy like this, generate £200bn, in new jobs, incomes, additional commuting to productive centres, taxes, dividends and so on. This kind of broad, historically informed analysis is in line with the sort of results that KPMG have generated. It can be improved, refined, redone, but it won’t actually produce a more rigorous estimate.
 Not quite like with like but ball park