Build a better model
Monday, July 30, 2012 | Charles Marohn We humans are eager to believe, in the face of overwhelming evidence to the contrary, that large changes in the trajectory of human progress are infrequent. There is comfort in pretending that we are far more in control of things than we really are. We don't understand, or perhaps respect, the mystery and complexity of the world we operate in.
If you are in Pennsylvania and would like to have the Strong Towns message brought to your community, we have an ongoing fundraiser to help us visit your state and hold 8 to 10 Curbside Chats. Please consider supporting this effort and pass it along to those you know in PA. We'd love to bring this message back to the Keystone State and change the conversation on growth statewide.
Last week I blogged about transportation projections and how it really is silly that we rely so heavily on a system that we know gives us bad advice. In The Projections Fallacy, I pointed out that we don't need better projections but a model of growth and development that is robust to modeling error. This seemed fairly self evident to me; if something is not working, find something else that you think will.
I was not prepared for the number of people that had a difficult time wrapping their mind around this. There were three main counterarguments put forth to my premise. First, there were some that agreed that our models were bad, but they argued that what was needed is a better model. I'm going to address that belief today. The other two -- that bad models are better than no models and that, flawed as it may be, I have no better solution -- I will address later this week.
Earlier this month I read When Genius Failed: The Rise and Fall of Long-Term Capital Management by Roger Lowenstein. Long-Term Capital Management (LTCM) was a famous hedge fund that experienced unprecedented success before going bust in spectacular fashion in 1997, necessitating a federally-organized bailout. The story should humble anyone who does high stakes projections or modeling for a living.
The idea behind LTCM was an equation known as Black Scholes. It is a model for pricing options that, through use of statistics and normal distributions, allowed hedge funds to make "low risk" bets on the movement of markets, offsetting those bets with a "hedge" to limit the amount of loss. In theory, it is an amazingly powerful model, brilliant enough to earn a Nobel Prize in Economics. In practice, it failed spectacularly and almost brought down the entire global financial system.
It is not that the model was wrong -- much of what they were doing had been back tested and, given infinitely deep pockets, infinite time and markets that operated within their modeled range, they may have been able to weather the storm -- but more that they were incomplete. They assumed rational market behavior; that others working in an efficient market would not leave money on the table, opt for the lower performing (but safer) bet, voluntarily take a loss, default on debt obligations and a myriad of other things that happened in succession. Their models said this confluence of events should happen only once in the history of four universes. Obviously, that was not quite accurate. Their downfall came from market responses that could not be modeled.
In comparison to the modeling used by the quants on Wall Street, traffic modeling is so simplistic as to be almost laughable. I've seen more just straight linear projections than anything else, essentially taking a ruler and drawing a trend line out from past data. This is done digitally using a spreadsheet to give it a veneer of complexity. It is anything but.
More advanced models actually examine the behavior of traffic given certain changes to the system. If one road is opened up or widened, cars will opt for that route, reducing traffic in a different area. These models are more complex with many more variables. Like any model, the greater the number of variables the increase in the likelihood of modeling error and the greater likelihood of an outlier event.
So the idea of building a better model is being tried and there are many people -- in the financial world and in the traffic modeling world -- that believe it is possible. If you back test the data far enough, fine tune your variables, try to take into account all the intangibles you can think of, you can get a model that is reasonably close.
Until it isn't.
Let's go back to 1925 and apply this build-a-better model logic. Pretend we worked for a local trolley company. We needed to project how much to invest in new growth (overlooking for a minute that this is not how the trolley systems were built -- build-it-and-they-will-come is a modern phenomenon). We look back over the past 50 years and see a country that has been urbanizing through the Second Industrial Revolution, one that is in the midst of a great boom that appears to have no end, one where the demand for the trolley system projects as strong.
We can pull in any variable or intangible that we could reasonably identify at the time and we would not have been able to predict the Great Depression, the Second World War, suburbanization, the advent of buses, the rise of the automobile, the Interstate Highway Act and the forces that would relegate the trolley industry to a local novelty. And that is just in fifty years.
We're making investments today that we expect to be around at least that long, with projections of the future that extend out decades. I don't care how good the model is, no model will accurately predict the future. Nobody can tell you with any type of certainty what will happen with gas prices, technology innovations, consumer preferences, war and disease and any other myriad of game-changing variables that humanity has faced throughout history. And we're in a period of history where change is accelerating, something that modelers are totally blind to.
Now, none of this would matter if the bets we were making were small. You can bet a small amount of money on the probability of things happening in the distant future and it really doesn't matter if you are wrong. Unfortunately, we're not making little bets. We're making massive bets, not only in terms of the money we spend today but in the cumulative repercussions of the approach over time. That's why we do modeling; this is all really important because there is a lot at stake.
So if we can't build a better model -- and we know that to be a self-evident truth -- why do we do it? That is a psychological question that I'm not fully qualified to answer. My suspicion is that we humans are eager to believe, in the face of overwhelming evidence to the contrary, that large changes -- positive or negative -- in the trajectory of human progress are infrequent. There is comfort in pretending that we are far more in control of things than we really are. We don't understand, or perhaps respect, the mystery and complexity of the world we operate in.
Our models simply don't work and we can't build a better model. We need to face those facts, embrace them fully and do something positive with that knowledge.
Later this week I will address the two other contrarian responses I received, specifically, that bad models are better than no models at all and, if I'm so smart, what would I do differently in a world without models. Check back with the Strong Towns Blog for that.
If you want to chat with Chuck and many others about implementing a Strong Towns approach in your community, consider joining the Strong Towns Network. The Strong Towns Network is a social platform for those working to make their community a strong town. Get expert assistance, consult your peers and stay current on the latest techniques and analysis.
Reader Comments (12)
Very well stated, Chuck. And, we often overlook the key element in our models: human beings. We're not always so predictable as we think - prone to use emotion as much as logic. And, in the case of traffic models, that's not water flowing through pipes - it's human beings operating all those vehicles.
The issue I see here, is that traffic, congestion, roads - and the priority we assign to each - is a social choice and not a scientific question at all. How much traffic is too much? How far are people willing to drive? How long does it take to drive across town? How safe and walkable are our neighborhoods for walking and riding bikes? These are all social choices. Yes, we need better modeling to be able to make more informed decisions, but the 900lb gorilla is that we accept that a traffic model will tell us what should be built.
Excellent writeup
Except: The "build it and they will come" mentality grew out of the trolley companies. Most trolley companies were actually land development firms with a transportation arm--the trolley was seen as an amenity to help sell what were otherwise the outskirts.
You have a point. The key difference with trolley companies -- which I will touch on later this week -- is that they speculated in small increments. If they were wrong in their estimates or projections, the entire thing was not going to collapse.
I understand your point, but projections are a fundamental part of the planning process. Yes, they are flawed. Yes, we probably won't end up following them, but they are still crucial for one important reason: Those who fail to plan, plan to fail. I have found both as a mom and an engineer that a day unplanned is a day frittered away. I don't know that I've ever once completely stuck to the plan in the entire history of my planning, but the plan keeps me headed in the direction I need to go, even if I don't follow it perfectly.
Part of the issue I have with planning models is that they are so exclusive to vehicular travel. This is not easily overcome as the California experience is showing. I do know that what we measure and project is what we fund. If we fail to measure and project non-vehicular flows, we won't fund them adequately and they won't exist. It drives me crazy that the only pedestrian flows that we use in the Florida standard structure are those that support transit--People are not assumed to just walk or bike anywhere--so there are no facilities funded or constructed to walk or bike and people don't. This will require more data collection and refinement, but it has to get on the radar somehow.
Chuck, your post reminded me of another projection I read about many years ago -- that somewhere around 1890, officials were making plans to stave off a future crisis they'd identified in their projections: By 1961, the horse population in Hennepin County would outnumber people (with all the attendant mess, odor, and management issues implied by that fact).
Patricia's comment illustrates a point I wanted to add: That models and projections can be even worse that what you're stating when they are based on biased or incomplete data. And as Wall Street and traffic engineers have found out, both models and projections are prone to serious error if human behavior is involved in any of the variables.
On the other hand, again to Patricia's point, projects that don't have plausible metrics attached aren't likely to be funded. Elected officials need defensible rationales for the decisions they make. I'm looking forward to hearing your ideas about alternatives.
Thank you Nassim Taleb :)
Your premise is too simplistic. The idea that LTCM blindly followed their model into oblivion, or that their model is to blame for their demise is just wrong. Then to extrapolate that flawed reasoning to somehow justify regional traffic models are therefore bad? Not believable.
@Jeff Morrow
You can say it is not a fair analogy -- I would disagree -- but LTCM did follow their models to the point of no return. In the final weeks they understood that the world had turned against them in a way not predicted in their models, but at that point it was too late to do anything about it. They could hang on while they continued to bleed money and hope a miracle turned things around or they could dump their holdings in a desperate attempt to stay solvent, a move that would have guaranteed massive prices drops in every security they were trying to unload.
The act of predicting the future is full of complexity and uncertainty. In the end, I really don't think that is a debatable premise. That traffic models largely ignore this is also not debatable. What traffic model includes the possibility that we will all be riding jet packs in 25 years? Which model includes the possibility that we will be back to horse and buggy? Which model accounts for $10/gallon gas and the inability of Americans to finance new auto purchases every five years? The answer to all three of these questions is "none". They all assume a rough continuation of the status quo condition, something we can clearly see is not the case simply by looking back in time the lifespan of a major bridge investment.
You may say, "that doesn't matter -- we have to work with what we have -- who could possibly predict that," but you would be making my point.
@ Charles Marohn
I will direct you to Wikipedia about LTCM. 2nd paragraph under "Trading Strategies". Basically, they got away from their models when personalities/greed/investor pressure took over. They quit following the game that made them successful. Took bigger risks that were NOT supported by their model. Thus, they lost. Even so, there are no guarantees. They could have lost even following their model because there is inherent risk. Their model simply helped them choose the least risk BETS, but there was still risk. When they got so big their model no longer was valid is when they got in trouble. You can't blame their inappropriate use of an inapplicable model for that.
Even if we are riding jet packs in 25 years, the turnover cycle to convert from autos to jet packs will be longer than the design of most roadway projects in the pipeline. Also, models are not static. They require maintenance, like anything else, to reflect change. I'm sure no planners in Detroit envisioned the collapse of the auto industry and subsequent population loss associated with that. So, you revisit your model population and employments forecasts and update your model. There might even be a project in the pipeline you abandon because it no longer is supported by the updated data. O.K. That's what's supposed to happen. Course correction based on current conditions and revised forecasting.
I would weed out the jet pack and horse / buggy options out of hand. They don't seem realistic to me. Maybe your new model will include them. However, our MPO is assuming a shift from auto traffic to other forms of transportation, which is an implicit accounting for higher fuel prices. That is far from status quo. But, is that right? I don't think so. I think it will take $50-$100/gallon gas (or more) to pry Americans out of their cars. Why do you think we are developing hybrids and super fuel efficient cars? Even at $3-$5/gallon gas we are still driving. Mileage is down, but mostly discretionary mileage. Most people still commute in the single person auto. More likely scenarios are that we will be driving electric cars or hydrogen cell cars or some other alternative fuel driven vehicles. More likely is we will have computer driven vehicles with 1 second headways at 50 mph and unified central traffic management computers that will automatically route traffic to spread the load (They already have that technology, it's just not released yet for public use). At the end of the day, we will still need to provide streets and parking for cars. Americans LOVE their cars and the freedom it imparts. But, I just have my post WW2 prism blinders on.
I get the impression, and I may be wrong, that you link construction of roadways (actual expenditure of public dollars) with regional traffic models. I've said this before, regional models are only a very small piece of the overall decision making process. In fact they are really only useful at the very infancy of the regional planning process to help focus planning efforts on specific areas. Actual capital improvement projects are identified within City public works departments (typically). Sometimes, the models are consulted to see if the projects identified are roughly consistent with model projections, or possibly model projections are used to assist with federal aid funding applications, but that's about it. Anything more is inappropriate.
@ Charles Marohn
I will direct you to Wikipedia about LTCM. 2nd paragraph under "Trading Strategies". Basically, they got away from their models when personalities/greed/investor pressure took over. They quit following the game that made them successful. Took bigger risks that were NOT supported by their model. Thus, they lost. Even so, there are no guarantees. They could have lost even following their model because there is inherent risk. Their model simply helped them choose the least risk BETS, but there was still risk. When they got so big their model no longer was valid is when they got in trouble. You can't blame their inappropriate use of an inapplicable model for that.
Even if we are riding jet packs in 25 years, the turnover cycle to convert from autos to jet packs will be longer than the design of most roadway projects in the pipeline. Also, models are not static. They require maintenance, like anything else, to reflect change. I'm sure no planners in Detroit envisioned the collapse of the auto industry and subsequent population loss associated with that. So, you revisit your model population and employments forecasts and update your model. There might even be a project in the pipeline you abandon because it no longer is supported by the updated data. O.K. That's what's supposed to happen. Course correction based on current conditions and revised forecasting.
I would weed out the jet pack and horse / buggy options out of hand. They don't seem realistic to me. Maybe your new model will include them. However, our MPO is assuming a shift from auto traffic to other forms of transportation, which is an implicit accounting for higher fuel prices. That is far from status quo. But, is that right? I don't think so. I think it will take $50-$100/gallon gas (or more) to pry Americans out of their cars. Why do you think we are developing hybrids and super fuel efficient cars? Even at $3-$5/gallon gas we are still driving. Mileage is down, but mostly discretionary mileage. Most people still commute in the single person auto. More likely scenarios are that we will be driving electric cars or hydrogen cell cars or some other alternative fuel driven vehicles. More likely is we will have computer driven vehicles with 1 second headways at 50 mph and unified central traffic management computers that will automatically route traffic to spread the load (They already have that technology, it's just not released yet for public use). At the end of the day, we will still need to provide streets and parking for cars. Americans LOVE their cars and the freedom it imparts. But, I just have my post WW2 prism blinders on.
I get the impression, and I may be wrong, that you link construction of roadways (actual expenditure of public dollars) with regional traffic models. I've said this before, regional models are only a very small piece of the overall decision making process. In fact they are really only useful at the very infancy of the regional planning process to help focus planning efforts on specific areas. Actual capital improvement projects are identified within City public works departments (typically). Sometimes, the models are consulted to see if the projects identified are roughly consistent with model projections, or possibly model projections are used to assist with federal aid funding applications, but that's about it. Anything more is inappropriate.
@Chuck Marohn
Not sure if my last post made it through.
Refer to Wikipedia on LTCM. Under Trading Strategies, 2nd paragraph. They got away from the model that made them successful and started making larger bets (due to internal and external pressures) on things that were NOT supported by their models. Greed and arrogance sank their ship. I don't see regional traffic models being used for personal gain in the same vein of LTCM. That's just too much of a stretch for me. Plus, in the end they didn't follow their model.
You like analogies. Here's one. Google Earth is a free internet based program that allows you to access satellite imagery from about anywhere in the world. I can enlarge the aerial photo on my house and see my driveway, front and back yards, etc. But when I enlarge the photo like that, it gets fuzzy and pixelated, hard to see details. Google Earth has a ruler function that allows me to measure things on the ground. It says the sidewalk from my driveway to my front door is 4.83' wide. But I know when I measure it at home it's 5' wide. This model is wrong. Why would we use it? Maybe it's because I am using the Google Earth model inappropriately for something it wasn't designed to do. But if it has a measurement function, shouldn't it be right? Well, how right does it need to be? Is it appropriate if you want to know about how big something is, understanding there is some error? Yes. Is it appropriate to build something off of? No. It's not that accurate.
This is EXACTLY the same as regional traffic models. They are good for the high level, planning type analysis of regional features. They are nowhere close to accurate as the sole source of information for project specific use. In fact, most regional models don't even have all the streets included in them. Usually most local streets are omitted. I've seen a regional model where one arterial actually represented 3 arterials. So in reality, the streets in the model are a surrogate for the actual existing supporting network around those modeled streets. You have to understand what the model data represents before you can use it. (or disparage it)
Now, let's get into the whole debate about whether you can accurately predict the future or not. Well, back to the Google analogy, how accurate does it need to be? Can we only tolerate 100% accuracy? No. Since we don't use regional models to make design decisions such as how many lanes or do we need turn lanes, etc. we can tolerate error in the regional model.
When we get to project specific traffic projections and assuming we have done our homework, then it gets tougher. So we use sensitivity analyses to vary our growth projections up AND down typically by 10%, 25%, 50% and 100% increments. 0% meaning we use our growth projection as is on top of existing traffic, +100% meaning double the growth we assume, and -100% meaning no growth - just use existing traffic. (I have even looked at decreasing traffic over time if the data supports it). Then you can get a feel for how much RISK is involved with the various design decisions. This gives an indication of how robust is the design relative to changes in the assumed growth. So, we do take into account errors in the projections and then make the safest "bet" based on the results. (Safest as defined by my client.)