Better to be lucky than good
Tuesday, July 31, 2012 | Charles Marohn We frequently surrender power to experts, trading off our discomfort with uncertainty for the illusion of certainty that they, the experts, provide. This illusion of certainty is that -- an illusion -- but when it comes to traffic projections, many would prefer the comfort of that illusion over coming to grips with the notion that we can't accurately predict the future. Understanding this tradeoff is key for those wanting to abandon current thinking and embrace a Strong Towns approach.
We're building some momentum with our ongoing fundraiser for bringing the Strong Towns message to Pennsylvania. Our plan is to visit the state and hold 8 to 10 (or more) Curbside Chats. I anticipate some communities committing in the next few days, but we still have a long ways to go to make this happen. Please consider supporting this effort and pass it along to those you know in PA. We'd love to bring this message back to the Keystone State and change the conversation on growth statewide.
Probably the most remarkable feedback I received in response to my analysis in The Projections Fallacy was the notion that, while our models may be wrong, they are better than nothing. Without disrespecting any of the people who made comments or sent me emails (and thus I will omit their names), here is a smattering of this thread of thought:
There should be some model, some theory, that underlies that; or if we lack such a model we should create one. Maybe the results of that model can’t be summarized in a bar graph, maybe it does not converge on one ideal response, but that doesn’t mean we must make decisions blind.
---
Broadly, I would not expect naively applied models to have a high degree of precision over the lifespan of these long term infrastructure investments. If we're building a road, railway, waterworks, whatever that will last 20, 30, 50 or more years, you should understand that things can be wildly different from the projections. Nonetheless, number crunching can give an otherwise fuzzy discussion a much better context.
---
If you're about to make an investment that will last 50 or 100 years, you have to have some idea of what's gonna happen, because otherwise, you can not make a decision. And that's where modeling comes in.
---
I think everybody here agrees that current models suck. The question then is what to do. Get rid of models and just take a swing, or invest in better models? I think the latter. Taking a swing has not proven to be a good model to move forward.
When I was much younger, I watched a news show where the reporter picked a portfolio of stocks by throwing darts at a list of all of the stocks traded on the New York stock Exchange. Then they ran the portfolio for 12 months and beat the performance of nearly every mutual fund available at the time. The point was, if you are paying fees to mutual fund managers, what are you supposed to be getting? For the extra money you pay to be in the fund, you would assume the performance would be superior.
Today there is a fairly broad consensus that nobody is going to beat the S&P 500 index fund over time. The S&P 500, a broad index of 500 companies, is essentially "the market". In fact, consider the following quote:
According to the folks at the Motley Fool, only ten of the ten thousand actively managed mutual funds available managed to beat the S&P 500 consistently over the course of the past ten years. History tells us that very few, if any, of these funds will manage the same feat in the decade to come. The lesson is simple: Unless you are convinced you are capable of selecting the 0.001% of mutual funds that are going to beat the broad market, you would best be served by investing in the market itself.
So if nobody beats the markets over time, why are there mutual funds? Why would anyone pay more for someone to underperform in managing their money?
The same reason we ask for traffic projections from engineers: it gives us comfort in the face of uncertainty.
People who don't know anything about investing but know they need to save money will most often give their money to a manager to take care of. The manager is the expert and their advice will provide peace of mind. I'm not suggesting that is a bad choice; you save and invest to have peace of mind. If you are going to gain a percentage point or two of return but not be able to sleep at night from the tension, paying someone more knowledgeable than you is probably worth the expense.
Let's not kid ourselves, though. Our need for projections is based largely on our need to feel comfortable with our decisions, not on any real value those projections provide. We feel more justified when following the advice of an expert, even when we know that advice to be flawed.
To me, the sad thing about this human tendency is that we collectively cede decision making over to others. We do this particularly when the stakes of the decision are highest. As economist Noreena Hertz states in this amazing TED talk on our over reliance on experts, "We've surrendered our power, trading off our discomfort with uncertainty for the illusion of certainty that they [the experts] provide."
I strongly recommend that you listen to this powerful video. In it she talks about how experts form very rigid camps where a dominant perspective emerges that often silences opposition, how they hero-worship their own gurus, how experts are governed by social norms of their times, are captured by funding streams and cling to dogma.
My favorite quote from her talk:
"Progress comes about not only from the creation of ideas but also from their destruction."
Am I saying that experts are bad and all projections worthless? Absolutely not, but I do think we need to understand the limitations. An engineer can predict with some relative certainty what the traffic counts will be over the next few months, or even a couple of years. For my transit friends, the phenomenon is much the same. We only demand more aggressive projections -- and expect engineers to speak with certainty -- because our system requires us to place such huge bets on the future.
Later this week I will deal with that aspect of the suburban development pattern; the requirement that we place such large bets on the future. In the meantime, I urge those of you that would rather follow a flawed projection than have no projection at all to embrace the reality of a more uncertain future. Instead of being scary, you may actually find it empowering, especially if you take the logical next steps and build some resiliency into your life and community.
If you want to chat with Chuck and many others about implementing a Strong Towns approach in your community, consider joining the Strong Towns Network. The Strong Towns Network is a social platform for those working to make their community a strong town. Get expert assistance, consult your peers and stay current on the latest techniques and analysis.
Reader Comments (7)
Chuck,
Great piece. If you haven't already, check out Nassim Teleb's Black Swan and upcoming Antifragile. Much along these very same lines.
~Joe
Love this. 'Street Parking' in my town is totally being stifled because it seems no one on the city staff seems interested in challenging their experts.
Actually, you have part of this backwards. You say "An engineer can predict with some relative certainty what the traffic counts will be over the next few months, or even a couple of years." It's the opposite. You can't know what is going to open/close tomorrow or next year. But you can estimate overall growth over a period of years (at least 10 years to have any credibility). It's like your stock market analogy. You have no idea what stock will go up or down tomorrow. But we do know historically, over long periods of time, you can expect an overall stock market growth around 3-5% annually. Doesn't mean there won't be dips or peaks in the meantime. Traffic projections are the same. If someone is telling you they can make accurate traffic projections at 1 to 2 year intervals they are full of it. The assumptions are just too rough for that fine of detail. Likewise, trying to project much beyond 20 years is unrealistic as there are too many other variables that come into play.
@Jeff Morrow
Please cite your data for your conclusion. I think what you are saying is common belief, but it is based on myth and not reality. Unless you believe that the entirety of humanity should be scored through a post-WW II prism, the notion that the world works on a predictable bell curve type of statistical modeling simply doesn't comport with reality.
Data. OK. Let's say you have an 8 block corridor with 3-4 acres of property that could be redeveloped. Based on the regional computer model provided by the MPO, looking at historic traffic growth rates, looking at the City's approved land use plan, and talking to the property owners on the corridor, I come to the conclusion that over the next 20 years this 3-4 acres (as well as some other dispersed sites along the corridor) could easily convert from a collection of old used car lots, some shell retail/office, a church, and miscellaneous vacant property into a regional medical corridor (there are already two hospitals on either end of this corridor). Based on what the property owners are telling me, I expect the land use to change and can therefore estimate the traffic volume difference for the new land uses they have in mind. Let's say it's a 25% increase in traffic on this 8 block stretch of roadway. Over 20 years, 25% increase works out to an annual average increase of about 1%. If I just project out 2 years, that's only a little over 2% growth in traffic. If, however, one of these property owners decides to pull the trigger at two years, then I may actually realize over half of that projected growth immediately. You just don't know what people are going to do or when they are going to do it. Now, if they don't pull the trigger within 20 years (the design life of the pavement) who knows what the economic climate will be at that time, and I wouldn't feel comfortable projecting beyond that for the exact reasons you cite in many of your blogs. So, my options are to ignore what the stakeholders on the corridor are telling me and just recommend do nothing - hold the status quo in the face of threatened redevelopment and infill, or tell my client, the City, that their constituents are signaling they are willing to invest in this downtown location and they are believable. AND they want the traffic accommodated.
This is a real project I have just completed. The 3-4 acre site is now under construction as a regional medical mall scheduled to open next year. They have already rented out the entire site and are planning their expansion (which I padded my numbers a little bit and accounted for this). Just down the street, the other hospital has just opened a regional cancer center that they didn't tell me about, and I didn't specifically include that in my calculations. However, not to worry, some of the regional background traffic growth identified in the MPO model provided some cushion to cover that. Is it a perfect 100% accurate traffic projection? Don't know - too early to tell yet. Does it have to be perfect? No. If anything, it might be low as the burst of redevelopment absorbed a lot of the expected growth over the next 20 years, which is what I told my client. Had I just gone with two years of growth and built the street back the way it was (or narrower as was suggested at one point), there would have been a serious backlash about why we are reconstructing a brand new street while these medical facilities that just opened are trying to operate. There would have been some very pointed questions I would have had to answer. We would have been very foolish and short sighted.
So, in this instance, myth did comport with reality. In my experience this is more often the reality than not.
@Jeff Morrow
That is not data. What data suggests that we have been accurate in predicting events decades into the future? All of the data in regards to traffic modeling says that it is simply not true. You can claim today that we've improved the models, but that is also a belief I'm sure our predecessors had as well.
Your story is fine and I see where you are coming from, but you are missing the larger point. Sure, if you are trying to function in today's marketplace as an engineer, you are stuck with the current (flawed) approach to development. Okay, let's stop justifying that flawed approach and instead adopt a different approach.
The idea of zoning everything into separate pods and then creating a hierarchical road network was a good idea in theory. In practice it has been a failure on many, many levels. You can do good, competent work within this system. That doesn't make it a good system that ultimately creates good outcomes in enough instances to justify keeping it.
@ Charles Marohn
I believe the report you reference in your blog about the projections fallacy suggests traffic models tend to consistently under predict future traffic (although I have a little problem with their methodology, but that's outside the scope of this discussion). However, nothing that is actually built is based on model projections. Models are only used for planning purposes and providing general guidance on possible relative growth in the system as a whole. Occasionally they are used to compare two competing regional alternatives. They are only appropriate for 10,000 foot level stuff.
For project specific, design level traffic projections, you review the model projections as one possible source of information. But I would say model data only accounts for 5-10% of the design project traffic projection. In fact, I have found models so ridiculously wrong at the MICRO level I don't even use the model data for the project corridor when developing the project traffic projection. But again, the models are totally inappropriate for micro level use. In that case, I might consider looking at the overall expected jobs growth or population growth as provided by the local Chamber of Commerce, or other business groups (keeping in mind they have their own agendas too).
The models have gotten a little better. However, the models are only as good as the competence of the staff who assemble them. There are good and bad models out there not so much because of the models themselves but because of the level of expertise of the modelers. That's why you have to delve deeply into the model data when using it and have pretty lengthy discussions with the modeler on how they built, calibrated, and developed the model data before you dare consider using it. This is just responsible engineering.
I agree we can't predict specific events decades into the future. However, in practical application, much of it comes out in the wash, per my story. My point is modeling is useful for some general guidance, but that's it. A real traffic projection takes a lot of different sources (Models, historic data, local stakeholder input, City vision / master planning, etc) to develop. Some sources you find credible, others you don't. I don't know of any reputable traffic engineer that would design a roadway and expend public funds to build it based solely on model data. That would be negligent.
By the same token, I am wide open to new approaches. But since I'm an implementation guy (actually getting stuff built) I will want to delve into the specifics of how we take the 10,000 foot level concept and apply it to an actual project. As you can probably tell from my writing, I'm a bit of a skeptic, but I can be convinced.