Superforecasting: The Art and Science of Prediction

by Tetlock, Philip E.

Published: September 29, 2015

Read: December 13, 2019

★★★★★

Review

The questions this books asks are so golden: How can you better predict the future? When pundits make predictions, why do we not care about accuracy? The intro is solid, the middle was a little slow covering some standard ideas in a standard way, but it culminated into a pretty grand conclusion. Approaching the world with an analytic, Bayesian mind and avoiding human cognitive biases are powerful ideas that are slowly changing society. Though focusing on more political/social predictions, the same approach could be applied throughout your life. Did you know fame and accuracy are inversely correlate? The ideas and conclusion are so valuable, but this book might be best read through a long summary. "People can with considerable effort make accurate forecasts about things that matter"

Notes

#Book by [[Philip E. Tetlock]] and [[Dan Gardner]]

Chp1 Optimistic skeptic

Pundits are not judged based on the accuracy of their predictions, but they tell good stories. Baseball coaches ask to see statistics, but we don't about the people making predictions we trust. 3-5 years out experts are random. Most experts are like monkey throwing darts, not all. Author says it I possible to predict and learn how to. Butterfly effect, there are limits to predictions and as we know more we are less confident in perfect prediction. But many things are. Bill gates is amazed at how much progress can be made if you make a metric and move towards it. But no one cares about accuracy of forecasting. Good judgement progjrct beat control by 80% for ARAPA. 10% jump in prediction by reading an hour long idea book. Superforecasting requires open minded, careful, focused, and self critical. Commitment to self improvement is the strongest predictor of performance. Human, computer pair probably future.

Chp 2 illusions of knowledge

Early physicians were like blind men arguing about the rainbow. If someone died no or not just meant the lead and bloo letting didn’t work. Random trials are a modern concept. Just trust physician ideas. Physicians didn’t need scientific validation they just knew and resisted from their authority. Controlled trials that give real insights vs uncontrolled trials that give the illusion of knowledge. If you implement the policy, then whatever happens can be argued away or solidified from didn’t perspectives. Maybe something else caused the change. Introspection doesn’t provide everything. System 1/2. 1 treats evidence as true. What you see if all there is. Sane people are expected to have reasons for doing things. Doubt, what would convince me I am wrong.

Chp3 Keeping Score

Randomized controlled trials? For history? Can't be ambigious about what prediction means. Ballmer said iPhone wouldn't get significant market share. But what is significant wnd what market and on what time scale. It is all obvious if you look at it with the right retrospective light. Professional Forecasting often given without a timeframe and with implicit understanding of terms and fuzzy meaning of words like fair chance or serious possibility. People look at right side of maybe, if true prediction is right if over 50% chance. But the prediction could be spot on but you can't rerun the event. Need many probabilistic forecasts to judge forecaster. Bieher scores measure distance between what happened and what you predicted with much bigger hits to 19:1 vs 9:1 i.e 95% vs 90%. 0 is perfect 0.5 is monkey random. But, some things predicted are more variable and some easier to predict.

Research study over long study: The average expert is like dart throwing monkey. Info wasn't important, how they thought. Hedgehog big ideas. Ideological squeezers often wrong, furthermore. Accuracy declines the more they know about subject. Inverse correlation between fame and accuracy, people like stories and certainty. Foxes are other group changed minds, probably on the other hand. Complex stories. Aggregate perspectives Wisdom of crowd, valid information pushes to truth and invalid information random so cancels out.

Chp4 Superforecasters

"Conclusion is reasonable, but it was wrong" This makes sense. Iraq WMD. iC was shook and very wrong though so it started trying to change. Evidence based obv, even if obvious like teaching cognitive biases. Track analyst accuracy. Iarpa funded Good Judgement Project. Ordinary people using aggregating wisdom of crowds idea and then extremizing it so pushing 80 to 90. Because info is spread, but if you had all the info you would be more confident. This beat professionals with priveliged info. Superforecasters top 2% see triple as far and beat professionals with priveliged info by 30%. And year after year the gap got wider. Brier of .25 vs.38. Baltimore stock broker, you can expect some people to be that good.

So was there regression to the mean? Actually the opposite! So convincingly less about luck. After year one, they put the super forecasters on teams and recognized them as super. Roughly 30% dropped year to year.

Chp5 Supersmart?

Knowledge and natural intelligence? Volunteer forecasters were 70% above average. Superforecasters were 80%. Fermi prediftions like jow many piano tuners in Chicago.

Priors, called outside view here. Inside view is details to update. Watch anchoring. Then you synthesize the outside view with the inside virw. Is there a better base rate? Asking yourself to seriously consider why it's wrong, almost as good as getting a second person perspective. Counter confirmation bias by flipping question. Need for cognition is a thing, these free forecasters are some. Actively open minded, try to actually change your mind.

Chp 6 Superquants

Superforecasters ace little math tests. Most Superforecasters don't use data science models, not even using math mostly intuition though. Getting agreement on a decision is a bad sign of group think. Probabilities are correct, even if seen weak and confidence of 95% is unreasonable. Flip of a coin often means maybe. Some people take ignorant prior when it seems uncertain and decide you can't know so don't think about it. Most people have three settings yes, no, maybe. Use 50% very often. Makes sense in ancient world People will pay 2-3x more than for a reduction from 5 to 0% then 10 to 5%. People misjudge competence and confidence. Science says no provable certainty too and so is everything in this book, but easier to take in terms of certainty. Epistemic vs alien uncertainty. Epistemic you can know, but you don't. Alien you can never know like long term currency or weather. Superforecasters stay very uncertain with alien things. Ordinary people stick to 10s and love 50. Superforecasters go to ones and even 5 in casual conversation and it actually changes their Brier score. nIC, gov, goes by 5s though. Only naive ask why, those who see reality more clearly don't bother. Meaning brings happiness and lowers PTSD, chance and fate don't mix. People like choosing fate. Thinking Counterfsctuals for school attendance, thinking of other possible world's, gave students more meaning and more fate thinkkng for the school they went to. Logic something like prob it happens randomly is low it must have been meant to be. They had to take a path, and this is the one it took. Why me, why not me? Superforecasters basically antifate, "randomness is often present in our lives". Finding meaning in events is positively correlated with wellbeing, but negatively correlated to foresight.

Chp 7 Supernews Junkies

Steps of Superforecasters: unpack questions in components, seperate known and unknown, take outside perspectives that classifies it as a regular thing, take inside view, notice differences between your views and others paying particular attention to crowd wishing like prediction markets, synthesize, express in fairly grained probability. Google alerts to update. SF did have 50% initial forecasting so not just unemployed people updating. Updating from auxillary data like a test lab delaying release of results relating to process of positive test. Spin wheel hard when you get new information, but can't overreact too. Take perspective of person not yours. Must be falsibable. Publically committing to belief is great way to keep it in place. When the facts change I change my mind. Identity with beliefs people can't even imagine evidence that would change their mind, so could better to not be expert. Dilution effect, stereotypes/certainty weakened if you get irrevalant info. Many small updates from best forecaster. Bayesian updating. Start with prior. Given new event how likely are you to see event if true. How likely if false. Adjust accordingly. Equation not used explicitly by forecasters. All this is broad guidelines

Chp 8 Perpetual Beta

Growth mindset. Failure especially surprising means one of your fundamental assumptions could be wrong. Don't stop trying new things. Informed deliberate practice. Learning to forecast means trying to forecast. Knowledge empirically helps those with practice even more. Need clear feedback. Officers grow confident much faster than they get better. Probabilities are much harder to judge as accurate. Hindsight bias. Shooting free throws in dark. Type of forecasting important to practice. Caution, humble, non-deterministic philosophy. Actively open minded, need for cognition, introspective, numerical, analytic, value diverse views, probably, grit. Strongest predictor is perpetual beta, commitment to belief updating and self improvement. 3x indicator than intelligence.

Chp9 Superteams

Group think when no one has complained you imagine there must be nothing wrong. But can also push bounds. Need independent judgments though. GJP created online teams with people having individual and team scores. 23% team improvement. CEO disease when success and hubris stops you from patterns that got you there. Corporate ideas are to suspend hierarchy, quiet leaders views, bring in outsiders, premoterm. Usually 5/6 nucleus. Superforecasters got 50% more accurate. Prediction markets. Ordinary teams beat average wisdom of crowd by 10%. Prediction beat ordinary teams by 20%. Superteams beat prediction markets by 15-30%. Obviously these prediction markets weren't highly liquid without money at stake. Teams culture is emergent process, better teams said our more than my. High open minded people didn't create a high opened team. Givers. Diversity makes ability if different perspectives. Remember wisdom of crowd is average, but more diversity means more scrape that on average push towards direction it was pushed to. So extremized the wisdom of crowds. Extremizing Superteams didn't help much because they share info.

Chp10 The leadership dilemma

Leaders must be confident and decisive and visionary. Forecasting is humble. No plan survives contact with the enemy. In war, no two cases are the same. Germany encouraged critical thinking. If you absolutely must, ignore orders. Once decision has been made, no room for doubt (externally). Top down tell units what goal is not how to do it, allow units to figure it out if map is different then territory. Soldiers were told what commander wanted. German success attributed to this mission command. Top University recommended not for education, but for seeing intelligent different perspectives. Preserve and promote iconoclast. Leaders do think they are the best, but that doesn't mean you have to not think. Important to wuestion should you and shouldn't you be confident about? The test of a intelligent mind is holding two contrary ideas and still function.

Chp11 Are they really so super?

Do you see them as different kinds of people or people that do different kinds of things? The feeling of knowing is so seductive. System 2 constantly countering system 1. Egocentric world vieez : what you see is what there is WYSIWTI. Knowing it's a illusion doesn't stop it. Scope insensitivity. 2k birds, 20k, 200k basically same donation, people ask how bad do I feel. Time frame, probability it falls in 3 vs 6 months made no difference for most forecasters. Superforecasters p good. Change scope to get this. Habitual stepping back. System 2 to habits would be ideal.

Black swan book. History is judged by huge shocks and so forecasting is misguided. B but, History also crawls look at 2% growth of economies. Everyone including author and taken agrees forecasting 5 or 10 years out beyond obvious because of butterfly effect of nonlinear systems. Every 4, dod commissioned 20 year defense forecast. Instead, just plan for surprises. Set priorities with forecasting though, cost analysis. Taken in black swan argues we live in extremistan world that is hard to predict. What if Hitler, Stalin, and make were women. 50% chance right. "But people can with considerable effort make accurate forecasts about things that matter"

Chp 12 What's next?

What does one do to update after surprising event? Like the Scottish voting no for independence. Start keeping score with a precise prediction. Forecast, measure, revise. Evidence based change. Accuracy is often only one of many goals of forecasts. Goal to advance interests of forecaster and forecasters tribe. Calling Holocaust will happen without nuclear dissrment, will push people to join which means it accomplished its goal, so maybe forecasting won't change. Ernest End result system, track hospital entrance and exit and end result. Initially heavily resisted and fired, never fully adopted but became Nationwide movement. Rigorous anaylsis pushing charity and government and sports. Evidence forecasting couple be part of this shift, what took so long? Intelligence community changing is surprising.

Humanistic perspective

"Not everything that counts can by counted. And not everything that can be counted counts."

"There are metrics for phenomena that can't be metrical measured. Numerical value signed to things that can't be counted by numbers"

Brier score has weaknesses. Credit score advance from nothing and loan officer decision. Big important questions is how does this all turn out? Not does North Korea launch a rocket next month? Can do Bayesian clustering of similar questions to see how something turns out?

Important to ask good questions. Slam the forehead tests, if only I had thought of that before. Like is Saddam cause of Iraq dictatorship or is Iraq cause of Saddam? Super questioner might need hedgehog approach instead of uncertain fox. Some pundits like Milton Friedman pointing to falling oil prices leading to unrest in middle East isn't a forecast perhaps but a question focusing attention.

Epilogue

10 commandments

Triage, focus on questions where hard work can pay off
Break seemingly intractable problems into tractable ones
Strike balance between inside and outside views, no event is unique
Balance between over and underrating to evidence, sneak out leading factors with subtle updates
Look for clashing casual forces in each problem. Look for counterarguments and list things in advance that would change your mind. Then synthesize
Strike to distinguish more degrees of doubt. Expert poker players see 60-40 as 65-35. Don't reserve rigorous thinking for sports betting
Balance between under and over confidence
Look for errors and watch out hindsight bias and protecting your ego. Postmortem on fainted and even successes.
Bring out best in others and let them bring out the best in you
Master error balancing bicycle. Actually do and update.
Don't treat commandments as commandments.

Review:

The questions this books asks are so golden: How can you better predict the future? When pundits make predictions, why do we not care about accuracy? The intro is solid, the middle was a little slow covering some standard ideas in a standard way, but it culminated into a grand conclusion. Approaching the world with an analytic, Bayesian mind and avoiding human cognitive biases are literally game-changing ideas that I imagine the world will inevitably change to. Though focusing on more political/social predictions, the same approach could be applied throughout your life. Did you know fame and accuracy are inversely correlate? Maybe this book is best consumed through just a summary but the conclusion is grand enough that I think it deserves the deep internalization you get by reading all the supporting ideas.

Key Takeaway: "People can with considerable effort make accurate forecasts about things that matter"