Every day experts of all stripes compete for our attention, on the television, on the radio, and in the papers. Should we tune in and update our prior expectations about the future based on their commentary? Not unless we have evidence of their track record, which we rarely do. Philip E. Tetlock, author of Expert Political Judgment: How Good Is It? How Can We Know?, summarizes the conclusions from the data he collected over 20-years:
The sanguine view is that as long as those selling expertise compete vigorously for the attention of discriminating buyers (the mass media), market mechanisms will assure quality control. Pundits who make it into newspaper opinion pages or onto television and radio must have good track records; otherwise, they would have been weeded out.
Skeptics, however, warn that the mass media dictate the voices we hear and are less interested in reasoned debate than in catering to popular prejudices. As a result, fame could be negatively, not positively, correlated with long-run accuracy.
Until recently, no one knew who is right, because no one was keeping score. But the results of a 20-year research project now suggest that the skeptics are closer to the truth…
Boom and doom pundits are the most reliable over-claimers. Between 1985 and 2005, boomsters made 10-year forecasts that exaggerated the chances of big positive changes in both financial markets (e.g., a Dow Jones Industrial Average of 36,000) and world politics (e.g., tranquility in the Middle East and dynamic growth in sub-Saharan Africa). They assigned probabilities of 65% to rosy scenarios that materialized only 15% of the time.
In the same period, doomsters performed even more poorly, exaggerating the chances of negative changes in all the same places where boomsters accentuated the positive, plus several more (I still await the impending disintegration of Canada, Nigeria, India, Indonesia, South Africa, Belgium, and Sudan). They assigned probabilities of 70% to bleak scenarios that materialized only 12% of the time…
Here, then, is a modest proposal that applies to all democracies: the marketplace of ideas works better if it is easier for citizens to see the trade-offs between accuracy and entertainment, or between accuracy and party loyalty.