I think it is time for me to issue an unconditional intellectual surrender to Cosma Shalizi. Watching Nate Silver and his now http://fivethirtyeight.com over the past two election cycles has convinced me that the Bayesian framework he throws around his model is a major obstacle to people’s understanding what is going on.
What is going on is made up of three things:
- Polling–that is, asking people what they think of the election candidates in a structured way.
-
Aggregation–so that you are not just using one sample of 1000 to assess the current mood of the electorate but instead have something like 1/5 of the sampling standard error.
-
Smoothing–imposing structure on the time series, both that it ought to be close to “fundamentals” and that it ought not to change too quickly.
But next to nobody reading Nate Silver and company’s “nowcast”, “polls-only”, and “polls-plus” forecast probabilities as they evolve overtime gets any sense of how the sausage is made.
It remains the case that the decision theorist in the subbasement dungeon of my brain whimpers that Bayesian posterior probabilities are what we ultimately want.
But, these days, when it says that, I gag and shorten its chain:
- I point out to it that what we really want as decision theorists are not Bayesian posterior probabilities but rather the misnamed “risk neutral probabilities” that are posterior odds times the utility of the outcome.
-
I point out to it that if we are betting against other minds we need to know in what ways their information sets might be superior to ours and what disadvantage that puts us at: that invulnerability to a Dutch Book is a third-order consideration in a world in which others might will know of jacks of spades that will piss in your ear on command.
-
And I point out to it that the answer to the frequentest question, “how different might our conclusions have been had we drawn a different sample?” provides much more insight into whether our procedures are converging to something sensible than any ex-ante Bayesian proof that we knew in advance, before we start the analysis, that our procedures must converge.
So go visit Sam Wang: polls, aggregation, soothing, plus not unreasonable random drift strike zones are more helpful than three different sets of posterior odds–given my suspicion that there is right now no action from the 538.com stuff on the truck side of the polls plus odds…