Must-read: John Holbo: It is difficult to get a man to intuit p-values when his h-index depends upon his not intuiting them

Must-Read: John Holbo: It is difficult to get a man to intuit p-values when his h-index depends upon his not intuiting them: “What p-value < .05 basically comes to…

…1) If this coin is fair, odds are less than 1 in 20 that you could match or beat that 5-heads run I just got!…. Now, to go with, an informal gloss on what your average scientific paper reports/asserts. No such thing as the prestigious science journal Fluke, so when a striking regularity of coin flips presents itself… scientific papers say: 2) Probably this is a trick coin!… Now we can trade in the rather confusing question—‘how does that p-value < .05 thing relate to the substantive take-away we really care about?’— with a less confusing question. What’s the relation between 1 and 2?…

5-heads in a row is evidence your coin is trick, or not, depending on background conditions. It could be weak evidence – so weak as to be none – or actually quite strong. Let’s talk through it. We are immediately inclined to say it’s weak evidence because we assume we are talking about our world, or one like it, in which trick coins are… waaaaaaaaay more unlikely than plain old flipping 5 heads. Ergo a 5-head run is vastly more likely to have been a fluke.

But, obviously, if the world is different things change. Suppose you are running to the bank with your brimming mason jar of quarters, and you collide with Mysterioso the Mysterious, carrying his equally large, equally full jar of trick quarters…. Oh no! The coins are mixed up! What to do? Flipping each 5 times is a decent method….

To review: we’re on the street, coins everywhere, magician swearing, jars rolling. From an even mix of fair and trick coins (per above) you pick a coin (any coin!) and flip – 5-heads. What to conclude? There is a 1 in 32 likelihood that this happened just by (longshot) chance. That is, given 5-heads, there is a 1-in-32 chance that you happen to have picked a fair coin (as likely as the alternative); then (flukily) you flipped 5 heads with it. On the other hand, there is a 31 out of 32 likelihood that… you picked a trick coin….

So if you want to explain to someone why their ‘likelihood that this thing happened just by chance’ intuition about p-values is wrong, flip it and tell them what they are thinking could be right, but only if they just collided with Mysterioso, as it were. So you gotta ask yourself: do you have reason to believe you just collided with Mysterioso? (Well do ya? Punk!?) OK, I promised intuitive….

Informally, a ‘collision with Mysterioso’ case can be glossed as: 1) The alternatives are each equally likely. (Fair coins roughly = trick in number, on the ground.) 2) The alternatives are each pretty likely. (If there are 20 different kinds of differently-behaved trick coins, scattered in equal numbers, flipping one 5 times can’t give you confidence as to which kind you’ve got.) 3) The alternatives are each quite different. (If trick behavior is subtle, 5 flips won’t cut it.) The world does present you, from time to time, with situations you can reasonably believe meet conditions 1-3. In any such case, misusing 1) as a reverse mirror, to say what is true if 2) will not be wildly off. But be aware this is a heuristic way to live the life of the mind. Very sketchy.

Let’s illustrate with a realistic case where 1-3 don’t hold, but people are in fact likely to reason, wrongly, as if they do. I tell you formula XYZ was administered to 5 cancer patients and they all recovered soon after. Would you say formula XYZ sounds likely to be an effective cancer treatment? Many would say yes. But now I add that formula XYZ is water and everyone immediately sees the problem. They were assuming it was independently even-odds XYZ was curative, or not. But it’s obviously not. A cure for cancer is like a trick coin. You don’t find one everyday. They’re 1 in 10 million. But if you are reasoning as if you just collided with Mysterioso, you may trick yourself into thinking maybe you just cured cancer. Intuitive?