Why Oh Why Can’t We Be Good Bayesians?: Austin Frakt Compares the Power of the Massachusetts RomneyCare and Oregon Medicaid Studies

Austin Frakt does the math:

If the study of the impact of RomneyCare in Massachusetts on mortality “amenable to health care” had had the power of the study of the impact of the Medicaid lottery in Oregon, the Massachusetts study would have concluded that the effects of RomneyCare on mortality were not “significant”. A whole host of commentators would then have said that the study’s failure to find “significant” effects–never mind that effects would have had to be at least five times as large as an optimist would find plausible to be “significant”–was a point that should strengthen your belief that RomneyCare was a waste of money.

This really is not rocket science, people!

Austin Frakt: Comparing the Massachusetts mortality study to the Oregon Medicaid study: “Many people are contrasting the Oregon study…

…which didn’t find statistically significant effects of coverage on biomarkers associated with physical health—with the new Massachusetts study—which found a statistically significant mortality benefit of coverage. How could these two findings coexist in a rational world?… To illustrate the difference in power of the two studies…. What if we presume that the same mortality effect that was found in the Massachusetts study applied in the Oregon case? Would the Oregon study have been able to detect it with statistical significance? The answer is no, and it’s not even remotely close. The Oregon (OR) study had a sample size about a factor of 100 below that of the Massachusetts (MA) study. That means the error bars… are huge, about 10 times bigger than those of the MA study, overlapping the origin by a country mile….

I am not suggesting it’s not worthwhile to compare the MA and OR study results. I’m just saying we should be mindful of the tremendous differences in sample, as well as statistical methods and regional contexts, of the two studies. Burn the above chart into your head. If that fails, consider a tattoo.