Must-Read: Nancy Cartwright and Angus Deaton: The Limitations of Randomised Controlled Trials

Must-Read: Smart thoughts from Nancy Cartwright and Angus Deaton:

Nancy Cartwright and Angus Deaton: The Limitations of Randomised Controlled Trials: “A well-conducted RCT can yield a credible estimate of an ATE in one specific population, namely the ‘study population’…

…Sometimes this is enough…. Yet the study population is often not the population that we are interested in…. More generally, demonstrating that a treatment works in one situation is exceedingly weak evidence that it will work in the same way elsewhere; this is the ‘transportation’ problem: what does it take to allow us to use the results in new contexts, whether policy contexts or in the development of theory? It can only be addressed by using previous knowledge and understanding, i.e. by interpreting the RCT within some structure, the structure that, somewhat paradoxically, the RCT gets its credibility from refusing to use. If we want to go from an RCT to policy, we need to build a bridge from the RCT to the policy. No matter how rigorous or careful the RCT, if the bridge is built by a hand-waving simile that the policy context is somehow similar to the experimental context, the rigor in the trial does nothing to support a policy; in any chain of evidence, it is the weakest link that determines the overall strength of the claim, not the strongest….

We have a better chance of transporting results if we recognise the issue when designing the experiment–which itself requires the commitment to some kind of structure–and try to investigate the effects of the factors that are likely to vary elsewhere. Without a structure, without an understanding of why the effects work, we not only cannot transport, but we cannot begin to do welfare economics; just because an intervention works, and because the investigator thinks the intervention makes people better off, is no guarantee that it actually does so. Without knowing why things happen and why people do things, we run the risk of worthless casual (‘fairy story’) causal theorising, and we have given up on one of the central tasks of economics.