Randomised trials for policy: a review of the external validity of treatment effects

Abstract

The paper provides a first survey of the literature on external validity. The starting point for this are debates regarding the use of randomised evaluations to inform policy. Besides synthesising contributions to the programme evaluation literature we consider definitions of external validity from other subdisciplines within economics, such as experimental economics and the time-series forecasting literature, as well as the disciplines of philosophy and medicine. We suggest - following Cook and Campbell (1979) - that the fundamental challenge arises from interactive functional forms. This somewhat neglected point provides a framework in which to understand how and why extrapolation may fail. In particular it suggests that replication cannot resolve the external validity problem unless informed by some prior theoretical understanding of the causal relationship of interest. Finally, we argue that the assumptions required for simple external validity are conceptually equivalent to those required for obtaining unbiased estimates of treatment effects using non-experimental methods, undermining the idea that internal validity needs be rigorously assessed whereas external validity can be ascertained subjectively. Theory may play a role in aiding extrapolation, but the extent to which this will be possible in practice remains an open question.