Wednesday, 20 June 2012

'Test, Learn, Adapt' - thoughts on the use of RCTs in social policy

Today, the Cabinet Office's 'nudge unit' has produced a new document advocating the greater use of randomised control trials (RCTs) in the policy-making process.  With 'Bad Science' writer Ben Goldacre on the team, it's received a fair bit of press, so I thought I'd review it and offer my own opinion on their key argument.

The argument
The thesis behind the paper is simple.  RCTs have been used in a range of other fields for some time - medicine, international development and business for example.  However, they have not been adapted to the policy-making process so enthusiastically.  Despite the early New Labour focus on 'what counts is what works', governments maintain the habit of ignoring scientific evidence on policy interventions.  The authors argue that this should change: if it does, we might get better policies and at a significantly cheaper cost.

Sounds too good to be true?
That's because it is.  In my view, the report rehashes old arguments about the use of 'scientific' evidence  in social policy.  The authors appear to come from a very one-sided scientific perspective: unaware of the long-established difficulties - particularly in social policy - of the interaction between science and society, politics and morality.  In fact, in their discussion of the 'myths' about RCTs, the authors focus exclusively on practical or economic objections, rather than any philosophical doubt about using scientific evidence to decide on policies.

The authors offer a deeply tempting and seductive view: that we can determine what policies we choose by scientific methods, removing age-old ideological debates about the Good Society once and for all.  Not only is it wrong to equate the objectives of social policy to other institutions, like medicine, but to my mind it seems quite undesirable.  Sure, we can use scientific methods like RCTs to choose between different policy interventions.  But first we have to decide what type of interventions we want in the first place.  And that requires debates and deliberations that science simply cannot answer.


  1. RCTs can't answer the big questions in medicine either. I like the way that Wouter Bos gets this across, even though he isn't talking about evidence, just values.

    Thanks for your blog.

  2. Thanks for your comment Anne Marie. And apologies for making the assumption that RCTs are more easily adaptable to medicine; the limitations in this respect are not something I've read into. Thanks for the link to the Ted film, I'll make sure I watch it.

  3. It would be useful to apply RCT methodology to the "work for your benefits" schemes currently operated by the government. Would a Labour government adopt this methodology? I doubt it. Why? Because the "work for your benefits" schemes were designed by... er... the last Labour government!