Evidence-based policy making

A number of my tweets and retweets this past week have focused in different respects on the notion of evidence-based policy.

Most with any training in the sciences may well be confused by the very concept since it implies that there is, somehow, an alternative of designing policy without evidence.  So is evidence-based policy simply about wanting “a bit more science of the Royal Society in politics” as claimed in the recent BBC Newsnight segment?

Consider the view of three of those interviewed on Newsnight.  Lord Butler, former Cabinet Secretary, and now involved in the Better Government Initiative: “evidence is no substitute for political debate, but it is a good basis for it”, but he takes the view that in recent years it has gotten increasingly harder on politicians because of the media and other external pressures. Hetan Shah, executive director of the Royal Statistical Society, argues that the use of ‘evidence’ is much more challenging when dealing with ‘big structural changes’ such as an overhaul of the health care or education systems, but that it is much easier to bring in such evidence when discussing smaller interventions.  But even many ‘smaller’ subjects (including those that nominally hinge directly on scientific evidence) can be the subject of intense public controversy. Finally, Claire Fox of the Institute of Ideas disparages people for “hiding behind the evidence” and implicitly complains of technocrats underestimating the complexity of many political issues, which can’t just rely on ‘science’ or evidence to solve the problem.  Amusingly, her sister, Fiona Fox, heads up the excellent Science Media Centre, which helps engage scientists and seeks to provide scientific evidence into the policy process.

Many of the most contentious cases involve public health.  What guidance should government (or doctors) give to women about drinking during pregnancy?  What role does (or should) evidence play? For example, a recent Harvard study found that moderate consumption of alcohol in the first trimester did not have an adverse impact on outcomes. What is the importance of one such study or piece of evidence? An allegation pushing in the other direction is that alarmist rhetoric has gone so far as to encourage some women to seek abortions out of misplaced concern about the health of their fetus.  Another case is that of the PSA test for prostate cancer, which is so prone to false positives and overdiagnosis that a recent national task force in Canada argues for dispensing with it entirely because it tends to lead to more harms than benefits.

A favoured target (perhaps because it is so easy to disparage but remains stubbornly present) is homeopathy, as Ben Goldacre has written about for many years.  So who supports such as quackery or ‘witchcraft’ as the British Medical Association describes it? There is a large community of activists who fervently believe in homeopathy, led by HRH (and Trinity alum) Prince Charles.  Perhaps the best (or worst) example is the case of MMR.  As a result of a paper in the Lancet by Dr Andrew Wakefield (later retracted) and the surrounding media hype led by the Daily Mail, vaccination rates fell from 92% to under 80% and, unsurprisingly, cases of measles in particular have surged reaching 1,348 confirmed cases of measles in England and Wales in 2008, compared with 56 in 1998.  And yet, the same media outlets continue to foment controversy,as seen in the hyperbolic, faux-populist coverage of MMR in the Mail on Sunday as recently as 2012.

Another long-standing subject is smoking – one can know smoking is harmful but still not choose to ban it in public places.  A libertarian might believe the evidence to be compelling but still oppose bans and instead favour alternatives such as labeling or information provision.  In other cases, vested interests might simply adopt such language to oppose greater restrictions.  By contrast, after having banned smoking indoors in the UK and elsewhere, the next step, as suggested by the British Medical Association, is to ban sales to anyone born after 2000 in order to create a ‘tobacco-free generation’.  I won’t even begin to delve into all the recent controversies over the legalisation of cannabis or the link between enforcement and drug use

One particular focus of supporters of evidence-based policy making is that there should be more real-world experiments before policies are rolled out nationally.  Ben Goldacre along with the Behavioural Insights Team in the Cabinet Office wrote an excellent piece on Randomised Controlled Trials (RCTs),  In most areas of policy though, the use of RCTs remains scant.  Why do we not see more RCTs used?Perhaps the reluctance also stems from the fact that many/most politicians are ill at ease with statistics or the scientific method?!  One argument put forward is that on grounds of equity it is not acceptable to give a ‘preferred’ treatment to only a fraction of the selected sample.  Another deterrent is the politics of accountability and adversarial politics combined with ‘gotcha’ coverage of the populist media (i.e., fear of headlines such as ‘Government Wastes £5 MILLION on Experimental Trial and Finds Smart Meters are Ineffective’).    Using such logic, it is better to waste £5 Billion on a national roll-out rather than £5 Million (although costs for the UK rollout are actually several times higher and there are many good reasons to be sceptical of the smart meter rollout as Ross Anderson explains). Others have highlighted more technical problems with RCTs such as selection bias and external validity.

A thoughtful critique of evidence-based policy comes from Chris Dillow, who makes the case against evidence-based policy making on the grounds that:

(a) in some cases, greater knowledge may not produce better decisions, but may erroneously increase confidence in those decisions;

(b) it is inevitably conservative because there’s ‘no available evidence one way or the other about the effects of truly radical policies’ (a mirror image of the logic behind the precautionary principle — something we will hear about when Nick Ashford speaks to us in Week 7);

(c) since the public is often wrong in its assessment of the facts, as seen in a recent Ipsos Mori survey across 14 countries, then it would be necessary to override the public’s preferences in many policy areas to impose ‘evidence-based’ policy; and

(d) according to Dillow, “we do some things not because they fit a narrow cost-benefit calculus but because they symbolize who we are”.  A great example is the recent Sutton Trust report on education and what makes for great teaching argues that certain behaviour, such as lavishing praise on students, have no measurable impact on outcomes. Many have pushed back on this

So I leave you with a simple task to provoke your comments:

Choose ONE policy controversy that you are familiar with (or would like to learn about more) and describe how evidence has been used in the policy process.  You might also want to tackle such sub-questions as: Would RCTs have produced better policy? Has the notion of evidence-based policy making been used by one side in the debate to make their case more difficult to criticise? Is the reason for the controversy primarily to do with a lack of evidence, a misunderstanding of the available evidence or is the debate a proxy for deeper ideological and political disputes?