Evidence in policymaking

Edit (30th April 2013):  As has been pointed out in the comments, it doesn't look like Ben Goldacre's tweet I mention in the highlighted section below was actually aimed at Jeremy Hardie.  I've left it there, though, partly out of transparency but also because the debate between them is still what sets the context for this post, though obviously there's also plenty of others contributing to this too (notably, for me, the British Educational Research Association - BERA).

I’ve been noticing quite a few discussions of evidence and policymaking recently.  There’s been a bit of a spat between Ben Goldacre and Jeremy Hardie, after the publication of his book Evidence-based policy: a practical guide to doing it better co-authored with Nancy Cartwright.  At least, I assume that’s who Goldacre was getting at when he tweeted:

It is very sad - and odd - how some in the policy community hear "do more RCTs" and pretend they've heard "do only RCTs"

This is presumably in response to Hardie’s IPPR blog post, which itself was a response to Goldacre’s Building Evidence into Education report.  (And around we go…)

Also, this week has seen the publication of a study suggesting that the alcohol industry’s evidence to the Scottish Government was misleading.  This led to the inevitable exchanges on Twitter between Mark Baird and John Holmes, amongst others.  I’m going to write about that specific issue separately, but the point here is that there’s much to be discussed about evidence and policy.

What I want to do here is offer a model for understanding policymaking that, I hope, helps shed some light on – and cut through – some of these discussions.

I know that there are plenty of models of policymaking already available in social policy textbooks, and I’m sure I’m reinventing the wheel at some level.  However, I want to be clear about what I’m trying to do with the model.  This isn’t about modelling how policy is made in practice and the variety of influences on it (though there are plenty of models for this).

What I’m trying to do instead is think in a slightly abstract way about how policy is made: what thinking or ideas underlie policy?  That doesn’t mean that this thinking is overt, or that policymakers themselves are always conscious of it, but (even if it’s only implicitly) positions are taken on each of the following aspects.  I would suggest this model is helpful for looking at existing policies, or policy debates, in order to better understand some of the currents and influences that are underlying them.

I hope that you’ll be able to see how it’s helpful to identify (sometimes hypothetical) steps in policymaking, conceived of as a problem-solving process, from my next post, which will be about the role of the alcohol industry in alcohol policymaking.

It also helps shed light on my earlier post about alcohol and neoliberalism.  If we can see that a ‘problem’ and ‘solution’ have been defined in particular ways, we can start to think whether there is a broader approach to the idea of government and policy that is affecting how people approach certain issues.

However, I’m keen to stress that this is just some thoughts about how to think about policymaking that have come to me recently; this post shouldn’t be taken as an attempt to produce a definitive model of policymaking.  I’m sure people will be able to spot gaps in the argument, or perhaps misunderstandings of certain policy areas.  If you do, please let me know, as it will help my way of thinking about policy.  Certainly, there could be more or fewer steps to the model.

1. What is the problem?

‘Problem’ is a bit too specific a word here, really, but any policy has several aims, which might be understood as attempts to solve certain problems.  One of the things that fascinates and frustrates me about alcohol policy is that there’s any number of aims – and that’s crucial for all the stages that follow.  However, whether it's stated clearly or not, there will always be some perceived ‘problem’ underlying a policy.

2. What works?

This is seen as a key issue in current politics.  It’s what’s behind the Goldacre-Hardie spat and, having been a key part of Tony Blair’s rebranding of the Labour Party, this precise phrase is now the title of new independent policy advice centres set up by the Coalition Government.

How to judge ‘what works’ is what Goldacre and Cartwright & Hardie discuss, and with more detail and insight than I can here – or could even if I had more time and space.  The point I want to stress here is that this is not a neutral, value-free question, because it presupposes an answer to the previous one: what are we trying to do with the policy in question?  You can only judge whether something ‘works’ if you know what its aim is.  A computer doesn’t ‘work’ very well for cooking a roast dinner, but it might ‘work’ for looking up advice on how to do this.

There are two issues with this.  First, as I’ve already stated, there’s very often multiple aims of policies – and sometimes these can even be in conflict to some extent.  A drug treatment policy might want to increase the number of individuals who are free of any drug dependence, but you wouldn’t want to do this at any expense.  There’s a possibility that the same policy that successfully pushes a lot of people through the system successfully leads to others feeling alienated, and they therefore go on to disengage and commit more crime, suffer more health harms and so forth.  That is, you can’t judge the policy on one aim, or on one metric.

This brings me to the second point.  The measurement of ‘what works’ depends on choosing some way of measuring success and failure – ‘operationalising’ your concepts, as I was taught.  This is where there’s often disagreement about education policy: do rises in GCSE pass rates indicate a successful policy, or is this metric inadequate?

Ben Goldacre, in his paper on education and research, suggests that most of the time this operationalising isn’t an issue, as we should be able to find decent metrics that we can agree on.  To a certain extent this is true – we could perhaps develop better metrics than GCSEs or SATs, or we could contextualise them to understand what they ‘really’ tell us about the standards of education in the country.

However, this only brings me back (again) to question 1: what are we trying to do with education policy?  When people disagree about standards slipping, this isn’t simply because they don’t have faith in the exam system; they’re also often expressing an underlying difference in opinion about what examination is for and, more generally, what education itself is for.  It is impossible to agree on a metric to measure the quality of education if we can’t agree what we’re trying to do by teaching someone history, for example – and there’s any number of things we might be trying to do, perhaps all in combination (knowledge of dates, an understanding of the human condition, an understanding of the evolution of the British institutions of government, an ability to think analytically and assess claims of evidence or causality, etc etc).

I realise that this makes the idea of this sort of model questionable: if we can’t have a single measure of ‘what works’, then what is the help of this concept in a model like this?  My point is that by looking at a policy and thinking through what the phrase ‘what works’ might mean, we can begin to see the assumptions that underpin the policy area.  If we were to analyse the new curriculum, or the 2003 Licensing Act, or the recent Higher Education reforms, we’d find implicit answers to these questions (even if they were only: aim to save money, does it work measured by whether or not the deficit is technically reduced).  These help us understand what’s actually going on in policymaking, and I’d hope give us some idea about the process could be better.

3. How does it work?

This is really where Cartwright and Hardie attack Goldacre for his emphasis on RCTs (randomised control trials).  They say that RCTs can’t tell you whether something will work in a different context.  This is part of the more general point that RCTs can’t tell you how something works – that is, what the precise features are that deliver outcomes.  I don’t want to enter into this discussion now, but this is a key stage in the process, because it adds to the next stage…

4. A model for intervention

This is something that actually comes out as the ‘solution’ to the problem, and depends upon a belief that something ‘works’, with assumptions about how it works.  I’m thinking really of something like NICE guidelines for the medical profession:

• There’s a problem of heroin addiction, and we want to make sure individuals are no longer physically dependent on using heroin or other opiates.
• We’ve done studies of replacement therapies and found that in general prescribing methadone seems to ‘work’ towards this aim reasonably well.
• This is because it satisfies the body’s craving for opiates/opioids.
• Therefore, here are some recommendations for how to deliver this initiative (NICE guideline TA114)

However, in reality you might find that the guidelines aren’t actually that specific or tailored, and leave plenty of room for manoeuvre.  Certainly this is the case with the guidelines for the specific example I’ve just given.  This means that in practice we often have a further stage before there are actually interventions affecting people’s lives…

5. Application of guidelines by experts

In the methadone example, here we move to treatment staff actually putting a package of care in place for an individual service user, which will (we might hope) be based on NICE guidelines, but which we wouldn’t be able to exactly predict because it also depends on the application of context and clinical judgement.  In education policy, we might think of the national curriculum being the ‘policy’, and then the application of this being actual lessons.*

There is then something of a cycle taking us back to the beginning.  Doctors, in Goldacre’s example, implement guidelines, but on the basis of intuition, or performance data, identify new or continuing problems, which the NIHR can then investigate, taking us back to step 1 of the process.  Alternatively this might be done by some kind of policy review.

Thinking through these steps, I’m not sure that there’s always a 5, where guidelines/regulations are implemented by ‘experts’.  In terms of MUP, for example, I can only really think that this would imply some kind of inspection regime to ensure that prices in shops were indeed at least at the threshold.

Conclusion

I’ll finish this post in the same way I started it: by stressing that these are only some basic reflections about how implementing policy entails certain conclusions to have been reached.  I’d really welcome responses that critique this model, or potentially refine it, as something like it is the way I approach public policy discussions at the moment, and any improvement on my thinking would be welcome.

* This is where Hardie’s citing of Hayek’s Use of Knowledge in Society comes from.  (I didn’t really need to mention this, but otherwise there wasn’t going to be a footnote in this post, and that just seems a missed opportunity when all the others have them.)