Test, Learn, Adapt:Developing Public Policy with Randomised Controlled Trials(by Laura Haynes, Owain Service, Ben Goldacre & David Torgerson)It is a good report and I commend it to you. As the introduction to this report states:
Randomised controlled trials (RCTs) are the best way of determining whether a policy is working. They are now used extensively in international development, medicine, and business to identify which policy, drug or sales method is most effective. They are also at the heart of the Behavioural Insights Teamʼs methodology. However, RCTs are not routinely used to test the effectiveness of public policy interventions in the UK. We think that they should be.I think they should be too. I did a degree in psychology and as many people know, a huge focus of this degree is on scientific method. So when I arrived in my first 'proper' job in a local housing department, I was genuinely shocked that there seemed (to me) to be an almost complete absence of established methods or knowledge of best practice from around the country and beyond. I still have those concerns.
I have always been puzzled as to why politicians and senior public servants seem so reticent at using science to determine whether a new policy works or not? At a cynical level, I wonder if it is about ego, pomposity, fear of being found to be less than effective, worry about how the media will see such an experiment or even a knowledge that while the scientific understanding of many (most?) politicians is very low, it is even lower amongst most journalists... I don't know and I am cautious about speculating without some proper research. It is also the case that there are many academics in the social science field who are very uncomfortable with applying the rigour of RCTs to many socio-political interventions. I respect their positions, but I would hope that a middle ground can be found that means we get just a little bit more 'hard' science into the policies devised by politicians and senior public servants.
Which brings me round to highlighting the decision yesterday that nobody can be compelled (under current regulations) to participate in working 'for free' for companies while they are on benefits. The story can be understood here. For me the most telling comment came from one the people involved who said that her placement in Poundland was a "complete waste of time" and "the experience did not help" her get a job. From my limited knowledge of the case, it would seem that the regulations were designed in such a way that meant the court had no option but to award the case in favour of people who had been forced to take on a placement or face losing their benefits.
So here is an example of where an RCT could easily have been used to test and refine those regulations. If the intention by DWP was to assist participants in gaining new skills and confidence whereby they were more able to get a job (rather than merely massaging unemployment figures or punishing those on benefits or pandering to the Daily Mail etc..) then this could easily have been done through a randomised controlled trial. Had that been done, I would put money on there being no such court case decisions like the one from yesterday.
Meanwhile plans to roll out Payment by Results and privatisation of the probation services continues apace even when the pilots (by no means RCTs but an attempt to validate the policy) have been abandoned.
May I remind this Government that one of the people that Margaret Thatcher admired greatly was Karl Popper who was a strong advocate of good scientific method. To quote Wikipedia, Karl Popper was "known for his attempt to repudiate the classical observationalist / inductivist form of scientific method in favour of empirical falsification". Mrs Thatcher herself was a chemist and understood the importance of science.
So please can we have a little less populist policy making from this government and a bit more use of good science to determine whether various policies actually work or not!
Jon, My initial response to this is that there is no point in conducting RCT's in a system where you only have 5 years to make the changes you believe are right. By the time you have conducted your RCT and analysed the result it will be time for an election and possibly a change of government.ReplyDelete
Therefore in the time allowed changes have to be implemented without analysing the results of trials to ensure that you appease the people who voted you in so that you may get a second term.
A good example of not waiting for pilots to finish before implementing root and branch change can be found here: http://www.telegraph.co.uk/news/politics/9624955/We-will-push-ahead-with-prisoner-rehabilitation-plans-before-pilot-is-finished-Grayling-says.html
Perhaps you are correct Steve and I am whistling in the political wind. However, I do not think RCTs need to take that long: it is mostly about randomising the subjects between intervention and control group. That is not hard to do. Yes it takes a bit more thought & design but given that we are spending millions of public money - would that not be a good idea?Delete
Thanks for your input to the debateDelete
I actually think it is a good idea but do not believe that the political will is there. I'd much rather see RCT's as a standard part of the process so that when new legislation is enacted it has the desired impact. It requires a change of mindset from those in government and sadly there are too many vested interests at stake to enable that to take place.ReplyDelete