Skip to content

Getting to grips with replication

8 March 2013

At the end of last month, BIG convened a group of forward looking trusts and foundations for the first of three seminars on ‘Getting to Grips with Replication’. Alongside Geoff Mulgan of NESTA and Stuart Thomason from the Young Foundation, Tim Hobbs spoke to the group about his experiences of replication working at the Social Research Unit, Dartington. His guest blog reflects upon just a few of the key themes emerging from the rich discussion…

TIm Hobbs

Tim Hobbs

Not everything needs a robust evidence-base before you replicate it

There are some social interventions or services that you don’t need a robust evidence-base for prior to replication. These include, for example, safe and inviting playgrounds for young children in urban environments or schemes to get food, clothing or warmth to those desperately in need.

But there are many social interventions where robust evidence of impact is arguably a pre-requisite of replication and scale. The last thing anyone wants to do is propagate ineffective or harmful interventions.

Let’s face it, whilst almost all social innovations and services are well-intentioned, the efficacy of most is unknown despite the protestations that “we just know it works”.

But too few services or interventions are well-evidenced and ready for replication

Even in the relatively well-developed world of children’s services, there is not too much for a funder or commissioner to choose from and replicate that is well-evidenced. Of the 1,100 interventions reviewed by Blueprints for Healthy Youth Development the leading database of evidence-based programmes in the US –  only 44 meet the Social Research Unit’s high Standards of Evidence.

Of the 240 applicants to the Big Lottery Fund’s Realising Ambition programme, just 10 made the highest grade (for the remainder of the funded projects the evidence was slightly below this level). In both cases, this equates to just 4%. What’s more, this 4% figure relates to streams of work explicitly reaching out for well-evidenced interventions. The ‘real’ proportion of services with a robust evidence-base is certainly less than 1%.

What could funders to do improve the Innovation to Impact Pipeline?

There is a lot that a more coordinated effort from trusts and foundations could achieve in terms of strengthening what we at the Social Research Unit refer to as the Innovation to Proven Impact Pipeline. Here are some ideas discussed at the seminar:

  • Map the sectors they invest in, plot the degree to which their individual and collective investments are innovative or evidence-based and the scale to which such services are implemented
  • Coordinate – across Trusts and Foundations – where investments are made across the Innovation to Impact Pipeline, contributing to a more balanced spread of investment across the pipeline
  • Invest in the tools and support structures that organisations require in order to refine interventions and ensure they are implemented and replicated with fidelity
  • Invest less in light touch evaluation of social interventions that tell you little about impact, and more in robust experimental evaluation to help build the evidence base across the UK, plus smart data dashboards allowing real-time monitoring and evaluation allowing more reflective adaptation and testing of interventions.

The time has never been better for greater coordination between Trusts and Foundations. I hope this seminar series – and the conversations and actions that result from them – lead to a more functional innovation to proven impact pipeline.

Tim Hobbs is Post-doctoral Researcher, Social Research Centre at Dartington

What do you think of Tim’s guest blog? Join the discussion on Twitter using #BIGreplication or leave your comment below.

The second of three ‘Getting to Grips with Replication’ seminars will be held on Wednesday 13 March, and and the third on Thursday 21 March. There are a handful of spaces left – email if you would like to attend.

3 Comments leave one →
  1. 8 March 2013 7:56 pm

    Evidence2success in Perth and Kinross has had a robust impact but probably will not now lead to replication across Scotland, following widespread outrage at the way it was implemented in schools during teaching time and without explicit parental consent. Real time monitoring and evaluation of lessons to be learnt a must before this programme is ready to hit the proven pipeline!

    • Big Lottery Fund permalink*
      11 March 2013 11:33 am

      Thanks for your comment on the blog. Yes, how ‘robust’ is ‘robust enough’ evidence? To what degree do you replicate an intervention with 100% fidelity, or to put it another way how far should you be prepared to tweak an intervention to ensure it fits with new contexts or new beneficiaries?

      And absolutely, we need to find better ways of doing real-time monitoring and evaluation to enable us to effectively make those adjustments and understand the impact they have on outcomes.


  2. 12 March 2013 8:57 am

    In the replication of Evidence2Success, a fundamental difference between the US survey model and Scottish replication is that US parents were permitted access to the sensitive questions relating to sexual activity and drugs and were required to provide explicit opt in consent. In Scotland some parents (not all) were posted a general topic flyer without any reference to the sensitive question sets and were only given an opt out consent option.Real time monitoring and evaluation of sample size acheived in US vs Scotland would clarify the real world impacts of opt in vs opt out consent on sample size and provide effective learning for future research proposals of this type, particularly where children are the core participants and the data “harvested” from the children is of a sensitive nature.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: