Evidence: do or be damned?

UCLPartners co-founded the NHS Innovation Accelerator with NHS England and now hosts the programme. Evidence: do or be damned is part of a series produced by the NIA to give an insight into how it works.

In this second instalment of the NHS Innovation Accelerator (NIA) series, sharing research and real-world insight, Dr Amanda Begley, lead for the programme, considers evidence in decision-making, including how innovators should develop and use evidence for effective scaling.

For an innovation to reach its full potential impact, clinicians, providers, commissioners and patients must decide to adopt it – changing behaviour from the status quo and potentially stopping previous practice. Persuading decision makers to take a risk on a new innovation, and simply to make the time to consider it within the whirlwind of their busy day jobs, is a challenge.

So, what’s needed to help someone decide to adopt an innovation? The literature on scaling agrees that evidence is critical (albeit not sufficient on its own). Taking a risk on something new requires data that reassures and convinces. But the role of evidence to support the scaling of an innovation is not a simple one. For every RCT that has created buy-in, there are examples of personal patient stories unlocking commitment, as well as hard data that has failed to mobilise any change. Innovators often plough a huge amount of resource into evaluation and evidence production, but it does not always translate into meaningful change.

In this challenging context, NIA Fellows are working hard to scale their innovations. And they’re having great success. Since the NIA launched in July 2015, 469 additional NHS commissioners and providers have adopted NIA innovations.

Innovations that NIA Fellows are scaling are diverse. Some are technical devices that improve existing practice, like PneuX Pneumonia Prevention System. Others offer care that was not previously available, like Sleepio’s digital sleep improvement programme. Depending on the nature of the innovation, Fellows face different ‘evidence thresholds’ from the key decision makers they work with. The more an innovation differs from existing practice, the more complicated its theory of change, the more components it encompasses and the more invasive; the greater the volume of evidence that is typically required by decision makers to reassure them that adopting the innovation does not entail unwarranted risk.

Despite this variation, there is much in common as to what Fellows are learning about how to develop and use evidence effectively.

Fellows’ experiences offer three key insights:

Start with the decision makers and their context

Evidence is a tool that helps people decide to invest public money and/or time in adopting an innovation. Every decision requires the weighing up of significant opportunity costs when resources (time and money) are extremely limited and demands are high.

When considered in this light, it’s clear that the way innovators develop and deploy evidence must be grounded in the context within which decision makers for their innovation operate. What is the greatest problem the decision maker is trying to solve? What is their local context, and how does this relate to national issues? What do they need to know? What might their concerns be? What level of risk will they be carrying?

Commissioner Rachel Anthwal calls for greater understanding of the context in which they commission services when it comes to medical research:

“I would… implore researchers to come and talk to us, to understand our pressures, our areas of worry, and help us build research evidence that answers our commissioning questions, with us.”^[1]

With these questions in mind, an innovator can deploy the most relevant information from their evidence toolkit to respond to the decision maker’s most important questions and priorities.

Maryanne Mariyaselvam of the Non Injectable Arterial Connector (NIC), for example, recognised that evidence for the NIC did not have currency with people who did not believe they had a problem to solve. She has therefore focused efforts on using data to show potential adopters the frequency and extent of harm caused by wrong-route drug administration.

Fellows have worked with NICE to develop Medtech Innovation Briefings (MIBs) for their innovations, including AliveCor’s Kardia, Episcissors-60, Pneux and NIC. These engage decision makers because a review by NICE – as a trusted, independent source – reduces the feeling of risk associated with investment and use locally.

When presented with an innovation, decision makers will be able to envisage with ease the multiple, complex and deep rooted barriers in the adoption context, often asking for more or different types of evidence as their immediate request. Although this offers a culturally acceptable response, determining ‘proof’ is only one factor that potential adopters need to consider and may not be the most critical one for your innovation. Therefore, evidence must be seen as a tool to be deployed alongside other scaling activities – including prototyping, which allows people to test an innovation locally on a very small scale, helping the decision maker to manage risk.

Starting with the decision maker and their context to understand specific needs and challenges can both help identify what evidence will be persuasive – and where evidence alone will never be persuasive enough.

Evidence should be as compelling as it is rigorous

Often we talk of the need for quality evidence, perhaps marked by an ‘independent’ evaluation, giving a stamp of approval. But the quality of evidence for mobilising action comes through how compelling the evidence is, as well as how rigorous.

For example, Mark Campbell, Associate Director at NICE, in a recent presentation to NIA Fellows, talked about the ‘wow’ factor, whereby adopters may consider lowering the evidence threshold when the theory of change and narrative around an innovation’s impact are highly persuasive.

The power of stories and experiential evidence should not be underestimated when communicating the benefits of an innovation – this kind of evidence can mobilise people’s energy, enthusiasm and frustration into action.

Take the work of Paul Jennings, scaling a new model of supporting high intensity users of police and NHS services in mental health crisis, called High Intensity Network (previously Serenity Integrated Mentoring (SIM)). He has developed a panoply of materials that demonstrate the impact of his new model, including a one page summary with key headlines, a video with personal testimonies of adopters and service users, and a detailed evaluation report of an adoption site. As a police sergeant, Paul also recognises the need to communicate with NHS decisions makers through different ‘messengers’, working closely with a senior nurse who has joined High Intensity Network, funded through Wessex AHSN. These materials and channels enable both rapid and in depth communication of the innovation, the personal impact the work has, and its national potential.

Furthermore, it’s not just what evidence you share, or even how you share it – the tone you strike is also critical. The Fellows have found that evidence is rarely a finished product – it tells part of a story of growing understanding and experience that is not yet finished. Being open, invitational and humble in your use of evidence, rather than being insistent you hold all the answers, welcomes people to come with you on the journey.

Creating a great evidence base for an innovation is a collaborative process

Often, the responsibility for producing evidence that proves an innovation’s impact is given to innovators. In fact, collecting evidence must be a shared endeavour between adopters and those scaling an innovation – both can benefit from being able to articulate the work and its impact.

Rupert Dunbar-Rees of Outcomes Based Healthcare (OBH) has worked with one adopter to co-produce co-branded materials. Using the OBH platform, the adopter has developed a new approach to patient population segmentation and outcomes based commissioning. The materials showcase insights and learning from their collective experience. This demonstrates not only the capabilities and potential impact of the OBH platform, but also the pioneering thinking of its early adopters.

Finally, if evidence must be personal and compelling, an innovator’s best intel is those that have already adopted it. Working with them to consider retrospectively what their biggest challenges were, what they would have found most persuasive about what they now know the innovation has to offer, and why they would recommend the innovation to others, helps innovators to understand what the most powerful messages are for future potential adopters.

It’s clear that evidence plays a critical role in scaling innovation, and that evidence alone is not enough – how evidence is gathered, presented and communicated is key. Given that producing evidence can be time and resource intensive, it’s important that an evidence base is developed intelligently with purpose, person and context in mind: what action do we want the evidence to inspire, and from whom?

Remember that the country’s 15 Academic Health Science Networks can provide specialist advice on the role of evidence, your adopters and their context, and how to create a compelling case for innovation.

We’ll be sharing more insights from the NIA soon – next time focusing on the role of the patient in innovation in the NHS.

This is the second blog in the NIA series developed with Innovation Unit. Read the first blog about matching supply and demand for innovation in the NHS here.

Join the conversation on Twitter @NHSAccelerator using the hashtag #innovationspread

Dr Amanda Begley

^[1] Rachel Anthwal, ‘Evidence-based Policy-Making: The View From A Commissioner’, http://www.dc.nihr.ac.uk/blogs/evidence-based-policy-making-the-view-from-a-commissioner/5934