Evaluation of service transformation needs more money, staff and focus

The response of the healthcare system in England to the covid-19 pandemic has been impressive, with services innovating and adapting during the first wave at unprecedented pace and scale. Simultaneously, the National Institute for Health Research co-ordinated a national research programme that has been hugely successful.

But after the first wave started to recede, key questions emerged about the safety, unintended consequences, inequalities impact and value for money of this massive pivot for health and care services: specifically, which innovations were to be retained and which ditched?

The reality check at that stage was that no one really knew the answers to those questions. Funding and resourcing for service evaluation had not been factored into the planning of the NHS changes, nor had significant additional central funding been set aside for applied health research.

And whilst the service and planners might be forgiven for not having this foresight in the midst of a crisis, it highlighted a long-standing deficiency in the system: that evaluation of service transformation has never attracted funding and resourcing of the kind allocated to discovery science nor has it been implicit in the planning of service change.

To gain further insight on these issues and potential solutions, UCLPartners, an academic health science partnership, has led a piece of work as part of the AHSN Network Health and Care Reset campaign to explore how the system might prioritise and resource rapid service evaluations to enable more efficient and effective scale-up of health and care innovations.

Our colleagues at the London School of Hygiene & Tropical Medicine gathered evidence from a series of interviews with 18 key stakeholders. We then discussed these findings in a roundtable in December 2020 with 12 leaders from NHS bodies, NIHR national and local organisations, National Institute for Health and Care Excellence and third sector representatives.

Some outstanding examples of individual and organisational leadership in response to the need for rapid evaluation at the regional level were uncovered – Health Innovation Manchester, for example, led work across their local system to collectively define trials and diagnostics to evaluate and respond to national research priorities. But during the first wave there was no pre-existing national system to ensure understanding of the effectiveness of changes.

The Beneficial Changes Network – an NHS England/Improvement initiative to identify positive changes introduced during covid-19 – has helped address this; however, the network has been set up as a standalone initiative and runs on minimal funding so big questions remain about how service evaluation is funded, coordinated and delivered in the longer term and indeed who should undertake such evaluations.

Our conclusions, drawn from insights from our discussions and published in an AHSN Network white paper Rapid evaluation of health and care services – planning for a sustainable solution for the post-covid reset, point to the need for changes in how rapid service evaluation is funded, co-ordinated and delivered, not just during but beyond the pandemic.

Previous work shows that many evaluations of health services are poorly designed, fail to define clear research questions, and do not achieve their desired outcomes…

First, there should be a national policy to promote evaluation of all significant service changes, with an appropriate funding allocation to support evaluation. This requires more clarity on expectations of different funded entities regarding balance of research and evaluation.

For example, our conversations with stakeholders showed that the NHS was not viewed as an entity that systematically funds rapid service evaluations or training of staff to conduct them, with many seeing the NIHR as holding that function. However, most NIHR-funded organisations are geared towards longer-term research, and reward academic publication over rapid results.

In terms of co-ordination, while a range of organisations carry out service evaluations, including universities, consultancies, think tanks, trusts, Public Health England and others, there is often room for more collaboration among them. We recommend the creation of a more formal system at both national and regional level for ongoing dialogue between the NHS and those tasked with evaluation to identify priority needs for service evaluation.

The products of national and regional evaluations should then be collated with a single point of access to facilitate future commissioning and service design.

Finally, in terms of delivery, there is a need to bolster capacity and capability through increased staff training and deployment. Previous work shows that many evaluations of health services are poorly designed, fail to define clear research questions, and do not achieve their desired outcomes.

This is partly because the priorities of researchers and commissioners are often misaligned, particularly as regards what questions can be answered in short timescales and how likely work is to be publishable.

Now more than ever, there is a pressing need to quickly understand the benefits and potential harms of service changes. The covid-19 pandemic has shone a light on long-standing issues in the way that evaluations are funded, co-ordinated and delivered, and we must respond by taking a new approach to systematically understand patient benefit and value for money of our health and care system.

This blog was originally published in the Health Service Journal