Evaluation of new digital health innovations

Andy Scally, Bradford Evaluation and Trials Unit (BETU)

DHEZ festival, June 2016

3 May 2023 Evaluation of new digital health innovations1

Triple Aim

3 May 2023 Research and Evaluation in Healthcare2

Population Health

Patient experience Reduce cost

Adapted from Institute for Healthcare Improvement (USA)

An exercise in balance.

Stages of development• For pharmaceutical products

– Discovery – many compounds investigated– Development – pharmacokinetics, mode of delivery, manufacture, testing (in vitro, in vivo)– Phase I Trials – safety in humans– Phase II Trials – dose finding and safety– Phase III Trials – efficacy/effectiveness and safety– Phase IV Trials – effectiveness and safety post-marketing

• Average time from preclinical testing to approval is approximately 12 years and less than 10% of products make it to the end of this journey• For other (e.g. digital) interventions/products – similar number of phases, differing in precise characteristics

– but SME’s can’t afford this failure rate and can’t wait 12 years!

3 May 2023 Research and Evaluation in Healthcare3

What input can BETU offer?• You (small, medium and large scale enterprises) have the

ideas regarding innovation, application and the technical expertise for initial development

• BETU can offer advice and input to the development of all stages of evaluation, from initial evaluation through product refinement to large scale testing– Research design– Qualitative and quantitative evaluation– Process evaluation– Seeking ethical and governance approval (can link with

expertise on MHRA approval)– Data analysis– Writing for publication

3 May 2023 Research and Evaluation in Healthcare4

Research v. Evaluation• Whether we call it research or evaluation, the aim is to

generate evidence that informs us of the effectiveness/usefulness of an intervention, which could be a product, a process or a combination of both.

• Research – a rigorous process for assessing the effectiveness of an intervention before ‘rolling it out’.

• Evaluation - a rigorous (one hopes!) process for assessing the effectiveness of an intervention once it’s in place.

• The methods involved are conceptually broadly the same and some types of research, especially those involving complex interventions, require some form of ‘evaluation’, in its generic sense, of the internal components of the intervention – ‘process’ evaluation.3 May 2023 Evaluation of new digital health innovations5

Challenges in the digital world• The regulatory and approval systems for health care

innovations are largely based upon the pharmaceutical model– This can present many challenges for digital innovations

• We need to identify and develop time- and cost-efficient mechanisms for evaluating digital innovations before they are likely to be widely embedded in practice– Digital health technology companies lack the resources

of big pharma– Research funding is very competitive and can take

much effort, and a long time, to obtain– Technology rapidly evolves over time

• We can’t always rationally evaluate the ‘digital’ component independently – it is often embedded within a larger ‘process’3 May 2023 Evaluation of new digital health innovations6

Challenges in the digital world• Increasing shift in the nature of healthcare towards more

patient autonomy and pro-active engagement with their health management

• Exponential growth in ‘health apps’, with very little evidence base, that the public can try for themselves

• Health practitioners are increasingly ‘trying them out’ in their practice– By the time they have been ‘proven’, using the current

paradigm, they may have become obsolete• Can we ‘try them out’ and evaluate as we do so?

– Yes, in many circumstances, but the way in which we do this can significantly impact our ability to evaluate them rigorously

3 May 2023 Evaluation of new digital health innovations7

Addressing the challenges• Early user testing to establish a prima facie case for early

adoption and further development• Think clearly about the nature of the ‘benefit’ of the

product/innovation and how this can be measured• Be aware of the principles of good research design and threats

to validity• Design the evaluation at the time of implementation. If it is an

afterthought, it may be too late!– ‘To consult the statistician after an experiment is finished is

often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.’ - Ronald Fisher

• Make smarter use of routinely collected electronic data– Huge potential savings in time and cost, but there are data

governance, and research governance, challenges to address.

3 May 2023 Evaluation of new digital health innovations8

3 May 2023 Evaluation of new digital health innovations9

Thank you