How can research reveal the science underlying health informatics ?

So we can make HI more professional - like building bridges

Prof Jeremy Wyatt DM FRCP ACMI FellowLeadership chair in eHealth research, University of Leeds, UK

Clinical adviser on new technologies, Royal College of Physicians

From 1/1/16: Director, Wessex Institute of Health, University of Southampton

j.c.wyatt@leeds.ac.uk

Some tough questions

1. Why are usable EPRs so hard to engineer ?2. Why do one third of CDSS trials fail (Garg 2005) – when

those CDSS must be very well engineered for an RCT ?3. How can an ePrescribing system cause so much harm ?

(Koppel, JAMA 2005)

A clue: why don’t bridges fall down nowadays:a) There is a science of materials and construction methodsb) Engineers are professionals: they learn the science &

keep up to date

Tay Bridge disaster, 1879

Is HI / eHealth a “professional” discipline yet ?

Evolution of professionalism:

• Intuition – a craft • Mapping, taxonomy – a trade• Testing of predictive theories - research • Reliable engineering based on this – a profession

Heathfield H, Wyatt JC. Methods Inf Med, 1995

For HI:

1960-70s1980-1990s2000-20202020 on ?

What kinds of theories are relevant in eH / HI ?

User 2Health information system Decision Improved behaviour

& outcomeUser 1

Theories of communication

Theories of information retrieval

Theories of decision making

Behaviour change theories (personal / organisational)

Consider a simple eHealth system: an internet forum to support smoking cessation

How to carry out theory-based eHealth research

Identify a promising theory

Identify a common, important eHealth problem

Version of information system that ignores the theory

Incorporate this theory into an information system

Measureusage & impactof both systems

Analyse problem characteristics and possible solutions

New knowledge about the problem - and the theory

Literature review, systematic review

Example 1: Does Fogg’s theory help website persuade people to donate organs for transplant?

Persuasive features:1. URL includes https, dundee.ac.uk2. University Logo3. No advertising4. References5. Address & contact details6. Privacy Statement7. Articles all dated8. Site certified (W3C / Health on Net)

Result: 900 students recruited to RCT in 5 days; no diffe

rence in

NHS organ donation register sign-up rates (38% both groups)

Work of Thomas Nind, PhD Student, Dundee

Example 2: does feedback on group performance increase exercise ?

RCT with 32 students: all sent us daily txt msg of step countHalf (“Team B”) got weekly feedback on total step count of

“their” group vs control groupModest support for “group obligation” theory

Control (team A)

Intervention (team B)

Work of Sam Dhesi, Medical Student, Leeds

Intervention modelling experiments

Aim: to optimise the intervention before an RCT

Example methods:• Attitude surveys• Focus groups• Formal usability studies• Log file analysis• Eye tracking studies• Neuromarketing methods• Simulated decision studies

Example 3: How to improve the acceptability of prescribing alerts?

DSS are effective tools to improve prescribing (Garg 2005)

However, GPs usually turn off their prescribing alerts, because:• Too many alerts – no grading by severity• False positives: poor knowledge base, poorly coded data

Question: • Can we improve acceptability of alerts while still

reducing prescribing errors ?

Work of Greg Scott, ACF, London funded by NPfIT

Potential ways to improve clinical alertsAlert content:• Wording – signal words (“Warning !”)• Other material: symbols – alert triangles etc.• Clickable list of actions to perform

Alert accuracy:• Improve completeness, quality of coded patient data• Improve completeness, quality of drug knowledge• Improve underlying alert logic eg. calculate event probability

How the alert appears on screen:• Location, size• Persistence

Interruptive alert

Non-interruptive alert

Summary of results

Modal alerts: participants 12X (95% CI 6.0 to 22.3) less likely to make prescribing error than when not shown any alert

Non-modal alert: 3 times (CI 1.9 to 5.3) less likely to make prescribing error

Non-modal alert error rate 4 times higher (CI 1.9 to 7.0) than with modal alerts

“Safe” Dr = 0 or 1 error out of 24 scenarios

Some participant comments

“When you are in a rush, the one that pops up is better – forces you to click on OK”

“Pop-ups make you think more as you do it”“[I prefer] interruptive – likely to miss otherwise. But

recognise the problems, irritating in daily use.”“Interruptive tend to be annoying. But if it’s something you

don’t want to miss…”“Difficult to say what deserves one type or other of alert”“Didn’t notice it”

Published as: Scott et al JAMIA 2011

The MOST SMART approach

MOST: multiphase optimisation (of complex interventions):

1. Screen intervention components for effectiveness (lab expts on simulated decisions, RCTs, full / fractional ANOVA…)

2. Fine tune the combination of intervention components using SMART, qv.

3. Standard RCT to confirm effectiveness

SMART: sequential multiple assignment randomised trial (of time-varying interventions):• Randomise participants at each stage to competing

interventions, as suggested by theory

• Collins et al. Am J rev Med 2007

SMART: example for an exercise SMS programme

Assess stage of change

(Prochaska)

-ve / +ve framed msgs

Positive framed msgs better for

relapsers ?

Own name or not

Own name annoying after

a while ?

Individual / aggregate team

feedback

Risk of everyone

matching lowest performer in

group ?

Theories tested:

What is eHealth research really for ?

Relevant theory

Rigorous research

Generic, reliable, actionable knowledge

Safer, more reliable eHealth tools

Publication, dissemination

Health problem

Benefits of building the eHealth “theory base”

• No more trial and error or re-invention of ad hoc systems that seemed sensible at the time

• eHealth will evolve from an intuitive craft (reliant on experts and apprenticeship) into a professional discipline, making its decisions based on tested theories

• Systems will be safe, efficient & predictable (like bridges)• No need to evaluate every version of every app / website

/ serious game...

Conclusions

1. Professionalism requires sound theories

2. eHealth research should test theories from information, cognitive, organisational and computer science

3. Suggested procedure: • Define a question of generic importance to our field• Identify a candidate theory, relevant eHealth case

study & potential biases• Select the best evaluation method to test the theory• Carry out the study

4. Promote the results to students and eH practitioners

Even a tablet is a complex intervention

Doctor / nurse / pharmacist instructions

Leaflet insert

Packaging

Colour of the pills

Monitoring of drug levels, response to therapy

Pt expectations

Clinician expectations

Experience of others

eHealth mechanism of action

22/39

Clinical eHealth system eHealth system

Clinician

Outcome

Patientt1

action Disease activityt1

Patientt2

Disease activityt2

Patient eHealth system

Decisioninterval t2-t1

ii

data collection bias

placebo effectcontamination, checklist effect

TIDieR intervention reporting checklistHoffmann et al. Template for Intervention Description and Replication (TIDieR) checklist and guide. BMJ 2014

23/39

BRIEF NAME - name or a phrase that describes the intervention.

WHY Describe any rationale, theory, or goal of the elements essential to the intervention.WHAT: Materials: Describe any physical or information materials used, including those provided to participants or used in intervention delivery or in training of intervention providers. Provide information on where the materials can be accessed (e.g. online appendix, URL).Procedures: Describe each procedure activity, and/or process used in the intervention, including any enabling or support activities.WHO PROVIDED For each category of intervention provider (e.g. psychologist, nursing assistant), describe their expertise, background and any specific training given.HOW Describe the modes of delivery (e.g. face-to-face or by some other mechanism, such as internet or telephone) of the intervention and whether it was provided individually or in a group.WHERE Describe the type(s) of location(s) where the intervention occurred, including any necessary infrastructure or relevant features.WHEN and HOW MUCH Describe the number of times the intervention was delivered and over what period of time including the number of sessions, their schedule, and their duration, intensity or dose.TAILORING If intervention was planned to be personalised / adapted, then describe what, why, when, and how.

MODIFICATIONS If the intervention was modified during the course of the study, describe the changes (what, why, when, and how).HOW WELL:Planned: If intervention adherence or fidelity was assessed, describe how and by whom, and if any strategies were used to maintain or improve fidelity, describe them.Actual: If intervention adherence or fidelity was assessed, describe the extent to which the intervention was delivered as planned.

Cross disciplinary research

Chindogu device for restarting your PC

Neuromarketing – a food industry example

Theory: for behaviour, emotion > information (Kahneman’s System 1)Methods: FMRI; EDA; facial EMG; web-cam facial expression recognition

Study aim & methods

Aim: to help develop more effective SMS msgs for health promotion, by:

• Developing a reliable methods to capture EDA, facial EMG• Validate it against words & phrases of known emotional import• Use it to test & improve new phrases and txt msgs before an RCT

Methods - 40 volunteers:

• Measure EDA and facial EMG • Exposed to 20 words of known emotional import, 5 words about

exercise, 5 nonsense words & their own name in random order

Work of Gabriel Mata, Leeds PhD student funded by CONACYT, Mexico

Methods

Results

1 6 11 16 21 26

-0.04000000

-0.02000000

0.00000000

0.02000000

0.04000000

0.06000000

0.08000000

Series1

EDA reactivity

word

reac

tivity

in µ

S