Reflections of an editoron research and practice?

Richard SmithEditor, BMJ

Granada, May 2002www.bmj.com/talks

What I want to talk about

• The disconnect between practice and research

• A rough history of health research in Britain

• A vision of how to improve the connection

• The relation between disease burden and volume of research

• Setting research priorities

What I want to talk about

• Where does innovation come from?• Peer review of research• Measuring the value of research• Disseminating research• How to get from research to change?• Conclusions

The disconnect between practice and research

• Research is usually funded by the Ministry of Education, whereas health care and public health is funded by the Ministry of Health

• Research is run by researchers who value basic science, discovery, and original questions, thinking, and methodology

• The answering of practical questions is seen as dull, unoriginal, and “unimportant in scientific terms”

• Nobel prizes go to the discoverers of molecular mechanisms not those who work out the most cost effective method for treating incontinence

The disconnect between practice and research

• There is often no mechanism to transmit the questions of practitioners (and patients) to researchers

• Scientists are wary of directed research: “only scientists can know what is scientifically important”; “directed research leads nowhere”

• The results of research do not seem valuable to practitioners

• The idea that doctors are scientists is a myth

The disconnect between practice and research

• Most practitioners are not competent researchers• Nor are practitioners sophisticated consumers of

research• “Practice is one thing; research another. I

make decisions based on my experience and what clinical experts advise”

• Health policy makers sometimes boast that they don’t use research results

• Evidence based practice is a force for change, bringing research and practice together

The disconnect between practice and research

• It must be “evidence informed” practice not “evidence tyrannised” practice

• But we realise that fewer than 5% of studies in medical journals are both valid and relevant to clinicians or policy makers; in most journals it’s less than 1%

• We have good evidence on perhaps 10% of treatments and a smaller percentage of questions about diagnosis, symptoms and signs, and prognosis

• Evidence needed for health management and policy is even weaker

A rough history of health research in Britain

• 1900s--independent researchers• 1930s--Medical Research Council (MRC)

begins• 1940--Pharmaceutical companies begin to

do a great deal of research• 1980--MRC begins some health services

research• 1986--House of Lords realises that the

National Health Service has almost no research capacity

A rough history of health research in Britain

• 1990--NHS research and development directorate established– vision is a “knowledge based health

service”– aim is to spend 3% of NHS turnover on

R&D– programme attracts international interest

• 2001--NHS R&D programme still there but is less central than it once was

Bringing practice and research closer together: a vision

• Patient asks a question to a doctor • Doctors consults databases on what the

evidence says (Cochrane Library, Clinical Evidence, or an electronic decision support system)

• (Or, increasingly, patient consults the same knowledge sources as the doctor--besttreatments.org)

• If there is evidence, patient and doctor discuss best course of action

Bringing practice and research closer together: a vision

• If there is no evidence, then a systematic review may be needed

• Or the patient and doctors consult the meta-register of trials underway

• If there is a trial, the patient may enter the trial (knowing that patients treated in trials do better than others no matter whether they get the active treatment)

• If there is no trial, then the patient and doctor register the question with a central database

Bringing practice and research closer together: a vision

• Trials can then be conducted to answer the questions that are most important and arising most commonly

• The information sources needed to achieve this vision exist for questions on treatment

• (The culture and the infrastructure do not exist)• The information sources do not exist for

questions on diagnosis, prognosis, health policy, and much else--but could be created

The relationship between disease burden and the amount of research

• Examples from the US, Africa, and neurology

• The 90:10 rule--90% of research is on diseases affecting 10% of the world’s population

• The association is often small: some diseases with a small burden are highly researched, whereas some with a high burden are poorly researched

A system for prioritising research

• Consider disease burden• Consider questions generated by patients

and health care providers• Consider possible “research gain” (what are

the chances that an investment could result in real advances?)

• Incorporate social and professional values• Britain has had a system along these lines--

but only for NHS R&D programme

Where does innovation come from? Two models

• The linear model: curiosity driven research---applied research---experimental development---innovation

• The market pull model: market need---applied research---experimental development--innovation

Project Hindsight (1966)• Examined 20 weapon systems (including

Polaris)• Researchers identified 686 “research or

exploratory development events” that were essential for development of the weapons

• Only 9% were “scientific research” (0.3% basic research)

• Only 9% of research conducted in universities

Project Hindsight (1966)

• “Science and technology funds deliberately invested and managed for defence purposes have been about one order of magnitude more efficient in producing useful events than the same amount of funds invested without specific concern for defence needs.”

TRACES Study (1968)

• Technology in Retrospect and Critical Events in Science

• Origins of magnetic ferrites, video recorder, contraceptive pill, electron microscope, and matrix isolation

• Looked back 50 not 20 years, as did Project Hindsight

TRACES Study (1968)

• 340 events• 70% non-mission research, 20%

mission oriented, and 10% development and application

• Universities did 75% of non-mission and one third of mission oriented research

Comroe and Dripps (1976)

• Julius Comroe, physiologist, and Robert Dripps, anaesthetist

• The top 10 advances in cardiovascular and pulmonary medicine and surgery in the last 30 years

• Around 100 specialists selected the top 10

Top 10 advances• Cardiac surgery• Vascular surgery• Drug treatment of

hypertension• Medical treatment

of myocardial ischaemia

• Cardiac resuscitation

• Oral diuretics• Intensive care

units• Antibiotics• New diagnostic

methods• Prevention of

polio

Comroe and Dripps (1976)

• Went back to the dawn of time• 137 “essential bodies of knowledge”• 500 essential or key articles• 41% not clinically oriented• 37% “basic: not clinically oriented”• 25% “basic: clinically oriented”

Conclusions from studies of innovation

• The sources of innovation are numerous, varied, and scattered

• Both the science push and market pull models of innovation are oversimplified

• Research funders should not put all their eggs in one basket

• Attempts to force “more relevant” research may backfire

Conclusions from studies of innovation

• The coming together of different lines of research and and scientists from different disciplines seems to be important

• Promoting interdisciplinary research may seed innovations

• “Research into research” may be beneficial

Peer review of research

• Research grants are often given after peer review

• Which research will be published is often decided by peer review

• But there are problems with peer review

Problems with peer review

• A lottery• A black box• “Ineffective”• Slow• Expensive• Biased• Easily abused• Can’t detect fraud

Peer review

• But it is hard to find an alternative to peer review

• It’s like democracy--”the least bad system”

• The answer seems to be to improve peer review with training, openness, blinding, etc

Measuring the value of research• The point of health research is to improve health• But researchers are usually rewarded according to

measures of scientific value• These include the impact factor of the journal in which

they publish--despite there being little or no correlation for individual authors between the impact factor of the journal in which they publish and citations to their articles

• There are many other problems with impact factors--bias towards certain disciplines, US, methodology; data are often unreliable

Measuring the value of research

• Royal Netherlands Academy of Arts and Sciences is trying to devise a measure of the social impact of research

• Might include publications, software, products, press coverage, etc

• But it’s not easy to find a reliable measure

My suggestions for measuring influence/impact

– Level one (the highest): making change happen

– Level two: setting the agenda for debate– Level three: leading by example– Level four: being quoted– Level five: being paid attention to– Level six (the lowest): being known about

Dissemination of research• There are tens of thousands of journals• Millions of studies are published each year• Most studies are neither valid nor relevant• It’s hard--usually impossible--for clinicians

and policy makers to keep up• There is a need to review research results

systematically

Dissemination of research

• Evidence based journals--coverage of one off studies, not put into context

• Cochrane Library--treatments only, big gaps, researcher (not clinician) led questions, complex

• Clinical Evidence--treatments only, 160 topics, 400 needed

Dissemination of research

• Guidelines--cover only some topics, sometimes not evidence based, go beyond the evidence, tell people what to do

• Appraisals by National Institute of Clinical Excellence (NICE) --cover only a few topics, must incorporate evidence, cost, and “values,” insufficiently transparents

From information to change

ChangeKnow how

Know aboutInformation

Data

Failures to follow evidence• Aspirin underused in patients with vascular

disease• ACE inhibitors underused in patients with heart

failure• Inhalational steroids underused in patients with

asthma• Antibiotics overused in patients with upper

respiratory tract infections and acute otitis media• Enemas, pubic shaving, and episiotomies

overused in women in labour

From research to change

• “We should stop all research for two years and concentrate instead on implementing what we already know.”

• Somebody in, I think, the Lancet quite some time ago

From information to change

• Achieving change is hard• Information on its own rarely changes

practice• Combinations of audit and feedback,

computerised reminders, educational outreach, and interactive educational sessions will sometimes change practice

From information to change

• Interactive learning• Improvement methods• Organisational development• Consultancy• “Just in time” information

“The thing” that will save us

• Able to answer highly complex questions • Connected to a large valid database• Electronic - portable, fast, and easy to use• Prompts doctors - in a helpful rather than

demeaning way• Connected to the patient record• A servant of patients as doctors• Responds to the need for psychological support

and affirmation

Conclusions• Research and practice are currently not

well connected• It’s possible to envision how they might

be better connected• Some substantial health problems are

poorly researched, while some smallish problems are heavily researched

• Mechanisms are needed to set priorities in health research

Conclusions

• Research into sources of innovation suggests that different sorts of research in different circumstances are important

• Innovation often comes from interdisciplinary innovation

• Peer review has many problems but can probably be improved

Conclusions

• Better methods are needed for measuring the performance of health researchers

• The dissemination of research results is inadequate, but better means are appearing

• Moving from research to change is hard, but we can see how to do it better