Theme 5. Cochrane Reviews: updating

• 9:30 – 10:10 Theme overview and CRG experiences of using tools/frameworks• 10:10 – 10:30: Q&A• 10:30 – 11:00 Working groups• 11:00 – 11:15 Feedback

Overview

1. How often should Cochrane Reviews be updated?

2. Which Cochrane Reviews should be a priority for updating?

3. How should updates of Cochrane Reviews, and Cochrane Reviews no longer being updated, be classified in The Cochrane Library?

Updating frequency

• CRGs are not able to update all Cochrane Reviews every two years. The reasoning behind updating decisions is not transparent to readers.

• The burden of updating is ever increasing• Cochrane Reviews may become out of date at different times• Updating too soon may introduce bias• 80% of 617 respondents moderately or strongly supported a

move from the current policy of aiming to update all Cochrane Reviews every two years to a ‘needs-based’ approach

Prioritising Cochrane Reviews

Prioritising Cochrane Reviews

Classification framework

Experiences

Criteria area Question Yes(=1)/No(=0)  Strategic importance Is condition of strategic important to MSK Group or funders or users 0/1   Is intervention of strategic important to MSK Group or funders or users 0/1     Patient importance/ impact Does review measure patient important outcomes? i.e. clinical outcomes 0/1    Does intervention have a high potential impact on quality of life? 0/1    Does intervention potentially significantly lower mortality 0/1    Leading cause of burden of disease in Europe (WHO burden of disease DALY list) 0/1    National Spend High spend on condition (including societal costs)? 0/1    High spend on intervention or on currently used alternatives? 0/1    Presence on list of 20 drugs which had the greatest number of items dispensed, OR greatest Net

ingredient cost 0/1    Incidence prevalence High incidence or prevalence? 0/1    Recent increase in incidence/prevalence? 0/1  

      Emerging evidence/remaining uncertainty Current review conclusions reflect uncertainty? 0/1    Significant RCTs/other eligible trials published since last update? 0/1    SRs/HTAs covering same ground NOT published (or about to be published) since last update? 0/1     Additional Cochrane criteria Leading cause of burden of disease in the rest of the world i.e. outside of Europe (WHO burden of

disease DALY list) 0/1    Review content out of date 0/1    Review has attracted > 10 citations 0/1    Review in top 10 most accessed MSK reviews 0/1    Review addresses important equity or Global Health Issues 0/1    Author available for update 0/1       Total score: 0  

       

Original criteria

    Question

ConditionBurden of disease, spend, and incidence/prevalence related to the condition

Is condition of strategic importance to CCDAN Group or funders or usersLeading cause of burden of disease in Europe (WHO burden of disease DALY list)Leading cause of burden of disease in the rest of the world i.e. outside of Europe (WHO burden of disease DALY list)High spend on condition (including societal costs)?High incidence or prevalence?Recent increase in incidence/prevalence?

Evidence Emerging evidence / remaining uncertainty

Current review conclusions reflect uncertainty?Significant RCTs/other eligible trials published since last update?Are there new trials addressing a specific call for research in previous version of reviewSRs/HTAs covering same ground NOT published (or about to be published) since last update?

Stakeholder

NIHR / DOH / NHS

High spend on intervention or on currently used alternatives?

Presence on list of 20 drugs which had the greatest number of items dispensed, OR greatest Net ingredient costReview has contributed to a NICE guidelinesRelated NICE guideline pending / update due soonIs intervention of strategic importance to fundersHas intervention been highlighted in guidelines or elsewhere as requiring more research

Patients

Does intervention have a high potential impact on quality of life?Is the intervention recommended at all in NICE guidelinesIs the intervention a first line treatment according to NICE guidelines?Current treatment options have evidence of effectiveness but are not well tolerated by patientsDoes the intervention have significant known adverse events (score -1)Intervention discontinued / no longer prescribed (score -1)

Cochrane Collaboration

CCDAN

Review has attracted > 10 citationsReview in top 10 most accessed CCDAN reviewsReview never cited (score -1)Review rarely accessed (score -1)Could this contribute to an MTM / overview

Rest of the World (WHO as proxy)

Review addresses important equity or Global Health IssuesIs there potential for collaboration with or benefit for WHO or other similar body

Does this review directly contribute to Evidence Aid

The criteria below are to be considered once a review has been prioritised according to the above and so these criteria do not contribute to the scoring

Review Quality

Methodology

Full risk of bias tool used

Summary of findings tables used

Methods up to date and written in RevMan 5 headings

Is the scope of the review question sensible (should it be split/merged)

Does the review address its objectives

Patient important changes

Are patient important outcomes addressed - if not they must be.

Are adverse events dealt with - if not they must be

Revised criteria

Cochrane Airways Group Prioritisation exercise using the decision flowchart and

STATA program

Context

• ~250 reviews• 166 reviews over 2 years out of date• Don’t have a prioritisation policy and make

decisions to update based on author availability

• Looking to: – make decisions transparent– direct limited resources at most important topics

What we did

• Considered 21 reviews with the highest number of hits and cites for prioritisation

• TSC ran searches• ME/TSC/RA screened titles and abstracts• Used prioritisation flow chart and STATA tool• Extracted outcome data for the STATA tool

where possible even if review currently being updated

Flow chart

Issues with flow chart

• Seemed very complicated at first, but was OK to use• We created a spreadsheet to reflect the flow of work

from the decision tool and record comments on reviews

• We didn't go searching for abstracts or papers if they weren't available; however we felt that the new studies identified represent the bulk of new research for the review

• Some updates were already in progress and we consulted authors to try to get the number of hits

Innovated table to record decisions

Issues with STATA tool

Possibly too crude?– need to select a single outcome– extract outcome data often from abstract only.

• Unresolved issues with continuous data and GIV

• However, once data had been extracted the STATA tool was easy to use and may find application in some circumstances

Reviews that reached stage x in the flowchart

• Stage 1: 4 review questions appeared answered or partly answered

• Stage 2: 13 reviews needed some sort of work. 7 were unclear. Most needed SOF

• Stage 3: We did searches for all of them to test out the STATA tool (so could save time here in future). Some searches challenging to go through

• STATA: we got sufficient data to run 6 studies through STATA tool but could not analyse the continuous data or the GIV data for unresolved technical reason

Outcomes• 8 indicated for update for other reasons, 5

possible updates and 8 not to be updated (already up-to-date, irrelevant review question, not enough new data)

• We didn’t feel that the STATA tool was worth the additional workload and that it was too crude for our needs, however now we know how it works, may use in future

Issues/comments

• Estimate it took around 5 days in total to do this work. Plus 2 hours with a statistician on STATA.

• This is a reasonable amount of time compared to the editorial process for an ill-conceived update!

• The flow-chart tool provided – a helpful structure to assess each review – a transparent way to summarise evidence to

stakeholders

Issues/comments

• Other groups may need to modify tool and consider which of their reviews to put through the tool

• We would need a further conversation with an advisory board on the obtained information to rank them/decide relevance of review question

• Need a transparent way to select reviews for this tool• With more time, may be possible to screen

references carefully enough to be of use to authors

Objectives To assess whether CRGs other than the CIDG

could classify their Cochrane Reviews using the framework

To identify any problems with the framework and potential solutions

To determine how long it takes CRGs to classify Cochrane Reviews

Methods CRGs who nominated themselves classified a

sample of 20 Cochrane Reviews selected by CEU, and fed back experiences

Classification framework pilot

Results Thirteen CRGs completed the pilot CRGs were able to classify Cochrane Reviews of

interventions, and found the process manageable and useful

CRGs faced several issues when making judgements, and we adjusted guidelines and guidance

CRGs varied in the amount of time they took to classify Cochrane Reviews

Next steps Consider whether framework is suitable for non-

intervention reviews User test framework on readers of CDSR, with mock-ups of

potential views Implementation

Results and next steps

How should Cochrane Reviews be classified if there is more than one comparison, and the comparisons have different categories?

How should a Cochrane Review be classified if trials are ongoing? Could there be more subdivisions in the categories, for example,

to highlight whether an author is or isn’t available? When should a Cochrane Review be classified as up to date? If

there are no new studies, can the Cochrane Review be labelled ‘Current, Up to date’? And what should happen if a few small studies are identified that are unlikely to change the conclusions?

Is the term ‘Current, Up to date’ misleading when a full systematic update search as not been done?

How frequently should these statuses be applied? For the classification ‘Current, No update intended’, how should

CRGs make this judgement? Should there be a final search before a Cochrane Review is

classified as ‘Historical, No update intended’. Should authors be consulted before making a judgement about

the classifications?

Queries from CRGs

Revised framework

Revised framework in light of queries

Recommendations

“Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...”

“Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...”

“Ensure decisions around prioritisation are transparent to readers.”

“Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”

Working groups

Group 1: “Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...” AND “Ensure decisions around prioritisation are transparent to readers.”

Group 2:“Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...”

Group 3: “Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”

To consider in working groups• Agree/disagree? Consensus?• If agreement, high or low priority (compared with other strategic session

paper recommendations)? How urgent?• If disagreement, tweak, rework, or discount?• Implementation?

– Changes to tools or framework?– Affect on CRGs’ monitoring?– Changes to Review Manager and Archie?– Changes to The Cochrane Library website?– Changes to CRG entity websites?– Changes to resources (e.g. Cochrane Handbook)?

• Barriers?– Methods?– External perceptions?