systematic review module 11: grading strength of evidence interactive quiz kathleen n. lohr, phd...

Systematic Review Module 11:Systematic Review Module 11:Grading Strength of EvidenceGrading Strength of Evidence

Interactive QuizInteractive Quiz

Kathleen N. Lohr, PhDKathleen N. Lohr, PhDDistinguished Fellow Distinguished Fellow

RTI InternationalRTI International

Interactive Case Quiz: Interactive Case Quiz: InstructionsInstructions

Open this presentation as a slideshow. This will activate the hyperlinks.

When you come to a decision slide, choose the red box corresponding to the correct choice. If you are correct, you will be directed forward in the case. If you are incorrect, you will be directed back to the decision slide to choose again.– Click on forward hyperlinks (Red Boxes) to follow

through case based on your responses – Click on home hyperlink (Blue House Icon) to go

back to the last correct step in the series

Case Study Quiz: Case Study Quiz: Grading Strength of Evidence (SOE)Grading Strength of Evidence (SOE)

You are near the end of the process of developing a CER of the benefits and harms of a therapeutic intervention. This step entails grading the strength of one or more bodies of evidence that pertain to the key questions in your CER.

Participation Quiz: Question 1Participation Quiz: Question 1

Grading SOE is the same as rating the quality of studies.

They are the same thing.

You can grade SOE at the same time as you rate the quality of individual studies.

You can grade SOE only after you have rated the quality of individual studies.

They Are the Same ThingThey Are the Same Thing

This is incorrect. Individual studies are rated for their

quality (i.e., internal validity or risk of bias).

Grading pertains to entire bodies of evidence about important key questions, outcomes, or comparisons, and the quality of individual studies is just one element in grading SOE.

[Click on Blue Box to Go Back]

You Can Grade SOE and Rate You Can Grade SOE and Rate Quality at the Same TimeQuality at the Same Time

This is incorrect. Grading pertains to entire bodies of

evidence about important key questions, outcomes, or comparisons.

Because domains other than the quality of individual studies must be used in grading SOE, evaluating study quality and SOE together is insufficient.

[Click on Blue Box to Go Back]

You Grade SOE Only After You Grade SOE Only After Rating Study QualityRating Study Quality

Yes, you are correct. Grading SOE is a late step in the review

process. It requires judgments about several key domains, only one of which is quality of individual studies.

[SELECT RED BOX]

Participation Quiz: Question 2Participation Quiz: Question 2

Grading SOE specifies required domains. What are they?

Risk of bias, consistency, precision, and directness

Risk of bias, consistency, precision, directness, and publication bias

Risk of bias, precision, strength of association, and directness

Four Required Domains Four Required Domains

Yes, You are correct! The four required domains are

– Risk of bias: quality ratings for individual studies

– Consistency: degree of similarity in the effect sizes of different studies within an evidence base

– Precision: degree of certainty for estimate of effect with respect to a specific outcome

– Directness: whether evidence reflects a single, direct link between the interventions of interest and the ultimate health outcome under consideration, or for comparisons, whether the evidence base has head-to-head studies [SELECT RED BOX]

Four Required Domains and Four Required Domains and Publication BiasPublication Bias

This is incorrect. The four required domains are

– Risk of bias

– Consistency

– Precision

– Directness

Publication bias is an additional domain that can and should be used only when relevant.

[Click on Blue Box to Go Back]

Three Required Domains and Three Required Domains and Strength of AssociationStrength of Association

This is incorrect. The four required domains are

– Risk of bias

– Consistency, which is the missing required domain in this answer

– Precision

– Directness

Strength of association is an additional domain referring to magnitude of effect that can be used when relevant.

[Click on Blue Box to Go Back]

Participation Quiz: Question 3Participation Quiz: Question 3

Strength of evidence is graded according to the following levels or grades:

High, moderate, low, very low

Very high, high, moderate, low,

very low

High, moderate, low, insufficient

High, Moderate, Low, Very LowHigh, Moderate, Low, Very Low

This is incorrect. Very low is not an option. Use four SOE grades:

– High confidence that the evidence reflects the true effect

– Moderate confidence that the evidence reflects the true effect

– Low confidence that the evidence reflects the true effect

– Insufficient: Evidence either is unavailable or does not permit a conclusion

[Click on Blue Box to Go Back]

High, Moderate Low, High, Moderate Low, InsufficientInsufficient

Yes, this is correct. Strength of evidence can have three

“strength” grades relating to the confidence one has that evidence reflects the true effect and that future research might confirm or overturn the results.

Insufficient implies either that one has no relevant evidence or that the evidence is so mixed and inadequate that one cannot definitely label it as high, moderate, or low.

[SELECT RED BOX]

Very High, High, Moderate, Very High, High, Moderate, Low, Very LowLow, Very Low

This is incorrect. Very high and very low are not used.

The four correct options for scores are – High

– Moderate

– Low

– Insufficient

[Click on Blue Box to Go Back]

Participation Quiz: Question 4Participation Quiz: Question 4

In combining scores for domains into an overall grade for strength of evidence, you can use the following approaches:

Only the GRADE algorithm

Only your own weighting system

Only your own qualitative approach

Any of the above

Only One Approach is Only One Approach is AllowedAllowed

This is incorrect. Any of the three approaches can be

used. The critical element in selecting any

option is careful documentation of the methods.

[Click on Blue Box to Go Back]

GRADE, Own Weighting, or GRADE, Own Weighting, or Own Qualitative ApproachOwn Qualitative Approach

Yes, this is correct. Any of the three approaches can be

used. The critical element in selecting any

option is careful documentation of the methods.

[SELECT RED BOX]

Participation Quiz: Question 5Participation Quiz: Question 5

In scoring required and additional domains and in arriving at an overall grade for strength of evidence, you

Use only one senior reviewer to do these tasks and report this score.

Use two or more reviewers, resolve differences by consensus or adjudication by a third party, and report all scores.

Use two or more reviewers, resolve differences by consensus or adjudication by a third party, and report a consensus score.

Only One ReviewerOnly One Reviewer

This is incorrect. You should use at least two reviewers

with appropriate clinical and methodological expertise.

[Click on Blue Box to Go Back]

Two or More Reviewers and Two or More Reviewers and Report All ScoreReport All Score

This is incorrect. You should use at least two reviewers

with appropriate clinical and methodological expertise.

But Differences should be mediated or

adjudicated by a third party. You should report only the consensus

grade. [Click on Blue Box to Go Back]

Two or More Reviewers and Two or More Reviewers and Consensus ScoreConsensus Score

Yes, this is correct. You should

– Use at least two reviewers with appropriate clinical and methodological expertise and

– Mediate or adjudicate differences by a third party

But You should report only the consensus

grade.