Critically Examining the Evidence

Andrea M. Landis, PhD, RNUW LEAH

December 7, 2012

Learning ObjectivesDiscuss the importance of critically examining the

literature.Review key issues for evaluating the literature.Explore in detail each section of a manuscript.Appraise systematic review article.

Sources:“How to read a paper” (2010) Greenhalgh, T. [ebook]Centre for Evidence Based Medicine (Oxford)UW Health Sciences Library - Toolkit

Before any research project review the literature relevant to your RQ:Gain a full and in-depth understanding of a subject. See if your intended research subject has been

done before and avoid duplication. Avoid any errors made in similar research. Enable you to place your study within its context (ie

so that you can show how your research will add to the existing sum of knowledge).

Provide you with ideas to help you define or amend your own research topic.

Provide you with information with which to compare and contrast your findings.

IntroductionOne’s ability to review articles is something

that requires practice and experience. Process can be aided by following a checklist

of things to look out for and comparing the paper under review to the criteria.

The type of research will effect the information you are evaluating.

Standard journal format subheadings for research reports: title, abstract, introduction, method, results, discussion and conclusion.

Summary of key issues that need to be evaluated (APA, 1983):Is the RQ question significant and is the work original

and important? Have the instruments used been demonstrated as

reliable and valid? Do the outcome measures relate clearly to the variables

with which the investigation is concerned? Does the research design fully test the hypothesis? Are the subjects representative of the population to

which generalizations are made? Did the researcher observe ethical rules? Has the research reached such a stage that publication

is justified and the results are meaningful?

TitleVery important. Short and informative.Gives insight into what (was done), whom (it

was done to) and how (it was done). Gimmicky?

AuthorSome idea of the author(s) academic

background and ability to carry out valid researchjob title, qualifications, and where they work. Corresponding author

AbstractThe abstract should contain a brief statement

about the study's purpose, method, results, conclusion and clinical relevance.

Time-efficient way for readers to determine if the article suits their needs – return to the article later.

Do not to accept the conclusions before critically reading the entire article.

SourceAssessment of where the article was

published should give some clues as to its potential value.

Is it a peer reviewed journal?

Introduction & Statement of the ProblemResearch problem/clinical question should be defined clearly.Expect to find clear descriptions of the research aims, an

outline of theoretical issues and the hypothesis should be introduced.

Information should include the current state of knowledge about the research topic and an indication of the gaps in knowledge which the current study will hope to fill.

“Why was the research done?”

Statement of the Problem: should describe the questions and concerns that led the author to undertake the investigation.

“What question did the author try to answer?”

Literature Review → Purpose StatementThe literature review should establish a theoretical and historical

basis Survey of current knowledge highlighted by a thorough review of

the existing literature. Identify gap of knowledge between what is known (or

previously documented) and what is desired to be known. Find information in the literature that supports the concept and approach of the study.

Attempt close the gap by explaining why the study was conducted. Or point out flaws, inconsistencies or areas where no conclusions can be drawn.

Issues:Up to date referencesUnbiased

The purpose of the study should be described in a direct, clear statement.

MethodologyClearly explain how the study was conducted.

Critical readers should pretend they are

going to replicate the study: Is there sufficient detail in the method to

conduct the study and obtain similar results? Was the design of the study sensible?

Method can be divided into the following subsections: subjects, instrumentation and apparatus, procedure, and data analysis.

Subjects Summarize and describe the subjects who participated in the study in

terms of age, sex, diagnosis and other pertinent demographic characteristics (Table).

Inclusion/Exclusion criteria. How were they recruited?The extent to which readers are able to use the results (generalization)

of the study depends on how the sample of subjects was selected: Randomly selected Sample of convenience RCT - allocation of subjects to experimental and control groups

How many subjects were included in the sample: Small vs. Large Impact on power analysis Diversity (race/ethnicity reported)

IRB approval (includes confidentiality and anonymity assured)

Instruments or ApparatusAny special equipment or instruments (e.g.

questionnaires) should be described. Indicate validity and reliability.

Described in such a way that readers can replicate the study.

Specify model numbers, corporate names and addresses, and other pertinent details about the instruments.

Any apparatus (or questionnaire) designed and developed by the researcher should be fully described with a drawing, photograph and description.

Were the instruments calibrated? How were they calibrated? Are they repeatable day-to-day?

ProceduresHow and when the steps of the study were

applied. How the data were collected. Internal validity - changes noted during the study

are the result of the devices being studied and not the result of a sloppy procedure. Did the experimental treatment cause the

observed change in the dependent variable? Could other (extraneous) factors be

responsible for that change? Has the investigator taken steps to improve their

internal validity or control sources of secondary variance.

Data AnalysisSection should describe all testing applied to the data

NO results. Did authors chose the appropriate statistical tests for

the type of study and design. When analyzing data, arithmetic operations too

frequently are misapplied to data based on levels of measurement. Common error - analyzing ordinal data as though they

were quantitative (interval or ratio). Did authors screen data for errors in data entry,

outliers and distribution. Conventional parametric statistical analyses are

conducted on continuous data. descriptive, comparative, associative and predictive.

ResultsReports what has been discovered. Reported factually and formally without

commentary. Summary statistics may be presented in

tables/figures. Statistical tests and measures used should be

described allowing the reader to evaluate whether the appropriate tests were applied. Level of significanceStatistical versus clinical significance

Do the authors mention all the relevant results? - even those that actually go against the hypothesis.

DiscussionThe issues raised by the findings should be discussed

and resolved in this section. Should relate back to the literature/aims of the

research as outlined in the introduction. The author is expected to examine, interpret and

qualify the results and draw any inferences from them.Large enough/long enough/followed-up enough to be

credibleIs the research question answered? Has the author

given meaning to the results? Has the author considered broader implications of

his/her findings?

Conclusion & RecommendationBrief restatement of the experimental results and

describes the implications of the study.The paper should end with some conclusions

about the importance (or otherwise) of the findings.

The author should not make any statements here which are not supported by the facts found.

Recommendations on the basis of the findings are often stated here.comments on possible improvementsfuture areas for more study.

ReferencesConsistent citing of references.

Endnote, RefWorksAppropriately extensive and up-to-date.

Questions1. What question did the systematic review

address?2. Is it unlikely that important, relevant

studies were missed? 3. Were the criteria used to select articles for

inclusion appropriate?4. Were the included studies sufficiently

valid for the type of question asked?5. Were the results similar from study to study?6. What were the results? How are the

results presented?