Introduction to selecting and using critical appraisal tools/frameworks

How to interrogate research papers Alan Glasper and Colin Rees

How to Write Your Nursing Dissertation, First Edition. Alan Glasper and Colin Rees.© 2013 John Wiley & Sons, Ltd. Published 2013 by John Wiley & Sons, Ltd.

Critical appraisal tools/frameworks

• Sue is meeting with her supervisor to discuss the selection of an appropriate critical appraisal tool to evaluate her selected papers.

What is critical appraisal? What are critical appraisal tools?

• Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, value and relevance in a particular context.

Health care practitioners need to: • Decide if the research being

reported in the papers they read has been conducted in such a way to ensure that the finding are both valid and reliable

• Be able to understand and make sense of the results

• Decide if the research is strong enough to suggest changes to practice.

What is the best critical appraisal tool to use?

• There are many different critiquing tools available, some target specific types of study (e.g. qualitative) and some are generic and can be used to appraise any paper.

• Critical appraisal is the systematic and unbiased detailed examination of all the reported elements of a published paper or study to allow judgement of both the merits or strengths and the weakness or limitations in order to facilitate both the meaning and relevance to practice (Burns and Grove 2009).

• The University of South Australia International Centre for Allied Health Evidence (http://www.unisa.edu.au/cahe/Resources/CAT/default.asp) offers a full and comprehensive discussion of the attributes of many popular health care critiquing tools. Additionally the Scottish Intercollegiate Guidelines Network (SIGN) has developed evidence based clinical practice guidelines for the National Health Service (http://www.sign.ac.uk/methodology/checklists.htm).

What is the best critical appraisal tool to use?

Although there are many critiquing tools available, this presentation will only consider critiquing tools which have been developed by:• Crombie • CASP • Parahoo• Rees

Sam supervisor has advised him to use Parahoo’s critiquing approach, whereas Sue has been advised to use Rees’ model.

Using highlighter pens

• Sue has been advised by Sam to buy a large pack of children’s highlighter pens including some exotic fluorescent colours. She now believes that this will be useful as she now has a different colour for each of the critiquing tool questions.

Riley (1996) first used highlighter pens used as a method of identifying commonalities in transcribed qualitative interviews. Highlighter

pens

Using colour highlighter pens

Sam gave me a good tip

Using colour highlighter pens• Have at least three or four copies

of each your selected papers, one to carry with you in your handbag, briefcase or rucksack to read on the bus, train or plane or in your coffee break, one to identify specific details using the colour highlighter pens. (Some students like to use one copy of the papers to cut up with scissors after highlighting and reassemble piece by piece to match the criteria of the specific critiquing tool they are using). Finally, keep a spare copy for insurance – just in case!

glad the colours

work for you

Using Savage and Callery grids to undertake a preliminary critique

Author(Year)Country

Aim(s) of study Methodological Issues Relevant/key findings

Sample Design, data collection and analysis, rigour/reliability and validity

The Crombie Model of critiquing

1 Why was it done? (Objectives)What was the rationale for undertaking this study?What was the purpose of the study?Was the research question clearly stated?

2 How was it done? (Methods)

Was the research design appropriate to address the research question?How was the sample selected?How was the data collected?How was the data analysed?Were ethical issues discussed?

3 What has it found? (Results)

Are the findings clearly presented?Do the findings answer the research question?

4 What are the implications? (Conclusions)Were the findings discussed?Are implications for future practice summarized?

Crombie’s framework can help practitioners think about the process of critiquing a research paper (qualitative or quantitative)

Sam and his supervisor discuss a range of models

The CASP models of critiquing

• The Critical Appraisal Skills Programme (CASP) has been designed by Solutions For Public Health http://www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme and provides 7 different critiquing tools, each designed for different types of study and these are freely available to download from their website.

Yes I am using CASP

The Parahoo model of critiquing

Title of studyDoes the title convey the study clearly and accurately? AbstractDoes the abstract give a short and concise summary of the following aspects of the study? BackgroundAimDesignsResultsConclusionsLiterature review/BackgroundIs the importance of study justified?What is the context of this study?Does the literature review show the gap/s in knowledge which this study seeks to fill?Aims/objectives/research questions/hypothesesAre the aims of the study clear?Design of studyWhat is the design of the study? Is it the most appropriate for the aims of the study?Are the main concepts (to be measured) defined?What are the methods of data collection? Are they constructed for the purpose of the current study or do the researchers use existing ones?Who collected data? Can this introduce bias in the study? In studies where there are more than one group, is there a description of what intervention/treatment each group receives?Is the setting/s where the study is carried out adequately described? Who was selected? From what population were they selected? What was the precise method of selection and allocation? Was there a sample size calculation? Was ethical approval obtained? Are there any other ethical implications?•

Data analysisWas there a separate section in the paper that explained the planned analyses prior to the presentation of the results?Which statistical methods were relied on?Is it clear how the statistical tests were applied to the data and groups?ResultsAre the results clearly presented?Are the results for all the aims presented?Are the results fully presented?DiscussionIs it a balanced discussion? Has all possible explanations for the results given?Are the results discussed in the context of previous studies?Are the results fully discussed?Are the limitations of the study discussed?Are the results discussed in the context of previous studies?Are the results fully discussed?Are the limitations of the study discussed?Conclusions/RecommendationsAre the conclusions justified?Are there recommendations for policy, practice or further research?Are the results/conclusions helpful for my practice?Are the results generalisable?FundingIs there potential conflict of interest (if information on funding is provided)?

The Rees (2011) model for critiquing quantitative and qualitative research

• Focus• Background• Aim• Study design• Data collection method• Ethical considerations• Sample• Data presentation• Main findings

• Conclusion and recommendations

• Readability• Practice implications

NB Depending on the level of the course i.e. undergraduate or post graduate, the amount of information concerning statistical information will vary but for all dissertations, at least a preliminary understanding of the tests used by the researchers is mandatory.

Assessing the strengths and weaknesses of critiqued papers

• Sue and Sam have completed their critiquing chapter of their dissertations. They are now in the final aspects of their dissertations and are planning to write a short chapter summing up the relative strengths and weaknesses of the papers they have read and comprehensively critiqued.

Study Strengths Limitations 1

2

3

4