sac clinical reasoning tool - copy · “clinical reasoning describes the process by which members...

2018‐04‐05

1

1

Clinical ReasoningEvaluation and Remediation

Alex Carling

Director of Professional Practice and Quality Assurance

Sarah Chapman‐Jay

Advisor Professional Practice and Quality Assurance

SAC Conference 2018 2

• Project background

• Clinical Reasoning Tool development

• Testing the effectiveness of the Tool

• Next Steps

• Remediating clinical reasoning


2018‐04‐05

2

Project Background


Project Background

: Amendments ‐ 2009

Minimum Requirements of a Quality Assurance Program

Regulated Health Professions Act 80.1

• Self assessment and peer assessment

• (a) Continuing education or professional development designed to,

(i) promote continuing competence and continuing quality improvement among the members,

• A mechanism for the College to monitor members’ participation in, and compliance with, the Quality Assurance Program.

SAC Conference 2018

Project Background

The Self Assessment and Peer Assessment process did notevaluate clinical reasoning.

Why add clinical reasoning?

The College believes it is an essential component of quality practice.

Members told us it was missing.

SAC Conference 2018

2018‐04‐05

3

Project Background

Sample of Member Quotes:

“The chart review was fine, but I wanted more

feedback and discussion about the charts.”

“I feel that my actual clinical work was not explored”

“I am looking forward to the addition of a more

clinical part to the assessment process”

SAC Conference 2018

Project Background

Canadian faculties of medicine use Chart Stimulated Recall to:

– identify critical thinking and reasoning skills

– stimulate reflective practice

– provide feedback

– improve documentation skills

– help demonstrate and evaluate roles and competencies

– help structure a teaching session

– identify gaps in knowledgeDept. Family Medicine, University of Alberta (2009)

SAC Conference 2018

Project Background

Health regulated colleges across Canada use different methods to evaluate their members’ clinical reasoning.

SAC Conference 2018

2018‐04‐05

4

Project Background

Evidence to support the use of Chart Stimulated Recall for evaluating quality care.

• Schipper, S and Ross, S (2010) Structured teaching and assessment: A new chart‐stimulated recall worksheet for family medicine residents Canadian Family Physician Le Médecin de famille canadien Vol 56:

• Mills,E. Blenkinsopp,A. and Black, P. (2011) The Assessment of Observed Practice:

A Literature Review Department of Medicines Management, School of Pharmacy

Keele University

• Goulet, F. Jacques,A. Gagnon,R Racette,P., Sieber, W (2007)

Assessment of Family Physicians' Performance Using Patient Charts Interrater Reliability and Concordance With Chart‐Stimulated Recall Interview. Eval. Prof Health, vol 30, 4

10

Clinical Reasoning Tool Development



Goal:

To develop a reliable and valid tool to evaluate speech language pathologists’ and audiologists’ clinical reasoning across all practice areas as part of the Quality Assurance Program.

SAC Conference 2018

2018‐04‐05

5


Definition of Clinical Reasoning

“Clinical reasoning describes the process by which members collect and evaluate information, come to an understanding of a patient problem or situation, plan and implement interventions, evaluate outcomes, and reflect on and learn from the process.”

SAC Conference 2018


SAC Conference 2018

CLINICAL REASONING PROCESS

The member collected sufficient information.

The member applied background/clinical information in the decision making process.

The member linked information from one phase of intervention to the next.

The member took the patient’s context and situation into account in all areas of intervention decision making

Theoptions

member considered options and provided a reasonable rational to eliminate

The member was flexible in their approach regarding the patient , their needs or other intervention options

The member provided a reasonable rationale to explain why they did what they did

The member’s judgement was reasonable


SAC Conference 2018

“What a hypothetical, typical member who exercises average care, skill, and judgment would do in similar circumstances and thereby serve as a comparative standard.”

Definition of Reasonable:

2018‐04‐05

6


The Tool and the Guide were reviewed by 5 stakeholder focus groups:

• Managers and Administrators

• Speech Language Pathologists

• University Clinical Educators

• Members recently peer assessed (SLPs and audiologists)

• Peer Assessors (SLPs and audiologists)

The groups recommended changes and additions which were incorporated.

They agreed that the content Clinical Reasoning Tool was valid,

that is, it measures clinical reasoning.

SAC Conference 2018


Trial 1 ‐ 2016

Trial of the Clinical Reasoning Tool (CRT) with Peer Assessors and volunteermembers to determine:

1. The value of measuring clinical reasoning

2. Opinions regarding the ability to measure it

3. Effectiveness of the Tool to measure clinical reasoning

The PAs and volunteers administered the tool with each other, determined clinical reasoning and completed pre and post surveys.

SAC Conference 2018


Trial 1 – 2016 Results

Statistical results of pre and post surveys showed:

Importance of clinical reasoning skills

Include clinical reasoning in peer assessment

Tool is valid

Peer assessors can evaluate clinical reasoning

? Reliability of the Tool

SAC Conference 2017

2018‐04‐05

7


• Changes to the Tool:

– Reconsider “telling the patient’s story” too open‐ended

– Some wording changes

• Gather additional data

Document examples of inadequate reasoning.

– Collate all responses and discuss with the Quality Assurance Committee

– Refine with peer assessors to develop a reliable scoring system

• Develop a report template including clinical reasoning

SAC Conference 2018

Testing the Effectiveness of the Tool

SAC Conference 201820


2017 Peer Assessment Clinical Reasoning Tool Trial

Goal

To test the effectiveness of the Clinical Reasoning Tool in an authentic situation with both professions across all practice areas


2018‐04‐05

8


Participants

• 50 members were randomly selected for peer assessment

• 45 members completed the Clinical Reasoning Tool

• 3 French and 42 English administrations

• 9 Audiologists

• 36 Speech Language Pathologists– 9 provided adult services

– 8 provided preschool services

– 9 provided services to school boards

– 10 provided services through SHSS, LHIN and/or private



Participants

• The Quality Assurance Committee agreed that individual member results will not be disclosed.

• Aggregate data will be shared with the committee and the members participating in peer assessment at the conclusion of the trial.



Method

• Two Clinical Reasoning Tools were completed– The member selected one patient record (chart)

– The peer assessor selected another patient record

• Both administrations of the Tool were audio recorded

• The peer assessor scored the Tools


2018‐04‐05

9


Measures:

1) Pre site visit member survey

2) Post site visit member survey

3) Post site visit Peer assessor confidence survey

4) Inter‐rater reliability

5) Member and peer assessor comments




Member Survey Results

Surveys used a 5 point Likert Scale

2 Tailed t‐test used to measure change between pre and post

• Importance of Clinical Skills – pre and post 5/5

• Entry to Practice skill – lower post CRT (4.4)

• Part of PA Process – higher post CRT (4.4)

• Part of SAT Process ‐ higher post CRT (3.9)


2018‐04‐05

10



One question showed a significant change from pre to post

• Do you think a measure of clinical reasoning skills can be valid(it actually measures clinical reasoning)?

2 Tailed t‐test t = 2.2953, df = 44, p = 0.03153




• Reliable ‐ higher post CRT (3.3)

• Peer Assessor can evaluate ‐ higher post CRT (3.7)

Post Survey Questions

• Effective Tool – 3.5 (agree)

• Help Clinical Reasoning – 3 (not sure)



Member Comments:

• This is a critical skill which is not always reflected in our documentation, but should be going on ‘behind the scenes’.

• CRT could be an effective tool – will mainly depend on reliability. It was a positive experience with the peer assessor.

• I was initially nervous about the CRT, but the administration was straight forward and it provided me and opportunity to reflect on my clinical reasoning.

• I would really like to see it incorporated into the mentorship process with new graduates.

• I am curious about how sensitive and valid the tool is, and look forward to hearing more about it in the future.


2018‐04‐05

11


Inter‐Rater Reliability

The audio recordings were sent to a second peer assessor working in the same practice area for scoring.

Results



Peer Assessment Confidence Survey (Post administration)

“How confident are you in your determination of Clinical Reasoning?”

5 point Likert scale to measure responses

Results:

N = 44

Mean 4.6 – between ‘confident’ and ‘very confident’

Range 3‐5



Peer Assessor Comments:

• I did not feel as though I asked enough questions even though I determined this member's clinical reasoning.

• The process did not “flow” for me, however I believe that I have strategies to help with my next one, especially after listening to the recording.

• Going forward I need to focus more on active listening rather than commenting.


2018‐04‐05

12



The CRT had eleven questions

Number of CRT questions

Average Questions asked (M) 6

Average Questions asked ( PA) 6

Maximum asked (M) 11

Maximum asked (PA) 10

Minimum asked (M) 1

Minimum asked (PA) 2



Question Format

Questions verbatim

Questions changed intent kept

Questions changed intent changed

Average Questions (M) 4 2 0

Average Questions ( PA) 4 2 0

Maximum asked (M) 9 8 0

Maximum asked (PA) 10 7 1



Administration Timings

Time to complete CRT

Average time 12.29 (M+PA)

Maximum time (M) 32.22

Maximum time (PA) 25.06

Minimum time (M) 3.41

Minimum time (PA) 4.55

2018‐04‐05

13


Most commonly asked questions included:

• Tell me about the patient. 87%

• What makes this an interesting

or challenging case? 80%

• Knowing what you know now, is there

anything you might do differently? 95%



Clinical Reasoning Tool Results:

• 41 members found to have clinical reasoning

• 2 members inconclusive findings (PAs did not agree)

• 2 members were found not to have clinical reasoning



2018‐04‐05

14

Next Steps

2018 Peer Assessment Process

• Administer two CRTs (member and PA choice) per member

• Choose two current patients

• Simplify the layout of the CRT form

• Develop standardized introduction for PAs

• Simplify question 1 and 4

• Audio record all administrations


Next Steps


• If PA doubts clinical reasoning, audio recordings will be sent to a second PA in the same practice area for review

• After discussion, the original PA will make a determination and link reasons to the clinical reasoning constructs:

Collect Apply

Link Patient context

Options Flexible Reasonable


Next Steps


• Report will be sent to the member for review and comment.

• Report and member’s response will be sent to the Quality Assurance Committee

• The Committee will decide on remediation


2018‐04‐05

15

Next Steps

Nine Discussion Areas of the Tool:

1. Briefly give me some background information about this patient

2. What is/was unique about this patient?

3. How did the background information direct your assessment?

4. Lead me through your assessment process

5. How did the assessment results help you develop your recommendations and/or management plans?

SAC Conference 2018

Next Steps

Nine Discussion Areas (continued):

6. How did/will you decide to change or move onto another section of the

management plan?

7. Did you involve others in intervention? How did you come to that

decision?

8. Lead me through your discharge process. How do/did you decide

whether your patient needed further audiology/SLP or other professional

intervention?

9. Knowing what you know now, is there anything you might do differently?

SAC Conference 2018

Remediating Clinical Reasoning


2018‐04‐05

16




Our thoughts so far . . .

The College will:

• Develop an educational program on clinical reasoning

• Train the Quality Assurance Committee in the principles and potential methods of clinical reasoning remediation

• Train the Peer Assessors to provide direct clinical reasoning remediation



Potential Clinical Reasoning Remediation Program

• Discuss the processes that make up clinical reasoning:

Collect Apply Link Patient context

Options Flexible Reasonable

• Generate options and decide on the best solution

• Relate clinical reasoning to current patients

• Relate clinical reasoning to patients at different phases of intervention

• Use reflection as a process connecting it to reasoning


2018‐04‐05

17


Members will show one the following:

• Clinical reasoning

• Clinical reasoning in most but not all areas

• Minimal clinical reasoning

• No clinical reasoning



Remediation will differ for members who show:

• Clinical reasoning in most but not all areas

• Minimal clinical reasoning

• No clinical reasoning



Potential remediation strategy

a) Clinical reasoning in most but not all areas

• Send the member the Clinical Reasoning Remediation Program.

• Member prepares patient records focussing on the targeted area clinical reasoning.

• Connect with the peer assessor using the virtual communications phone/skype/etc.

• Administer the Clinical Reasoning Tool on the prepared records.

• Peer Assessor determines if the member shows clinical reasoning in all intervention areas.


2018‐04‐05

18


Potential remediation strategy

b) Minimal or no clinical reasoning

• A peer coach will be assigned to meet with the member

• The member will be sent the Clinical Reasoning Remediation Program.

• The peer coach will review and discuss the program with the member.

• The member will prepare current patient records at different phases of intervention.

• The Clinical Reasoning Tool will be administered.

• Peer Assessor determines if the member shows clinical reasoning in all intervention areas.




We would like to thank the clinical coordinators from the School of

Communication Sciences and Disorders at Western University for

their support, advice and knowledge.

PRECEPTOR EDUCATION PROGRAM

UNDERSTANDING AND FOSTERING CLINICAL REASONING

Moosa, T. & Ferguson, K. (2015). Understanding and fostering clinical reasoning. In Kinsella,E.A., Bossers, A., Ferguson, K., Jenkins, K., Bezzina, M.B., MacPhail, A., Moosa, T., Schurr, S.,Whitehead, J. & Hobson, S. Preceptor Education Program for health professionals and students.(2nd ed.) www.preceptor.ca. London, ON: The University of Western Ontario

Clinical Reasoning: Evaluation and Remediation


2018‐04‐05

19

Contact Information

Alex Carling, [email protected]

Sarah Chapman‐Jay, [email protected]

www.caslpo.com


sac clinical reasoning tool - copy · “clinical reasoning describes the process by which members...

Documents