prescribing skills assessment background, purpose and structure professor simon maxwell medical...

Prescribing Skills AssessmentBackground, purpose and structure

Professor Simon MaxwellMedical Director, Prescribing Skills Assessment

[email protected]

Slide compilation © 2011 The British Pharmacological Society & Medical Schools Council. All rights reserved.

http://www.chms.ac.uk/default.html

Background

Prescribing medicines

the most common intervention (for good or bad) that most doctors make to improve the health of their patients

Concerns about FY1 Prescribing• Illing et al (2008)

– How prepared are medical graduates to begin practice? A comparison of three diverse UK medical schools.

– Prescribing considered to be the key problem– Over 80% graduates failed a prescribing assessment

• Skills for Health Report (2009)– Junior doctors in the NHS: Preparing medical students for employment and

post-graduate training– 65% of respondents considered prescribing was an issue

• EQUIP Study (2009)– An in depth investigation into causes of prescribing errors by foundation

trainees in relation to their medical education. – 125,000 prescriptions in North-West England – error rate 9%

• PROTECT Study (2011)– 45,000 prescriptions in Scotland – error rate 7%

An in depth investigation into causes of prescribing errors by foundation trainees in relation to their medical

education. EQUIP study. Dornan et al, 2009

Available at http://www.gmc-uk.org/about/research/research_commissioned_4.asp

124,260 medication orders

http://www.gmc-uk.org/index.asp

Safe Prescribing Working Group (2008)

• Agreed learning outcomes of undergraduate medical education in relation to prescribing

• National e-Learning materials to support medical students

• Development of an assessment that might demonstrate whether the outcomes have been met

• National prescription chart

• Access to the BNF for students

Available at www.medschools.ac.uk/AboutUs/Projects/Documents


Those competencies (and more) included in the new outcomes-based version of Tomorrow’s Doctors

See paras on outcomes:3, 8, 12, 14, 16, 17, 18 (Appendix 1), 21, 23

See paras on process and assessment:27, 28, 29, 37, 21, 86, 108, 109, 112, 117, 171

“We realise that meeting these outcomes and standards will be challenging” – Foreword by Prof Peter Rubin

Purpose

Potential value of a national assessment

• No validated, reliable and widely accepted measure of prescribing performance currently exists

• All medical schools are required to demonstrate that their students are competent (Tomorrow’s Doctors para 117)

• NHS organisations are starting to set up multiple ad hoc assessments and would value a recognised assessment

• A national prescribing assessment would – pool academic resources– serve to raise and unify standards– be relevant prescribing groups: other grades of doctor, nurses,

pharmacists, dentists and other health professionals

Why assess prescribing?• Clinical governance/patient safety reasons

– Prescribing is a fundamental skill of a doctor

– Assessment might be a marker of competence to enter into (or continue in) clinical practice

– Protection of patients – patient safety

• Educational reasons– Measure the success of training

– Identify the need for improvements in training

– Contribution to training (formative assessment)

– Foster higher standards of attainment in prescribing

Prescribing Skills Assessment

• Developed jointly by the Medical Schools Council and British Pharmacological Society

• Summative (pass/fail) assessment of knowledge, judgement and skills related to prescribing medicines


Basic principles of the PSA

• It should be passed before qualification and subsequent assumption of NHS prescribing responsibilities

• It will be available to be taken during the final year and can be repeated if necessary

• It will be delivered online

• Candidates will have access to the British National Formulary

• It will assess competencies in prescribing relevant to Foundation doctors that map onto the outcomes identified in Tomorrow’s Doctors 2009

• Will test skills and deductive powers (as well as knowledge) relevant to early postgraduate practice

Development

Steering Group

• Medical Schools Council

• British Pharmacological Society

• GMC

• Postgraduate Deans

• MSC-Assessment Alliance

• BMA Medical Students Committee

• NHS Employers

• Prescribing Skills Assessment Project Team is responsible for delivering the key work streams

Steering Group

WorkStream 1ITEM BANK

Project Team

WorkStream 2

DELIVERY

WorkStream 3GUIDANCE

Funding

• The project has received initial funding from the Department of Health to support the development and piloting of the assessment

– Employment of an assessment officer to support development of the item bank

– Development of online delivery system

– Initial piloting

Progress

Progress• Steering group met 8 times

• Paper-based pilots with student feedback– 11 medical schools (summer 2010)

– Around 1,300 students

• Blueprint agreed

• Eight question item styles defined

• Templates for writing question items developed (Ppt)

• Department of Health funding secured (February 2011)

• Assessment officers appointed (summer 2011)

Student Feedback (Summer 2010)• This assessment was an appropriate test of the prescribing skills expected

of a final year medical student (4.1)

• The material in this assessment was relevant to the task expected of a Foundation Doctor (4.2)

• I was able to answer most of the questions in this assessment (3.8)

• The questions were set at the right level of difficulty (4.0)

• I do not have the knowledge and skills to enable me to pass a Prescribing Skills Assessment of this kind (2.0)

• Having access to the BNF during the assessment was very helpful (4.5)

• The layout and presentation of the questions was easy to follow (4.2)

Feedback based on Likert scale (1 = strongly disagree to 5 = strongly agree)

Structure

Make a diagnosis

Establish therapeutic goal

Choose the therapeutic approach

Choose the drug

Choose the dose, route and frequency

Choose the duration of therapy

Write the prescription

Inform the patient

Monitor drug effects

Review/alter prescription

PATIENTHistory - examination - investigations

Knowledge Judgement Skill






Knowledge Judgment Skill

Knowledge Judgment Skill



1

2

3

4

5

6

7

8

9

10

Prescribing sub-competencies

Prescribing Assessment

Station 12Data

Interpretation

Station 2Prescribing


Station 9Prescription

Review

Station 5Communicating

Information

Station 6Calculation Skills



12 stations

TOTAL = 60 mins

(100 marks)

Station 11Drug Monitoring

Station 3Prescription

Review

Station 10Adverse Drug

Reactions

Station 4Planning

Management

Prescribing Skills Assessment Blueprint

Station Description Marks Comments

1 Prescribing 1 10 1 question item of 10 marks


3 Prescription Review 1 8 2 question items of 4 marks each

4 Planning Management 8 2 question items of 4 marks each

5 Communicating Information 6 3 question items of 2 marks each

6 Drug Calculation Skills 8 4 question items of 2 marks each



9 Prescription Review 2 8 2 question items of 4 marks each

10 Adverse Drug Reactions 8 4 question items of 2 marks each

11 Drug Monitoring 8 4 question items of 2 marks each

12 Data Interpretation 6 3 question items of 2 marks each

TOTAL MARKS 100

Prescribing Skills Assessment – Format

Summary

• Prescribing is a key clinical skill for all doctors

• There is evidence of poor prescribing practice amongst new graduates and others

• A reliable and valid assessment of competence would be helpful for medical schools and the NHS, and serve to stimulate improved educational experiences for students

• The Prescribing Skills Assessment will enable graduates to demonstrate that they have achieved a basic level of competence

Prescribing Skills AssessmentQuality control, peer review and delivery

Professor Simon MaxwellMedical Director, Prescribing Skills Assessment

[email protected]


Work streams

• Question item bank development

• Online delivery platform

• Guidance and communications

Quality control

Item Bank and Test Development

• Aim to deliver around 560 items by April 2012

• Question item authors (around 60) currently being recruited– British Pharmacological Society

– Medical Schools

– Clinical pharmacy

• Training events (4) planned for September 2011

• Items initially collected as Ppt files but will move to an online authoring system (submitted to [email protected])

• Peer review event over 2 days in February 2012

• Assessment Board first meeting April 2012

• Processes for assembling (e.g. standards) and standard setting assessments being developed

mailto:[email protected]

Unformatted content (scenarios and MCQs) provided by medical schools

Steering Group members and volunteers apply content to PSA templates (ppt files)

Volunteer item writers write assessment items using the 8 station type PSA templates

Project team conducts initial review of items, checking for completeness, face validity, relevance to station type etc. Entered into item log with appropriate metadata. Sent for peer review.

Project team collects item performance data based on performance amongst randomly selected students.

Peer review workshops to (i) conduct face validity and cover up test of scenario and correct answer, (ii) review and edit the item to remove any ambiguity and eliminate detractors in the answer options, and (iii) decide whether to refer the item for standard setting.

Assessment Board standard sets V3 items with reference to performance data and the definition of a borderline student.

Project team enters V4 items into live bank

V1

V2

V3

V4

Assessment Board monitors and maintains the bank in relation to current prescribing policy and practice, current relevance to the FY1 role, and balance of items across station type, domain, patient group and drug type.

PSA Item Bank

Delivery

Technical capacity and delivery

• Online delivery system– developed by University of Edinburgh (based on OSCA system)

– Authoring and editing tools, delivery of question items, assessment construction tools, marking, reporting, item analysis

• Piloting strategy– Preliminary online pilots (2) held May 2011

– Further online pilots being offered (Feb-June 2012) - to test the online delivery in a range of schools including those with and without experience of delivering online assessments*

– Engaging with schools during 2011/2012 on local issues relating to the capacity to deliver this online assessment

– Full scale pilot involving all medical schools (2012/13)

Guidance and Communication

Guidance and communications• Engaging with all medical schools on the development

of national policies to guide the implementation– Running of assessment events

– Retakes and remediation

– Appeals

– Candidates with disabilities

• Development of appropriate contacts– Assessment and IT leads

• Website development – www.prescribe.ac.uk/psa

Medical Schools

Medical Students

Other stakeholders

www.prescribe.ac.uk/psa

Timelines

• Question item author training – Sept 2011

• Major peer-review event – Feb 2012

• Pilots to be offered to schools – Feb–Jun 2012

• Full national pilot – 2013

• Implementation - 2014