assessment zone issue 1 2013

8/13/2019 Assessment Zone Issue 1 2013

1/64

assess ment zone

Assessment and Evaluation in Health Professional Education

ISSUE 1 2013


2/64

FRONT ROW ( from left ): Deanka Preston, Kate Gray, Jodie Atkinson

MIDDLE ROW ( from left ): Simone Thurkle, Debra Jeavons, Lubna Al-Hasani,Sandy Presland, Alison Creagh

BACK ROW ( from left ): Christopher Etherton-Beer, Elisabeth Feher


3/64

ContentsEditorial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

FeaturesStudying at UWA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Evaluation metaphors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

When assessment goes wrong horror stories!. . . . . . . . . . . . . . . . 11

Blueprinting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Social media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Does your OSCE make the cut?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Direct observation checklists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Assessment on the go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

InterviewsDealing with difficult students (Christine Adams) . . . . . . . . . . . . . . . 12

Different perspectives on midwifery education . . . . . . . . . . . . . . . . 22(Sadie Gerarghty, Sarah Bayes and Dianne Bloxsome)

Assessing physicians of tomorrow (Claire Harma). . . . . . . . . . . . . . . 36

Learning issuesAssessing clinical competency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Making sense of portfolios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Assessing registered nurse performance. . . . . . . . . . . . . . . . . . . . . . 49

Puzzle page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Crossword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Useful websites and resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Quiz and crossword solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60


4/64

A S S E S S M E N T Z O N E

2

The authors acknowledge the Traditional Owners of the land on which we work and live.

Informed consent was obtained from each health professional interviewed for this magazine.

Copyright the authors

Assessment Zone is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 Australia License (2013).

Contact: Associate Professor Zarrin Siddiqui ([email protected])


5/64


3

Welcome to the inaugural issue ofAssessment Zone Assessment andEvaluation in Health Professional Education .The IMED 5802 student group for Semester2, 2013, are proud to present this magazinefor our Capstone Experience Group Project.This group project forms a major assessmentfor each student undertaking this unit. It isthe culmination of eleven interprofessional

students electing to work collaboratively todesign and compile a magazine featuringtopics related to assessment and evaluationgeared towards health professional educators.

The composition, core ideas and philosophyfor this magazine align with our learningoutcomes for the unit IMED 5802 Principlesof Assessment and Evaluation. These learningoutcomes are:

Explain the role of assessment in thelearning process

Discuss techniques of measurement Explain the validity and reliability in

assessment and evaluation Describe the disadvantages and

limitations of different assessmenttools

Design and critique assessment andevaluation tools

Assess the quality of an examinationby looking at its constituent parts

Explain the principles and models ofevaluation

We have also committed to applying andmaintaining the UWA educational principlesthroughout the magazine items and production.

The conception of this magazine has notbeen without its challenges. We formed ourstudent group early in the semester, manyof us meeting for the first time and with

varying work and study schedules. With atimeline in place, we met weekly after classto garner concepts, ideas, and view pointsfor the magazine items. Meetings weredocumented, and communication externalto the classroom was posted via Facebook,the UWA Learning Management System andstudent email accounts. The magazine name,design and format were decided upon by avoting system. We utilised Joomag to viewmagazine templates and Dropbox on lineapplications as a central work space to submitand review individual student creations. Wecompleted the process by accessing theexpertise of Uniprint to assist with formatting,design and publication of the final product.We would like to acknowledge Associate

Professor Zarrin Siddiqui, unit coordinator forproviding direction to us as required.

On behalf of the student group, I hope thismagazine resonates with you and your roleas an educator, and that you benefit from thepersonal stories, feature articles, interviews,and resource tools that we have produced.

Deanka Preston

Ed i tor i al


6/644

Jodie Atkinson is aRegistered Midwife andNurse. She currently worksas a Clinical Facilitator atKing Edward MemorialHospital and Edith CowanUniversity, supervisingboth undergraduate andMasters Midwifery students.Additionally she works asa sessional lecturer, tutorand research assistant atECU. Jodie is completingher Masters in HealthProfessional Education.

Alison Creagh has lovedmedical teaching since 1979,and has been employed asMedical Educator at FamilyPlanning WA since 2000. Shehas also worked for manyyears in general practiceand womens health. Alisonis completing her Mastersin Health ProfessionalEducation.

Christopher Etherton-Beeris a Geriatrician and ClinicalPharmacologist at TheUniversity of WesternAustralia and Royal PerthHospital. Christopher iscompleting the GraduateCertificate in HealthProfessional Education.

Elisabeth Feher is a MedicalEducation Registrar at SCGHand Advanced Trainee inGeriatric Medicine. Hercurrent educational projectsinclude running an educationprogram for Internationalmedical graduates andIntroducing simulation intoBasic Physician Training.Elisabeth is completing theGraduate Certificate in HealthProfessional Education.

Kate Gray is aPhysiotherapy Coordinatorof Medical and StudentServices at Joondalup HealthCampus. Completing theGraduate Certificate in HealthProfessional Education.

p r o

f i l e

s


7/645

Debra Jeavons is a StaffDevelopment Midwiferesponsible for clinicalexperiences of student nursesand midwives (undergraduateand postgraduate) andgraduate midwives, providesprofessional developmentand orientation of all staff.Facilitator of interprofessionalobstetric education withinthe health service. Debra iscompleting the GraduateCertificate in HealthProfessional Education.

Janet Vince is a StaffDevelopment Educator,Swan Kalmunda. Extensivebackground in Critical Carespecialising in ICU. Shehas experience in nursingeducation and is presentlyworking on the developmentof the interprofesionaleducation project for FionaStanley Hospital. Janet iscompleting the GraduateCertificate in HealthProfessional Education.

Lubna Al-Hasani is aninternational full timestudent in Master, HealthProfessional Education atThe University of WesternAustralia. Head, Training andStaff Development in RoyalHospital, Muscat, Oman.

Sandy Presland is aClinical Nurse RecoveryRoom, Fremantle Hospital.Responsibilities includeclinical supervision and teamleadership of undergraduate/ postgraduate nurses.Career direction: nursingeducation and leadership.Sandy is completing theGraduate Certificate in HealthProfessional Education.

Deanka Preston has beena Registered Nurse for 21years and in the last 10years has worked in thestaff development areain metropolitan teachinghospitals. She is currentlyActing ProfessionalDevelopment Educatorfor the Graduate NurseProgram at Swan KalamundaHealth Service. Deanka iscompleting her GraduateCertificate in HealthProfessional Education.

Simone Thurkle is a StaffDevelopment Nurse at SCGHEmergency Department.She has completed a PostGraduate Certificate inEmergency Nursing andis currently completing aGraduate Certificate in HealthProfessional Education.


8/64


6

s an international post graduatestudent in The University of WesternAustralia (UWA), my experience has beenenriched both personally and professionally.Coming from a different background has mademe value the experience from a differentperspective. Also, embracing the culturaldifferences has lead me to critically analyseand adopt the best practices.

The first thing about Australia, which I founddifferent than my country, was calling peopleby their first name. I still struggle doing this,even in emails.

Also, being a student with a differentbackground (both cultural and religious)was difficult at the beginning. But unlikeother countries I felt welcomed not only inthe university, but in the community too.However, I do find difficulties in gettingHalal food in the university campus, but it is

plentiful in the community.

In terms of professional growth, I have foundthe university environment collaborative and

Studyingat UWABy Lubna Al-Hasani

stimulating. UWA is full of resourcesto assist the students in their studies such asthe Study Smarter sessions. Nevertheless,although they are very helpful, I think UWAshould increase the number of tutors insessions like Write Smarter, to cover asmany students as they can. Compared tomy experience in my previous university,the number of tutors available for assistancewere more and for a longer time.

At UWA the teaching methods are varioussuch as face to face lectures, workshops,and the online modules. This integrated

curriculum creates a comprehensive andfruitful learning experience. Just being in asetting amongst a group of diverse healthcare professionals with different backgrounds

A

In terms of professional growth,I have found the universityenvironment collaborative andstimulating.


9/64


7

and experiences is a valuable asset. Besides,the lecturers are very supportive andcooperative, helping the students to reachtheir potential in professional growth.

The concept UWA is adapting in its teaching,is to create future professionals who are greatteam players. Most of the assignments areperformed in teams and it is this approachthat leads to professional maturity.

Moreover, I have been exposed to severalforms of formal and informal assessments.Examples of these assessments areassignments, student led presentationsand reflections. Interviewing professionals

from the real working environment andcritically analysing their organization isanother assessment example. I believe

that the assessments, even though theytake me away from my comfort zone, havebeen helpful to discover my capabilities,and therefore for me, to acknowledge mylimitations and improve on them.

Assessments at postgraduate level aremore in depth, comprehensive and basedon evidence-based practice. This promotescritical thinking and develops practicalimplications in the real world. Also receivingfeedback in the middle of the units assuresthe transparency of the university, as it isopen for constructive feedback.

Going through this experience is not aneasy path, and I have found it a difficult,yet an exciting experience. However, it isthe difficulties that makes a person andshapes their future. This is an extra-ordinaryexperience and it will assist me in my

journey as an educator and a health careprofessional.

I think UWA should increase

the number of tutors insessions like Write Smarter

Oman


10/64


8

rom the day my daughter sat behind a wheel of a car it was not just about starting thecar and going forward in a straight line. There were road conditions to evaluate such as

traffic hazards, road works and pedestrians. We had to consider the timing of the lesson andreadiness to learn on my daughters part. At the end of each learning session we would lookback and assess how she went, then analyse her progress. From there we would plan the nextlesson and make any changes to the teaching plan such as practicing reverse parking or perfectingparallel parking.

As my daughters driving skills improved we would give her more challenges to keep her engagedand increase her skill. But there came a point she reached a plateau and was not progressing.Evaluating our teaching abilities, we realised that as parents, we were not the best teachers. Sowe brought in expert teaching skills with the use of a driving instructor. With access to improvedteaching tools, my daughter went on to passing her driving assessment.

When I supervise students and junior nurses, its a little like learning to drive. They will often joke

that they are on their L plates. We take them on a learning journey letting them steer the wheel,but if we do not evaluate the teaching tools used along the way, they may not progress as well.Poor performance can be the result of an inadequate teaching program. Evaluating the teaching isjust as important as assessing the learning.

evaluat ion student thoughts, experiences and perceptions explained with the help of a metaphor

by Sandy Presland

F

todrive

Learning


11/64


9

ne day myhusband and Iset out to find

the perfect pup.

The first thing wehad to do was toresearch breeds thatwere appropriate. Weassessed a varietyof types in terms ofsize, types of hair,ease of training andtemperament. Afternarrowing the searchdown, much to myhusbands surprise,a toy poodle was thewinner! Next thing wehad to do was to assesswhere was the best

place to purchase apuppy from; breeders versus pet shops. Wereceived feedback from family and friendsand came to the conclusion that a breederwould be better to suit our needs. Finallywhen we found a breeder that suited, we hadto assess which particular dog would suit usbest. We looked from dog to dog and playedwith them one by one, spending time with

them to make our decision. After playing withthe hyperactive one, the yappy one, the sulkyone... We found him! The little fluffy, quietbut happy and playful one: Lenny.

The perfect pupBy Kate Gray

O After a few sleeplessnights, cleaning up a fewsurprises, puppy schooland park excursions, my

husband and I reflectedon our purchase and howour lives had changed.We evaluated thewhole experience andwholeheartedly came tothe conclusion that gettingLenny was one of the bestthings we had done!

I consider selection ofstudents supervisors as animportant task. I look at thesupervisors experience,training and motivationand assess their abilityto adequately supportthe students learning.

After the placement has finished, I reflecton the students experience, the feedbackthey received and their final assessmentresults. I receive feedback specifically fromthe student and the supervisor and afterconsidering all of this information, I canevaluate the effectiveness of our studentprogram. The ultimate outcome is that

hopefully we have positively shaped thestudent into becoming a competent andconfident health professional.



12/64


1 0

My DaughtersJourneywo years ago my daughter decided

to move schools. She was unhappywhere she was and had some ideas

as to where she wanted to go. Together welooked at why she wanted to move to makesome objective decisions.

Having identified various points, we listed thequalities we wanted to see from the schoolswe were to look in to. From here we beganour research.

We searched the internet, contacted variousschools to review their prospectuses, liaisedwith students and parents from various

by Janet Vince

schools and finally selected a few of whichwe wanted to visit.

From here we were able to make a formativeassessment of what each school couldoffer, guiding our choice. As I evaluated thechoice we made, I based it on continuousassessment of my daughters performanceand happiness.

When planning my education programs, Iam aware that everyone has a choice, justas we had. If I dont continuously assesswhat I am providing and change according tofeedback, my evaluation will reflect a lack of

interest and engagement

in the programs. Ivalue feedback throughsurveys, questionnaires,evaluation of studysessions and wordof mouth. This givesme vital informationto change and adaptaccording to students

wants and needs. Myevaluation is the successof overall outcomes.


T


13/64


1 1

The VIVA is mine and several of mycolleagues biggest horror stories.

Different students get different assessors. Inmy case the assessor interacted with me andargued about one of my answers when shewas not meant to interact at all. This got me offtrack during the exam and meant that I did notdo as well as I could have. My colleague had asimilar experience but in her case she knewthe answer the assessor wanted and whichwould have given her a pass but that is notwhat she wanted to say or what she believed.Another colleague got so distressed by an

oral exam she froze and got flustered andfailed even though she knew the correctresponses. Several assessments are doneat one time with the same question,however each assessor has their ownindividual philosophies and interpretationof the question. Students were aware ofwhich assessors marked easier or higherthan others. You knew that if you got

certain assessors you would receivea lower mark even giving the sameanswers. This assessment appears tohave caused the most distress.

Assessment can be a terrifyingexperience for both the assessor and the assessee. Our group reflected on their ownhorrifying experiences of the past. Read on if you dare!! What we recommend NOT doingto your students...

Assessment stories

My most awful and instructive experience was at the end of my medical degree, wheneverything (back then), depended on passing the final exams, of which there were

quite a number, both clinical and written. I saw a high proportion of my fellow students fallingapart in a variety of ways. Some used lots of alcohol, others took sedatives, some others tookillicit substances, and some became extremely anxious and depressed, including one who wasadmitted to hospital. I did learn from this: personal coping skills included taking a fatalist approachI will do my best, and if thats not enough, the worst thing that can happen is to have to repeat,professionally I learnt, amongst other things, that continuous assessment would be better!

H o r r o r

My hor r or s t or y i s about a f r i end of mi ne. She w as a f i nal y ear s t ud ent

and on pr ac at a hospi t al. She had t hought a c omment she had hear d f r om a pr ev i ous s t ud ent say i ng t hey w ould r at her sc r ub t oi le t s w as an ex agger at i on unt i l she ex per i enc ed i t

f or her se lf . T her e w er e 2 super v i sor s . T hey w ould c ont r ad i c t w hat t he ot her one sai d , and ev en c ont r ad i c t t hei r ow n ad v i c e t he f ollow i ng d ay . T hey gav e mai nly negat i v e f eed bac k . F r om a c onf i d ent , happy per son my f r i end

bec ame v er y s t r essed and d r ead ed eac h d ay w or r y i ng t hat she w ould f ai l and t her e f or e be unable t o g

r ad uat e . She w as not t old unt i l t he las t d ay t hat she had passed sub j ec t t o a spec i al c ons i d e r at i on f or m f r om uni . C onsi d er i ng she got a hi gh d i s t i nc t i on f r om ev er y ot her pr ac bef or e and af t e r w ar d s i t must say somet hi ng about t he ex per i enc e! T her e w er e 2 t hi ngs she lear nt f r om t hi s - pos i t i v e f eed bac k ( or at leas t c ons t r uc t i v e f eed bac k ) and a suppor t i v e lear ni ng env i r onment

i s w hat s t ud ent

s need t o f lour i sh, sec ond ly she lear nt how not t o t r eat s t ud ent s i n t he f ut ur e . ( by t he w ay t hi s happened r ec ent ly , not a long t i me ago).

by Jodie Atkinson, Alison Creagh and Debra Jeavons


14/64


1 2

Christine, we are writing a magazineon assessment and evaluation in HealthProfessional Education. Can you explainto our readers your current role inassessment and the learning process?

Primarily my role involves lecturing andsupervision of undergraduate nursingstudents. I am also involved in problemsolving particular cases on site that aredifficult for our facilitating staff. I am generallycontacted when a student is struggling with

their learning objectives on practicum, ormore importantly if there has been a nearmiss event such as a potential drug error.Also I am contacted by clinical facilitators todeal with student behaviour problems andstudents failing to attend practicum or arrivinglate. I generally go outto speak to studentsand address issues as

they arise. I rely on theassessment the clinicalfacilitator has made,and the documentation

provided. I cannot stress enough theimportance of documentation and that wehave evidence of a chain of events, not justa single record of an incident.

What issues do you experience regardingunderperforming students during clinicalplacement?

Generally the issues Im called out to can besorted out on site with a meeting and

developing a learning contract with someclear learning objectives. Specific goals aregraduated through their practicum, so by mid-way or end of practicum, the student knowswhat they need to achieve. If its behaviouralissues, such as attitude problems, the student

responds very quicklyto the threat of removalfrom practicum. I

always keep a recordof the issue, and thatit has been addressed.This is important in

Dealing with difficult students

An Interview with Christine AdamsChristine Adams is currently a Clinical Placements Officer andLecturer for the Notre Dame University Undergraduate Nursing

Program. She also works clinically as a registered nurse in abusy post anaesthetic care unit. She previously held a positionat Curtin University as a Clinical Facilitator for undergraduatenursing students.

I cannot stress enough the importance of documentation

and that we have evidence of achain of events, not just a single

record of an incident.


15/64


1 3

terms of the student appealing, should theyfail their practicum. Documentation is crucialbecause we have students come back and

say they didnt get a visit from anyone, theywerent aware that there was a problem orthey dont see why they should be failed. I have had a couple of recent problems wherestudents have been involved in near missevents involving potential administrationof the wrong medication. Any incidentthat involves patient safety or an adverse

event means removal of the student frompracticum. I will inform the student thatthey are to leave the health site premisesimmediately and make an appointment withme onsite at the university, and that itspotentially a fail of the practicum and mayinvolve repeating the unit. This is actuallya legal requirement because as healthprofessionals we have a duty of care toprotect the safety of our patients.

Is there any advice you can give to ourreaders regarding strategies to supportfailing students?

In terms of supporting a failing student, or anystudent for that matter, the most importantprinciple is formative assessment. Theassessor must sit down mid-way through thestudents practicum and give feedback on howthey are going, if their learning objectives arebeing met and if not, areas they can improve. Ithink the importance of formative assessmentcertainly supports your final decision. At theend of a practicum when you are grading the

students summative assessment, you havethe backup of feedback discussions mid-waythrough the practicum to support the finalassessment. I can say to the student that

you did not meet the learning objectives weset nor improve in the areas we identified asbeing weak and for that reason I am failing

you as I am unhappy with your level ofcompetence.

Without identifying and documenting astudents weakness early, it can give thestudent grounds for appeal. I think this isprobably the biggest advice I can give readers,to make sure you identify any areas of poorperformance early in their placement. I really

think formative assessment supports thestudent in learning but also supports theclinical supervisor or assessor in the processof dealing with students who areweak or failing.

How do you deal with student reactionswhen they are informed they areperforming unsatisfactorily or failing?

I dont get too much into the emotional sidewhen it comes to informing a student theyare failing. I try to be empathetic but if itsa behavioural issue Im very clear about theexpectations when they go on practicum.All students sign a clinical practicum policywhich outlines behaviour we expect. If astudent is removed from practicum due topoor behaviour, they are reminded they havenot met the expectations of the university.I appreciate they may be upset and forthat reason I ask them to write a reflectivepaper on how they could manage thesituation better, prior to making a follow-upappointment to see me. In that way they are

coming back to me when the emotionhas settled down and they have had time

continued overleaf


16/64


1 4

to reflect on theirbehavioural issues.The same principle

applies when dealingwith students removedfrom practicum dueto unsafe practice.When they are upsetthey have a barrier to receiving information.By waiting for the emotion to settledown they are more open to receivinginformation at the follow up meeting and

thats when I will formulate an action plan.At the meeting I will emphasise its not somuch a disciplinary measure but a learningmeasure and I want the student to be asafe and competent practitioner. I advisethe student that by reflecting on the criticalelements where they failed, and identifyingappropriate learning opportunities to meettheir objectives, they will make a betterprofessional down the track.

A lot has been said about failing to failstudents. Why does this happen and whatdo you think the consequences are?

Contributing factors for failing to failstudents include lack of mentor experience,time burden and unwillingness to dealwith a situation potentially fraught withemotion. Ive heard registered nursessuggest they cant be bothered failing astudent as it involves more work identifyingweaknesses, setting objectives andcreating learning contracts. Recently aregistered nurse said to me he had signed

off an incompetent student as competentbecause he wanted to avoid hurting herfeelings. I really think that is what it comesdown to a lot of the time.

It doesnt serve thefailing student to passthem. Instead it puts

them in an arena asa trained practitioner,where they are underpressure to perform.They do not have

the skills or knowledge to deal with anenvironment they are not ready for, and weend up with nurses out there practicing inan unsafe manner. By giving the student the

benefit of the doubt, we are not doing theprofession any service. It is worth noting thatin the United Kingdom trained nurses areheld accountable by the registration board forsigning off student competencies.

Christine, is there something you wishyou had learnt about assessment andevaluation earlier in your career?

We are now realising that the qualities thatmake a good clinical supervisor are quiteoften characteristics such as personalitytraits and I think research has caught up withthis. Students are more inclined to use theirinitiative and seek out learning opportunitieswhen their supervisor is welcoming andmakes the student feel part of the team. Soin education we are looking for people with apassion to teach, and the tools we can teachlater. Experienced health professionals whoare open to educating students need to beoffered opportunities to develop their clinicalsupervision and assessment skills. I think thatis what I really didnt get offered earlier on in

my career.

In terms of supporting afailing student, or any student

for that matter, the most important principle is formative

assessment


17/64


1 5

Are there any resources that havechanged your approach to assessmentand evaluation?

I was fortunate to attend the Art of ClinicalSupervision program offered by WA Health.I found it useful, especially focusing ondifferent learning styles and competencyassessment. Consolidating the knowledgeand tools the program provided, I now applythese strategies into practice. In the case of astruggling student I look at how that studentlearns so I can adapt the environment tomaximise their learning potential and applya structured framework for assessment ofcompetence.

Christine, as someone whose roleinvolves assessing students in clinicalpractice, what might be the one mostimportant piece of advice youd like toshare with our readers?

In terms of assessing students, rememberthey are students, not practitioners. I think aspractitioners we might have to step back andaccept students may take longer to completetasks, stumble along a bit and seek support.

What Im looking for when assessingcompetence is if the student is askingall the right questions and reflecting on

their practice. We need to assess studentcompetence for the level of training theyare at.

If you would like more information about theArt of Clinical Supervision program pleasecontact: [email protected].

Interview by Sandy Presland

Practical advice for supervisors: Provide formative assessment Develop an action plan Identify problems early Set achievable learning objectives Provide opportunities for feedback Document evidence to justify actions


18/64


1 6

BackgroundIn 2012, I introduced an activity at aclinicians training day at my work place. Afterviewing short video clips of sexual healthconsultations, each participant (doctors andnurses providing clinical training) rated thehealth professionals level of competencein the relevant skill, using the same methodof assessment as for trainees in our clinics.Assessment was made according to thecurrent four level scale:

Needs assistance

Developing competence

Competent

Mastery

A Learning Issue is a focused questiongenerated about issues related to assessmentand evaluation. Alison has examined the issueof the reliability of the assessment of clinicalcompetence.

Assessingclinical

competencyby Alison Creagh

For each skill, the same list of ideal attributeswas used as in our clinics.

Amongst the 15 clinicians present, there wasan unexpectedly wide range of assessments

for each of the five video clips. This led to aconsideration of the reliability of assessmentsof competence, and whether reliability couldbe improved.


19/64


1 7

does

shows how

knows how

knows

DiscussionMillers pyramid 1 of health professionalassessment is:

It seems widely accepted 2 that assessing

actual practice (does, in Millers pyramid) ismore predictive of work skills than assessingperformance in simulated or standardisedsettings (the shows how level). However,assessing competence in the workplaceappears more complex than assessing thelower layers of the pyramid.

A number of ways to improve the reliability

of assessments of competence have beensuggested in the literature.

Assessors can be provided with training toincrease consistency between them. 3 Forexample, all would observe aparticular scenario, rate the competence ofthe trainee, and then discuss any differencesbetween their assessments.

A commonly accepted strategy is to ensurethat assessments are made on multiple

occasions by multiple assessors. 4 Thishas the effect of smoothing out bothindividual assessor variations, and thedifferences in assessment resulting fromdifferent levels of complexity in the casesseen. Supervisors could identify traineeswho are not having sufficient assessmentsperformed, and facilitate further training andassessment sessions. 4 A practical difficultywith this strategy may be limitations onthe number of assessments that can beperformed, as they can be time consumingand costly.

Some authors have discussedgeneralizability theory as a method ofcalculating the reliability of competencyassessments. These calculations dependon a number of assessors seeing the samestudents performing in a number of cases,and then looking at the variation in marks

due to individual assessors, the variationbetween students, and the variationbetween cases, to work out the degreesof variation due to each of these factors.Using some standard rating scales, andsome devised for the study, they foundthat using more specific rating scales,aligned more closely with the assessorsways of thinking about trainees, were

more reliable than the standard, genericones. For example, some of the genericscales included features such as whethera trainee was at or above the expectedlevel. Assessors tended to avoid morepejorative ratings, and infrequently ratedtrainees as below the expected level.Scales that were found to be more reliableincluded those where the degree oftrust in the trainee was rated, and those

continued overleaf


20/64


1 8

which assessed the trainees level ofindependence to perform certain tasks.

Another recommendation is to considerwho the assessors should be. Shouldthere be 360 degree assessments, witheveryone working with a trainee providinga point of view? It seems that the mostreliable assessors are those who observe theparticular skills required on a regular basis. Forexample, ward clerks may usefully provideassessment information on communicationskills or on team work, but would not providereliable assessments of clinical skills. 3

Finally, what information should be providedto assessors to guide their ratings oftrainees? Some organisations have developeddetailed checklists, in an attempt to reducethe subjectivity of assessments. 3 Examplesof checklist items are trainee introduced

themselves or trainee scrubbed handscorrectly. In a comparison between theuse of detailed checklists and a global ratingof trainees, it was found that the globalrating was more reliable. 3 This is conciselysummarised:

perhaps performance is more than the

sum of its parts 3

ConclusionAssessing competency in practice is abetter predictor of actual work skills thanassessments of knowledge or of skills instandardised situations.

In summary, useful strategies include: use assessors who both observe and

perform the relevant skills on a regularbasis.

ensure that a broad range of assessorsprovide feedback and assessment onmultiple occasions.

train assessors in the consistent use ofrating scales.

use rating scales that are specific to thetype of competence being assessed,rather than generic ones.

ensure that rating scales are aligned toassessors ways of thinking.

ask assessors to make judgements, rather

than use detailed objective checklists.As Crossley and Jolly 3 say,

scraping up the myriad evidential

minutiae of the subcomponents of the

task does not give as good a picture

as standing back and considering the

whole.

References1. Miller G. The assessment of clinical skills/competence/performance. Academic Medicine.1990;65:s63-7.2. Crossley J, Johnson G, Booth J, W W. Good questions, good answers: construct alignment improves the

performance of workplace-based assessment scales. Medical Education. 2011;45:560-9.3. Crossley J, Jolly B. Making sense of work- based assessment: ask the right questions, in the right way, about

the right things, of the right people.Medical Education. 2012;46:28-37.4. Homer M, Setna Z, Jha V, Higham J, Roberts T, Boursicot K. Estimating and comparing the reliability of a suite

of workplace-based assessments: an obstetrics and gynaecology setting. Medical Teacher. 2013;35:684-91.


21/64


1 9

OPTIONSEACH OPTION MAY BE USED ONCE,MORE THAN ONCE, OR NOT AT ALLA. Educational impactB. Practicality

C. ReliabilityD. StandardisationE. Student satisfactionF. UniformityG. Validity

Lead in: For each assessment of aninstrument, select the characteristic ofeffective assessment being measured.

STEMS1 Item scores from a 100-item multiple

choice question paper are divided intotwo sets of 50 questions each that arecompared.

2 Scores on a written examination in thefinal year of the medical course are

analyzed to determine whether they areassociated with successful completion ofthe intern year.

3 The feasibility and costs (includingmaterials and staff time) of twoinstruments being considered for use inthe new MD curriculum are compared.

4 Student feedback, and overallperformance in a unit, is compared before

and after addition of a formative ObjectiveStructured Clinical Examination at mid-semester.

SELECT THE CORRECT OPTIONSTEM:Concurrent validity in assessment:

OPTIONS

A measures the correct learning outcomeB measures a large enough sample of the

intended learningC is appropriate to the learners stageD predicts success after graduationE gives the same results as another test

measuring the same outcome

SELECT THE CORRECT OPTION

STEMKirkpatricks model of evaluation has fourlevels which measure:

OPTIONS

A. effect, knowledge, performance, outcomesB. feedback, education, conduct, gradesC. reaction, learning, behaviour, resultsD. response, outcome, behaviour, results

Solution page 60.

Created by Alison Creagh, Christopher Etherton- Beer and Deanka Preston.

The Pu !zz ? le PageTest your knowledge


22/64


2 0

Blueprinting (or creating an assessment Blueprint ) is an approach linking assessment withcurricular content and learning outcomes. 1 The Blueprint itself is usually in tabulated format,which is then used to assist planning of teaching and assessments. One type lists learning

outcomes on one axis, and on the other, the assessment methods and the timing and weightingof assessments. It often includes the timing of teaching and the type of learning to be achieved.Like learning experiences and courses, blueprints come in all shapes and sizes! To illustrate thesepoints, here we share a Blueprint for assessment in a course for preceptors.

Course : Preceptor-ship Course Unit : Providing Constructive Feedback Students : 10 Health care professionals,

who have at least 3 - 4 years ofexperience

Course Duration : 5 Days (combiningclasses and practical exams at the end)

Course Requirements : Pre-readingassigned articles

Unit Duration : 3 Hours / 3 days

Learning Outcomes :At the end of this unit, the students will beable to:1 Identify the different methods of providing

constructive feedback2 Discuss the advantages of constructive

feedback3 Construct real examples from the working

environment and criticize4 Demonstrate how to provide constructive

feedback effectively5 Analyze different situations and evaluate

the best ones

Continuous

Assessment

Marks Time

1 Multiple ChoiceQuestions (Pre andpost the unit)

5% 1st day of the unit

2 Student ledpresentation (15minutes/eachstudent)

15% 2nd day of the unit

3 Assignment 30% End of thecourse

Linking assessmentthe Blueprint approach

by Lubna Al Hasani

Assessment Continuous Assessment

Summative Assessment : Practical Exam-(50%) at the end of the course.

The Course Outlines (Criteria) : Thestudent must pass the practical exam, inorder to pass the unit.


23/64


2 1

Learning Outcomes Recall Interpretation Application Skills Total1. Identify the different methods of

providing constructive feedback5% 5%

2. Discuss the advantages ofconstructive feedback

15% 15%

3. Construct and critique real examplesfrom the working environment

30% 30%

4. Demonstrate how to provide

constructive feedback effectively.

50% 50%

100%

to the curriculum

References

1. McLaughlin K, Lemaire j & Coderre S. Creating a reliable and valid blueprint for the internal medicine clerkshipevaluation. Medical Teacher 2005; 27(6):544547

Unit Content :1 What is preceptor-ship2 Preceptor and preceptee roles and

characteristics3 Achieving Competencies4 Assessment5 Clinical Supervision6 Constructive Feedback7 Evaluation8 Conflict Resolution

Teaching and learning processes forthe course : Pre- reading, presentation,workshop, practical exams and seminars.

Assessment Blueprint


24/64


2 2

We are writing a magazine on assessmentand evaluation in health professionaleducation what would be the one mostimportant piece of advice youd like toshare with our readers?

I would suggest stop taking advice go withyour gut feelings what works for others maynot work for you. Trial and error is appropriatefor most people. I get up at 0500 and workuntil 0730 before work when I have deadlinesin Summer I am a morning person. Thisworks for me but not everyone.

Is there something you wish youd learntabout assessment and evaluation earlier inyour career?

I would have benefited from knowing aboutRubrics and Marking Guides early in mycareer I never knew what different lecturerswanted from assignments. A rubric / markingguide shows what is required and what marks

can be obtained for work produced. I wouldhave been a better proof reader and criticalanalyst if I had known this sooner.

Different Perspectives onSadie Geraghtyis a Midwifeand Midwifery

Educator. Sheis Co-ordinatorPost GraduateMidwifery,

Edith CowanUniversity.

Tell us about the structure of assessmentused in post graduate midwifery educationat present.

Postgraduate midwifery uses blendedlearning / teaching which is reflected inassessments. Students are assessedusing a variety of methods writtenexams, presentations, reflections, OSCEs,

essays, creative assignments (poetry/ painting/ drawing/modelmaking) in whichthe students usually excel at one type ofassessment.

Do you think these assessmentsadequately determine whether a student iscompetent for registration as a midwife?

They are part of the assessments forregistration the clinical assessmentsare equally important. The theoreticalassessments compliment the clinicalassessments by providing knowledge andcritical thinking abilities without both clinicaland theoretical assessments a student wouldnot be adequately assessed for registration.

What advice would you give in workingwith difficult learning situations?

There is no one piece of advice that isappropriate in difficult learning situations but I would suggest remaining grounded. Lifeexperience helps us to deal with situations,and most of us learn from experience.

Watching how others deal with difficultsituations also is beneficial we often learnby example.


25/64


26/64


2 4

What advice would you give in workingwith difficult learning situations?

Be confident and get support from superiorsif needed. A problem shared is a problemhalved!!

Speak with the student alone, get advice andsupport from lecturer, put the student on alearning contract and be sure to review it.

How do you deal with student reactionwhen they are informed that they areperforming unsatisfactorily by failing?

I have found the students will either beaccepting or defensive, either way they

Dr. Sara Bayes is

currently a SeniorLecturer andMidwifery ProgramDirector at EdithCowan Universityand a Post Graduate

Research Fellow at University ofNottingham. She is a RegisteredNurse and Midwife

are upset and annoyed that you havehad the courage to pick them up on theirincompetence.

Any final words of wisdom?

When deciding on whether to pass or faila student in question, ask yourself thequestion Would you want to work withthem?, and Would you like them tolook after you or your loved one?. If theanswer is no, you fail them.

Sara, can you tell our readers if there issomething you wish you had learnt aboutassessment and evaluation earlier in yourcareer?

Well, my early experience of assessing andevaluating students was that I was extremelywell supported by the more experiencedacademics around me who constantly,intensively and proactively checked mymarking to make sure I was doing the right

thing by the students - so my only contributionhere, if that hadnt been the case, would be

to ensure that when youre new to it, you setyourself up with an effective mentoring system.

Are there any resources that havechanged your approach to assessmentand evaluation?

Only the other academics that Ive come intocontact with and milked for their experienceand wisdom over the years!

Do you think these assessmentsadequately determine whether a student iscompetent for registration as a midwife?

For my setting, I do. Assessments of alltypes in each theory and practicum unitthroughout each midwifery preparationprogram are structured to enable the studentto demonstrate their developing competencein a number of the NMBA Midwifery


27/64


28/64


2 6

Welcome to the Web 2.0 ageIn this digital age, we are all familiar with theweb. But for a second, think about how farthings have come in a relatively short time.Web 2.0 describes the new Internettechnologies that now allow users to activelycreate content, share material and interact. 1

This is in contrast to the earlier, largely static(Web 1.0) Internet sites with relativelylittle user content. Web 2.0 now offerseducators the opportunity to design and buildinnovative custom applications to supportteaching, learning and assessment in specificdisciplines. 2,3

The rise of social media has been one of thegreatest results of the popularity of Web 2.0technologies. Few people, from studentsto Professors, havent at least heard ofFacebook and Twitter. Popular socialmedia sites now have an astonishing numberof users (Box 1).

Among the enormous numbers of users,

many are avid users, checking theirFacebook or Twitter accounts daily. 5,6 Socialmedia are particularly popular among themillenial generation (Generation Y) born

Social mediaopportunities forassessment andevaluation in theWeb 2.0 age

Are you familiar with these socialmedia?

Blogger (Web based journaling[Blogging] site)1

Facebook (Social networking site with 1

billion users) 4

PebblePad (e-Portfolio application)

Twitter (Micro-blogging site for sharingof ideas and conversation with 200 millionusers) 5

Wikipedia (Encyclopedia wiki [websitecreated, edited, and developed by acommunity of individuals working on

collaborative projects])1, 4

Youtube (Video sharing site with 800million users)4

Box 1: Popular social media sites

by Christopher Etherton-Beer


29/64


2 7

from the early 80s onwards. These youngfolk have grown up in a digital age and arecomfortable using social media in both theirprivate and academic lives. 1,6 In line withthese trends, the use of social media is veryhigh among health professionals and studentpopulations, 7 and its now recommendedthat medical educators be familiar withsocial media technologies. 5 Despite thisenthusiasm for social media use, manypeople continue to be concerned about theboundaries, oversight and privacy of datashared via social media. In this article wewill consider some of the opportunities andpotential pitfalls, associated with the use ofsocial media in health professional education.(Box 2)

Potential benefits of social media ineducation and assessmentIt is not just access to social media sitesthemselves which has increased at astaggering rate, it now seems that portableelectronic devices (including smartphones, tablets and lap top computers withthe capability of wirelessly accessing theInternet) are ubiquitous among Australianuniversity class attendees. Thus social media

can be used without special organizationby faculty. Compare this to more traditionaltechnologies designed to facilitate learnerparticipation (such as audience clickers,

which require specially enabled classroomsand are limited to use in one physical site).Combined with portable devices enabledwith Internet access, Web 2.0 technologiesoffer remarkable flexibility to teachers andlearners alike.

Instead of diminishing attendance attraditional teaching activities (such aslectures), social media can actually enhanceparticipation. Social media can also beharnessed to encourage participationby learners who may be intimidated byparticipating in traditional class settings,but may be more comfortable initiallybuilding their confidence by participatingelectronically. 1 Electronic participation

can also be anonymous - when this is thelearners preference, and it is appropriatein the context of a particular use of socialmedia. For example, students may contributeto a Twitter feed during a group lecture, orrespond to a poll of learners via Twitter, 5 under a non-identifying username.

There is now a growing experience, and

evaluation, of the use of social media in healthprofessional education. The available datahave been reviewed 4,8 and may surprise someeducators. Certainly it seems that judiciousand carefully considered use of social mediacan facilitate collaboration and learning, withpositive feedback from learners. 4,9 Socialmedia presents a flexible tool (Box 3) thatcan be used to engage learners, facilitatecollaboration and provide feedback. 8

Web 2.0 technologies and social mediaoffer exciting opportunities to promotelearner participation, engagement, andeffective assessment

Use of social media can enhancetraditional pedagogical approaches

Teachers and facilitators haveopportunities to model, promote andrequire e-professionalism amonglearners

Box 2: Key points

continued overleaf


30/64


2 8

Use of social media tools can be associatedwith improvement in the knowledge, skillsand attitudes of learners. 8

Have you considered using social mediato: establish a virtual community of

practice

increase participation in lectures (trydisplaying a live Twitter feed! 1,5

provide real time questions, orfeedback, linked to your class blog 5,10

informally poll learners 5

create a shared class blog 9

provide a discussion forum 4

create, discuss, and edit shared studentarticles on class topics in a Wiki 1

promote communication betweentutorial group members outside oftraditional classes 10

create e-portfolios of student work,allowing them to share some or all oftheir portfolio

create custom Web 2.0 applicationsto support your teaching, learning andassessment 2

Box 3: Applications of social media inhealth professional education

Utilizing social media for studentassessmentThere are many opportunities to supportassessment and evaluation in healthprofessional education using social media.Provision of feedback is a frequent andimportant use of social media in healthprofessional education. 8 Similarly, the provisionof opportunities for formative self-assessment was one of the most highly ratedaspects of a medical education Facebookpage in anatomy. 11 Twitter can also be usedto provide formative feedback. 5 Web 2.0technology can be used to offer authenticassessments (i.e. that reflect activities thelearner will need to undertake in their actualpractice). For example, a custom web basedapplication has been designed that facilitates theassessment of nursing students in searchingand critically reviewing the literature. 2

In addition to the provision of feedback andassessment activities, the extent and qualityof electronic participation by students ingroup work or discussion can be rated bytutors or peers (mirroring the assessmentof more traditional face-to-face group work).Learners can also be prompted to self-reflecton their on-line contributions. Similarly,electronic material authored by students can

be assessed by tutor, peer or self-rating.On-line environments offer the potential for

combinations of self, tutorand peer rating, as materialis easily shared. Provisionof feedback in an on-lineenvironment can providestudents with experience inconstructing and deliveringconstructive feedback, whichin turn can be assessedformatively and summatively.


31/64


2 9

Evaluating the use of social media in yourcourseThe social media aspects of your teachingcan be assessed in tandem with otherfeedback collected from learners. Inaddition, teachers can consider usingspecific resources provided by somesocial media sites. For example, Facebookprovides the Facebook Insights functionthat can automatically provide pagemetrics (such as numbers of visitorsand de-identified demographic data). 11

Google Analytics offers a free service toanalyse Internet site statistics (such asthe number of visits to a course Web 2.0site). Depending on the learner cohort andparticular social media application it mayalso be appropriate to measure the learnersattitudes towards the use of social media, 12 and there is much potential for educationalresearch in this field!

Challenges of using social mediaCommon challenges when using socialmedia are technical problems and ensuringconsistent user participation. 8 Learnersmay find the array of social media optionsbewildering, or find the presentation ofcommercial advertising distracting. For thesereasons, teachers may focus on using oneor two social media tools in each class. Theuse of social media can present learnersand teachers alike with practical, ethical andlegal challenges 6,13 relating to privacy and the

blurring of private and professional lives. Someof the risks associated with social media use(such as breaches of patient confidentialityor the patient-doctor relationship) have nowbeen well publicized. 7 These challenges haveled many professional groups and educationalinstitutions to consider policies regarding theuse of social media. E- professionalism isdefined as the attitudes and behaviors that

reflect traditional professionalism paradigmsbut are manifested through digital media. 6

continued overleaf


32/64


3 0

References

1. McAndrew M, Johnston AE. The role of social media in dental education. J Dent Educ. 2012;76(11):1474-81.

2. Eales-Reynolds L-J, Gillham D, Grech C, Clarke C, Cornell J. A study of the development of critical thinkingskills using an innovative web 2.0 tool. Nurse Education Today. 2012;32(7):752-756.

3. Watson N, Massarotto A, Caputo L, Flicker L, Beer C. e-ageing: Development and evaluation of a flexible

online geriatric medicine educational resource for diverse learners. Australasian Journal on Ageing. 2012;Forthcoming (online ahead of print DOI: 10.1111/j.1741-6612.2012.00622.x).

4. Hamm MP, Chisholm A, Shulhan J, Milne A, Scott SD, Klassen TP, et al. Social Media Use by Health CareProfessionals and Trainees: A Scoping Review. Acad Med. 2013;88(9):1376-1383.

5. Forgie SE, Duff JP, Ross S. Twelve tips for using Twitter as a learning tool in medical education. MedTeach.2013;35(1):8-14.

6. Kaczmarczyk JM, Chuang A, Dugoff L, Abbott JF, Cullimore AJ, Dalrymple J, et al. e-Professionalism: a newfrontier in medical education. Teach Learn Med. 2013;25(2):165-70.

7. George DR, Green MJ. Beyond Good and Evil: Exploring Medical Trainee Use of Social Media. Teaching andLearning in Medicine. 2012;24(2):155-157.

8. Cheston CC, Flickinger TE, Chisolm MS. Social media use in medical education: a systematic review. AcadMed. 2013;88(6):893-901.

9. George DR, Dellasega C. Social media in medical education: two innovative pilot studies. Med Educ.2011;45(11):1158-9.

10. Wells KM. Social media in medical school education. Surgery. 2011;150(1):2-4.

11. Jaffar AA. Exploring the use of a facebook page in anatomy education. Anat Sci Educ. 2013; DOI 10.1002/ ase.1404.

12. Wang AT, Sandhu NP, Wittich CM, Mandrekar JN, Beckman TJ. Using social media to improve continuingmedical education: a survey of course participants. Mayo Clin Proc. 2012;87(12):1162-70.

13. Cain J, Fink JL. Legal and ethical issues regarding social media and pharmacy education. Am J Pharm Educ.

2010;74(10):184.

A faculty can take many practical steps to limit risks, and maintain e-professionalsim, associatedwith use of social media in their courses such as:

consider setting up separate pages, groups or accounts for each class advise students to select maximal security and privacy settings have separate profiles, accounts or pages for professional and private interaction


33/64


3 1

ACROSS

6 Way of matching learning outcomes andassessments

8 Systematic way of checking that learningobjectives are achieved by learningactivities

9 A number of stations to assess differentclinical skills

10 An easy to mark, standardised type ofassessment

11 Means of ensuring that learning has beenachieved

12 Type of assessment that often determinesprogression to next stage

1 2 3

4 5

6

7

8

9 10

11

12

DOWN

1 Life like learning or assessment2 An assessment measures what it intends

to measure3 Collection of work by learner4 Sufficient knowledge and skill to work in

particular clinical scenarios5 Results of an assessment are reproducible7 Type of assessment primarily to provide

feedback to learner

Solution on page 48

The ASSESSMENT ZONE Crossword


34/64


3 2

Sandy Presland reviews the strengths and limitations of using portfoliosto assess competency and learning of nursing professionals The use of portfolios within the nursing profession for the purpose of assessment and professionaldevelopment has become accepted widely. 1 Portfolios are considered as a portable and visiblerecord of professional contribution and credentials. 2 Portfolios consist of an organised collectionof evidence that the nursing professional can use to showcase details of professional education,practice experience, formative and summative assessments, and achievements.

Overseeing a post anaesthetic care unit within the South Metropolitan Health Service, myrole focuses on ensuring that nursing staff are effectively assessed as competent to deliversafe patient care and to encourage them to uphold their professional responsibility to maintaintheir competency, knowledge and skills relevant to current practice. The present assessmentstructure within the unit I supervise involves nursing staff completing competency skillschecklists, which are entered into a data base. An annual performance review ensures the nurseis meeting professional development requirements. With the current reconfiguration within theSouth Metropolitan Health Service, and the opening of a new hospital in 2014, it seems timelyfor a review of current assessment practice. Portfolios may be well worth exploring for futureimplementation to enhance learning and promote professional development.

According to the Australian Nursing and Midwifery Council, competency is defined as acombination of skills, knowledge, attitudes, values and abilities that underpin effectiveperformance. 3 Current competence assessment methods have been reported as only measuringa nurses competence levels when they focus on skill, not knowledge. 4 However, it has beenreported that portfolio assessment allows a nurse to reflect on both academic and clinical practice,thereby applying theory to practice. 4 Portfolios offer a collection of evidence that demonstrates anurses skills, knowledge, attitudes and achievements that reflect a nurses ongoing professionaldevelopment. 5 They offer an opportunity for individuals to document evidence of learning

outcomes, areas of development and reflect upon areas that require further development. 1 Portfolios are also a mandatory requirement for nurses registered with the Nursing and MidwiferyBoard of Australia, to keep a record that demonstrates professional competence. 3,6

Making sense ofportfolios


35/64


36/64


3 4

be developed. 1 It is reflective thinking thatenables the learner to move from surfacelearning to deeper understanding and this isperhaps where the value of portfolios lies.

A range of limitations creating a barrier for theimplementation of portfolios use has beenidentified in nursing literature. As discussed,there is limited evidence that portfolios canmeasure competence with reliability andvalidity. Lack of clarity about expectationsand outcomes can lead to confusionand inconsistency with interpretation ofrequirement. 7 Standard output of portfolioscan also vary as it is driven by eachindividuals motivation and experienceand the self-directed approach does notsuit all learning styles. 1,7 Furthermorecollecting evidence for portfolios can be timeconsuming, and this may put extra burdenon the time deficient nursing professional.

Portfolios can also be time consuming toassess due to the quantity of material,and pose difficulty in assessing due to thesubjectivity of the material. 11

Taking these limitations into consideration, itis evident that portfolios may be resisted bywork pressured nurses. To assist in reducingan extra burden on nurses and increasing

feasibility of use, portfolio guidelines would

need to stress quality not quantity ofevidence. Provision of templates could assistwith making explicit the type of evidencerequired and setting the limits of content .11 Adequate support and feedback would needto be provided to both nurse and assessor, ifthe portfolio system is to be effective.

Summary

It is clear that portfolios can contribute to theprofessional development of the qualifiednurse by enhancing deep learning throughreflection, showcasing achievements, andpromoting lifelong learning by engagingthe individual to keep up with change.Evidence on whether portfolios can measurecompetency is inconclusive, however Ibelieve the value of the portfolio lies indocumenting the assessment as a process,

rather than measuring it as an end product.

Portfolios incorporating both evidence andreflection aligned to competency standardsmaybe worth embracing in the future as away of promoting improved nursing practiceand meaningful learning, but their processmay require considerable investment of timeand effort by all parties involved.


37/64


3 5

References

1. Endacott R, Gray MA, Jasper MA, McMullan M, Miller C, Scholes J, et al. Using portfolios in the assessmentof learning and competence: the impact of four models. Nurse Education in Practice [Internet]. 2004 [cited 2013Sept 2];4(4):250-257. Available from: ScienceDirect.

2. Meister L, Heath J, Andrews J, Tingen MS. Professional nursing portfolios: A global perspective. MedsurgNurs [Internet]. 2002 [cited 2013 Sept 2];11(4):177-82. Available from: ProQuest Health & Medical Complete;ProQuest Nursing & Allied Health Source.

3. Australian Nursing Midwifery Council Continuing Competence Framework 2009: Available from: http:// equals.net.au/pdf/73727_Continuing_Competence_Fram ework.pdf.

4. McCready T. Portfolios and the assessment of competence in nursing: A literature review. Int J Nurs Stud[Internet]. 2007 [cited 2013 Sept 5];44(1):143-151. Available from: ScienceDirect.

5. Hughes E. Nurses perceptions of continuing professional development. Nurs Stand [Internet]. Jul 6-12, 2005[cited 2013 Sept 2];19(43):41-9. Available from: ProQuest Nursing & Allied Health Source.

6. Timmins F, Dunne PJ. An exploration of the current use and benefit of nursing student portfolios. Nurse EducToday [Internet]. 2009 [cited 2013 Sept 10];29(3):330-341. Available from: ScienceDirect.

7. Evans A. Competency Assessment in Nursing: A summary of literature published since 2000 NationalEducation Framework Cancer Nursing Education Project (EdCaN), Cancer Australia [serial on the Internet]. 2008:Available from: www.edcan.org/pdf/EdCancompetenciesliteraturereviewFINAL.pdf.

8. Webb C, Endacott R, A Gray M, Jasper MA, McMullan M, Scholes J. Evaluating portfolio assessmentsystems: what are the appropriate criteria? Nurse Educ Today [Internet]. 2003 [cited 2013 Sept 2];23(8):600-609.Available from: ScienceDirect.

9. Anderson D, Gardner G, Ramsbotham J, Tones M. E- portfolios: developing nurse practitioner competenceand capability. Aust J Adv Nurs [Internet]. 2009 [cited 2013 Sept 2];26(4):70-76. Available from: Informit.

10. Redfern S, Norman I, Calman L, Watson R, Murrells T. Assessing competence to practise in nursing: a

review of the literature. Research Papers in Education [Internet]. 2002 2002/01/01 [cited 2013 Sept 4];17(1):51-77. Available from: Routledge.

11. Driessen E, Van Tartwijk J, Van Der Vleuten C, Wass V. Portfolios in medical education: why do they meetwith mixed success? A systematic review. Med Educ [Internet]. 2007 [cited 2013 Sept 4];41(12):1224-1233.Available from: Wiley.

photo: Jodie Atkinson


38/64


3 6

Dr Claire Harma , Advanced

Trainee in Gastroenterology withthe Royal Australasian College ofPhysicians, shares her thoughtson the assessment of physiciantrainees.

Dr Harma (right) assisted by endoscopyregistered nurse, Belinda Brohman

We are writing a magazine on assessmentand evaluation in health professionaleducation what would be the one mostimportant piece of advice youd like toshare with our readers?

You need to make sure that it is clear totrainees at the start of a program what thelearning outcomes are and what is going to beassessed. When providing feedback it needsto be FIT & ABLE (Frequent, Interactive,Timely, Appropriate for learner level,Behaviour specific and balanced, Labelled andEmpathetic) 1, which is a message that those

interested in education need to pass on toothers acting as supervisors and mentors sothat trainees get more learning out of theircontinuous assessment

Is there something you wish youd learntabout assessment and evaluation earlier inyour career?

Something I wish I had a bit more insightinto earlier in my career is that all too oftenthe trainee who is performing well in theworkplace is never given detailed feedback,and often these are the individuals whoactually want to find out the areas they canimprove in rather than just hearing yeah,good job. More specific feedback andextension advice for how someone couldimprove is warranted, even for those people

who are performing well.

Assessing Physicians of tomorrow


39/64


3 7

Are there any resources that have changedyour approach?

The Teaching On The Run program has hadquite a big impact on how I provide feedbackand how I structure assessment, and thesimple and very practical approaches taughtin this program have been very helpful duringthe course of this year working as a medicaleducation registrar. Also, in setting up a mockwritten exam for basic physician trainees thisyear, weve been using an online question banktool in Moodle, which has been very helpfulin collating questions and delivering the examonline and performing statistical analyses.

Tell us about the structure of assessmentused in physician training at the moment.

I think the best way of explaining this is that

physician training is done in two components.We have basic physician training (BPT) andadvanced training (AT). BPT typically takesthree years and during that time, basicphysician trainees are expected to havegained skills and knowledge in a broadspectrum of physician areas from generalmedicine through to sub-specialty areas, andthen there is a barrier exam to get through to

advanced training.

The exam is performed in two parts. There isa written exam, which has two papers lookingat basic sciences and clinical applications.Then everyone is ranked on a bell curve, andif you are in the top two-thirds of candidatessitting the exam, you are then eligible to sitthe clinical exam.

The clinical exam tries to simulate real patientinteractions to try and get a better idea of

how capable and competent the individual isin their day to day practice. The clinical examincludes long cases, where the candidate hasan hour with the patient, and has to interviewand examine the patient and formulatetheir major medical problems; as well asshort cases where a clinical examination isperformed in seven minutes while examinerswatch and a diagnosis and management planhas to be deduced from that. So they are thebig exams, the big barriers that we have inphysician training.

Throughout BPT and AT, there are also anumber of formative assessments performed,and these include Mini-CEXs, case-baseddiscussions and learning needs analyses. Inthe three years of advanced training, traineesmove into a subspecialty area, and eachsubspecialty has a different assessmentstructure. Some of the subspecialties

have formalised, summative examinationsthroughout their advanced training, but thevast majority have no barrier assessmentto becoming a consultant. But they dohave a series of mid-term and end-of-termassessments and formative feedbacksessions.

Do you think this is the best method fordetermining whether a trainee is ready tobecome a consultant physician? And doesit assess all different facets of practice fora consultant physician, aside from justclinical practice?

Physician training is moving away from theapprenticeship model toward a competency-based model. Competency was previously

continued overleaf


40/64


3 8

thought to be gained by a certain amount oftime in a particular area, which is what theapprenticeship model works on, but in thismodel you may only be exposed to a certainrange of conditions depending on where youare training and so you may miss out on thebroader realms of things that fall within yourspecialty; whereas the competency-basedmodel requires the trainee to prove theircompetence before going on to become aconsultant. I think by having the barrier exambetween BPT and AT it determines a level of

competence that seems to be fairly rigorous,but in moving on to become a consultantphysician there is the potential for traineesworking in this apprenticeship model to nothave the degree of exposure they needto become a competent consultant. Andthere is some talk of going on to having exitexaminations for physician training.

The expectation of the role of the consultanthas changed a lot in the last 10 years, wherebeing a good clinician isnt necessarilyenough to secure employment, and assuch, consultants are now expected tohave experience in research, education andmanagement. I think these are three keyareas that at the moment arent adequatelyassessed. Although in most specialty areastrainees are expected to perform researchprojects over the course of their training,there isnt necessarily any assessmentassociated with that or a quality standardupheld. Education and management are areaswhere there isnt any training or assessmentat the moment.

Can you tell me a bit about the differentmodalities of assessment currently usedin gastroenterology training and whetheryou think that is adequate to meet theassessment needs of the trainees?

Work-based assessment is essentially whatthe RACP is trying to use, particularly inadvanced training, but I think there are anumber of limitations with this. Continuousassessment is utilised in the mid and endof term feedback that the trainee receives.Depending on the interaction between thesupervisor and trainee, the quality of thatfeedback may be different depending on thesupervisor and the frequency of interactionthat those two individuals have. Mosttrainees will try and nominate a supervisorthat they work closely with to give them thebest possible feedback, but within somesubspecialties that is not always possible

and there may be a supervisor who doesnthave close contact with the trainee dayto day, and so the reliability and validity ofthat assessment may be diminished. Inprocedural specialities, performance-basedassessment is more based on volume ratherthan competence. Within my specialty,gastroenterology, its based on the numberof procedures including gastroscopies and

colonoscopies. The training program is athree year advanced training program, andat the end of that you are expected to haveperformed 100 colonoscopies unassisted.Realistically, most will have performed thatmany colonoscopies within the first 6-12months of training and if over the courseof the three years youve only just madethat level, I am not convinced you wouldactually be capable of going out to practiceindependently. The chances of the rarercomplications cropping up in that time is


41/64


3 9

limited, so you may not feel confident inmanaging those complications, which wouldbe expected of a consultant.

One possibility would be to have a supervisorwho has monitored you closely over aprolonged period of time who determinesyour competence, and that person needs tohave undergone standardised accreditationby the college. However, I think, ideally,there needs to be a volume component toit as well, but I think the current volumerequirement is probably too low.

You mentioned before the possibility ofexit exams being introduced as a barrier tocompleting advanced training. What sortof format might that take? And what areyour views on the issue? I think with the increasing number of

trainees that are coming through, the ratioof supervisors to trainees has the potentialto become more dilute, and as a resultthat training experience may vary betweentrainees, and between training locations.With increased numbers of trainees, I thinkthat there is merit in an exit exam. You wouldwant to assess the broadest range of theaspects of the job you possibly could. I think

there needs to be a knowledge component,which would likely be in the form of a writtenexamination. I think there needs to be aclinical component, so potentially a patient

interaction, OSCE- styled component. AndI think within procedural specialties, theremay need to be a procedural simulationcomponent as well to demonstrate proceduralcompetence.

I can foresee some problems with introducingan exit exam. It will be costly for the college,and ultimately for the trainees as the cost willbe passed on to them in increased trainingand exam fees. It will be very time consumingfor both organisers and exam candidates toprepare for the exam. It will still not assessthings like interprofessional communication,unless there are specific OSCE stations onthese topics. And I think the biggest challengewill be what to do with the trainees whodont pass. Will they repeat another year oftraining, and will they ultimately be asked toleave the program? It would all add to theincreased number of trainees who need to be

accommodated in the system. I think all theseissues need to be worked out before thiscould be implemented.

1. Beckman TJ, Lee MC. Proposal for a collaborativeapproach to clinical teaching. Mayo Clin Proc. 2009

Apr;84(4):339-44.

Interview by Dr Elisabeth Feher

For more information about the Teaching onthe Run program, go to tellcentre.org


42/64


4 0

OSCE stands for Objective StructuredClinical Examination and assesses mainlycognitive and psychomotor domains oflearning. OSCEs focus on the students abilityto synthesise and apply their knowledge intopractical tasks.

OSCEs were first introduced into medicaleducation in 1975 by Ronald Harden in

Scotland at the University of Dundee. Theywere utilised as an alternative to traditionalclinical examinations which had significantlimitations due to subjectivity, inconsistencyand poor structure. OSCEs examine clinical competence andaim to assess a students communicationskills, ability to think critically and problem

solve. OSCEs are structured performancetests that are standardised, objective andif designed well, can have high levels ofreliability and validity .3,4

Does your OSCE

by Kate Gray

OSCEs are used across a variety of healthdisciplines including nursing, medicine andallied health extensively in Australia, NorthAmerica and the United Kingdom.

OSCEs are designed as a series ofexamination stations that the students rotatearound. Each station is designed to assessone competency by using a pre-determined

guideline or checklist. The students are assessed on not only theresults they obtain from their performance ofthe practical skill but also how they go aboutobtaining these results.

Very clear instructions are provided tominimise ambiguity for the student and

standardise the responses and promptsthat the examiner and simulated patientcan provide.

OSCE Fast Facts


43/64


4 1

make the cut?Assessing the quality of OSCEs

ReliabilityOSCEs have been demonstrated to havemuch higher levels of reliability than otherforms of practical assessment and havehence become the mainstay of practicalclinical competence in medical education. 2

Moderate to good inter-rater reliability hasbeen demonstrated when multiple assessorsscore one station independently of oneanother. Caution should be used whenthere is only one single assessor for eachstation. 5 A correlation of 0.6 or more hasbeen suggested as a good level of agreementbetween raters. 5

The ability of an OSCE score on one stationto determine the performance on anotherstation has been found to be low in somestudies. The number of stations and diversityof the stations impacts on the internalconsistency of the OSCE. Increasing thesimilarity of skills to be tested can improvethe internal consistency of the OSCEand reducing the number of stations andinadequate sampling threatens the reliability.

Test retest reliability is the consistency inwhich a student receives the same score inan OSCE if they repeated it several timesover. By using simulated patients, who actin a standardised manner improves the

consistency of OSCEs as it is easier andmore feasible to train subjects to respond ina particular way multiple times, rather than totrain a real patient. 3

StandardisationA marking checklist is provided to allexaminers with appropriate training toensure they score each component of thetask reliably against a set of pre determinedcriteria. Aspects of the students performance

with higher importance such as safetycan determine an automatic fail grade inspite of the other aspects of the studentperformance. 6 The use of global rating scales(GRS) can allow the examiner to make ajudgement about student performance thatis based on aspects of the performance thatare difficult to quantify with a checklist, suchas interpersonal skills. 2 Combining the GRS

and checklist give a more holistic and reliableassessment on performance. 6

Passing a borderline student also needsto be standardised to ensure the OSCEis fair and reliable. Standards need to beset to determine if a borderline student inone or more stations will pass the overallexamination. Research has demonstratedthat in borderline students, all station scoresshould be combined to determine their overallclinical competence. The standard passscore for each OSCE can be determined byexpert opinions of the examiners or expertsin the field and all of the students actualperformance scores combined. 6

continued overleaf


44/64


4 2

ValidityContent validity can be high in OSCE examswhen the outcomes being assessed arereviewed by experts in the field. 3 OSCEsassess a wide variety of clinical skills by usingmultiple assessment stations using a range ofwell trained examiners to consistently assessthe students. 3 The marking criteria shouldonly entail aspects of performance thatare directly related to the skill that is beingassessed. 6

The ability of an OSCE score to correlateto other examination scores on the sametopics, or the OSCE score correlation to a finalcourse score has been demonstrated as beinggood, especially when checklists and GRSare utilised. 6,7 Statistically significant criterionvalidity has also been demonstrated in OSCEexams by using correlation statistical studiesto compare students OSCE results with

their written examination, oral examinationand clinical evaluation scores. 3 This suggeststhat OSCE scores correlate well to otherexaminations taken at the same time.

The predictive validity of OSCEs has beenestablished when comparing students resultsin an OSCE to their actual clinical performanceand performance in other examinations suchas written or case based interdisciplinaryassessment. 8 Students who demonstratestrong analytic and problem solving skills,whilst also showing good communicationand practical skills in OSCEs will most likelyperform well in a real life clinical scenario.

OSCEs can effectively identify students whomay struggle with clinical experiences andalso highlight significant concerns around astudents professionalism and ability to copeunder stress. 2

EducationalOSCEs can be used not only in summativeassessment but also in formative assessmentas it provides feedback to students abouttheir areas for improvement and providesthem with an opportunity to practiceclinical skills and prepare them for finalexaminations. 4

OSCEs are generally well received bystudents, who perceive OSCEs to be arealistic reflection of real life scenarios theyencounter in clinical situations. It has beenreported that students feel they can be fairand useful assessment methods. It must be

noted that students may complain of highstress and anxiety levels with this type ofexamination. 2,3

PracticalityThe cost of OSCEs has been found to bewithin acceptable limits if compared to otherforms of practical assessment. HoweverOSCEs are initially costly to set up, not onlyin amount of examiner time and facilities butalso in examiner and simulated patient training,with some reported as requiring up to fourhours of training per actor. 2 These costs aregenerally reduced when the same subjects areused in multiple examination periods.

a success skills for life, will impact on your practice it is so practical like the job.

First and second year nursing students reflections an OSCE assessment.


45/64


4 3

Development of an OSCE 9,10

Your Checklist

Establish purpose

of test.

To ensure good content validity and reliability ensure your OSCE stations sample an

adequate array of topics that have been covered in the learning material during thecourse of the study. The amount of stations required depends on the volume of content and level of skills to

be assessed. A formative OSCE will require fewer stations than an end of year/graduateentry examination.

Definecharacteristics oftarget population

Ensure the OSCE only assesses the clinical skills that are required for that stage in thestudents training.

Create a testblueprint

Specifically define all of the competencies you wish to assess and ensure there is a goodrepresentation of all disciplines to ensure all appropriate content is covered.

To ensure adequate sampling, create a two dimensional matrix outlining all of the cont

assessment zone issue 1 2013

Documents