Train the Trainer Assessment Workshop October 27-31, 2003

Train the Trainer Assessment Workshop October 27-31, 2003

Participant Feedback:Participant Feedback:

from

Australia, Bahrain, Canada, China, Fiji, Indonesia, Nepal, New Zealand, Pakistan, South Africa, United Arab Emirates,

United Kingdom, United States

from

Australia, Bahrain, Canada, China, Fiji, Indonesia, Nepal, New Zealand, Pakistan, South Africa, United Arab Emirates,

United Kingdom, United States

Hong Kong International Consortium

Arabian Gulf University

Flinders University

Peninsula Medical School

The Australian National University

The Chinese University of Hong Kong

The University of Alberta

The University of British Columbia

The University of Hong Kong

The University of Sydney

United Arab Emirates University

University of Otago

University of the Witwatersrand

Arabian Gulf University

Flinders University

Peninsula Medical School

The Australian National University

The Chinese University of Hong Kong

The University of Alberta

The University of British Columbia

The University of Hong Kong

The University of Sydney

United Arab Emirates University

University of Otago

University of the Witwatersrand

Members

Non Member Participants

Fiji School of Medicine

Kathmandu University

Monash University

Shifa College of Medicine

The Hong Kong Polytechnic University

The University of Adelaide

The University of Auckland

The University of Melbourne

Universitas Padjadjaran

University of the Free State

Fiji School of Medicine

Kathmandu University

Monash University

Shifa College of Medicine

The Hong Kong Polytechnic University

The University of Adelaide

The University of Auckland

The University of Melbourne

Universitas Padjadjaran

University of the Free State

4.69 4.724.84 4.78

1

2

3

4

5

Will use in my ownassessments

Will use ideas & aidsto teach colleagues

Workshop wellorganized

Recommendworkshop tocolleagues

StronglyAgree

Agree

NoOpioion

Disagree

StronglyDisagree

Overall Quality & Impact of Workshop

4.76 4.69

4.44 4.404.22

3.81

4.48

1

2

3

4

5

A-Type R-Type MEQ / SAQ M R OSCE OSPE FACS

StronglyAgree

Agree

NoOpioion

Disagree

StronglyDisagree

Quality of Workshop by Item Format

4.40

4.06 4.14

1

2

3

4

5

Item Analysis Test Blue Prints Use of IDEAL

StronglyAgree

Agree

NoOpioion

Disagree

StronglyDisagree

Quality of Workshop by Test Construction Topic

Written Comments

Introduction

– The following slides subdivides comments into those that

• apply to entire workshop

• apply only to a particular session

• suggest how to improve a further workshop

– If a respondent offered comments applicable to more than one of the above categories, his/her multiple comments were appropriately separated and grouped respectively.

Comments pertaining to overall workshop

– “Content – very good; appropriate level for varied audience; Process – workshop to apply sessions helpful; organization – has been VERY well run. Excellently done.”

– “Incredibly well-run. Hosts were exceptional. Very high level people here”

– Organization – “very strongly agree”

– “It is difficult to be critical of any aspect of this workshop. I strongly feel that a similar workshop should be offered again to other medical educators”

– Organization – “strongly agree +++++ Wow!”

– Recommend to others – “absolutely”

– “Whole thing was fantastic”

– Superb organization

– “The conference was great. It takes a few days for all the information to percolate in the brain. My frustration of not getting hundreds of items written & submitted is dampened by having a “template of how to proceed. So this is my greatest take home. Thank you.”

– “I have very much enjoyed this workshop & have learned a great deal. You have been exemplary hosts & made the week run like clockwork … Banquets: a. the social activities were fabulous, I have had an excellent time. b. The transport has been excellent. c. Facilities excellent d. Hotel very pleasant e. Very nice group of participants, friendly, non-threatening”

Written Comments (cont’d)

Comments pertaining to overall workshop (cont’d)

– “The arrangements were fabulous & I salute you all for your very hard work!”

– Organization – “Superb!!”

– “Hugely valuable as networking”

– “Hugely valuable as personal & own-University team motivator”

– “Fantastic effort & enthusiasm from all involved”

Written Comments (cont’d)

Comments pertaining to specific sessions

– Multiple Response

• “Attended lecture only”

– FACS

• “Hands on session was strongly agree”

• “Attended demo only”

– OSPE

• “We decided OSPEs only have relevance to anatomy & possibly pathology/histology”

– IDEAL

• “IDEAL workshop attended elsewhere”

• “Bit difficult to follow the lecture”

• “Previously saw and it was excellent”

– Several sessions

• “Difficult to evaluate both items as I attended the big group sessions of all but not all small groups”

Written Comments (cont’d)

Written Comments (cont’d)

Suggestions for Improvement

1 “Pressure to produce questions may have been counter-productive for people new to the process; maybe have a 2nd ‘work’ workshop at a later or separate date for people now experienced in the formats would serve better; overall the schedule was demanding & [sic] productivity of attendees. May have been better if there was more FLEXIBLE time (to exercise for example! or work alone on items discussed)”

2 “Please try & ensure opportunities for inclusion of non-medics – was easy for us to feel excluded by our lack of medical knowledge”

3 “Could be shortened”

4 “One session left free to allow follow-up with colleagues, not working – the days were so long it was difficult to do this at the social events”

5 “More forums for open discussions”

6 “More time for exercise would have been appreciated”

7 “Clear indication it is an item writing workshop; participants to bring in more items; find a way to ease template use. Thanks.”

8 “As, I believe, participants have different experiences and ways of student assessment it would be very useful if there is time for the participants to have ample time to know certain terms that they maybe do not know. I think it will be very useful if we are provided with a Glossary of terms.”

9 “I would have appreciated one free afternoon in the middle of the week – I found the first 2 days very long & it was difficult to maintain attention.”

Written Comments (cont’d)

Suggestions for Improvement (cont’d)

10. “In terms of suggestions, I can only offer the following (and some of this I understand may not be in your control): a. the lecture theatres & rooms have been very cold – perhaps advise participants to bring a warm jacket;

b. I am feeling very tired after this week – I don’t know that you can do anything about this, but I know I am not the only one who is feeling worn out;

c. please include DSM-IV or ICD on the resources list for workshops– the texts were geared heavily towards physical medicine, not psychological;

d. . . .[one presenter] made some very derogatory remarks in the presentation about psychiatry & mentally ill – this sort of behavior is extremely inappropriate in a setting such as this (or anywhere in fact).”

11. “Very important to mix clinicians with basic scientists. The latter cannot write clinical vignettes without help! Further suggest that nothing is specific, for example:

Physiologists -- anaesthetists, intensive care, physician (general), physician (renal)Anatomist -- surgeonMicrobiologist -- infectious disease physicianImmunologist -- oncologist, hematologist, pathologistBiochemist -- physician (general), chemical pathologist, etc.”

12. “I would suggest building in a bit of free time, since it was difficult to maintain enthusiasm from 9 – 6 every day.”

13. “Would like to see more use of valuable insights from assessment program coordinators (‘process’ people) who are now trying to manage IDEAL’s integration with existing local university systems/mechanisms, i.e. for reality check/collegial strategy seeking.

14. “Would like to see schools encouraging and supporting funding attendees in numbers proportionate to curriculum representation (e.g. why did we have many more clinical than non-clinical people here this time?”

Appreciation

This information will be used in the planning of further assessment workshops run under the auspices of the

Hong Kong International Consortium for Sharing Medical Student Assessment Banks. The organizers are thus grateful to those participants who completed the workshop evaluation form.

We are indebted to all the facilitators that contributed throughout the week-long workshop to enhancing the participants’ skills in writing medical and nursing student assessments. These internationally recognized experts included Susan Case, David McHaffie, David Prideaux, Gordon Page, Kathy Boursicot, Ernie Skakun, Brian Jolly, Neil Spike and were ably aided by local contributions from Shekar Kumta, James Ware and Simon Yip.

Finally, the organizers gratefully acknowledge the Hong Kong University Grants Committee whose Teaching Development Grants made it possible for:

- Hong Kong’s two medical & three nursing schools to host this workshop, and

- creation of the Consortium whereby its members can better develop, share and have improved access to quality medical and nursing school student assessments. Our goal is that all our assessments will now better meet internationally recognized standards for quality of

assessment, have utility for enhanced student learning and thereby help in our efforts to ensure that the patients, who will be served by our future graduates, receive even better care.

John Nicholls (University of Hong Kong)

Clarke Hazlett (Chinese University of Hong Kong)