blueprinting and drafting questions, liz norman, savs-csu learning and teaching extravaganza,...

Blueprinting and drafting questions

Liz Norman

Massey University

Overview

1. Blueprinting

2. Question difficulty

3. Wording questions

4. Question timing

What is a blueprint?

"blueprint, n.". Something which acts as a plan,

model, or template

Specifies what’s going to be in the exam

– Content/topics – breadth of the exam

– Level - depth of the exam

OED Online http://www.oed.com

Why we need to blueprint

• Its important that we sample representatively

from the content domain (all that it is possible to

examine)

• This is so we can extrapolate performance to the

whole content domain

Breadth by topic

PathophysiologyInvestigation and

diagnosis

Treatment and

management

Gastrointestinal P1Q1 P1Q1, P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

Nervous P1Q3, P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5

Breadth across species

species number of Qs percent

small animal 15 52%

farm 5 17%

horse 4 14%

exotic 3 10%

lab 1 3%

all 1 3%

Level - depth

• Surface – deep

• Fact recall – applied

• Blooms taxonomy

• SOLO taxonomy

Fact recall vs applied

Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)

Applied (higher order)Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact

From: Peitzman et al. (1990). Academic Medicine, 65(9), S59-60.

create

evaluate

analyse

apply

understand

rememberRecall

Application

Blooms taxonomy

Blooms instructional verbs

• Create: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare.

• Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate

• Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise.

• Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch.

• Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell

• Remember: define, repeat, record, list, recall, name, relate, underline.

SOLO taxonomy

http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/

Prestructural Question may be rephrased as the answer; almost completely

misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such

as being able to describe aetiology, clinical features, management

of thrombotic stroke) but unable to explicitly explain causes for

observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally

explain or elaborate observations into cause-effect relationships;

able to compare similarities and differences between apparently

distinct phenomena. This level is taken as suggesting that the

learner has understood.

Extended

abstract

Highly developed; able to explain mechanisms of phenomena and

apply this information to a novel context — able to develop novel

hypotheses, theories, and deduce principles; creative thinking.

Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

Prestructural



Multistructural

Relational

Extended

abstract

Quantitative change

Qualitative change

Prestructural



Multistructural

Relational

Extended

abstract

surface

deep

Prestructural



Multistructural

Relational

Extended

abstract

recall

application

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple

instructions, calculate, reproduce, arrange, recognise, find, note,

seek, sketch, pick

Multistructural Combine, classify, structure, describe, enumerate, list, do

algorithm, apply method, account for execute, formulate, solve,

conduct, prove, complete, illustrate, express, characterise

Relational Analyse, compare, contrast, integrate, relate, explain causes,

apply theory (to its own domain), argue, implement, plan,

summarize, construct, design, interpret (some senses), structure,

conclude, substantiate, exemplify, derive, adapt

Extended

abstract

Theorise, generalise, hypothesise, predict, judge, transfer theory

(to new domain), assess, evaluate, interpret (some senses),

critically reflect, predict, criticise, reason

Potter & Kustra (2012). http://www1.uwindsor.ca/ctl/system/files/PRIMER-on-Learning-Outcomes.pdf

Topic 130%

Topic 230%

Topic 340%

total

Knowledge of terms 2 5 5 12

Comprehension of principles 4 3 4 11

Application of principles 3 3 3 9

Analysis of situations 3 2 5 10

Evaluation of solutions 3 2 3 8

Total questions 15 15 20 50

In this example

• Topic 1 represents 30% of the whole curriculum being assessed

so 15 of 50 questions are to address it.

• Four questions are to address comprehension of the principles

of Topic 1.

Scoring office, Michigan State University, Writing Test Items, http://scoring.msu.edu/writitem.html

Blueprints that include depth

PathophysiologyInvestigation and

diagnosis

Treatment and

management

recallhigher

orderrecall

higher

orderrecall

higher

order

Gastrointestinal P1Q1P1Q1,

P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

NervousP1Q2,

P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5

Blueprint

• Enables you to plan and check that an exam, or

set of exams, covers the content it is meant to

cover

Question difficulty

Should questions to be difficult?

• Norm-referenced vs standards-based grading

Question difficulty

• Familiarity - obscurity

eg:

– Name the two most significant muscles involved for

mastication.

– Explain the physiological actions of insulin

– Explain the physiological actions of ghrelin

Question difficulty


• Level of task

– Complexity

– Abstractedness

Question difficulty


• Level of task

– Complexity

– Abstractedness

• Structure

Structure

Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)

a) Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats (10 marks)

b) Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)

c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).

Strategy guidance

You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario.(20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide

Question difficulty


• Level of task

– Complexity

– Abstractedness

• Structure

• Resources

Question difficulty


• Level of task

– Complexity

– Abstractedness

• Structure

• Resources

• Wording

General issues with question

wording

CommunicationThe examination question is the question setter’s expression

of the question setter’s task.

The student’s answer represents the student’s expression of

the student’s interpretation of the question.

The marker evaluates the marker’s interpretation of the

student’s expression of the student’s answer.

The marker uses the marker’s interpretation of the setter’s

expression of the setter’s task to evaluate the student’s

answer.

Modified from Pollitt & Ahmed (1999) New Model of the Question Answering Process. IAEA. Bled, Slovenia

Expectations and stereotypes

Examples:

• male animal case

• all differential diagnoses vs those only applicable in a particular case

• “treatment”

• expectation of hard questions

• expectation that Qs will ask about what something is rather than what it is not

Contextualising Qs

• Context is good because it brings relevance and

authenticity

• Allows assessment of concrete or specific examples

not abstract concepts or generalisations

• Allows assessment of applied learning (doing not

just knowing)

• All these carry with them a potential for bias.

Real world context

• Advantages– Assessing application to a real problem

– “Shows how” rather than “knows”

• Disadvantages– More words used

– More intended/unintended demand

– Familiarity – schemas/sterotypes

– Focus may direct candidates to the wrong aspects

– Images can be particularly distracting

Guidelines for context

• If you are going to use context use a natural real

one

• Contextualise purposefully for what it brings to

the task

• Only use images when the Q could not be asked

without them

• Don’t decorate!

Unpacking or sifting

• A crate of 12 cans of cola costs $4.20. How

much do 7 crates of cola cost?

• A ski pass costs $4.20. How much would it cost

for 7 days?

Ahmed & Pollitt (2007) Assessment in Education: Principles, Policy & Practice 14:201-232.

A client rings you up to arrange his quarterly visit as he is frustrated by ongoingproblems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear.

On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present.

You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)

Other aspects

• Distractors

• Everyday language

• Highlighting of words and phrases

• Showing the mark allocation

Wording questions - examples

Don’t write questions; write tasks

What is your diagnosis?

State the most likely diagnosis orState the most likely diagnosis and explain your reasoning orDiscuss the differential diagnoses you would consider in this caseor…..

Instructional verb examples

Compare: to find similarities between things, or to look for characteristics and features that resemble each other.

Contrast: to find differences or to distinguish between things.

Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.

Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.

Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.

Illustrate: to use a picture, diagram or example to clarify a point.

Specify boundaries of the answer

Species

e.g. “in both dogs and cats…”

Quantities and amounts

e.g. “Provide 5 reasons why…”

With reference to

e.g. “ With reference to the published research from ..”

List the clinical signs of hypothyroidism in dogs.

List the three most common owner-observed clinical signs of hypothyroidism in dogs andexplain how thyroid hormone deficiency leads to each of these signs.

Q parts – the scope

Examples of problems….

Name two (2) diagnostic tests you would run next to

investigate the cause of this dog’s current illness.

Examples of problems

Outline your approach to confirming the initial

clinical diagnosis and a management and

prevention plan for this problem. This discussion

should include an outline on further observations

taken about ….


…list in dot point form: the gross pathological

features, the characteristic histopathological

changes, and the clinical pathology changes. In

your discussion, list one antemortem

test/procedure that can be used to aid in the

diagnosis …


A veterinarian asks you for assistance in designing

a protocol for the delivery of a vaccine for cats in

their practice. What factors would you take into

consideration in designing this protocol?

Examples of question problems

Are there any clinical features which can help you

determine a patient’s prognosis?


Describe and discuss the following:

a) preparedness


State what you believe is your most likely

diagnosis.


Discuss commonly found tumours and tumour-like

disorders associated with the oral cavity and

dental tissues of the horse.


How would you localise the site of the lesion?

Answer provided in the marking scheme:

Spinal lesion between T3 and L3


Write notes on

a) considerations in the selection of stockpersons

and animal attendants


• Describe and give reasons for your further

clinical examination (if any), and

recommendations to the owner/trainer for

diagnostics and treatment. Explain what you

think is really important and why. Explain if you

think there is any controversy in treatment plans.


Version 1

Using examples, describe the categories of ways that the productivity and profitability of a livestock enterprise may be compromised by disease.

Version 2

Using examples, describe at least five different ways that the productivity and profitability of a livestock enterprise may be compromised by disease.

Sentence-level clarity

• Simple sentence structures

• Grammatically correct

• Subject verb object sequence

• Remove superfluous words

Question timing

How long it will take the candidate

• Unstructured tasks – open ended

• Need to control the demands of the Q carefully

• Question reading time

• Writing speed

Klatt & Klatt (2011) Acad Med. 86:1079–1083

Summers & Catarro (2003) Australian Occupational Therapy Journal 50(3): 148-157

Effect of time stress

• Time stress

– Increases the use of schemas (sterotyping)

– Decreases working memory processing capacity

– Decreases the ability to maintain relevant information

and suppress irrelevant information

http://www.slideshare.net/liznorman