1 blueprinting
TRANSCRIPT
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting and drafting questions
Liz NormanMassey University
Sharanne Raidal, BVSc, PhD, FANZCVS, GradDipEd
http
://co
nsul
tatu
torb
log.
files.
word
pres
s.co
m/2
010/
06/e
xam
stre
ss.g
if
Australian and New Zealand College of Veterinary ScientistsExaminer Workshop, 6 – 7th February 2016
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Overview
• Blueprinting and drafting questions• Grading criteria and marking schemes• Clarity, timing; oral examinations
http://www.slidehare.net/liznorman
http://toonclips.com/600/4997.jpg
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What is a blueprint?
blueprint, n. something which acts as a planmodel ortemplate
OED Online http://www.oed.com https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What is a blueprint?
• Specifies what’ going to be in the exam Content / topics (breadth) Level (depth)
https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Purpose of blueprinting
• To document sampling• To show that your
examination is representative of all that could be examined and is sufficient
Content domain
Exam 2
Oral
Essay
Credentials domain
Exam 1
Prac
Practice domain
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Why blueprint?
• To show that we have a representative sample
• To permit extrapolation to content (practice) domain
• To plan how each component assessed
Content domain
Exam 2
Oral
Essay
Credentials domain
Exam 1
Prac
Practice domain
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Content of questions
• Scope of topics• Other relevant considerations - eg. species
body systems• Scope of skills and abilities
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Subject guidelines
• Represent the entire content domain (ie. they define all that is examinable)
• Specify both scope (breadth) and level (depth) of the knowledge, skills, attitudes and judgements required
• For Fellowship, some aspects are assessed in the credentials document (and process and training)
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Subject guidelines
The candidate will have a detailed knowledge of:
The aetiology, pathogenesis and pathophysiology of cardiac, renal, respiratory, alimentary, musculoskeletal, endocrine, ophthalmological and neurological organ dysfunction in the cat and the dog.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Level – depth Levels of knowing and understandingThinking processes required by the discipline
•Surface vs deep•Fact recall vs application•Blooms taxonomy•SOLO taxonomy
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring several facts).
Applied (higher order):Questions that require the use of facts or concepts, the solution of a diagnostic of physiologic problem, the perception of a relationship or other process(es) beyond recalling discrete fact.
From: Peitzman et al. (1990) Academic Medicine , 65(9): S59-60
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Bloom’s taxonomy
http://julietovar.edublogs.org/files/2011/05/blooms-taxonomy-1k4snjn.JPG
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Bloom’s taxonomy
http://www.psia-nw.org/wp-content/uploads/Blooms_Taxonomy.jpg
Recall
Application
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities requiredBloom’s instructional verbsCreate: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare.
Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate.
Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise.
Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch.
Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell.
Remember: define, repeat, record, list, state, recall, name. Recall
Apply
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
http://www.pescholar.com/wp-content/uploads/2013/03/Solo-SS.png
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
SOLO taxonomy
http://litre.ncsu.edu/solowrittencomposite.gif
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions skills & abilities required
SOLO taxonomy
http://www.mbms.org.uk/faculties/science/assets/SOLO/solo-taxonomy-explained-full.jpg
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking
Quantitative change
Qualitative change
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking
Deep
Surface
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking
Application
Recall
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise
Multistructural Combine, classify, describe, enumerate, list, do algorithm, apply method, account for, execute, formulate, solve, conduct, prove, complete, characterise
RelationalAnalyse, compare, contrast, integrate, relate, explain causes, apply theory, argue, implement, plan, summarise, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt
Extended abstract Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, critique, reason
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Skills and abilities
•What is contained in subject guidelines?•Skills beyond cognitive?•Recognition / interpretation?
•Communication?
- diagnostic images- cytology / pathology- reports - statistics
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Mapping to scope and level
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by learning outcome
Written paper 1
Written paper 2
Practical examination
Oral examination
LO1 x x x
LO2 x x
LO3 x x x
LO4 x
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by topic
Patho-physiology
Investigation & diagnosis
Treatment & management
Gastrointestinal P1Q1 P1Q1, P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by topic and depthPatho-
physiologyInvestigation &
diagnosisTreatment & management
recall higher recall higher recall higher
Gastrointestinal P1Q1 P1Q1 P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by species
Species Number of Qs Percent
Small animal 15 52%
Farm animal 4 14%
Horse 5 17%
Exotic 3 10%
Lab 1 3%
All 1 3%
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Content coverage x depthTopic 1
30%Topic 2
30%Topic 3
40% Total
Knowledge of terms 2 5 5 12
Comprehension of principles 4 3 4 11
Application of principles 3 3 3 9
Analysis of situations 3 2 5 10
Evaluation of situations 3 2 3 8
Total questions 15 15 20 50
http://scoring.msu.edu/written.html
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting
Enables you to plan and check that your exam, or set of exams, covers the content (and skills) it is meant to cover, at an appropriate level.
Without this you cannot extrapolate performance on this exam to performance in the whole domain.
Validity
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Should questions be difficult?
Norm-referenced vsstandards-based grading
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Should questions be difficult?
Appropriate demandvs irrelevant difficulty
Practice domain
Content domain
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
What makes questions difficult?
•Familiarity / novelty•Complexity – components, links•Scale – components, links•Resources utilised (provided, generated)•Abstraction – concrete experience, concepts, principles•Task strategy – simple, stepwise, integrated•Guidance
Adapted from Hughes et al (1998) The development of a tool for gauging demands of GCSE and A level exam questions. Brit Ed Res Assoc Conference.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Familiarity / novelty
Question 1Explain the physiological actions of insulin.
Question 2Explain the physiological actions of ghrelin.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Familiarity / novelty
Question 1List four (4) clinical signs of diabetes mellitus in dogs.
Question 2List four (4) clinical signs of feline hypersomatotropism.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Complexity, scale – components, links
Question 1Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs.
Question 2Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Complexity, scale – components, links
Question 1Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats.
Question 2Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Resources utilised – provided, generated
Describe the skull.(30 marks)
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Abstraction – concrete experience, concepts, principles
Discuss the concept of quality of life in terms of biological function, “feelings” and natural existence.
Describe how both classical conditioning and operant conditioning are involved in cows confidently entering the milking shed and letting down.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise, integrated
Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats.
(25 marks)
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise, integrated
Question 2a)Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats. (10 marks)b)Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)c)Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus. (10 marks)
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise, integratedGuidance
Describe the skull. (30 marks)
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise, integratedGuidance
You are presented with a 7 year old male Warmblood gelding with exercise intolerance. You auscultate an arrhythmia with a variable pulse and order an ECG. Define the type and give a specific name for the arrhythmia demonstrated in the rhythm strip below. (2 marks)
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise, integratedGuidance
You have been contracted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)
Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider, and detail how you might further investigate potential causes, and what advice you would provide.
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What’s different about oral questions?
• Thinking on your feet• Can’t revisit or edit• Tests verbal communication skills (as well as
knowledge)• Interactive – can prompt and redirect, can ask
candidate to explain their answer
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Key points
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting
• Ensure the examination (as a whole) representatively samples from the domain specified for the examination
• Membership – this is all the learning outcomes
• Fellowship – this is all the learning outcomes, except those that can only be sampled in the credentials document
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting
• Ensure the examination (as a whole) representatively samples from the domain specified for the examination
• Consider - topics- level- species, body systems, modalities- skills - other things (eg. Dx vs mgt, principles vs application)
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Structuring the exam
• Ensure you follow subject guidelines explicitly• Shorter vs longer questions• Scenario vs direct questions• Difficulty, guidance
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Next session’s task:
• Check your blueprint - content- level
• Classify your draft Qs according to SOLO taxonomy
• Redraft Qs if necessary
TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT
SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise
Multistructural Combine, classify, describe, enumerate, list, do algorithm, apply method, account for, execute, formulate, solve, conduct, prove, complete, characterise
RelationalAnalyse, compare, contrast, integrate, relate, explain causes, apply theory, argue, implement, plan, summarise, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt
Extended abstract Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, critique, reason