![Page 1: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/1.jpg)
Question setting
CIEA Insight is a key support for CIEA members.
Through Insight you can tap into a range of materials in one place that can:
• help you develop your own knowledge and skills in specific areas of assessment• help you develop your team’s knowledge and skills • give you access to information that we hope you will find stimulating and informative.
![Page 2: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/2.jpg)
Question setting
This short CIEA Insight will give you a glimpse of some key aspects of question setting
More detailed information on the subject is available to CIEA members in the members area of www.ciea.org.uk
This Insight will cover
Question typesSome general points on good practiceThings to avoid
![Page 3: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/3.jpg)
Question setting
Whether you are setting questions for assessment in your own centre, class or tutor group or involved in setting questions for public examinations, there are still good and bad approaches to the process.
![Page 4: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/4.jpg)
Question setting
Compare two main characters from different works of literature from different centuries.
Good question?
![Page 5: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/5.jpg)
Question setting
Compare two main characters from different works of literature from different centuries.
“There is no evidence that Lady Macbeth liked fish; Tarka the Otter loved it.”
“Whereas Tarka was despatched with a spade,it was a balcony wot done for Lady Macbeth.”
![Page 6: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/6.jpg)
Question setting
Poorly written questions have a negative impact on the learners who are faced with them
Poorly written questions also create frustration amongst assessors
![Page 7: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/7.jpg)
Types of Question
Summative AssessmentMultiple choiceShort answerExtended answer / essay
Continuous AssessmentTasksEnd of unit testsExtended essays
![Page 8: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/8.jpg)
Types of Question
Knowing which type of question to use, when, is a skill in itself
![Page 9: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/9.jpg)
Multiple Choice Questions
Benefits•Objective marking – requires no interpretation from marker•Quick to answer•Test a breadth of knowledge quickly•Marking is fast and consistent
Disadvantages•Limited testing of skills•Need very tight focus•Longer to set than other question types•Setting is a specialist skill
![Page 10: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/10.jpg)
Multiple Choice Questions
Terminology
Question = stem
Correct answer = key
Choices = options
Incorrect answers = distractors
![Page 11: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/11.jpg)
Multiple Choice Questions - Example
The capital city of Spain is (STEM)
A Barcelona (DISTRACTOR)
B Madrid (KEY)
C Valencia (DISTRACTOR)
D Aragon (DISTRACTOR)
![Page 12: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/12.jpg)
Multiple Choice Question Stems
Stems can be written in various ways
Among these are:
Completing a statementThe capital city of Spain is ...
Logic and reasoningWhich of the following statements can be justified using the information in the passage alone?
Odd one outEach of the following is a cause of drought except ...
![Page 13: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/13.jpg)
Negative Stems
Using the negative in a Stem can be confusing
There are three things wrong with this question
Which of the items in the following list is not one of the major organs of the human body?
A SkinB HeartC LegD Pancreas
![Page 14: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/14.jpg)
Vague and Confusing Stems
Which of the items in the following list is not one of the major organs of the human body?
A SkinB HeartC LegD Pancreas
Some learners will focus on the word “organ” and simply pick one out
The word “major” has not been defined
The answer “Leg” is too obvious
![Page 15: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/15.jpg)
Vague and Confusing Stems
This question stem is confusing
Which of these is not a good reason for the world economy to slow down in the event of a major natural disaster?
![Page 16: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/16.jpg)
Vague and Confusing Stems
Which of these is not a good reason for the world economy to slow down in the event of a major natural disaster?
It uses the negative
The main thrust of the question is separated from the opening
It is difficult to word a correct answer and even harder to write distractors
![Page 17: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/17.jpg)
Carefully Worded Odd One Out Questions
It is valid to test the ability to recognise which items in a list stand out in some way
Using the format:
Each of the following is ... except
can be a good way around the confusing negative stem problem
![Page 18: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/18.jpg)
True / False Questions
True / False questions are not true multiple choice questions
e.g. A basic principle of physics is that energy can neither be created nor be destroyed
True or False?
![Page 19: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/19.jpg)
True / False Questions
A better approach would be to ask:
A basic principle is that energy
A can be created but not destroyedB can be destroyed but not createdC can be destroyed and createdD cannot be destroyed or created
![Page 20: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/20.jpg)
True / False Questions
A basic principle is that energy
A can be created but not destroyedB can be destroyed but not createdC can be destroyed and createdD cannot be destroyed or created
Even here the correct answer is the only one to contain the word “cannot” so it might need further revision
![Page 21: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/21.jpg)
Multiple Choice Questions Summary
You have now seen some of the benefits of multiple choice questions.
They can:
•be objective •be answered quickly•test a breadth of knowledge•be marked quickly and consistently
![Page 22: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/22.jpg)
Short Answer Questions
SAQs can be marked
By human markers - possibly subject specialistsBy human markers who are not specialists
On screenOn line
Automatically - if the list of possible outcomes is finite and easily referenced
![Page 23: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/23.jpg)
Short Answer Questions
Be clear in your demands
Avoid ambiguitye.g. Say how glaciers move
SLOWLY
And the old maths classic:
![Page 24: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/24.jpg)
Short Answer Questions
are
useful for developing points slightly further than multiple questions would allow
a good way to explore a learner’s knowledge quickly
easily administered and marked - but not always automatically
are not
Multiple Choice QuestionsEssay titles
If the point you wish to test is ideally suited to another method then use the other method
![Page 25: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/25.jpg)
Extended Answer Questions
Traditional essay style questions
Used to test knowledge in depth
Need to be carefully worded to test what is actually required
![Page 26: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/26.jpg)
Writing Extended Answer Questions
e.g. Give an account of “Macbeth”
This pretends to be an essay but is nothing more than a request for a candidate to write out the plot.
A low order skill would actually be tested
![Page 27: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/27.jpg)
Writing Extended Answer Questions
e.g. Macbeth is a good man who allows himself to be tricked by the forces of evil.
Discuss the above statement with reference to at least three scenes from the play.
This is clearer in its demands and gives an element of structure to the task which was not present in the previous question.
![Page 28: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/28.jpg)
Structuring Extended Answer Questions
Having said that ...
Don’t turn an Extended Answer Question into nothing more than a series of SAQs (Short Answer Questions)
This can be the case if the question has
• too many parts
e.g. Give two reasons why Britain sent troops to France in 1914 and three reasons why the USA entered the war and two outcomes of the treaty of Versailles.
![Page 29: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/29.jpg)
Deriving Extended Answer Questions
The question is being asked in order for a candidate to demonstrate knowledge and/or skills
The question should come naturally from a programme of study and not be tacked on artificially
![Page 30: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/30.jpg)
Extended Answer Questions
can
assess knowledge in depthprobe into skills
should not
lead to repetitive low order tasksbe unconnected to a programme of study
![Page 31: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/31.jpg)
Showing Off
It might be interesting to set questions for the local pub quiz which test esoteric knowledge
BUT
Question setters should resist the temptation to show the world how clever they are by setting candidates ridiculously difficult tasks
If there is a clear form of words then use it
![Page 32: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/32.jpg)
Remember what you are testing
The question is being asked for a clear reason
It is in clear language
It asks what it seems to ask
Deciphering the question should not be the test
![Page 33: Question setting CIEA Insight is a key support for CIEA members. Through Insight you can tap into a range of materials in one place that can: help you](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ccf5503460f9499b58b/html5/thumbnails/33.jpg)
Teaching to the tests or testing what is taught?
Whilst the role of assessment remains very important in school, college and workplace education and training, we should all remember that assessment is not an end in itself.
Keep a sensible balance between teaching / training and the various forms of assessment.
The CIEA hopes to engage members in wide ranging discussion about the nature and the future of assessment.
You can find out more about question setting in the members’ area of the CIEA website - www.ciea.org.uk