Creation, Development, and Analysis of Test

Questions for Educators BECKI KEITH, MS, RT(R)(CT)

ASSISTANT PROFESSOR AND DIRECTOR OF ADMISSIONS

DEPARTMENT OF RADIATION SCIENCES VIRGINIA COMMONWEALTH UNIVERSITY

AUSTIN TURNER, MS, CNMT, PET, RT(MR)

CLINICAL COORDINATOR DEPARTMENT OF CLINICAL HEALTH SCIENCES

SAINT LOUIS UNIVERSITY

a

Objectives

By the end of this session, the attendee will: Understand the concept of test-wiseness and its importance in

item writing and development

Apply Bloom’s taxonomy to the development of test questions

Identify differences in the application and illustration of knowledge in online and traditional learners, and how this may be different in different courses

Complete an interactive exercise using exam questions to reach a higher level of thinking and understanding of content

a

Outline

Part I: Write some questions

Part II: Learn/review some stuff

Part III: Evaluate/improve questions

a

Part I: Write Some Questions

a

What has changed since YOU went to school?

a

How did YOU learn in school?

See one/do one/teach one? ‘Sage on the Stage’ vs ’Guide on the Side’? How you learn may not be how others learn

Visual? Did you read the textbook only? Oral? Did you listen to lecture only? Tactile? Did you learn by doing? Read/Write? A mix?

Same goes for assessment

a

Where do you get your questions?

How many of you “inherited” your test questions? How many of you use the sample questions from

the text/online resource? How many of you periodically review/rewrite your

test questions?

a

What is a good question?

Effective test questions assess: whether a particular learning objective was achieved

or subject mastered by learner whether the instruction was successful

a

Question types

Question types include: multiple choice / multiple multiples FITB short answer / essay matching calculation true/false etc.

a

When to use what?

What kind of test is being delivered? For formative tests, perhaps questions that require recall and application

of facts are appropriate

For final exams, you might want to emphasize questions that assess the higher levels of thinking (Bloom’s)

What kind of technology is to be used for delivery? Some question features may not be appropriate, depending on the type of

technology to be used for delivery (e.g., slow data connection, differences in internet browsers and operating systems)

How much time is available to create/grade test questions? Certain types of questions require more time to create/grade than others

Depending on the time available, you may tend to create more of one type than another

a

Ask Yourself:

What am I testing?

Why am I testing?

How am I testing?

What results am I getting (or hoping to get)?

How am I going to use the results?

o What interpretations do you want to make with the scores?

o Can you change/improve content delivery?

a

Goal of quality item-writing (in a nutshell)

Examinees should get an item… Right - because they know the correct answer

or

Wrong - because they don’t know the correct answer That means the item has to work for you AND them!

a

Issue #1: Test-wiseness

What is test-wiseness? Test-taking skills, tips or tricks

Test prep or coaching

Test wisdom or sophistication Extra advantage gained from taking many tests

in multiple formats May be inherent, learned, or taught

a

“…any skill which allows a student to choose the correct answer on an item without knowing the correct answer. Students who are test-wise: - look for mistakes in test construction - make guesses based on teacher tendencies - search for unintentional clues found in a test Scores from a student who got lucky and guessed his or her way to a high score are meaningless and not valid. It is important to distinguish between this sort of guess, which good tests are designed to protect against, and an ‘educated guess,’ which is not nearly as harmful to the validity of a test.”

a

History

Robert L. Thorndike, educational psychologist and psychometrician

Conceptualized "test-wiseness" as a general and lasting characteristic of the individual that affects test scores (1951)

2nd most prominent source of score variance on cognitive tests (behind cognitive ability)

a

Definitions

1965: Millman, Bishop, and Ebel Defined test-wiseness as the capacity to utilize

the characteristics and formats of the test and/or the test-taking situation to receive a high score

Test-wiseness is logically independent of the examinee's knowledge of the subject matter for which the items supposedly measure

a

Test-wiseness Cues

1) What is the primary purpose of the cluss in frumpaling? a. to remove cluss-prangs b. to patch tremalls c. to lossen cloughs d. to repair plumots

- Answer stem is in the question

a

Test-wiseness Cues

2) The fribled breg will minter best with an: a. mors b. ignu c. derst d. sortar

- Grammar (consonant/vowel agreement)

a

Test-wiseness Cues

3) Why does the sigla frequently overfesk the treslum? a. all siglas are mellious b. siglas are always votial c. the treslum is usually tarious d. no tresla are directly feskable

- Absolutes, categorical exclusives

a

Test-wiseness Cues

4) Trassing normally occurs under what conditions? a. when dissels frull b. when luss trasses the vom c. when the belgo lisks easily d. when the viskal flans, if the viskal is zortal

- Number agreement/multiple conditions

a

Test-wiseness Cues

5) The mintering function of the ignu is most effectively performed in connection with: a. fribbled breg b. arazma tol c. groshing stantol d. frallied stantels

- Giveaway answer (from #2)

a

Test-wiseness Cues

6) What probable causes are indicated when tristal doss occurs in a compots? a. the kredges roted with the rots b. the sabs foped and the doths tinzed c. the rakogs were not accepted in the sluth d. the polats were thenced in the sluth

- Number agreement/multiple causes

a

Test-wiseness Cues

7) Which of the following are present when trossels are being gruven? a. rint and vost b. trelsum and shum c. shum and vost d. vost

- Convergence strategy

a

Test-wiseness Cues

8) The total population of fribled breg immediately prior to overfeskation is: a. 60 b. 6,000 c. 500 d. 600

- Convergence strategy & middle values

a

Test-wiseness Cues

9) Bonus question:

a. a b. b c. c d. d

- Answer patterns

a

Test-wiseness

Additional Cues

Absurd/Implausible Answer: Distracters are unrelated to the stem; some incorrect choices may be more obviously wrong and easier to eliminate

Precision/Specificity: Correct answer is more precise, clear, or qualified than the distracters; its level of detail and lack of ambiguity identifies it as the answer

Length: Correct answer is longer/shorter than the distracters

The acronym PACS stands for: Picture Archiving and Communications System Picture Array and Controlling System Photographic Array of Communication Systems Pretty And Curvy, Son!

a

Test-wiseness

Additional Cues

Odd Man Out: If it’s difficult to distinguish between two or more choices conceptually, that helps rule them out

Spelling: Test authors are more likely to make spelling errors on incorrect choices, since they mainly review the correct answers

a

Test-wiseness

Number of Options and Guessing:

Out of 100 multiple-choice items with 3-answer options each, a student with no knowledge of the content will get 33 correct simply by guessing blindly

The more choices available, the more susceptible the question becomes to test-wiseness (which is why 4 is common)

a

Bloom’s Taxonomy

a

Issue #2: Bloom’s Taxonomy

1956: Benjamin Bloom worked with educational psychologists to develop three domains of objectives for educators:

cognitive (knowing/head)

psychomotor(doing/hands)

affective (feeling/heart)

Facts

Skills

Critical reflection

a

Bloom’s Taxonomy

Nouns referring to increasing levels of

intellectual ability/effort

a

Bloom’s Taxonomy Revised

Verbs requiring increasing levels of

intellectual ability/effort (Anderson & Krathwohl, 2002)

a

Bloom’s Taxonomy

Knowledge/remembering learned info Define, describe, state Label, list, match, arrange Identify, recognize Recall, reproduce

Ex.: Describe the proper technique for a PA chest X-ray

Ex.: Label the components of the X-ray tube

a

Bloom’s Taxonomy

Comprehension/understanding of facts Classify, convert Explain, give examples

Ex.: Convert absorbed dose in cGy to rads

Ex.: Explain the process of DR image acquisition

a

Bloom’s Taxonomy

Application/applying knowledge Apply, change Calculate, solve Predict

Ex.: Apply the Inverse Square Law

Ex.: Calculate penetration depth

a

Bloom’s Taxonomy

Analysis/analyzing information or ideas Analyze, examine, appraise Compare, contrast, criticize Differentiate, distinguish, categorize

Ex.: Analyze the following image for artifacts

Ex.: Compare/contrast ionic & non-ionic CM

a

Bloom’s Taxonomy

Synthesis/creating new ideas or solutions Combine, assemble Develop, devise, prepare, generate Reconstruct, reorganize

Ex.: Devise a comprehensive department radiation safety plan

a

Bloom’s Taxonomy

Evaluation/evaluating evidence, judgments Defend, justify, support Judge, evaluate

Ex.: Justify the purchase of new lead aprons.

Support your conclusions with examples.

a

Additional Tips

Use multilevel thinking Questions that include wording such as “the most

appropriate” or “most important” can test a learner’s judgment skills or understanding of an in-depth subject

When verifying patient risk for renal effects prior to a contrast-enhanced CT exam, which of the following is the most important?

Patient's medication list

Patient's lab work results

Patient's history of shellfish allergy

Patient's history of receiving gadolinium-based contrast agents

a

Additional Tips

Avoid using excessive humor in the responses can get old quickly and can easily be offensive

List choices vertically for increased readability

Arrange responses logically, either in alphabetical, chronological or numerical order

a

Additional Tips

Course objectives should correlate to questions Objectives language may suggest a particular

type of test question Learner will be able to “discuss x” suggests an essay Learner will be able to “identify x” suggests a multiple

choice, short answer, or true/false question

Doing the work “up front” when developing learning objectives significantly reduces the work required to develop effective test questions

a

What about Online Courses?

How to improve attention/ability to learn while online? Researchers found that online lectures with periodic short

tests decreased mind-wandering 1/2, increased note-taking 3x, and improved overall retention of material

Testing at end of each segment --> incentive for students to pay closer attention because they'll have to answer questions at the end!

a

What about Online Courses?

How do I ensure that students don't cheat? Software may use randomization and question banks to limit

the possibility of cheating

Use or purchase 3rd party proctoring service

How many times to take the test? Depends on the purpose of the test

Allow multiple attempts so students retain the most information?

One attempt may force student to focus more

Remember: no right or wrong answer, only what works for you/them. The bottom line is always that the student actually understands and retains the information!

a

Considerations

Are you asking the right questions?

Are you testing all levels of knowledge?

Can you write a question that negates test-wiseness?

a

Considerations

Do your test scores reflect subject knowledge or test-wiseness? Or both?

What if a student learns to pass tests according to the test writer’s idiosyncrasies?

Is it fair if under-prepared students perform better than students who are prepared?

a

Part III: Evaluate/Improve Questions

a

Questions?