p2p cadre meeting september 29, 2009 welcome! enjoy some breakfast and networking!

P2P CADRE MEETINGSEPTEMBER 29, 2009

Welcome! Enjoy some breakfast and networking!

Updates

Modules and kit status Teaching timeframe Calendar review for the year Upcoming in-district meetings Today’s meeting

ATLAST ITEM WRITING WORKSHOP

Horizon Research, Inc.May 15-16, 2009

Made possible by a grant from the National Science Foundation (Grant no. EHR-0335328). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the presenters and do not necessarily reflect the views of the National Science Foundation.

Classroom Vignette

After a unit on seasons, a science teacher included the following item on a test:

Which of the following locations would you expect to have the least

temperature variation throughout the year?(a) Toronto, Canada(b) London, England(c) Sydney, Australia(d) Manta, Ecuador

The teacher had done a thorough job teaching the unit, and based on student responses to questions she asked during class, she was confident they would recognize a location on the equator as the correct answer. She was disappointed and confused when three-fourths of the class missed the item. As she returned the tests, she asked several students why they got the question wrong. Almost without exception, each student asked, “Where’s Ecuador?”

Excerpted from: Taylor, M. & Smith, S. (2009). How do you know if they’re getting it? Writing assessment items that reveal

student understanding. Science Scope, 32(5), 60–64.

Meeting Goals

1. Familiarity with a process for developing student assessment items.

2. Familiarity with some principles for developing student assessment items.

3. Opportunity to apply the process and principles for developing student assessment items.

Overview of Day

Overview an assessment development process

Clarifying the content domain Principles for writing student

assessment items Practice applying principles Write items in small groups Revise items in large group Orient to interviewing

A PROCESS FOR DEVELOPING ASSESSMENT ITEMS

Identify Science Topicsand MisconceptionsAbout Those Topics

Write MultipleChoice Items

InterviewStudents andRevise Items

Pilot Items

CreateAssessment

CLARIFYING THE SCIENCE CONTENT TO BE ASSESSED

Assessment Development

Clarifying the Science Content

In ATLAST, we started with middle grades benchmarks.

Flow of matter and energy in living systems

Force and motion Plate tectonics

Overview of Benchmark Clarifications

Within each benchmark, we identified narrowly defined sub-ideas.

Sub-ideas are the smallest assessable piece of information

Within each sub-idea, we identified relevant common student misconceptions.

Why define the content domain?

It affects the validity of the assessment.

The content domain determines the boundaries of the assessable content.

CULLING STUDENT MISCONCEPTIONS RELATED TO THE SCIENCE CONTENT

Why identify common student misconceptions?

They are a source of plausible distractors for assessment items.

Sources of Student Misconceptions

Syntheses of researched misconceptions Content experts and educators with K-12

experience K-12 students

Eliciting Student Ideas

Answers to open-ended questions inform teachers of how students think about the content

Sample prompts for each content area are included in binder

Key questions organize the prompts by the particular aspect of the content they target

SOME PRINCIPLES FOR ITEM WRITING

Some Principles for Item Writing

Content Facilitating comprehension Structure Format Avoiding test-wiseness

CONTENT

Content

1. Target the item; one sub-idea per item if possible.

Why? Precision Good coverage of all sub-ideas within the

benchmark Cognitive load gets higher as more sub-ideas

are being assessed within one item

Content

2. Necessity principle: The knowledge in the sub-idea is needed to answer the item, so the right answer should be about the sub-idea.

The Necessity Principle

The knowledge in the sub-idea is needed to answer the item.

OR

The item cannot be answered correctly without knowing the sub-idea.

Practice Applying the Principle: Items 1A, 1B, and 1C

For each item,… Read the targeted sub-idea, correct

answer, and principle(s) being violated

Review the item

Identify/highlight why the item violates the principle(s)

Item 1A-Flow of Matter and Energy

Which one of the following energy transformations happens in a plant?

A. Light energy is transformed into chemical energy.

B. Heat energy is transformed into chemical energy.

C. Heat energy is transformed into motion energy.

D. Motion energy is transformed into heat energy.

Item 1B-Force and Motion

A student throws a ball straight up. Which one of the following best describes the motion of the ball as it falls down?

A. The ball moves faster and faster because of the force due to gravity.

B. The ball moves slower and slower because the force from the kick is acting against gravity.

C. The ball moves slower and slower because the force of motion is opposing the force due to gravity.

D. The ball moves at a constant speed because gravity is constant.

Item 1C-Plate Tectonics

In which one of the following places would it be most likely for an earthquake to happen?

A. Along the Pacific coast of North America

B. Along the Atlantic coast of North America

C. In the middle of the Pacific Ocean

D. In the middle of North America

Content

3. Sufficiency principle: The knowledge in the benchmark is ALL a student needs to know to answer the item correctly.

The distractors may be about other sub-ideas as long as they’re within the benchmark, but it’s best to have distractors also be in the sub-idea.

The Sufficiency Principle

The knowledge in the benchmark is ALL a student needs to know to answer the item correctly.

OR

Answering the item correctly requires NO knowledge outside the benchmark.

Force and Motion Benchmark

An unbalanced force acting on an object changes its speed or direction of motion or both.

Example:

Which of the following is a force that opposes motion?

A. AccelerationB. InertiaC. MomentumD. Friction




Review the item



Which one of the following is NOT food for humans?

A. SugarsB. FatsC. ProteinsD. Vitamins


A girl slides a box across the floor. Which of the following is a force that acts on the box?

A. The energy the girl transfers to the box

B. The momentum of the box

C. The inertia of the box

D. The friction between the floor and the box


Why do earthquakes occur more often in some areas than in others?

A. Earthquakes tend to occur where there are large oil deposits underground.

B. Earthquakes tend to occur in desert regions, where the ground is unstable.

C. Earthquakes tend to occur near the edges of plates.

D. Earthquake location is currently not well understood by scientists.

Content

4. Item should not teach content.

Example:

A girl gets on her skateboard and pushes off with her foot. As she moves across the gym floor, she moves slower and slower and eventually comes to a stop. Why does she come to a stop?

A. Because the force due to her foot runs out.B. Because the force in the skateboard runs out.C. Because the force of friction is acting on the skateboard.D. Because the force due to gravity is greater than the force due to friction.

Correction:

A girl gets on her skateboard and pushes off with her foot. As she moves across the gym floor, she moves slower and slower and eventually comes to a stop. Why does she come to a stop?

A. Because the force due to her foot runs out.B. Because the force in the skateboard runs out.C. Because the force of friction is acting on the skateboard.D. Because there are no forces acting on the skateboard.




Review the item



Which one of the following do plants NOT need to make food?

A. WaterB. LightC. Carbon dioxideD. Soil


A boy slides a salt shaker along a table toward the right. Friction acts to oppose the motion of the salt shaker. As the salt shaker slides, in which direction does the force of friction act on the salt shaker?

A. To the left

B. To the right

C. Upward

D. Downward


Which one of the following best describes how fast Earth’s plates move?

A. A few centimeters a yearB. A few centimeters a monthC. A few meters a yearD. A few meters a month

Content

5. Mirroring: Answer choices should reflect the question being asked.

Example:

How do plants get their food?

A. Plants make food from minerals, water, and sunlight.

B. Plants make food from carbon dioxide, water, and sunlight.

C. Plants get food by absorbing water through their leaves.

D. Plants get food by absorbing nutrients through their roots.

Example:

A sailboat is traveling in the ocean, as shown in the diagram. The wind exerts a force that pushes the boat forward (toward the right side of the page), and the water exerts a weaker force to the left.

How do these forces interact?

A. The forces combine to create a total forward force that is smaller than the force due to the wind.B. The forces combine to make a total forward force that is larger than the force due to the wind.C. The forces cancel each other out.D. The force to the left doesn't matter because the force to the right is stronger.

AVOIDING TEST-WISENESS

Avoiding Test-wiseness

Students tend to…

Choose the longest answer

Choose the most “science-y” answer

Rule out answers with strong qualifiers (e.g., only, never, always)




Review the item



Grass is food for cows. How does the cow use the grass?

A. All the grass leaves the cow’s body after the energy is absorbed.B. All the grass leaves the cow’s body after the water is absorbed.C. The grass is broken down and some of it becomes part of the cow’s body.D. The grass is broken down, but none of it becomes part of the cow’s body.


A school bus is slowing down as it comes to a stop sign. What is true about the forces acting on it while it is slowing?

A. The forces in the forward direction are greater than the forces in the backward direction.B. The forces in the forward direction are less than the forces in the backward direction.C. The only force acting on the bus is friction.D. There are no forces acting on the bus.


Which one of the following is true about Earth’s plates?

A. Earth has one large plate of solid rock.B. Earth has one large plate that has some solid parts and some liquid parts.C. Earth has a number of plates of mostly solid rock.D. Earth has a number of plates, all of which are liquid.

FACILITATING COMPREHENSION

Facilitating Comprehension

7. Keep sentences brief and straightforward, with a simple phrase structure and no additional clauses.

Example:

The South American plate and the Nazca plate are neighboring plates, with no other plates in between. Scientists say that the South American plate and the Nazca plate are moving toward one another.

Which one of the following best explains what they mean when they say that these two plates are moving toward one another?


12. Use present tense and active voice only.

Example:

A shopping cart is pushed so that it moves faster and faster. What is true about the situation?

Correction:

A man is pushing a shopping cart. The cart is moving faster and faster. What must be true about the situation?


12. Visuals should facilitate the understanding of what is being asked; don’t include a visual just to include a visual.

No extraneous information Should complement the presented scenario Labels should mirror wording in the item

STRUCTURE

Structure

Restricted to four answer choices

Avoid multiple correct answers (i.e., do not use “Both A and B are correct”)

Avoid using “None of the above”

FORMAT




Review the item



When a baby chick develops inside an egg, the yolk in the egg is its only source of food. As the chick grows, the yolk decreases in size. Why does the yolk decrease in size?

A. Matter is destroyed during the growth of the chick. B. The yolk provides the building blocks the chick is made of. C. The yolk is a source of energy for the chick but not a source of body parts. D. The yolk decreases in size to make room for the growing chick.


A car is moving faster and faster along a straight, level section of road. Which of the following must be true about the forces on the car?

A. The total force in the direction of the car’s motion is greater than the total force in the opposite direction.B. There is an increasing total force in the direction of the car’s motion.C. The total force in the direction of the car’s motion is greater than the car’s force.D. There are no forces in the direction opposite to the car’s motion.


Locations 1, 2, and 3 are shown on a map of Iceland to the left. The arrows indicate directions of plate movement.

Where would you expect to find the youngest rock?

A. Location 1B. Location 2C. Location 3D. The rock is the same age at all 3 sites.


Locations 1, 2, and 3 are shown on a map to the left. The arrows indicate directions of plate movement.

Where would you expect to find the youngest rock?

A. Location 1B. Location 2C. Location 3D. The rock is the same age at all 3 sites.

Most Important Rules

1. Target the item; one sub-idea per item if possible.

2. Necessity principle: The knowledge in the sub-idea is needed to answer the item, so the right answer should be about the sub-idea.

3. Sufficiency principle: The knowledge in the benchmark is ALL a student needs to know to answer the item correctly. The distractors may be about other sub-ideas as long as they’re within the benchmark, but it’s best to have distractors also be in the sub-idea.

Taylor, M. & Smith, S. (2009). How do you know if they’re getting it? Writing assessment items that reveal student understanding. Science Scope, 32(5), 60–64.

Science Scope Article

Content-Specific Guidelines

Developed during the process of item writing to address issues that arose in student and/or content expert feedback

Please review prior to writing items

WRITING ITEMS

Writing Items

Goals and Guidelines Helpful hints Logistics

Goals and Guidelines

Goal: Write 2-3 items per pair/group If you want to work independently for a

portion of the time, please make sure you move toward collaboratively drafting, reacting to, and/or editing items after the first 20 minutes.

Use the item writing checklist to “interview” your own items

“Interview” Your Items

Have you clearly defined the content you are assessing? Do you have a good sense of what is fair game?

Does the item clearly target a specific idea? Do students have to know the content to answer

the item? Could they answer it correctly without knowing the content?

Is the target content sufficient for students to be able to answer the item? Do they need to know anything outside the target content?

“Interview” Your Items

Does the item inform students about the correct answer to other items?

Do the answer choices mirror the wording of the item, or is there a mismatch?

Does the item: keep sentences as brief as possible? use present tense and active voice only? make appropriate use of visuals?

Can students answer the item without knowing the content by using common test-taking strategies?

Helpful Hints

Use existing items as fodder for new items (your own or samples).

Use the “related prior student conceptions” as a source of distractors.

Use textbooks, diagrams, etc. to get ideas for items around specific sub-ideas.

GROUP EDITING OF ITEMS

Attack the item; affirm the writer.

OVERVIEW OF STUDENT COGNITIVE INTERVIEW PROCESS

Purposes

Cognitive interviews help…

1. Determine the item’s validity2. Determine the item’s clarity3. Determine the item’s plausibility; and4. Identify specific improvements/

revisions to the item

Purposes

1. Validity Does the student think the item is asking

what we intended it to ask? Does the student use the knowledge we

are trying to measure to answer the item? Is the student getting the item

correct/incorrect for the appropriate reason (i.e., understanding/not understanding the knowledge being measured rather than the wording being tricky or because of test wise-ness).

Purposes

2. Clarity• To make sure the item is clear and the

wording and/or graphics are not confusing

3. Plausibility• To determine if the scenario is realistic

4. Improvements• To elicit suggestions for alternative

distractors (e.g., one that may be more realistic or better aligned with a common misconception)

Interview Protocol

Prologue

Purpose of the prologue Helps explain what we are doing; Helps put the student at ease:

Interview is voluntary; Interview is anonymous; Stresses that we aren’t testing him/her, rather

we’re trying to determine if the items are any good.

Background Information

Helps put responses into context; Allows us to internally track:

who has been interviewed; around which items; and when.

Procedure

Ask student to read and think aloud as they read the item and consider the answer choices. Gives you a sense if wording is tricky (e.g.,

s/he re-reads the item several times); and Helps you determine why student is

answering the way s/he is (e.g., real understanding, guessing, test wise-ness).

INTERVIEW: PROCEDURE

Item-Specific Questions

1. Why did you choose that answer? [Validity]

2. What did you think of each of the other answer choices? [Validity]

3. Was there an answer choice you were expecting to see, but did not? [Improvements]

Item-Specific Questions

4. Were there any words or diagrams you did not really understand, or situations that made the question confusing? [Clarity and Plausibility]

5. Is there anything about the question that did not confuse you, but that you think might confuse other middle school students? [Clarity and Plausibility]

INTERVIEW NOTES

Interview Notes

Place to capture student thinking Record of which answer choices were

selected, considered, and/or ruled out Coding of item—Did the student answer

the question correctly or incorrectly for the right or wrong reason? (record after the interview)

STUDENT INTERVIEW

INTERVIEW: FORCE AND MOTION ITEM

Force and Motion Item

Sub-idea E: If an object is slowing down, then there is a net force acting on the object in the direction opposite to the object’s motion.


Tim stretches a rubber band between two posts and uses it to launch a toy car, as seen in the picture. He pulls the car back against the rubber band (as shown in 1), then lets go and the car moves across the floor (as shown in 2 and 3). The car slows down and eventually stops after a few meters.

Why does the car slow down and stop?


Why does the car slow down and stop?

A. The force due to the rubber band gets used up.

B. The car’s force of motion eventually runs out.

C. There is a force acting in the direction opposite to the car’s motion.

D. The car starts to slow down once the backward forces are equal to the forward forces.


The student chose the incorrect answer for the right reason—she did not fully understand the content

Content probing re: friction

Asked about all answer choices

PREPARING FOR STUDENT COGNITIVE INTERVIEWS

Interview Tips

Reassure/reinforce good responding, especially early in interview.

Don’t let the interviewee know what the correct answers are during the interview; you can review items after the interview if student desires.

Remember the main goal of interview is to check validity of item, not to uncover student thinking about the content.

Interview Tips

Are there any answer choices that you can definitely rule out?

Are there any new, unfamiliar, or confusing words in the question or answer choices?

Let’s just talk about each answer choice and what you thought they meant by it.

Let’s say you were taking one of those standardized tests and you just had to pick one answer to bubble in on the answer sheet. Which one would you choose?

I’m curious about what they meant by “X.” What do you think they meant by “X”?

I noticed they used the word “X.” If you had to explain to a younger student what “X” meant, what would you say?

Interview Forms

Includes all three content areas Flow of Matter and Energy 1-Green Flow of Matter and Energy 2-Blue Force and Motion-Purple Plate Tectonics-Tan

When interviewing, … Start with your own items, Then move on to other items in your content area, and Talk about items in the other content areas, as time

allows.

TAKE HOME MESSAGES ON ITEM WRITING

Item Writing

There are many item writing principles; a few are key to writing valid assessments.

1. Clarify the content domain2. Target item3. Necessity 4. Sufficiency

Next Steps in the Assessment Development Process

Identify Science Topicsand MisconceptionsAbout Those Topics

Write MultipleChoice Items

InterviewStudents andRevise Items

Pilot Items

CreateAssessment

Next Steps for P2P Item Writing Write science items for upcoming test Conduct 1 student interview of items by

March Reflect on how student interview

assisted/did not assist with item revision

Reflection

On an index card, write 3 things you learned today 2 things you plan to try 1 question you still have

Next Meeting

December 14th

Blue Licks Focus of meeting:

Score student work from PoM unit

Revise PoM units Prepare for

mentee training on units

p2p cadre meeting september 29, 2009 welcome! enjoy some breakfast and networking!

Documents

student responses

student understanding

culling student misconceptions

science teacher

science contentwhy

science scope

principleswrite items

assessable content