dec. 11, 2012, sung- youn choi

Students Learning in the IQWST & LP

Improving middle school students’ understanding of core science

ideas using coherent curriculum

Dec. 11, 2012, Sung-Youn ChoiPost-doc of CREATE 4 STEM Institute, Michigan State University

IQWST & LP

What will we do today?

• Examine why we should redesign curriculum materials• Introduce the IQWST curriculum design; the next generation of

middle school science materials• Introduce the test design; aligned with a LP approach• Discuss key design principles and the CCD process• Share some of our findings; using IRT analysis• Discuss about improving middle school students’ understanding of

core science ideas using coherent curriculum

– Investigating and Questioning our World through Science and Technology (IQWST)

– Learning Progression (LP) – Construct-Centered Design Process (CCD) – Item Response theory (IRT)

2

IQWST & LP 3

Why redesign curriculum?

• Inadequate Science Materials– Cover many topics at a superficial level– Focus on technical vocabulary– Fail to consider students’ prior knowledge – Lack coherent explanations of real-world phenomena – Provide students with few opportunities to develop explanations of

phenomena– Materials lack coherence and as a result don’t focus on building

understanding over time• Students need to be prepare for the world in which they live now

and will live as adults– Deeper understanding of the core ideas becomes instrumental for

competitiveness in a global economy, as well as for the personal well being of each citizen

– Students need deeper knowledge in order to use understanding to solve problems

IQWST & LP 4

Design the Next Generation of Science Learning Environments

• Solution• The Investigating and Questioning our World through Science and

Technology (IQWST)– Utilizing a coherent approach for building curriculum materials across the

school years– Using learning goals as a focus– Supporting students in developing understandings of the core ideas of science

and scientific practices

– Applying what we know about student learning– Promoting literacy practices – Engaging students in complex tasks – Using a project-based approach

IQWST & LP 5

IQWSTDesign Principles

IQWST & LP

Key Design Principles

• Curriculum Coherence

A. Inter-unit coherence• Build links across units among the grades or content areas• Provide opportunities to develop, reinforce, and use their understanding

during their three-year IQWST experiences

B. Intra-unit coherence• Use driving-questions as support to link ideas together • Create alignment by iteratively aligning learning goals with tasks

C. Learning-goals coherence• Use big ideas• Unpack standards from a learning perspective• Create learning performances as a way to specify knowledge in use

6

IQWST & LP 7

A. Inter-unit Coherence: Structure of IQWST• The IQWST is built on scientific core ideas.

Physics Chemistry Biology Earth Science

6th gradeLight & its

interaction with matter

Particle nature of matter, phase

change

Survival: From organisms to ecosystems

Water & rock cycles

7th gradeConservation & transformation

of energy

Nature of chemical reactions

Biological organization & development:

Cells to systems

Surface & atmospheric

processes behind weather

and climate

8th grade Laws of motionPhotosynthesis and respiration(carbon cycling)

Heredity, Natural selection

Large-scale geological

processes on earth & other

planets

Units

IQWST & LP 8

A. Inter-unit Coherence: Scope and Sequence• The core ideas are coordinated across the grades and the contents.

Physics Chemistry Biology Earth Science

6th gradeLight & its

interaction with matter

Particle nature of matter, phase

change

Survival: From organisms to ecosystems

Water & rock cycles

7th gradeConservation & transformation

of energy

Nature of chemical reactions

Biological organization & development:

Cells to systems

Surface & atmospheric

processes behind weather

and climate

8th grade Laws of motionPhotosynthesis and respiration(carbon cycling)

Heredity, Natural selection

Large-scale geological

processes on earth & other

planets

IQWST & LP 9

B. Intra-Unit Coherence: Driving Questions

• Driving Question: – Links content in a context– Links tasks, investigations, and assessments to learning goals– Ties the unit together– Builds intra-unit coherence– Uses rich and open-ended questions with everyday language

IQWST & LP 10

C. Learning Goals Coherence

• Developing Learning Goals• The construct-center design

approach (CCD)– Step 1: Define the constructs

• Correcting ideas and concepts to learn

• Identifying potential students difficulties and alternative ideas

– Step 2: Unpack the ideas and concepts

• Creating a set of claims• Specifying evidences

– Step 3: Design tasks and materials

– Step 4: Review Products

IQWST & LP

IQWSTExample lesson

11

IQWST & LP 12

IQWST Smell Unit

• 8-week, project-based unit for 6th grade students• Driving Question: How can I Smell Things from a Distance?

• Three Learning Sets, 15 Lessons– Learning Set 1: Students construct models to help them understand the

particle nature of matter while focusing on the behavior of gases

– Learning Set 2: How models help to explain why different materials have different properties

– Learning Set 3: Using the particle nature of matter model to explain phase changes

Ideas from all three learning sets are brought together through a culminating final task in which students use their knowledge of the particle nature of matter.

IQWST & LP 13

Driving question for UnitScientific Core Idea

Particle Nature of Matter

Properties of Matter

Phase Changes

IQWST & LP 14

Example (1) activity

• Learning Goal: Lesson 13 – Students will explain phase changes from gas to liquid to solid and from solid

to liquid gas at the molecular level.• Scientific Principle

– Evaporation

IQWST & LP 15

Driving question

Example (2) reading

• Scientific Principle– Condensation

• Alternative Conceptions– Condensation on the outside of a container is water that seeped through the container

itself (or sweated through the walls of the container).– The coldness comes through the container and produces water.– Condensation is when air turns into a liquid.

Prior knowledge

Everyday experiences

Question & writing opportunity

Set purpose for reading

IQWST & LP

Example test items

16

IQWST & LP 17

Example item (#1) PC02

• Contents area– Phase change, Structure

• Claim– Students can explain how condensation occurs (without a particulate model).

• Unpacking/evidence– Level 1. Matter can exist as a solid, liquid or gas. Generally, the same pure

substance can exist in all three states. The state of matter depends on the temperature.

IQWST & LP 18


• Contents area– Phase change, Structure


• Unpacking/evidence– Level 1. Matter can exist as a solid, liquid or gas. Generally, the same pure

substance can exist in all three states. The state of matter depends on the temperature.

PC02

Gas Liquidd = 0.958 (hard)

18.4% 29.0%N1=1092, N2=399

IQWST & LP 19


• Contents area– Phase change, Conservation

• Claim– Students can explain the mass of matter are conserved in transforming from

solid to liquid or solid to gas (without a particulate model).• Unpacking/evidence

– Level 1. Matter is not created or destroyed when substances change form or change into other substances (e.g., crushing, dissolving, phase change, chemical reaction).

IQWST & LP 20



• Claim– Students can explain the mass of matter are conserved in transforming from

solid to liquid or solid to gas (without a particulate model).• Unpacking/evidence

– Level 1. Matter is not created or destroyed when substances change form or change into other substances (e.g., crushing, dissolving, phase change, chemical reaction).

PC18

Solid Gasd = - 0.253

23.6% 34.9%N1=368, N2=175

IQWST & LP 21




• Unpacking/evidence– Level 1. Matter is not created or destroyed when substances change form or

change into other substances (e.g., crushing, dissolving, phase change, chemical reaction).

PC19

Solid Liquidd = - 0.780 (easy)

35.8% 41.9%N1= 358, N2=201

IQWST & LP 22

Example item PC02, PC18, PC19

• Can we give same credit to each item?• Do students need to take every item?

PC02 PC18 PC19

Phenomena Phase Change

Contents area Structure Conservation Conservation

Concept Gas Liquid Solid Gas Solid LiquidItem difficulty d = 0.958 (hard) d = - 0.253 d = - 0.780 (easy)

% of correct answer(pretest) (posttest)

18.4% 29.0% 23.6% 34.9% 35.8% 41.9%

# of students(pretest), (posttest)

N1=1092, N2=399 N1=368, N2=175 N1= 358, N2=201

IQWST & LP

Item Response Theory: IRT analysis vs. CTT (classical test theory)

Creating test forms

23

IQWST & LP 24

A. Scoring system

• CTT (raw score)– O X O O O = 4 (total score)– How many correct answers did you get? – The original result obtained by a student on a test. (% of correct answer, mean

score, total score, or etc.)

• IRT (latent ability score/logit)– ? ? ? ? ? = 1.3 – What is the probability of a correct response to an item?– Using the IRT model (latent trait models), we can estimate overall latent ability

score/ logit. – In three parameter logistic model (3PL)

• difficulty• discrimination• pseudo-guessing

IQWST & LP 25

Example: PC02 (ice cup, condense)

• The item difficulty is determined by the latent trait logit of a person who has 50% of right answer. In this graph, 0.5 probability meets the trait graph at 0.958.

• The MNSQ fit is around 1. It has good fitness between observed responses and latent trait model.

Item difficulty = 0.958

Observed response

Latent trait model

IQWST & LP 26

Example: PC18 (solid gas)

• The item difficulty is determined by the latent trait logit of a person who has 50% of right answer. In this graph, 0.5 probability meets the trait graph at -0.253.

• MNSQ fit is around 1. It has good fitness between observed responses and latent trait model.

Item difficulty = - 0.253Latent trait model

Observed response

IQWST & LP 27

B. Test form

• CTT – (missing data) O X O O O _ _ = 4, or deleted– The raw score, which is the number of items correct, does not present a broad

picture of test performance because it can be interpreted only in terms of a particular set of test questions.

• IRT – (missing data) ? ? ? ? ? _ _ = 1.7– Linking item– Latent student ability score allows direct comparisons of student performance

between specific sets of test items.

– IRT generally brings greater flexibility and provides more sophisticated information.

IQWST & LP 28

B. Test form (LP test design)

• Learning Progressions– LPs are research-based descriptions of how students build their knowledge

within and across disciplines (topics) over a broad span of time– Explaining many important phenomena requires incorporating ideas from

many topic areas– Focusing on the growth of sets of ideas instead of individual topic

PhaseChange

ChemicalReaction Others

Phenomena

Structure of matter

Conservation

Interactions

Topic areas

IQWST & LP 29

• Learning Progressions– LPs are research-based descriptions of how students build their knowledge

within and across disciplines (topics) over a broad span of time– Explaining many important phenomena requires incorporating ideas from

many topic areas– Focusing on the growth of sets of ideas instead of individual topic

PhaseChange


Phenomena

Structure of matter

Conservation

Interactions

Topic areas


Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

IQWST & LP 30


• Test form design– Total 62 items; 52 multiple-choice items and 10 open-ended or short-answer

items

• A-version Form (1A, 2A, 3A 4A)– Each containing 14 items– For 6th, 7th grade

• B-version From (1B, 2B, 3B, 4B)– Each containing 15 items – For 8th grade– Three items were used to adjust the difficulty between A- and B-version

• Linking items– Each test form contained three pairs of linking items– Total six linking items were contained– Linking items cover a range of contents areas along levels 1 to 3 on the LP

IQWST & LP

Research Questions &Findings

31

IQWST & LP 32

Research Questions

• Do the chemistry IQWST units, which link content within and across units, build understanding of targeted chemistry ideas?

– Comparing students’ performances and learning gains in Phase Change and Chemical Reaction

– Comparing Students performances among Topic Areas

• Do student apply what they learned from IQWST Units to solve new problems?

– Comparing between IQWST items and non-IQWST items in Chemical Reaction

IQWST & LP 33

Data Collection: pre- and post-test

• Participants– Total 1,519 students

• 481 students in 6th grade, 519 in 7th grade, 519 in 8th grade– 7 teachers– 6 schools

• Two of the schools represent middle- to upper-middle class socioeconomic status

• Four of the schools were from neighborhoods that are lower-SES, with a majority of students eligible for free and reduced lunch.

• Research period– Schools used IQWST materials during national field trials (2007~2010)– Pretest (fall, 2010), Posttest (Spring, 2011)

IQWST & LP 34

Test Instrument

• Hypothetical LP based test– We developed test instruments based on a Learning Progression (LP) for the

transformation of matter– The assessment instruments measured how students applied ideas within and

across topics to explain phenomena involving transformations of matter.

• Test forms– total 62 items, 8 different forms, each form contained 3 pairs of linking items for a

total of 6 linking items.

• Validation of instrument– Piloted with students from grade 6th -15th (N=792). – Content validity: semi-structured interviews with a subset of students

supplemented the pilot test data.– The LP research team is contributing to validate instruments. (Namsoo Shin,

Shawn Stevens)

IQWST & LP 35

Framework (1)

• Phenomenon– Phase Change: 6th grade IQWST unit, 21 test items (20 with IQWST)– Chemical Reaction: 7th grade IQWST unit, 16 test items (11 with IQWST)

PhaseChange


Phenomena

Structure of matter

Conservation

Interactions

Topic areas

IQWST & LP 36

Framework (2)

• Topic areas– Conservation– Structure of matter; matter and materials, properties and periodicity, atomic

structure– Interactions; kinetic energy, forces, energy, processes

PhaseChange


Phenomena

Structure of matter

Conservation

Interactions

Topic areas

IQWST & LP 37

Item analysis

• The purpose of item analysis is to provide how well assessments work, and how well individual items on assessments work.

• Item Response Theory (IRT, Wu, et al., 2007)– IRT allows us to evaluate students’ ability and to describe how well items on

the test are performing. In IRT, ability and item parameters are both estimated based on students’’ response patterns on the test.

– Item parameters (difficulty, discrimination, pseudo-guessing)were used to determine whether an item displayed sound psychometric properties

• General item analysis criteria• Items difficulty (estimate)• MNSQ fit (the weighted means square) ~ 1• T (pseudo-guessing) <3 • Item correlation > 0.3• % of correct answer (total)

IQWST & LP 38

Analyzing data set

• Phase Change (20 IQWST items + 1 non-IQWST item)– Conservation (4)– Structure of matter (8+ 5 overlapped with interactions + 1 non-IQWST)– Interactions (3+ 5 overlapped with structure)

• Chemical Reaction (11 IQWST items + 5 non-IQWST items)– Conservation (6+ 2 overlapped+ 1 non-IQWST)– Structure of matter (2+ 2 overlapped+ 5 non-IQWST)– Interactions (2+ 1 non-IQWST)

IQWST & LP 39

Result fromPhase Change

IQWST & LP 40

Item analysis (PC)

• Accepted items range • - 0.988 < d <1.174• MNSQ fit <1.17 • - 4.0 < T < 3.0• 0.35 < r

Problematic item

IQWST & LP 41conservation

structure

Interactions

conservation

structure

Interactions

Students distribution Item diff distribution

Item difficulties & Latent distribution (PC)

IQWST & LP 42

Scoring

• Students latent ability logit– The partial credit, multi dimensional model – The expected a posteriori (EAP) parameter was standardized to

present student ability score

• Students ability score– To facilitate the interpretation of their ability scores assigned to

students– Normalization as t2 so that the mean and S.D of scores was 500

and 100

IQWST & LP

• Phase Change; Topic- 1. Conservation

43Distribution of Students

6th gradePretest Posttest



Stud

ents

’ abi

lity

scor

e

R1. Do students develop understanding of Phase Change?

IQWST & LP

• Phase Change; Topic- 1. Conservation

44

% of variance= 0.46 **


% of variance= 0.22


IQWST curriculum

IQWST & LP

• Phase Change; Topic- 2. Structure of Matter





Stud

ents

’ abi

lity

scor

e


IQWST & LP

• Phase Change; Topic- 2. Structure of Matter

46





IQWST & LP

• Phase Change; Topic- 3. Interactions





Stud

ents

’ abi

lity

scor

e


IQWST & LP

• Phase Change; Topic- 3. Interactions

48





IQWST & LP 49

Result fromChemical Reaction

IQWST & LP

Item difficulties & Latent distribution: (2) CRconservation

structure

Interactions

conservation

structure

Interactions

Students distribution Item diff distribution

IQWST & LP 51

• IQWST aligned items• Non-IQWST

R2. Do Students apply their learning to solve new problems?

IQWST & LP 52

Students’ development for IQWST items and non-IQWST



IQWST & LP 53

R1. Do students develop understanding of Chemical Reaction? (with IQWST items)

• D1. Conservation



% of variance= - 0.08

IQWST curriculum

IQWST & LP 54


• D2. Structure of Matter

% of variance= 077 **


% of variance= - 0.05

IQWST & LP 55


• D3. Interactions



% of variance= 0.10

IQWST & LP 56

Interpretation of the finding

• Demonstrate that students’ latent ability changes across grade level, supporting the hypothesis that coherent curriculum materials can support students in developing integrated understanding.

• Provide support that curriculum coherence may increase student achievement and build integrated understanding

• Suggest that curriculum coherence can help students develop further understanding then when ideas are not linked

IQWST & LP 57

Acknowledgments

• Thanks to the IQWST and the LP research team • Thanks to the participating schools, teachers, and students• Email me; [email protected]

dec. 11, 2012, sung- youn choi

Documents