design experiments for assessment of “21st century” learning · 2012-01-23 · design...
TRANSCRIPT
Design experiments for assessment
of “21st century” learning
ROSEMARY HIPKINS, New Zealand
Presentation at VUW Assessment Conference, Wellington, September 1-3, 2011
An “evidence-centered assessment architecture”:
1. Select and develop tasks based on construct
2. Present tasks to learners
3. Learners generate evidence with respect to constructs
4. Evidence used to make inferences about construct of
interest
(Almond et. al. 2002, cited in Scalise and Wilson, 2011)
OECD Key competencies
• Holistic - integrate knowledge, skills, attitudes and values, including a focus on dispositions
• Demonstrated in authentic contexts when students adapt the competency to do something in a new setting.
• Participatory in intent: consistent with situated and socio-cultural views of learning.
• A focus on doing and being, not just knowing
How should the two parts of NZC relate to each other?
The “front end”
• Vision
• Values
• Principles
• Key competencies
• Effective pedagogy
The “21st century” vision
The “back end”
8 levels
8 learning areas
8 sets of achievement
objectives per level
The more traditional “content”
New ‘nature of science’ integrating strand
(based on generic Key Competencies)
Understanding about science Investigating in science
Communicating in science Participating and contributing
Living
WorldMaterial
World
Physical
World
Planet
Earth and
Beyond
Content included through integration
Working “like a scientist”: two inter-related roles
Constructing claims via age appropriate
investigations and development of simple
explanations
Critiquing claims of others
(ask questions, cite evidence, defend
arguments etc)
Research to explore teaching for key competencies
(and hence potential transfer of learning)
• Reframing traditional Assessment Resource Bank
tasks to support formative assessment
• ARB items trialed with at least 200 students and in a
range of schools
• Each item is a small “design experiment” to identify
and support “next learning steps”
• The question here: what can students do with
classification (is there simple inquiry potential?)
Item analysis
• Students from 16 schools in trial
• 169 @ year 6, 171 @ year 8
• First part of item: basic frequency counts
• Second part coded: code combinations (A, AA, ABB, BC, etc) recorded in data base
• Some cross-tabulation to address emergent questions
• “Next steps” advice worked out and published with results
Features of reptiles
have a backbone
are cold-blooded
breathe with lungs
have dry skin covered with
scales
(some also have a hard shell)
have ear holes but no ears
have 4 legs or no legs
lay eggs which have a strong,
soft shell. A few reptiles have
live babies.
Yr 6 Yr 8
yes 88% 91%
no 9% 6%
I can’t
tell
2% 2%
Features of reptiles
have a backbone
are cold-blooded
breathe with lungs
have dry skin covered with
scales
(some also have a hard shell)
have ear holes but no ears
have 4 legs or no legs
lay eggs which have a strong,
soft shell. A few reptiles have
live babies.
Yr 6 Yr 8
yes 53% 60%
no 37% 28%
I can’t
tell
10% 11%
Features of reptiles
have a backbone
are cold-blooded
breathe with lungs
have dry skin covered with
scales
(some also have a hard shell)
have ear holes but no ears
have 4 legs or no legs
lay eggs which have a strong,
soft shell. A few reptiles have
live babies.
Yr 6 Yr 8
yes 9% 8%
no 75% 79%
I can’t
tell
14% 12%
Scoping the potential for key competency/
Nature of Science development
What would the teacher learn from only assessing
for “correct” answers?
What if the “next steps” were about the use of
evidence?
What role could “I can’t tell” play in developing an
inquiry focus for classification?
What is discipline-specific about classification?
• For many students the task was not seen as one of skills-based observation and decision-making
• From what I have learnt at school I’ve learnt frogs are amphibians.
• I just know it’s an amphibian. • I know a weta is an insect. It has 6 legs, no
scales, no hard shell, doesn’t have a backbone (emphasis added).
• Many students took “I can’t tell” to mean “I don’t know”, but one or two did see its intended purpose
• I can’t tell if it has a dry skin.
Understanding the affordances that students
could bring to the task: some or all of….
• Perceived purpose of activity
• Own knowledge of each animal
• Observation skills
• Ability to read and use list provided
• Disposition to suspend answering until all
possibilities considered
Many students used a combination of
observable and not observable features (own
knowledge as one affordance)
Fewer cited only observable features (coded
as A, AA, AAA etc combinations)
They were most likely to resort to use of their
background knowledge for the non-reptilian
vertebrates (where disconfirming evidence
becomes a bit less obvious)
Nature of
evidence
cited
Fish
%
Frog
%
Bird
%
Weta
%
Snake
%
Lizard
%
Turtle
%
Used own
knowledge
only
43 36 34 29 27 24 10
Used
observable
plus own
knowledge
25 30 25 23 38 34 36
Used
observable
only
16 15 23 22 26 27 34
Missing or
other 15 20 18 26 9 15 20
Year 6 students’ responses
• Cross-tabulation of the two parts of
the task confirmed that “I can’t tell”
was likely to mean “I don’t know” for
many but not all students
Evidence cited
(2nd part of
question)
Year 6 classification
(1st part of q)
Year 8 classification
(1st part of q)
Yes No Can’t
tell
Yes No Can’t
tell
Observation
only
2 35 2 45 1
Observable plus
own
knowledge
1 37 1 4 39 2
Own knowledge
only
10 34 3 4 40 3
No evidence
cited
2 20 19 3 11 15
X-tab data for responses to weta photo
A “thinking curriculum” for the 21st century (Laura
Resnick)
• Intellectually rigorous
• Discipline specific so requires deep teacher expertise
• For all students not just elite
The investigative potential in this science task:
• evidence-based decision making;
• recognition of disconfirming evidence;
• re-examination of own “just knowing”
• Active practice of both constructor and critiquer
roles
©NZCER
Acknowledgements
My science team colleagues: Chris Joyce, Lorraine
Spiller, Ally Bull, and statistician Hilary Ferral
New Zealand’s Ministry of Education for research funding
Teachers who help and students who complete ARB trials