an iterative approach to designing scalable and adaptive computer-based science instruction mari...
TRANSCRIPT
![Page 1: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/1.jpg)
An iterative approach to designing scalable and adaptive
computer-based science instruction
Mari Strand Cary, David Klahr,
Stephanie Siler, Cressida Magaro,
Kevin Willows, Junlei LiCarnegie Mellon University & University of Pittsburgh
![Page 2: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/2.jpg)
2
Goal of this EdBag?
• Discuss this iterative research process and its role/prominence in curriculum design, HCI, etc.– Usefulness to researchers and designers
(and, more importantly, end-users)– How different is it from “design research”– Difficulties we’ve encountered– Other?
![Page 3: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/3.jpg)
3
Overview of the TED projectCurriculum: Experimental design, evaluation,
and interpretationAge: 5th-8th grade studentsSchools: 6 inner city
– 4 low SES & challenging classroom environments– 2 mid-high SES
End goal: Computer-based adaptive tutor– 1 student : 1 computer in classroom environment
– Provides individualized, adaptive instruction– Supplements (does not replace!) teacher
![Page 4: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/4.jpg)
4
“Experimental design” = CVS (Control of Variables Strategy)
1. Simple procedure for designing unconfounded experiments
(Vary one thing at a time)
2. Conceptual basis for making valid inferences from data
(Isolating the causal path)
![Page 5: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/5.jpg)
5
Does SURFACE affect how
far balls roll?
A
B
Variable Ramp 1 Ramp 2
Surface Smooth Rough
Track length Short Long
Height High Low
Ball Golf Rubber
Ramp 1 Ramp 2
Smooth Rough
Short Short
High High
Golf Golf
Confounded Unconfounded
![Page 6: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/6.jpg)
6
What’s the best way to teach CVS?
• As a society (educators, researchers, and legislators), we don’t know
• Our research team knows of one effective way…
![Page 7: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/7.jpg)
7
Our basic CVS instruction (used in various forms in past studies)
• Students design experiments
• Students answer questions
• Instructor provides explicit instruction about CVS
• One domain
• Short instructional period
![Page 8: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/8.jpg)
8
But…
Our brief, focused CVS instruction is differentially efficient and effective for different student populations, settings, and transfer tasks.
We want to reach ALL students!
To improve our instruction for the entire student population, we must engage in modification & individualization
![Page 9: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/9.jpg)
9
A computer tutor could facilitate differentiated instruction
• Instruction that suited for full-class setting could be provided by teacher
• Remaining instruction could be provided by computer tutor– Individualized & self-paced– Provides instruction, practice, and
feedback– Teacher freed to provide coaching as
needed
![Page 10: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/10.jpg)
10
How are we building our tutor?
4 development phases
&
Iterative design process
![Page 11: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/11.jpg)
11
4 development phases:1. Information gathering: What are the
novice models students hold and how can we address those?
2. Refining the basic instruction and “going virtual”
3. Building a computer tutor with a few “paths”
4. Building an adaptive computer tutor with a “web” of paths
![Page 12: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/12.jpg)
12
Improve current version &
Inform next version
Compare against previous version
Our iterative design process:
Version n
Pilot testing
Delayed post assessment
One-on-one human tutoring
Classroom validation study(+ pre, post, and formative
assessments)
![Page 13: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/13.jpg)
13
An evolving CVS computer tutor
Version 1 Version 2 Version 3 Version 4
Instructional mode
Class (teacher) 2a) Class (teacher) Class (teacher)
Individual (computer)
Class (teacher)
Individual (computer)
InflexibleFlexibility Limited flexibility(differentiation points)
Flexible (multiple paths)
Adaptive (“web” of paths)
Stimuli Simulations
Computer interface
Physical apparatus
Overhead transparencies
Simulations
Computer interface
Simulations
Computer interface
Instructional components
(domain)
Procedural & Conceptual (Ramps)
Prereq. skills (Auto sales)
Procedural (Study habits)
Conceptual (Ramps)
TBD TBD
DiscussionFeedback Discussion, paper exchange, researchers
Discussion, Computer, researchers TBD
2b) Small groups (us!)
![Page 14: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/14.jpg)
14
What are we learning from each version that will help us design the final, adaptive tutor?
VERSION 1 (Completed)
• Initial list of student biases, misconceptions, errors & areas of difficulty
• Inventory of successful tutoring approaches
• familiar domains
• instruction in prerequisite skills
• step-by-step approach
• Student-friendly terminology, definitions, and phrasing
• Requiring explicit articulation by student
![Page 15: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/15.jpg)
15
A sampling of what students do wrong:
Common errors:• Vary everything• Hold target variable
constant and vary other variables
• Partially confounded• Nothing varied (identical)
Common justifications:• “I don’t know”• You told me to test x!• Describe their set-up• Want to see if x happens• Want to see if this setup
is better than that setup
![Page 16: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/16.jpg)
16
Why?• By accident
– misread question– working carelessly
• Are led astray– by saliency of physical apparatus (e.g., ramps) – don’t understand written representations (e.g.,
tables)
• On purpose – different goals (e.g., “engineering”)– misconception of experimental logic– think other variable(s) don’t matter
• Just guessing
![Page 17: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/17.jpg)
17
VERSION 2a (Complete) & 2b (Fall 2007)
Information regarding:
• Addressing most common problems in full-class instruction using successful tutoring approaches
• Instructional effectiveness of switching domains
• effect of emphasizing domain-generality
• interface usability
• worksheet usability
• 2b: Implementation of successful tutoring approaches with small groups (groups will differ in the “paths” they take)
![Page 18: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/18.jpg)
18
VERSION 3 (being developed)
Information regarding:
• division of instruction between teacher and tutor
• individual tutor usability and pitfalls
• comparative efficacy of set learning paths
• efficacy of immediate computer feedback
![Page 19: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/19.jpg)
19
The adaptive tutor will include:
• Pre-testing and ongoing monitoring of student knowledge
• Self-paced instruction
• Diverse topics matching student’s interests
• An interactive and engaging interface
• Teacher-controlled and/or computer-controlled levels of difficulty
• Level of scaffolding, feedback, and help aligned with student’s needs
• Computerized assessments
• Logging capability (“level” of output TBD)
![Page 20: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/20.jpg)
20
Discuss!
• Discuss this iterative research process and its role/prominence in curriculum design, HCI, etc.– Usefulness to researchers and designers (and,
more importantly, end-users)– How different is it from “design research” (and
would it have gotten funded if we had labeled it that?)
– Difficulties we’ve encountered– Other?
![Page 21: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/21.jpg)
Questions? Comments?
Funding provided by:Institute of Education Sciences (IES _____)
![Page 22: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/22.jpg)
![Page 23: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/23.jpg)
23
V1 learning examples:
VERSION 1
• Database of student biases, misconceptions & areas of difficulty
• Inventory of successful tutoring approaches
• familiar domains
• instruction in prerequisite skills
• step-by-step approach
• Student-friendly terminology, definitions, and phrasing
• Requiring explicit articulation of understanding and reasoning
Ignore the data or Biased by expectations
Create “best” outcome or Most dramatic difference
Learn about all variables at once
Pets, Sports drinks, Cars, Study habits, Running races
Variable vs. Value
Experiment
Result vs. ConclusionRead carefully, Identify question, Identify variables…
Good vs. Fair vs. Informative vs. True
“Variable” = something that can change
Table format
Remembering the target variable
Drawing conclusions based on the experiment
![Page 24: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/24.jpg)
24
Stand-alone, detailed lesson plan with
visual aids
Examples of exp. designs
(good and bad)
Assessments (formative and
summative)
Students designing experiments
Asks students to explain, justify,
and infer
Feedback
Every version
![Page 25: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/25.jpg)
25
Increasing complexity and adaptiveness
Physical apparatus Virtual simulations
Full class Full class & individual computer use
Inflexible Individually-adaptive & self-paced
One domain Multiple domains
![Page 26: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/26.jpg)
26
What if later versions are less effective than earlier versions?• “Stop the presses!”
• Look for obvious reasons
• Examine lesson components individually
• Consider what is missing
![Page 27: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/27.jpg)
27
“Procedures”
Test one variable at a time
1. Make the values for the variable you’re testing be DIFFERENT across groups.
2. Make the values for the variables you’re not testing be the SAME across groups.
![Page 28: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/28.jpg)
28
“Concepts”
• You need to use different values for the variable you’re testing in order to know what effect those different values have.
• You need to use the same value for all the other variables (hold all the other variables constant; “control” the other variables) so that they can’t cause difference in the outcome.
• If you use CVS, you can know that only the variable you’re testing is causing the outcome/result/effect.
![Page 29: An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro,](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56649e6f5503460f94b6cc2e/html5/thumbnails/29.jpg)
29
Beyond our classroom instruction…
• Where on the contextual / abstract continuum should this type of instruction be focused? When?
• Single vs. multiple domains?
• Static pictures vs. simulations vs. tabular representations
• Best mix of explicit instruction, exploration, help, feedback, etc.