california educational research association annual meeting san francisco, ca – november 19, 2009...
TRANSCRIPT
California Educational Research Association Annual Meeting
San Francisco, CA – November 19, 2009
Richard Wainess, Rebecca Buschang, & Deirdre Kerr
Using Human Help to InformAutomated Help in Games-BasedLearning Environments
2 / 23
Our Goal
Create a self-containedpre-algebra instructional game
as platform to examinewhat works in games for learning
3 / 23
Puppetman (a pre-algebra game)
4 / 23
Current Situation
CONDUCTED DESIGN STUDYTo examine [prototype] game/instructional
features and outcomes [learning/attitude]
Two Conditions: Mathy Version and Gamey Version
5 / 23
Issue
Players needed helpnot provided by game
6 / 23
1. Captured help needed
2. Coded the help (3 ways)
3. Analyzed game/test/help data
4. Report Results
5. Discuss Next steps
Process
7 / 23
Phase 1: Capture & Code HelpID Game Help Math Help Comments
- game mechanic - other (pls note)
- basic (e.g., job aide)
- instruction - solution - other (pls note)
Had her look up "unit" and green dots. Helped her with counting number of fraction (numerator). And that 1 1/2 is same as what else? (i.e., 3/2)
- game mechanic - other (pls note)
- basic (e.g., job aide)
- instruction - solution - other (pls note)
1. Basics of "unit" and green dots. 2. Counting fractions (numerator). 3. How many 1/6 is 1/3?
- game mechanic - other (pls note)
- basic (e.g., job aide)
- instruction - solution - other (pls note)
1. Recognizing 1/2 rather than thirds. 2. Counting 1/2s.
Plus game level and helper’s initials
8 / 23
Phase 1 Coding Results
Help Type Count
Game Help: Game Mechanic 12
Game Help: Game Other 8
Math Help: Math Basic 20
Math Help: Math Instruction 30
Math Help: Math Solution 8
Math Help: Math Other 1
46 participants need help (out of approx 120 participants)• 30 needed help 1 time• 11 needed help 2 times• 4 needed help 3 times• 1 needed help 4 times
9 / 23
Phase 2: Examined Help Elements
Comments Coded As
Had her look up "unit" and green dots. Helped her with counting number of fraction (numerator). And that 1 1/2 is same as what else? (i.e., 3/2)
1. Asked guiding Questions2. Prompted help seeking
1. Basics of "unit" and green dots. 2. Counting fractions (numerator). 3. How many 1/6 is 1/3?
1. Asked guiding Questions
1. Recognizing 1/2 rather than thirds. 2. Counting 1/2s.
1. Can’t code
1. One rater examined 120+ help elements and created rubric2. 3 rounds with 2 raters (1/3 of data each round)3. Modified/clarified/reduced rubric after round 14. Inter-rater reliabilities of 71%, 64%, 63% (100%
w/discussion)
10 / 23
Phase 1 coding resultsHelp Type Count
Can’t code 31
Asked guiding questions 17
Explained game control 10
Stated math concept in terms of game feature/mechanic 6
Gave solution step 6
Prompted help seeking 5
Explained game mechanic 4
Asked for explanation 2
Explicit math instruction 0
Stated math concept as math 0
Gave solution 0
Gave solution strategy 0
11 / 23
Phase 3: Recoded Help Elements
Comments Coded As
Had her look up "unit" and green dots. Helped her with counting number of fraction (numerator). And that 1 1/2 is same as what else? (i.e., 3/2)
• Help w/ unit• Help converting fractions• Help w/ green dots• Help w/ coil size (denom)• Help w/ coil num (numer)
1. Basics of "unit" and green dots. 2. Counting fractions (numerator). 3. How many 1/6 is 1/3?
• Help w/ unit• Help converting fractions• Help w/ green dots• Help w/ coil num (numer)
1. Recognizing 1/2 rather than thirds. 2. Counting 1/2s.
• Help w/ coil size (denom)• Help w/ coil num (numer)
1. One rater examined 120+ help elements and created codes
2. Second rater coded using same codes (matched 49%)3. Reviewed, clarified rubric, and reached consensus
12 / 23
Phase 3 coding resultsHelp Type Count
Clarify what one “unit” is 25
Determine appropriate coil size (denominator help) 19
Convert fractions 16
Determine number of coil pieces to use (numerator help)
11
Help with moving trampolines/coils 8
General help with fractions 7
Help with what green dots meant 5
Help with placing trampolines in correct grid position 5
Help on Level 1 4
Help combining fractions 4
Help with scrolling 4
Other help 4
Determine some aspect of distance 2
Help with understanding arrows 2
Help with undo/reset button 2
Help breaking coils into smaller parts 1
13 / 23
ANALYSIS & RESULTS
14 / 23
Differences in Gameplay
One-way ANOVA on Level Reached
• Only 27% of those reaching or exceeding level 17 needed help
• Only 35% of those reaching levels 12 to 16 needed help
• 78% of those who did not reach level 12 needed help
15 / 23
Differences in Learning
Regression predicting post-scores (results of ANOVA)(Note: Four models were run. Values below are from fourth model.)
•No difference in post scores between gamey/mathy, B = .023, p = .879
•Difference in post scores between help/no help, B = -.587, p = .001
Kids who got help also scored lower on post-test, when controlling for gender, gameplay, and pre-test
•Model r2 = .490, p = .000 is a significant predictor of post-test.
Adding interaction (help & condition) was a significant predictor of post-test scores (when added gender, pre-test, gameplay, condition, help, and measures of in-game learning), p = .033
16 / 23
Differences in Learning (cont’d)
• Interaction between help/no help and condition
Those who received help did worse those who didn’t receive help (p = .001)
B = -.587 for gamey B = -.564 for mathy
Mathy with help did better than everyone (p = .033) B = 1.095 better than gamey with help B=.508 better than gamey no help B = .485 better than mathy no help
17 / 23
Help by Condition
Help Type Gamey (N) Mathy (N)
Game Help: Other (3) Game Help (5)
Math Help: Instruction Math Help (15) (10)
PHASE 1
Help Type Gamey (N) Mathy (N)
Explained Game Controls (3) Game Help (8)
Prompted for Explanation Math Help (2) (0)
Game Solution Step Math Help (4) (2)
PHASE 2
18 / 23
Help by Condition (cont’d)
Help Type Gamey (N) Mathy (N)
Moving Trampoline or Coil Game Help (5) (3)
Level 1 (getting started) (1) Game Help (3)
How to Scroll (1) Game Help (3)
Defining a Unit Math Help (14) (10)
How to Reset or Undo (0) Game Help (2)
Other Help with Distance (0) Game Help? (2)
PHASE 3
19 / 23
Help by Condition Trends
Help Type Gamey (N) Mathy (N)
Math Help: Instruction Math Help (15) (10)
Defining a Unit Math Help (14) (10)
Explained Game Controls (3) Game Help (8)
Largest Ns
Condition Game Help Math Help
Gamey 1 4
Mathy 6 0
Most help by condition
20 / 23
Next Steps
Created the following help topics and instruction to address all areas needing help
1.Determining the Level’s Denominator
2.Determining the Distance Between Jump Points
3.Changing the Denominator of a Coil
4.Common Denominators
5.Reducing the Denominator
6.Least Common Denominator of Two Fractions
7.Matching the Level’s Denominator
21 / 23
Next Steps (cont’d)
• Next Steps
Validate Assumptions through Think-Alouds(beginning December, 2009)
Embed more instruction into game
As direct instruction (January, 2009)
As contextualized help/feedback
Through an in-game help system
22 / 23
Review
• Collected “help given data” during design study
• Coded data three ways
• Analyzed codes, gameplay, & test scores for themes
• Used analysis results to define instructional needs
• Created and are about to test instructional parcels
• Will begin testing embedded instruction January 2009
23 / 23
BACKUP
24 / 23
In-game Help Comparison: Level 6
Start Help Make Bucket
Place Bucket Wrong Unit End Death
Mathy We need to bounce PuppetMan 3 spaces to reach safety! Click on the big coil and scroll up on the mouse wheel to break the WHOLE into more but smaller EQUAL pieces.
The WHOLE unit (the space between the red lines) is broken into TWO equal pieces but it will take THREE of THOSE EQUAL pieces to get to the goal. Recall - two HALVES are ONE WHOLE
When you are finished loading the trampoline, place it on the blocks to help save PuppetMan!
Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you count how many of THOSE pieces you need to reach the goal? Remember ONE WHOLE UNIT is the distance between the red lines.
Remember, the coils must all be exactly the SAME size when ADDING them to the trampoline. If you cannot drag a piece in, break the coils into pieces of the SAME size (same denominator).
You are mastering the art of trampoline making! Just remember, addition requires that all the coils have to be the SAME size to be put into the same trampoline!
Make sure you see how many pieces the WHOLE (between the red lines) is broken into, THEN count how many of THOSE pieces you need to get PuppetMan to the goal.
Gamey
We need to bounce PuppetMan three half-units to reach safety! Click on the big coil and scroll up on the mouse wheel to break it in half, and then drag the 3 half pieces over!
Sorry, no help on this level.
When the trampoline says 3/2, place it on the blocks to help save PuppetMan!
This is the text that appears the player places some number of trampolines onto the grid
If you cant drag a piece in, click on the trampoline and scroll the wheel to change the units! You can also use the scroll wheel to break up the coils into the right units!
You are mastering the art of trampoline making! Always remember, all the coils have to be the same size to be put into the same trampoline!
DISABLED
25 / 23
In-game Help Comparison: Level 11
Start Help Make Bucket Place Bucket Wrong Unit End Death
Mathy You know the goal, get PuppetMan safely across the spears. But now the WHOLE unit has been broken down differently. Check it out and good luck.
The WHOLE UNIT has been broken into THREE pieces and it looks like you need FIVE of them in total. Since you only have HALVES break them into a piece that can also be made into THIRDS.
When you are finished loading the trampolines, place them on the right blocks!
Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you change HALVES into a unit you can also use to count THIRDS?
If you can't drag a piece in, click on the trampoline or coil and scroll the wheel to change the units into the units you need. You can only ADD IDENTICAL pieces.
Genius! Sometimes, you'll have some coils leftover! Don't worry at all about that, because PuppetMan is alive and well.
Break everything into SIXTHS to make it easier! Two SIXTHS is one THIRD and Three SIXTHS is one HALF. Then you can count both coils and how far you need to jump.
Gamey You know the goal, get PuppetMan safely across the spears. But now the WHOLE unit has been broken down differently. Check it out and good luck.
The WHOLE UNIT has been broken into three pieces and it looks like you need five of them to get to the goal (two to the first block and three from there to the goal).
When you are finished loading the trampolines, place them on the right blocks!
Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you count how many of THOSE pieces you need to reach the first block? How many more do you need to reach the goal?
If you can't drag a piece in, click on the trampoline or coil and scroll the wheel to change the units into the units you need.
Genius! Sometimes, you'll have some coils leftover! Don't worry at all about that, because PuppetMan is alive and well.
Break everything into thirds to make it easier! Then make sure you have enough thirds to reach the first block and enough to reach from there to the goal.