A progress report--Computer coaches for introductory physics

problem solving

Qing Xu, Ken Heller, Leon Hsu, Andrew MasonUniversity of Minnesota

Supported by NSF DUE #0230830 and DUE #0715615 and by the University of Minnesota

10/30/2010 MAAPT WAPT

Computer coaches for Introductory Physics

• Enhance Problem-Solving skills

• Possible Advantages:

- Provides individualized guidance and feedback outside of class; on-demand help

- Effectiveness through good design and pedagogy

- Customizable by instructors

Research Background

• Theory:

- Cognitive Apprenticeship1

• Design:

- Reciprocal teaching2

- Context-rich problems3

3 types of coaches

• 1st: Computer coaches the student– computer decides, student implements, computer assesses

• 2nd: Student coaches the computer– Student decides, computer implements, student assesses

• 3rd: Student works more independently– Computer provides help as necessary; Scaffolding

gradually withdraw

PILOT STUDY –Fall 2010• ~66 students, 1 lecture session of intro calculus-based physics

• Assign into 3 statistically matched groups (~22 for each group)

• Variables for matching: background information, e.g. HS physics & math level, FCI/CLASS/math pretests

• Subset of tutors available – energy, momentum (4 weeks)

• 1 Treatment group , 1 Control group and 1 placebo group• Treatment- computer coaching (on Web, outside of class), 4 problems per week • Placebo group – Work on Computer tutor problems on paper• Control group -- normal class setting

• Data collection• Written solutions on quizzes & final exam • 2×4+5=13 for each student

Today’s topics • Can written solutions well represent their Problem-

Solving skills? -- Yes.

• Interviews – (methodological triangulation) -- Triangulation: the use of two or more methods of data collection in the study of some aspect of human behavior. 4

-- An important source of evidence for validity. 5

• How to score the written solutions?-- Problem-solving Rubric. 5

UD: Useful DescriptionPA: Physics ApproachSA: Specific ApplicationMP: Mathematical ProcedureLP: Logical Progression

• Rubric Training– Inter-rater reliability

INTERVIEWS – Fall2010 5 volunteering students from introductory calculus-based mechanics course for scientists and engineers

Interviews were taken place around week 6 (Dynamics)

1st-Kinematics/2nd-Dynamics/3rd –Kinematics

In an interview, students are asked to solve physics problems while their actions and voice are recorded (30mins). After completing the problem, they are asked to explain their reasoning to an interviewer (30mins).

TriangulationTriangulation

student written workstudent written work self-reported thought processes

self-reported thought processes

correspond

correspond

5 rubric category processes

5 rubric category processes

self-reported problem-solving

processes

self-reported problem-solving

processes

characterize

characterize

Rubric scores Rubric scoresproblem-solving skills

from other measures of their performance

problem-solving skills from other measures of

their performance

inferences

inferences

Chart1: Triangulation 5

General information

1 student actually talked out loud while he/she was working on the problem

1 out of 5 students finished all 3 problems (1st-Kinematics/2nd-Dynamics/3rd –Kinematics)

1 student asked for help during problem solving process

3 out of 5 students gave up on Problem1 after the first 10 minutes; 1 student was

persistent until getting a final result (15mins); 1 student got stuck and moved on the the

2nd problem(15mins)

The students didn’t use the equation sheet too much, they either saw the equations on it

but remembered it anyway, or they just ignored the equation sheet.

ExpectationsExperts

Problem solving is a process of making a series of judicious decisions

Framework• Create useful description• Plan solution based on general

principles• Carry out plan • Evaluate solution (also

intermediate steps)

Novices

Plunge directly into mathematical calculations

Do not have a very useful or complete description (picture)

Focused on the superficial features of the problem

Transcripts/interview results

S6: The first thing I did is the angle of 30°, the initial velocity. I broke it down into components. Whenever there is an angle, I would do that first, so I don’t have to worry about it later on.

I: After I gave you the problem, What was the first thing that you did?

S6: Well, then I read the problem the second time. tried to absorb all the information. And, actually, I started to do my equations and backtracked and say that i don’t really know what the problem was saying. So I draw picture to kind of help me visualizing it. I kind of jumped into cracking some numbers, and then I realized, oh , wait, I don’t really get what the problem was asking, I actually go back to draw a little picture to help me visualize.

I: Okay, after that, what did you do?

interview results

S6: so I went back to work on the equations, using the equation sheet.

I: After you drew the picture, what did you do next?

Not a bad picture

Not complete

Not complete

Rubric Training & Scoring—Summer 2010

• 8 sets of previously scored student solutions• 1 undergraduate and 2 graduate students of physics• 1 experienced assessor and 2 novice assessors.

• Goal: For novice assessors to achieve an increased understanding of problem solving with measurable agreement.

Training

Qing and Talia

0

0.2

0.4

0.6

0.8

1

1.2

P1 P2 P3 P4 P5 P6 P7 P8

Solution Set

Agr

eem

ent w

ithin

1Agreement between Raters overtime

Figure 2: Agreement within 1 Rate 1& 36

Figure 1: Agreement within 1 Rate 1& 26

Qing and Karl (Before)

0

0.2

0.4

0.6

0.8

1

1.2

P1 P2 P3 P4 P5 P6 P7 P8

Solution Set

Agr

eem

ent w

ithin

1

Rater 1& 2 Rater 1& 3

Rubric Training & Scoring—Summer 2010

• 20 Students and 13 written solutions/student (4 quizzes & finals)• 1 undergraduate and 1 graduate students of physics• 2 assessors who completed training.

• Goal: Establish a baseline for future stages of research trends, dependencies( topics, gender, etc)

Scoring

Gender Gaps?

0

1

2

3

4

5

UD PA SA MP LP

Ave

rag

e R

ub

ric

Sco

re

Category

Quiz 1

Female

Male

Figure 5: Average score in each category for male and female students. Data is for the first quiz of the semester , however no statistically significant gender gap is apparent in any of the tests throughout the semester nor in student improvements from the first quiz to the final. 7

Improvement Overtime?

Figure 6: Changes in average rubric scores for each category from the beginning to end of the semester. Quiz 1 Problem 1 and Final Exam Problem 2 test the same physics concepts. Significant (p<0.05) gains are seen only in Physics Approach between Quiz 1 Problem 1 and Final Exam Problem 2. 7

References

1. Collins, Brown, & Newman, 1989. Cognitive apprenticeship.

2. Palincsar, A.S., & Brown, A.L. (1984). Reciprocal teaching of comprehension- fostering

and monitoring activities. Cognition and Instruction, 1, 117-175.

3. Heller, P., & Hollabaugh, M. (1992). Teaching problem solving through cooperative

grouping. Part 2: Designing problems and structuring groups. American Journal of Physics,

60(7), 637-644.

4. Louis Cohen, Lawrence Manion, Keith Morrison, Keith R. B. Morrison, (2007). Research

methods in education.

5. Docktor, 2009; Docktor & Heller, 2009

6. Talia Clark, Rubric Training Assessment Presentation, Summer 2010

7. Talia Clark, PER Final Presentation, Summer 2010