“My personal experience indicates that improvement requires constant experimentation in methods of delivery, the different ideas found in our classes, and evaluating the results so that I understand what works and what doesn't work and why.”

- Dr. Jose Garcia

Scientific Education

However:• Manual not always clear to students nor new instructors.• Most common complaint: lack of connection to lecture.

Physics 241 E&M Current Lab Status:• Excellent E&M lab curriculum for majors & non-majors with excellent lecturers.

The test-manual is an attempt to improvethe communication of the current curriculum

with zero impact on resources.

Test-manual Design Goals:• Teacher-proof the manuals from endless cycle of new TAs.

• Zero workload impact on staff by using current excellent lab design (and not reinvent wheel).

• Better connection of lab concepts to lecture.

• Specific instruction on scientific writing.

• Fairness in student assessment.

• Improve diversity friendliness and gender equity.

Format of Test-manuals

Theory with conceptual questions. (~45 minutes)

Directed experimentation with measurementquestions and conceptual questions.

(~65 minutes)

Authentic assessment. (~10 minutes)

Open-ended question / creative lab design.(with no help given by TA) (~45 minutes)

(~2.75 hours)

Writing guidelines: (~3 hours)

(compare manuals)

Website Supportedhttp://bohr.physics.arizona.edu/…/phys241lab.html

Some Qualitative Observations

Students work diligently for 2.5-3.0 hours.

Increased peer-peer physics discussions.

Improved lab report quality.

Less student confusion / greater enjoyment.

(see reports)

Student Opinions

Test-manual connects lab to lecture - YES

Test-manual makes learning productive -YES

Test-manual helps learning concepts - YES

Test-manual helps report writing - YES

Test-manual vs. Current manual 98% 0%

(see surveys at: http://bohr.physics.arizona.edu/~leone/fall_2008_phys_241_lab/new_manual_survey_results.pdf)

Spring 2008(previous)

Fall 2008(current)

125

T1+T2+T3

175

200Matt’s

Students187.7

125

150

175

200

Others’Students

176.6

Others’Students

152.7

Matt’sStudents

156.4 150

p=.1025(not significant)

Spring 2008(previous)

Fall 2008(current)

125

T1+T2+T3

175

200Matt’sMales187.9

125

150

175

200

Others’Males179.5

Others’Males155.1

Matt’sMales148.8

150

p<0.0001

Spring 2008(previous)

Fall 2008(current)

125

T1+T2+T3

175

200

125

150

175

200Matt’s

Females186.8

Others’Females

154.8Others’Females

142.8

Matt’sFemales

188.2

150

p<0.001p<0.001

T1+T2+T3

T1+T2+T3

Other Matt

MattOther

Psych “evidence”: High IQ female students may not learn well with high levels of initial confusion or some initial failure (Licht & Dweck, Dweck).

p<0.001

Personal statement: Students are not as likely to continue in SEM if they underperform (unnecessary female attrition).

MALE

FEMALE

Instructor Assessment Averages Female Male Both Instructor Assessment Averages Female Male BothArif # students 5 29 34 Steve # students 3 23 26

Sum Lab Reports 276.7 264.7 266.5 Sum Lab Reports 277.7 271.8 272.5Lab Practical 15.2 16.8 16.6 Lab Practical 17.3 16.6 16.7

Sum Homework 156.5 153.6 154.0 Sum Homework 146.7 156.2 155.1Exam 1 53.8 54.4 54.4 Exam 1 37.3 53.2 51.3Exam 2 66.2 69.3 68.9 Exam 2 50.7 66.0 64.2Exam 3 57.6 56.8 56.9 Exam 3 31.3 52.1 49.7

E1+E2+E3 177.6 180.6 180.1 E1+E2+E3 119.3 171.3 165.2Final 57.2 65.3 64.0 Final 48.3 59.7 58.4

Dan # students 4 28 32 Tony # students 11 11Sum Lab Reports 258.3 231.0 234.4 Sum Lab Reports 261.7 261.7

Lab Practical 16.8 17.3 17.2 Lab Practical 16.2 16.2Sum Homework 160.0 128.0 132.0 Sum Homework 138.1 138.1

Exam 1 38.5 53.1 51.3 Exam 1 45.8 45.8Exam 2 63.8 72.4 71.3 Exam 2 58.0 58.0Exam 3 56.5 63.1 62.2 Exam 3 48.7 48.7

E1+E2+E3 158.8 188.6 184.8 E1+E2+E3 152.5 152.5Final 59.5 65.8 65.0 Final 63.3 63.3

Michal # students 1 10 11 Matt # students 10 42 52Sum Lab Reports 266.0 253.3 254.5 Sum Lab Reports 290.8 259.2 265.3

Lab Practical 18.0 15.8 16.0 Lab Practical 17.3 16.8 16.9Sum Homework 170.5 139.2 142.0 Sum Homework 178.6 128.5 138.1

Exam 1 35.0 53.1 51.5 Exam 1 58.9 56.3 56.8Exam 2 65.0 73.3 72.5 Exam 2 73.0 69.3 70.0Exam 3 31.0 59.2 56.6 Exam 3 54.9 62.0 60.6

E1+E2+E3 131.0 185.6 180.6 E1+E2+E3 177.6 180.6 180.1 Final 51.0 70.2 68.5 Final 68.4 64.9 65.6

Combined # students 23 143 166 14% of class is female.Sum Lab Reports 279.3 256.6 259.8 Females do better on lab reports.

Lab Practical 16.8 16.8 16.8 Females perform equally well on lab practical.Sum Homework 166.0 139.4 143.1 Females do better on homework.

Exam 1 50.4 53.8 53.3Exam 2 66.7 68.8 68.5Exam 3 51.7 58.3 57.4

E1+E2+E3 168.7 180.9 179.2 Females underperform on exams. (Stereotype threat?)Final 61.0 64.6 64.0 Females underperform on final exam. (Stereotype threat?)

Matt’s StudentsSpring 2008

Matt’s StudentsFall 2008

€

TFall 2008Spring 2008

= 3.45Means test: (p=0.03).

€

x Spring2008

= 74.8 ± 5.2 and x fall2008

= 84.6 ± 3.095% confidence intervals:(Note: females trend same as males)

LAB PRACTICALS(Matt: Spring vs. Fall)

70

65

75

80

85

90

Score (%)

Improvedpracticals.(p=0.03)

Proposal Timeline:Spring 2009: Form writing team and make test-

manuals of outstanding quality.

Summer 2009: Prepare for formal PER experiment.

Fall 2009: Run experiment, publish, advertise.

Cost Analysis (already 220-250 of my hours)

Spring 2009: $100.00 paper + TA and faculty time.Summer 2009: $0.00 paper + TA and faculty time.Fall 2009: $200.00 paper + TA and faculty time.

Some writing team duties:• Create written curriculum to accompany test-manuals. Cite PER research & national/state standards(?).

• Add written learning goals to first page of test-manuals.

• Find and include more links to instructional applets.

• Teaching tips for TAs on last page.

• Minor revisions to writing guidelines to switch to biweekly lab reports (?).

PER Experiment Prep:• Research PER literature.• Investigate using measures (BEMA, MPEX, etc.) for actual data collection. • Write experimental design/proposal using literature and measures.• Get Human Subjects Board approval.• (Same process Tim just went through for 151.)• Examination of diversity friendliness & gender equity (how?).• Other measurements? (Now is the chance.)

extra post-presentation slides

Failsafe change?• Since test-manual is already in second draft (has been used for two semesters) most bugs have been removed.

• Not really changing anything. The labs are still set up the same with the same equipment. TAs still teach the same material. Only difference is communication style of manual.

Costless change?• Majority of time cost for writing already paid: (~100 hours Spring 2008, ~150 hours Fall 2008).

• Initial usage well received with no catastrophes or complaints.

Other writing team duties:• Incorporate UA Assessment learning goals.• Gender/diversity-neutral spotlights on physicists.• Affirmative gender-neutral statement beginning each goals.• Discussion about how physics can help others.• Malleability of IQ statement beginning each goals.• Arif’s quizzes and HW derivations. • Statement about studying together.• Statement about joining professional groups.• Modify authentic assessment to be group driven.• TA signature line to quickly examine worksheet completion.• Include specific prompts for open-ended questions.• Graphing instructions included on each writing guidelines.• Include notes for few beginning test-manuals.• List of necessary supplies and equipment.• Derivative explanations in RC & RL labs.• “…in SI units” in every numerical question.• “sketch” goes to “quickly sketch” where appropriate.• Begin each lab with fitting poem.• End each lab with Einstein chalkboard cartoon.

Spring 2008(previous)

Fall 2008(current)

125

T1+T2+T3

175

200Matt’sMales187.9

125

150

Others’Males179.5

Others’Males155.1

Matt’sMales148.8

p<0.0001

Matt’sFemales

188.2

Others’Females

142.8

Matt’sFemales

186.8

Others’Females

154.8

p<0.001

p<0.001p<0.001

p>0.10

p>0.10

p<0.001

p<0.001p<0.001

p<0.0001

Some Quantitative Observations

Statistics Used (Hogg & Tanis, 4th ed.):

€

TA-B =x A − x B − 0

SA-B ⋅1

nA

+1

nB

(i.e. normalized) with rA-B = nA + nB − 2

€

x A ± tA, 0.025 ⋅SA

nA −1 and x B ± tB, 0.025 ⋅

SB

nB −1

€

SA-B ≡nA −1( )SA

2 + nB −1( )SB2

nA + nB − 2Define

To hypothesis test difference of two means:

To report separate means with 95% conf:

Matt’s StudentsSpring 2008

Others’ StudentsSpring 2008

€

TMatt vs other = 0.34Means test: (p=large).

€

x MattSpring

=156.4 ± 21.3 and x OtherSpring

=152.7 ± 8.995% confidence intervals:

SUM 3 MIDTERMS(Previous Semester)

140

130

150

160

170

180

T1+T2+T3

No difference.

Matt’s StudentsFall 2008

Others’ StudentsFall 2008

€

TMatt vs other =1.24Means test: (p=.1075 trend).

€

x MattFall

=187.7 ±14.9 and x OtherFall

=176.6 ± 9.995% confidence intervals:

SUM 3 MIDTERMS(Current Semester)

170

160

180

190

200

210

T1+T2+T3

Someimprovement

(+17.3, 6% per test).

Matt’s MaleSpring 2008

Other’s MaleSpring 2008

€

TMatt vs other = 3.79Means test: (p=0.057 trend).

€

x MattSpring

=148.8 ± 22.6 and x OtherSpring

=155.1±10.495% confidence intervals:

SUM 3 MIDTERMS(Male Previous)

130

120

140

150

160

170

T1+T2+T3

Someweakening

(-6.3, -2% per test).

Matt’s MalesFall 2008

Others’ MalesFall 2008

€

TMatt vs other = 6.12Means test: (p<0.0001).

€

x MattFall

=187.9 ±16.7 and x OtherFall

=179.5 ±10.595% confidence intervals:

SUM 3 MIDTERMS(Male Current)

170

160

180

190

200

210

T1+T2+T3

Someimprovement

(8.4, 3% per test)

Matt’s FemaleSpring 2008

Others’ FemaleSpring 2008

€

TMatt vs other =15.2Means test: (p<0.001).

€

x MattSpring

=188.2 ± 75.1 and x OtherSpring

=142.8 ±16.195% confidence intervals:

SUM 3 MIDTERMS(Female Previous)

140

110

170

200

230

260

T1+T2+T3

Muchimprovement

(+45, 15% per test).

Matt’s StudentsFall 2008

Others’ StudentsFall 2008

€

TMatt vs other =10.6Means test: (p<0.001).

€

x MattFall

=186.8 ± 41.7 and x OtherFall

=154.8 ± 31.195% confidence intervals:

SUM 3 MIDTERMS(Female Current)

145

120

170

195

220

245

T1+T2+T3

Muchimprovement

(+32, 10% per test).