professional development activity log: comparing teacher log and survey approaches to evaluating...
TRANSCRIPT
Professional Development Activity Professional Development Activity Log: Log: Comparing Teacher Log and Comparing Teacher Log and Survey Approaches to Evaluating Survey Approaches to Evaluating Professional DevelopmentProfessional Development
AERA Annual Meeting
Montreal, April 11, 2004
Symposium - Evaluating the Quality of Professional Development: Symposium - Evaluating the Quality of Professional Development: Implications for Districts and StatesImplications for Districts and States
Kwang Suk Yoon
Reuben JacobsonAmerican Institutes for Research
OverviewOverview
Comparisons between teacher logs with survey
Teachers’ experience with logs Lessons learned Next steps
Professional Development Activity Log Professional Development Activity Log (PDAL)(PDAL)
The PDAL is a web-based, self-administered, longitudinal data collection tool for teachers to record their professional development experiences in detail with the assistance of a series of structured prompts
Teachers log on to their password-protected web account and fill out their PDAL at regular intervals
Visit www.PDAL.net for more information
PDAL Entries PDAL Entries
Name of activity Number of hours spent on each activity and its duration Whether the activity is a one-time or continuous event (e.g.,
recurring over a number of months) Type of activity (e.g., workshop, summer institute, study group) Purpose of activity (e.g., strengthening subject matter knowledge) PD quality features (e.g., active learning, coherence, collective
participation) Content focus (e.g., algebraic concepts: absolute values, use of
variables, etc.) Instructional practice – instructional topics covered in each activity
(e.g., use of calculators, computers, or other educational technology)
Why PDAL?Why PDAL?
Collects disaggregate information about specific PD activities – Increases the level of specificity of PD data and reduces bias introduced by gross data aggregation
Gathers accurate, reliable, and time-sensitive information – Minimizes recall problem with retrospective reports
Tailors technical assistance to teachers based on their response patterns
Allows teachers to review their own logs – Teachers can reflect on their own PD experiences
Generates context sensitive questions
““Black BoxBlack Box”” of of SurveySurvey Data Data
Hypothetical Data
Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04
Mr. Anderson A,B,C 32 30 12 - 21 - 14 - 109 0.8
Ms. Lopez A,D 32 32 - - 10 10 12 10 106 1.3
Mrs. Kelly E - 20 20 - - - - - 40 0.5
Mr. Lee E,F 18 18 44 18 - - - - 98 0.8
Mrs. Smith G - - - - 8 - - - 8 1.0
Aggregate 0 82 100 76 18 39 10 26 10 361 0.9
TeacherTime Total
Contact Hours
Mean Active
Learning
Activity
Fine-grain Fine-grain log-level data log-level data on con contact ontact hhours : ours : DDisaggregated isaggregated by tby teacher by activity by timeeacher by activity by time
Hypothetical Data
Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04
Mr. Anderson A 32 30 - - - - - - 62 2 31.0
B - - 12 - 13 - 14 - 39 3 13.0
C - - - - 8 - - - 8 1 8.0
Ms. Lopez A 32 32 - - - - - - 64 2 32.0
D - - - - 10 10 12 10 42 4 10.5
Mrs. Kelly E - 20 20 - - - - - 40 2 20.0
Mr. Lee E 18 18 16 - - - - - 52 3 17.3
F - - 28 18 - - - - 46 2 23.0
Mrs. Smith G - - - - 8 - - - 8 1 8.0
Total Contact Hours 82 100 76 18 39 10 26 10 361 20 18.1
No of Logs 3 4 4 1 4 1 2 1
Mean Contact Hours 27.3 25.0 19.0 18.0 9.8 10.0 13.0 10.0
Mean Contact
Hours
Teacher ActivityTime Total
Contact Hours
Mean Active
Learning
Validation of Teacher Logs as an Validation of Teacher Logs as an Alternative Data Collection MethodAlternative Data Collection Method
Comparing PDAL and exit survey results – Measurement properties– Correlations– Mean levels
Assessing the relative efficacy of teacher logs and survey
MethodMethod
Instruments– PDAL
• Conduced over 15 months – Exit Survey
• Follow-up questions about PD activities over the same period of 15 months
• Questions about teachers’ experiences with PDAL Sample
– 4 Math-Science Partnership Program projects– Sample
• 476 math and science teachers mostly in middle or high schools
– Participants• 326 teachers completed at least for a month in PDAL• 165 teachers participated in the Exit Survey
Measurement Properties of PDALMeasurement Properties of PDAL
# of # of
Variable items factors PDAL Survey
Active learning 8 2 0.86 0.76
Coherence 5 1 0.83 0.85
Collective participation 2 1 0.62 0.76
Reliability (a coef.)
Agreement between Log & Survey Methods:Agreement between Log & Survey Methods:Correlations between methodsCorrelations between methods
VariableCorrelation
between methods p
Active learning 0.44 ***Coherence 0.42 ***
Collective participation 0.56 ***Contact hours in WIC 0.55 ***
Frequency of PDAL use 0.79 ***Note: Comparisons are restricted to teachers who participated in PDAL for 12 months or longer.WIC denotes workshop, summer institute, and college course.
Agreement between Log & Survey Methods:Agreement between Log & Survey Methods:Mean LevelsMean Levels
Log SurveyVariable Mean Mean (s.e.) t d.f. p
Active learning 1.04 1.42 0.38 0.06 6.05 96 ***
Coherence 2.21 2.11 -0.11 0.06 -1.75 94 ns
Collective participation 0.65 0.93 0.28 0.07 3.80 96 ***
Contact hours in WIC 68.60 68.29 -0.30 5.26 -0.06 89 ns
Frequency of PDAL use 4.82 4.57 -0.25 0.07 -3.54 111 ***
Note: Comparisons are restricted to teachers who participated in PDAL for 12 months or longer.
Survey effect
Teachers’ Experience with or Opinions Teachers’ Experience with or Opinions about PDAL:about PDAL:
Results from the PDAL Exit SurveyResults from the PDAL Exit Survey
Variable N Mean SD Min Max
On the scale of 1=strongly disagree, 2=disagree, 3=agree, and 4=strongly agree.
Usability of the PDAL 148 2.78 0.50 1.3 4.0Support for PDAL Use 148 3.00 0.64 1.0 4.0Technical Difficulties 159 2.03 0.67 1.0 4.0Technology Obstacles 161 1.58 0.55 1.0 3.0Lack of Interest in Surveys 161 2.23 0.67 1.0 4.0Benefits of the PDAL 148 2.60 0.60 1.0 4.0
On the scale of 1=yes and 2=no.
Prefer a paper version? 161 1.75 0.43 1.0 2.0
Unique Benefits of PDALUnique Benefits of PDAL
Rich, in-depth data with a high level of specificity– Differences in features between PD sponsored by
MSP vs. other PD– Topic intensity (i.e., amount of contact hours per
topic) Time-dependent measures
– Percent of months with PD – Average contact hours by month– Average span of activity
Can be used for on-going formative evaluation to continuously improve PD – Episode-specific comments and feedback
AverageAverage Contact Hours by Month Contact Hours by Month
0
5
10
15
20
25
30
35
40
Jul-03
Aug-03
Sep-03
Oct-03
Nov-03
Dec-03
Jan-04
Feb-04
Mar-04
Apr-04
May-04
Jun-04
Jul-04
Aug-04
Month
Hou
rs
Lessons Learned & ImplicationsLessons Learned & Implications
Complementary uses of logs & surveys for different purposes– Globally estimating the mean level of PD activities– Investigating the variability of specific PD elements
and relating it to other outcomes – Improving PD design
Cost and benefits of different data collection methods
Improving survey method: Increase the level of specificity
Next StepsNext Steps
Final phase of data analysis– Using both PDAL and survey data to assess
change in teaching practice and assess their relative predictive validity
PDAL users focus group (May 2005) PDAL usability study Need for follow-up studies
– Such as a new CCSSO-AIR study on improving evaluation of professional development in math and science at state and local levels
Contact InformationContact Information
Kwang Suk Yoon
(202) 403-5358
Reuben Jacobson
(202) 403-6925
Visit us
www.PDAL.net www.air.org