maximizing evaluation impact by maximizing methods:
DESCRIPTION
Maximizing Evaluation Impact by Maximizing Methods:. Social Network Analysis Combined with Traditional Methods for Measuring Collaboration Carl Hanssen, PhD & MaryAnn Durland, PhD American Evaluation Association Baltimore, MD November 7, 2007. Agenda. Social Network Analysis: The Method - PowerPoint PPT PresentationTRANSCRIPT
Maximizing Evaluation Impact by Maximizing Methods:Social Network Analysis Combined with Traditional Methods for Measuring Collaboration
Carl Hanssen, PhD & MaryAnn Durland, PhDAmerican Evaluation AssociationBaltimore, MDNovember 7, 2007
11/07/2007 2
Agenda
Social Network Analysis: The Method
SNA Results and Interpretation
Next Steps
11/07/2007 3
SNA Methodology
Network Analysis is the study of the relationships formed by the interaction or links between components in a “set”.
MMP sets are schools The components are individuals:
Faculty, both math and non math MTL (School level Math Teacher Leaders) MTS (District level Math Teacher Specialists)
Relationship is communication about math
11/07/2007 4
Measures
Indegree – popularity Density – how “thick”, how much,
out of potential
11/07/2007 5
Role in Evaluation
How much does the communication structure actually fit the theory and the design of the project
Can the structure be correlated with other measures of implementation and impact? Activities Proximal measures
11/07/2007 6
MMP Evaluation Logic ModelStudent
Achievement
Teacher Content& Pedagogical
Knowledge
Math FacultyInvolvement
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
NewCourses
DistrictBuy-in
MPA Ownership
MATCBuy-In
UWMBuy-In
ClassroomPractice
MMPActivities
ProximalOutcomes
DistalOutcomes
11/07/2007 7
MMP Report Card Indicators
19 indicators in 7 domains derived from in-school data collection, online surveys, and MPS data
1. MTS Assessment2. Collaboration3. Learning Teams4. Classroom Practice5. Professional Development6. Teacher MKT7. Student Achievement
11/07/2007 8
SNA In Context: Evaluation Results
StudentAchievement
Teacher Content& Pedagogical
Knowledge
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
ClassroomPractice
WKCEMean % Proficient = 44%
Overall rating = 3.5Gap MTL v. other teacher = .2Teacher Engagement = 3.2
Overall IRT = -0.34Algebra IRT = -0.18
Team Functioning = 3.5MMP Principles = 3.6LT Quality = 3.1
PD Hrs. = 17.8Facilitation Hrs. = 1.0PD Quality = 3.1
Network density = 6.7% / School density = 17.6%MTL Role = 13.8 / MTS Role = 5.3
SR MTL Engagement = 4.4 / MTS Quality = 3.0
MTS Assessment = 38.3 of 55
11/07/2007 9
Data Collection
Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics
Statistical analysis focused on1. Network and in-school density2. Importance of MTL and MTS
11/07/2007 10
MMP Impact Continuum
Low High
Loose WebMTL Not CentralFew Links to MTLMTS OutsideFew Links to MTS
Tight WebMTL Central
Many Links to MTLMTS Inside
Many Links to MTS
11/07/2007 11
LowSchool n Total Named
Network density
Density in school
MTL Role--In Degree
MTS Role--In Degree
G 17 40 6.2% 9.0% 17.31 3.85
Average 21.1 54.0 6.7% 17.6% 13.81 5.31 SD 6.8 17.6 2.6% 9.6% 7.2 4.9 Median 19 48 6.2% 15.4% 13.07 3.75
11/07/2007 12
MediumSchool n Total Named
Network density
Density in school
MTL Role--In Degree
MTS Role--In Degree
I 28.0 75.0 4.0% 12.2% 23.31 0.33
Average 21.9 57.1 6.3% 12.2% 18.84 2.69 SD 8.0 16.7 2.6% 5.0% 6.90 3.70 Median 22.0 51.0 5.7% 11.4% 17.56 0.92
11/07/2007 13
HighSchool n Total Named
Network density
Density in school
MTL Role--In Degree
MTS Role--In Degree
B 23 55 11.4% 31.1% 28.24 18.52
Average 21.1 54.0 6.7% 17.6% 13.81 5.31 SD 6.8 17.6 2.6% 9.6% 7.2 4.9 Median 19 48 6.2% 15.4% 13.07 3.75
11/07/2007 14
Student Achievement & In-School Network Density
11/07/2007 15
Student Achievement & MTL In Degree
11/07/2007 16
Conclusions
Distributed leadership—a key program goal is manifested by a tightly webbed network
School-level adoption of program principals is manifested by positioning of key individuals within the network
There may be a natural evolution of school networks that is indicative of program impact in that school
11/07/2007 17
Next Steps Continue school-level analysis to
strengthen our hypothesis about the relationship between social networks and other proximal and distal outcomes
Develop cross-school (or district-wide) networks
11/07/2007 18
Contact Information
Carl Hanssen, PhDHanssen Consulting, [email protected]
MaryAnn Durland, PhDDurland [email protected]