Are We Making Progress Yet?

Linda A. Reddy, Ph.D.Rutgers University

Are We Making Progress Yet?

Linda A. Reddy, Ph.D.Rutgers University

A rogue wave smashed the bridge of the ship, destroying all the instruments except the speedometer.  The captain announced to the badly shaken crew, "We have no idea where we are and we don't have a clue where we are going, but we do know we are making record time."

A rogue wave smashed the bridge of the ship, destroying all the instruments except the speedometer.  The captain announced to the badly shaken crew, "We have no idea where we are and we don't have a clue where we are going, but we do know we are making record time."

Progress Monitoring is All About Change and Change is Hard

Progress Monitoring is All About Change and Change is Hard

What is Progress Monitoring?What is Progress Monitoring?

“A scientifically based practice that is used to assess students’ academic performance and evaluate the effectiveness of instruction” (National Center on Students Progress Monitoring)

A “seamless and flexible” approach ….. that can be used across students of different age and skill levels, different settings and curricula, and across school years (Wallace et al., 2007)

“A set of assessment procedures for determining the extent to which students are benefiting from classroom instruction and for monitoring effectiveness of curriculum” (The National Research Center on Learning Disabilities)

“A scientifically based practice that is used to assess students’ academic performance and evaluate the effectiveness of instruction” (National Center on Students Progress Monitoring)

A “seamless and flexible” approach ….. that can be used across students of different age and skill levels, different settings and curricula, and across school years (Wallace et al., 2007)

“A set of assessment procedures for determining the extent to which students are benefiting from classroom instruction and for monitoring effectiveness of curriculum” (The National Research Center on Learning Disabilities)

Historical Context of Progress MonitoringHistorical Context of Progress Monitoring

Traditional progress monitoring has been used to identify one’s relative position within a group rather than evaluating individual progress across time (Deno, 1997)

The early practice was a means of establishing eligibility for certain classifications/programs (e.g., special education, gifted)

Originally, progress monitoring was based on “mastery measurement” Criterion-referenced tests used to assess the mastery of specific

curriculum objectives Following mastery of specific objectives (skills), the teacher would provide

instruction and assess the skill taught This method of progress monitoring was reflected in common teaching

practices (e.g., Wisconsin Instructional Design System, Precision Teaching)

Traditional progress monitoring has been used to identify one’s relative position within a group rather than evaluating individual progress across time (Deno, 1997)

The early practice was a means of establishing eligibility for certain classifications/programs (e.g., special education, gifted)

Originally, progress monitoring was based on “mastery measurement” Criterion-referenced tests used to assess the mastery of specific

curriculum objectives Following mastery of specific objectives (skills), the teacher would provide

instruction and assess the skill taught This method of progress monitoring was reflected in common teaching

practices (e.g., Wisconsin Instructional Design System, Precision Teaching)

Historical Context of Progress MonitoringHistorical Context of Progress Monitoring

■ Questions arose - mastery measurement's assumption that a series of short-term objectives directly leads to broad-based competence (false sense of progress)

Resulted alternative ways of conceptualizing progress monitoring

Curriculum-Based Measurement (Deno, 1985) Requires students to simultaneously integrate the various skills required for

competent yearend performance on weekly test As students learn the necessary components of the curriculum, their CBM score

gradually increases Slope can be used to quantify rate of learning and gauge students’ responsiveness

to an instructional program or OTL When “inadequate” responsiveness is revealed, program can be revised

NCLB (2001)

IDEA (2004) – LD assessment - removal of discrepancy criteria Advent or “rebirth” of RtI

■ Questions arose - mastery measurement's assumption that a series of short-term objectives directly leads to broad-based competence (false sense of progress)

Resulted alternative ways of conceptualizing progress monitoring

Curriculum-Based Measurement (Deno, 1985) Requires students to simultaneously integrate the various skills required for

competent yearend performance on weekly test As students learn the necessary components of the curriculum, their CBM score

gradually increases Slope can be used to quantify rate of learning and gauge students’ responsiveness

to an instructional program or OTL When “inadequate” responsiveness is revealed, program can be revised

NCLB (2001)

IDEA (2004) – LD assessment - removal of discrepancy criteria Advent or “rebirth” of RtI

What is Student Social Behavior?What is Student Social Behavior?

Broad global construct Heterogeneous Complex in nature and context Lack of agreement

Broad global construct Heterogeneous Complex in nature and context Lack of agreement

How “Student Social Behavior” is Defined?How “Student Social Behavior” is Defined?

■ School and interpersonal connectedness (attachments) (Bond et al., 2007)

Cooperation, assertion, self-control (Rutherford, DuPaul, & Jitendra, 2008)

Self and social awareness, responsible decision-making, self-management and relationship-management skills (Zins et al., 2004)

Physical and intellectual actions (e.g., hygiene, nutrition, physical activity, avoiding harmful substances, decision-making skills, creative thinking), social/emotional actions for managing oneself responsibly (e.g., self-control, time management), getting along with others (e.g., empathy, altruism, respect, conflict resolution), self-honesty, integrity, self-appraisal and continuous self-improvement (e.g., goal setting, problem solving, courage to try new things, persistence) (Snyder et al., 2010)

Cooperation (i.e., puts work materials or school property away), Assertion (introduces him / herself to new people without being told), and Self-control (control temper in conflict situations with adults). Level of aggression towards peers, helpfulness with other children, interactions with peers, involvement or exclusion by peers, level of anxious-fearfulness, and hyperactivity-distractibility (Rimm-Kaufmann & Chiu, 2007)

Awareness of self and others, positive attitudes and values, responsible decision making, social interaction skills (e.g. active listening, expressive communication, cooperation, negotiation, refusal, help seeking) (Payton et al., 2000)

■ School and interpersonal connectedness (attachments) (Bond et al., 2007)

Cooperation, assertion, self-control (Rutherford, DuPaul, & Jitendra, 2008)

Self and social awareness, responsible decision-making, self-management and relationship-management skills (Zins et al., 2004)

Physical and intellectual actions (e.g., hygiene, nutrition, physical activity, avoiding harmful substances, decision-making skills, creative thinking), social/emotional actions for managing oneself responsibly (e.g., self-control, time management), getting along with others (e.g., empathy, altruism, respect, conflict resolution), self-honesty, integrity, self-appraisal and continuous self-improvement (e.g., goal setting, problem solving, courage to try new things, persistence) (Snyder et al., 2010)

Cooperation (i.e., puts work materials or school property away), Assertion (introduces him / herself to new people without being told), and Self-control (control temper in conflict situations with adults). Level of aggression towards peers, helpfulness with other children, interactions with peers, involvement or exclusion by peers, level of anxious-fearfulness, and hyperactivity-distractibility (Rimm-Kaufmann & Chiu, 2007)

Awareness of self and others, positive attitudes and values, responsible decision making, social interaction skills (e.g. active listening, expressive communication, cooperation, negotiation, refusal, help seeking) (Payton et al., 2000)

Student Social Behavior and Academic Outcomes

Student Social Behavior and Academic Outcomes

A comprehensive K-5 school-based program targeting student behavior and character was linked to academic achievement, school attendance, and positive disciplinary outcomes (Snyder et al., 2010)

ADHD 1-4 grade students - Teacher ratings of reading gains following academic interventions related to improvements in social skills on the SSRS Peer-mediated math interventions produced improvements

in math fluency and self-control (Rutherford, DuPaul, & Jitendra, 2008)

Social problem-solving skills are related to cognitive problem-solving skills (Ben-Avie, & Ensign, 2003)

A comprehensive K-5 school-based program targeting student behavior and character was linked to academic achievement, school attendance, and positive disciplinary outcomes (Snyder et al., 2010)

ADHD 1-4 grade students - Teacher ratings of reading gains following academic interventions related to improvements in social skills on the SSRS Peer-mediated math interventions produced improvements

in math fluency and self-control (Rutherford, DuPaul, & Jitendra, 2008)

Social problem-solving skills are related to cognitive problem-solving skills (Ben-Avie, & Ensign, 2003)

Student Social Behaviors and Mental Health Outcomes

Student Social Behaviors and Mental Health Outcomes

Low school connectedness and high interpersonal conflict in early secondary school are linked to increased mental health problems and substance use in later schooling. The presence of both school and social connectedness was

associated with the lowered risk of depressive symptoms in students (Bond et al., 2007)

Good Behavior Game (GBG) universal prevention intervention in first grade classrooms predicted lower levels of antisocial behavior by middle school (Kellam & Anthony, 1998)

GBG intervention provided in 1-2 graders (n=1,196) significantly reduced the risk (50%) drug abuse/dependence, smoking, and antisocial behavior for adolescent and young adult (Kellam et. al., 2008)

Low school connectedness and high interpersonal conflict in early secondary school are linked to increased mental health problems and substance use in later schooling. The presence of both school and social connectedness was

associated with the lowered risk of depressive symptoms in students (Bond et al., 2007)

Good Behavior Game (GBG) universal prevention intervention in first grade classrooms predicted lower levels of antisocial behavior by middle school (Kellam & Anthony, 1998)

GBG intervention provided in 1-2 graders (n=1,196) significantly reduced the risk (50%) drug abuse/dependence, smoking, and antisocial behavior for adolescent and young adult (Kellam et. al., 2008)

Three Presentations: Some Shared IngredientsThree Presentations: Some Shared Ingredients Comprehensive and rigorous programs of research Emphasis on student social behavior in schools Informant is the teacher (general education or special

education) Reliance on teacher accuracy and cooperation All tools are brief, user friendly and easy to administer Present integrative assessment-intervention models Assessment is dynamic, fluid, and continuous Framed in a three tiered RtI model Emphasis test scores (raw or scale score) used as a method for

determining efficacy of interventions

Comprehensive and rigorous programs of research Emphasis on student social behavior in schools Informant is the teacher (general education or special

education) Reliance on teacher accuracy and cooperation All tools are brief, user friendly and easy to administer Present integrative assessment-intervention models Assessment is dynamic, fluid, and continuous Framed in a three tiered RtI model Emphasis test scores (raw or scale score) used as a method for

determining efficacy of interventions

Enhancing Individual Education Plans for Children with Attention-Deficit/Hyperactivity Disorder Using a Daily Report Card

Enhancing Individual Education Plans for Children with Attention-Deficit/Hyperactivity Disorder Using a Daily Report Card

Disorder-specific tool ADHD representing a highly comorbid population Assessment is fluid for daily use Students receive immediate teacher feedback on their efforts Can be used to develop positive student-teacher and student-

parents interactions/relationships Fosters parent and teacher communication and partnerships Scores linked to home-based privileges Tailored for individual student needs – linked to IEP and 504 Plan

goals – critical for buy in Assesses targeted behaviors within specific academic lessons (e.g.,

art, math, language arts, science etc.) Includes benchmarks of specific social behavior within context DRC raw scores discriminate between treatment conditions

Disorder-specific tool ADHD representing a highly comorbid population Assessment is fluid for daily use Students receive immediate teacher feedback on their efforts Can be used to develop positive student-teacher and student-

parents interactions/relationships Fosters parent and teacher communication and partnerships Scores linked to home-based privileges Tailored for individual student needs – linked to IEP and 504 Plan

goals – critical for buy in Assesses targeted behaviors within specific academic lessons (e.g.,

art, math, language arts, science etc.) Includes benchmarks of specific social behavior within context DRC raw scores discriminate between treatment conditions

Development and Validation of Progress Monitoring Tools for Social Behavior: Lessons from Project VIABLE

Development and Validation of Progress Monitoring Tools for Social Behavior: Lessons from Project VIABLE

■ DBR - undergone significant development and investigation, until now there has been limited attention to the psychometric properties

■ Highly reliable and valid alternative to SDO, interviews, and behavior rating scales

■ Emphasis on specific time, place (context) and intended rater for assessing the behavior

■ Excellent tool for IEP Goals and 504 Plans for K-12 grade students

■ Terrific tool for minimizing effects of rater inference and retrospective judgments about student social behavior

■ Considerations for developing other DBR measures: Rater bias; rater skill (error) Feasibility Consideration of base rates within context Wording (molar/molecular, negative/positive) Visual cues Anchors General outcome or individualized targets

■ DBR - undergone significant development and investigation, until now there has been limited attention to the psychometric properties

■ Highly reliable and valid alternative to SDO, interviews, and behavior rating scales

■ Emphasis on specific time, place (context) and intended rater for assessing the behavior

■ Excellent tool for IEP Goals and 504 Plans for K-12 grade students

■ Terrific tool for minimizing effects of rater inference and retrospective judgments about student social behavior

■ Considerations for developing other DBR measures: Rater bias; rater skill (error) Feasibility Consideration of base rates within context Wording (molar/molecular, negative/positive) Visual cues Anchors General outcome or individualized targets

Scale gradients – first study to examine a single item scale compared three scale gradients (categories – 6, 10, 14) Chafouleas et al. (2009) Educational and Psychological Measurement

Illustrates that DBR are highly useful for measuring the effects of intervention on individual and classroom behavior problems

DBR-SIS: 2 part web-based training program – frame of reference and rater error (accuracy)

DBR-BASIS: web-based program for data entry, analysis, and graphing

Scale gradients – first study to examine a single item scale compared three scale gradients (categories – 6, 10, 14) Chafouleas et al. (2009) Educational and Psychological Measurement

Illustrates that DBR are highly useful for measuring the effects of intervention on individual and classroom behavior problems

DBR-SIS: 2 part web-based training program – frame of reference and rater error (accuracy)

DBR-BASIS: web-based program for data entry, analysis, and graphing

Developing Change Sensitive Brief Behavior Rating Scales as Progress-Monitoring Tools for Social Behavior

Developing Change Sensitive Brief Behavior Rating Scales as Progress-Monitoring Tools for Social Behavior

■ Significant research on social skills (SSRS) serves as the spring board for the development of 4 brief behavior ratings scales for social skills, conduct, anxiety, and ADHD

■ Construct specificity - detect progress in specific area (skills) of social emotional functioning

■ Emphasis on change-sensitivity of constructs and items

■ Proposes that change sensitive item are identified by four data analytic methods [Odds ratio, Effect sizes, Paired and 2-sample T-tests,Interaction effect from a mixed-factorial ANOVA]

■ Emphasizes the assessment of both strength and problem behaviors “Dual lens” assessment approach – particularly important in the area of

ED

■ Significant research on social skills (SSRS) serves as the spring board for the development of 4 brief behavior ratings scales for social skills, conduct, anxiety, and ADHD

■ Construct specificity - detect progress in specific area (skills) of social emotional functioning

■ Emphasis on change-sensitivity of constructs and items

■ Proposes that change sensitive item are identified by four data analytic methods [Odds ratio, Effect sizes, Paired and 2-sample T-tests,Interaction effect from a mixed-factorial ANOVA]

■ Emphasizes the assessment of both strength and problem behaviors “Dual lens” assessment approach – particularly important in the area of

ED

Where to go from here?Where to go from here?

Opportunities for measure development are endless!

Adopt a broad iterative test development and validation approach: Unified model of validation (Messick, Cronbach) and… Rigorously examine and validate test utility

Utility of test score(s) for informing decision making for intervention plans that improve students’ lives (meaningfully)

Opportunities for measure development are endless!

Adopt a broad iterative test development and validation approach: Unified model of validation (Messick, Cronbach) and… Rigorously examine and validate test utility

Utility of test score(s) for informing decision making for intervention plans that improve students’ lives (meaningfully)

Test Utility Framework for Future Test Development and ValidationTest Utility Framework for Future Test Development and Validation

“In developing a test, one must first decide on the proposed interpretations and uses - and the assumptions embedded in the proposed interpretations and uses. The chosen interpretations and uses then provide guidance on how to proceed in developing the test” (Kane, 2008; pp. 77).

“In developing a test, one must first decide on the proposed interpretations and uses - and the assumptions embedded in the proposed interpretations and uses. The chosen interpretations and uses then provide guidance on how to proceed in developing the test” (Kane, 2008; pp. 77).

Before Test Development and Validation Before Test Development and Validation

What is the specific construct(s) we intended to measure?

What is the intended purpose of the test?

Include a strong consumer participatory model for test development

What is the "end game” goal?

Test score utility (inference):

How can the test score(s) generalize to the function and process of the targeted social behavior in the classroom?

How can the test score(s) be used by the consumers to inform decision making for intervention plans:

individual student classrooms schools school district level

We must go beyond student level assessment and focus on system level assessment and change

What is the specific construct(s) we intended to measure?

What is the intended purpose of the test?

Include a strong consumer participatory model for test development

What is the "end game” goal?

Test score utility (inference):

How can the test score(s) generalize to the function and process of the targeted social behavior in the classroom?

How can the test score(s) be used by the consumers to inform decision making for intervention plans:

individual student classrooms schools school district level

We must go beyond student level assessment and focus on system level assessment and change

Where to go from here?Where to go from here?

Adopt a Strong Program of External Validation

Acceptability and usefulness of consumers

Carefully examine consumers’ intended and unintended test score inferences

What are the extraneous factors that impact test score utility (inferences) for the consumer?

Do test scores change with targeted practical school interventions and how do test scores change?

Incremental validity over business as usual

What are the pragmatic factor (barriers) that may impede development and validation (e.g., time, cost, teacher sensitivity or bias towards constructs and items)?

Adopt a Strong Program of External Validation

Acceptability and usefulness of consumers

Carefully examine consumers’ intended and unintended test score inferences

What are the extraneous factors that impact test score utility (inferences) for the consumer?

Do test scores change with targeted practical school interventions and how do test scores change?

Incremental validity over business as usual

What are the pragmatic factor (barriers) that may impede development and validation (e.g., time, cost, teacher sensitivity or bias towards constructs and items)?

Example Barriers for Future Measurement Development Example Barriers for Future Measurement Development

Barriers: Consumers’ perceptions of time, work, and cost Consumers’ perceived utility Progress monitoring instruments may not be sufficiently sensitive

to monitor progress among high school students because of small growth rates in this age range

Progress monitoring tools must be validated for different age groups (e.g., secondary students)

Absence of school staff buy-in School personnel misunderstanding and ineffective use of data

Barriers: Consumers’ perceptions of time, work, and cost Consumers’ perceived utility Progress monitoring instruments may not be sufficiently sensitive

to monitor progress among high school students because of small growth rates in this age range

Progress monitoring tools must be validated for different age groups (e.g., secondary students)

Absence of school staff buy-in School personnel misunderstanding and ineffective use of data

Example Facilitators for Future Measurement Development Example Facilitators for Future Measurement Development

Facilitators: Committed administrative leadership and support District-wide support Ongoing professional development Cohesive team membership Team efficiency (e.g., clear operating standards) Stakeholder agreement and shared vision regarding the change Autonomous teachers School psychologists serving as leaders, change agents, and

consultants Financial resources committed to the cause Organizational restructuring – system change

Facilitators: Committed administrative leadership and support District-wide support Ongoing professional development Cohesive team membership Team efficiency (e.g., clear operating standards) Stakeholder agreement and shared vision regarding the change Autonomous teachers School psychologists serving as leaders, change agents, and

consultants Financial resources committed to the cause Organizational restructuring – system change

Where to go from here?: How about teacher behavior?

Where to go from here?: How about teacher behavior?

What teachers do in the classroom influences students’ learning and behavior.

GBG intervention with 570 2-3 graders - reduced use of negative teacher remarks predicted improvements in on-task behavior and talking out behavior. Improved student behavior mediated the impact of the intervention on the

development of hyperactive and oppositional behavior (Leflot et al., 2010).

Teachers’ social and emotional competence are key to “creating a classroom climate that is more conducive to learning and that promotes positive developmental outcomes among students” (Jennings & Greenberg, 2009).

Teachers who interact with their students in supportive ways promote positive student social behavior (Wentzel, et al., 2010).

Problematic relationships between Kindergarten teachers and their students with behavior problems predict academic and behavior problems through eighth grade (Hamre & Pianta, 2001).

Students with ED are significantly influenced by teacher-student interactions (Sutherland et al., 2008; Reddy et al., 2009).

What teachers do in the classroom influences students’ learning and behavior.

GBG intervention with 570 2-3 graders - reduced use of negative teacher remarks predicted improvements in on-task behavior and talking out behavior. Improved student behavior mediated the impact of the intervention on the

development of hyperactive and oppositional behavior (Leflot et al., 2010).

Teachers’ social and emotional competence are key to “creating a classroom climate that is more conducive to learning and that promotes positive developmental outcomes among students” (Jennings & Greenberg, 2009).

Teachers who interact with their students in supportive ways promote positive student social behavior (Wentzel, et al., 2010).

Problematic relationships between Kindergarten teachers and their students with behavior problems predict academic and behavior problems through eighth grade (Hamre & Pianta, 2001).

Students with ED are significantly influenced by teacher-student interactions (Sutherland et al., 2008; Reddy et al., 2009).

Future Measurement Development Projects Future Measurement Development Projects

Tools for preschool, elementary school, and middle and high school that inform supports and interventions

Tools that are dynamic and flexible for 6-12 grade - multiple teachers and the complexity of settings (classroom, hallway, study hall, lunch, after school activities such as clubs and sports)

Tools tailored for specific disorders and conditions - example Aspergers, Autism, and Anxiety/Depression

Brief tools for teacher and student interactions (specific aspects of classroom and school climate)

Tools for teacher classroom practices – general and special education settings

Tools assess aspects of teachers' social emotional competency (e.g., anger management, efficacy, perceived risk of violence)

Tools that can link meaningful goals for IEPs and 504 plans

Tools for preschool, elementary school, and middle and high school that inform supports and interventions

Tools that are dynamic and flexible for 6-12 grade - multiple teachers and the complexity of settings (classroom, hallway, study hall, lunch, after school activities such as clubs and sports)

Tools tailored for specific disorders and conditions - example Aspergers, Autism, and Anxiety/Depression

Brief tools for teacher and student interactions (specific aspects of classroom and school climate)

Tools for teacher classroom practices – general and special education settings

Tools assess aspects of teachers' social emotional competency (e.g., anger management, efficacy, perceived risk of violence)

Tools that can link meaningful goals for IEPs and 504 plans

Future Measurement Projects - Opportunity for School ReformFuture Measurement Projects - Opportunity for School Reform

By 2015 over 1/3 of the nation’s veteran teachers and school administrators (baby boomers) will be leaving the field of education.

This change in school personnel offer the nation tremendous opportunity for comprehensive school reform and innovation in teacher preparation and training across the nation.

The time is right for new progress monitor tools for students and teachers.

By 2015 over 1/3 of the nation’s veteran teachers and school administrators (baby boomers) will be leaving the field of education.

This change in school personnel offer the nation tremendous opportunity for comprehensive school reform and innovation in teacher preparation and training across the nation.

The time is right for new progress monitor tools for students and teachers.

Thank you!Thank you!