opening the black box of individualization in programs for young children
DESCRIPTION
Opening the Black Box of Individualization in Programs for Young Children. Division of Early Childhood Conference San Francisco, CA October 18, 2013. The Black Box. How Do we use Progress Monitoring to help us individualize?. Panelists. Judy Carta, Senior Scientist, University of Kansas - PowerPoint PPT PresentationTRANSCRIPT
OPENING THE BLACK BOX OF INDIVIDUALIZATION IN PROGRAMS FOR YOUNG CHILDREN
Division of Early Childhood ConferenceSan Francisco, CAOctober 18, 2013
The Black Box
HOW DO WE USE PROGRESS MONITORING TO HELP US INDIVIDUALIZE?
Panelists• Judy Carta, Senior Scientist, University of Kansas
• Sally Atkins-Burnett, Senior Researcher, Mathematica Policy Research
• Charlie Greenwood, Senior Scientist, University of Kansas
• Alisha Wackerle-Hollman, Research Associate, University of Minnesota
• Jane Squires, Professor, University of Oregon
Questions we want to discuss• How do we do progress monitoring and
how do we use it to individualize for young children in our programs?
• How do we know when progress monitoring and individualizing are being done well? What are our quality indicators?
• Do we have evidence that progress monitoring and individualizing leads to better outcomes?
• What do programs need to know to determine when progress monitoring is done well?
Format for today’s session• Sally will describe a recently completed review of
literature describing what we know about progress monitoring and how it’s used for individualization
• Charlie will describe a general outcomes approach to progress monitoring with infants and toddlers, describe how it’s used for individualization and ways to determine the quality of implementation of progress monitoring
• Alicia will describe a general outcomes approach to progress monitoring with preschool-aged children
• Jane will describe a curriculum-based approach to progress monitoring and individualization
• We will have time to hear from YOU.
We want to hear from you!
Assessing Early Childhood Teachers’ Use of Child Progress Monitoring to Individualize Teaching Practices
DEC ConferenceOctober 18, 2013
Sally Atkins-Burnett, Lauren Akers, Patricia Del Grosso, Shannon Monahan, Judith Carta, Barbara A. Wasik, Kimberly Boller
Evaluate the existing evidence– Are there existing measures of teachers’ use of assessment
for individualization? – What is important to measure?
Develop evidence-informed conceptual model of ongoing child assessment for individualizing instruction
Develop a plan to efficiently assess implementation of the model– What methods and modes of data collection are needed?– What is feasible?
Develop and pretest a measure that could inform early childhood research and practice
Project Aims
9
Literature Review Approach and Preliminary Findings
10
Identify the key areas to include when evaluating the activities involved in the process of monitoring child progress and using that information for instruction and individualization
Find examples of how others have measured teachers’ implementation of ongoing assessment and progress monitoring
Identify gaps in the literature
Inform the project’s conceptual framework
Purpose of the Literature Review
11
To identify studies we:– Conducted a structured library search of last 10
years of research– Solicited recommendations from the project team
and expert consultant group
We identified 1,322 studies; of those 198 met relevance criteria and were screened into the review– Studies screened out for being off-topic, not an
eligible target population, not a relevant document type, duplicate studies, or not published in English
Methods and Results
12
To identify studies we:– Conducted a structured library search of last 10
years of research– Solicited recommendations from the project team
and expert consultant group
We identified 1,322 studies; of those 198 met relevance criteria and were screened into the review– Studies screened out for being off-topic, not an
eligible target population, not a relevant document type, duplicate studies, or not published in English
Methods and Results
13
10 studies reported on teachers’ perceptions or experiences with progress monitoring and using data to inform instruction– Commonly cited barriers to implementation:
• skill needed to use data for individualization and • knowledge of the subject matter
– Teachers reported the need for additional training
Teachers’ Perceptions of Progress Monitoring
14
No studies focused on how teachers actually select observation and assessment targets and methods
Over one-third of studies described methods for documenting children’s progress and systems for organizing information – Most only briefly mentioned the types of methods
used– Studies frequently discussed web-based or
technology-enhanced systems – 8 studies explored implementation experiences
Activities Involved in Using Progress Monitoring
15
Few studies described how teachers interpret data and/or apply it to instruction and individualization
17 studies discussed ways to engage families in progress monitoring– Studies described using it to provide regular
feedback to parents on children’s progress or engaging families in collecting data and using data for goal-setting
– One study focused on teaching primary caregivers to conduct formative assessments
Activities Involved in Using Progress Monitoring (cont’d)
16
Studies most often measured what teachers do– 14 of 18 studies measured whether teachers
implemented a specific progress monitoring tool with fidelity or reliability
– 5 studies examined the instructional modifications teachers made
4 studies measured what teachers think – 1 study used teacher written reports; another study
used a series of teacher interviews, including a think-aloud data analysis scenario
2 studies assessed what teachers know about child development, assessment, and/or instruction
Measures Used to Assess Implementation
17
Web- or computer-based systems can assist teachers in documenting, organizing, interpreting, and planning how to use data – Implementation issues can hinder the utility of these
systems
Ongoing professional development and support for teachers may assist teachers with using data to individualize instruction
Families are essential partners
Features of Successful Implementation
18
Very limited research exists about the use of progress monitoring in domains other than language/literacy, social-emotional, and math
Minimal research has focused on – Using progress monitoring in home visiting programs – Supporting families to in collecting assessment
information and observing their children
Research points to the importance of ongoing support for teachers– Although much of this research has been conducted
with teachers in K-3
Gaps in the Literature
19
Few studies have assessed implementation of progress monitoring and the individualization process– No studies have assessed implementation across a
range of progress monitoring tools – Studies typically only looked at one or two of the
activities involved in the process of using child progress monitoring for instruction and individualization
Gaps in the Literature (cont’d)
20
Few studies have assessed implementation of progress monitoring and the individualization process– No studies have assessed implementation across a
range of progress monitoring tools – Studies typically only looked at one or two of the
activities involved in the process of using child progress monitoring for instruction and individualization
Gaps in the Literature (cont’d)
21
Continuous, frequent, and standard assessment of child progress toward a long-term goal or outcome
Repeated measurement of a set of indicators predictive of a later outcome– Increasing proficiency indicated by rate of growth– Trend line compares expected versus actual rates of
learning
Brief and quick to administer
General Outcomes Approaches
22
Conceptual Model for the
GOM Approach
23
Information often collected within the context of the delivery of the curriculum
Assessments closely aligned to the curriculum
Assessments intended to be authentic in context
Curriculum-Embedded Approaches
24
Conceptual Model for the Curriculum-Embedded
Approach
25
A GENERAL OUTCOMES APPROACH TO PROGRESS MONITORING AND INDIVIDUALIZING FOR INFANTS AND TODDLERS
Charles Greenwood,Division of Early Childhood Conference
San Francisco, CAOctober 18, 2013
My Topics for Today• How is individualization defined?• What is my approach to progress monitoring?• What framework guides data-based individualization?• How is progress monitoring data used to track a child’s
response to intervention?• How is implementation quality addressed?• Is it working to improve children’s outcomes?
How is Individualization Defined?• A long held tenet of educational and behavioral
psychology and early intervention is that instruction (intervention) should be “adjusted” based on its observed effects on the learner• If effective, continue and improve the intervention• If not effective, change the intervention and try something else
• Individualization is occurring when this kind of dynamic “adjusting” is happening for all children in a program when needed
What is My Approach to Progress Monitoring?
• General outcome measurement in the form of infant/toddler Indicators of Individual Growth and Development (IGDIs) • Seasonal universal screening (quarterly)• Monthly progress monitoring for children receiving intervention• IGDI benchmarks and trends over time (decision points) are used
for making intervention decisions
Individual Child’s Growth Chart with Benchmark Indicators
Child’s Scores
Normative Trajectorie
s
Yes
What Framework Guides Data-Based Individualization?
NoQuarterly ECI Assessments
Is the intervention being implemented?
What intervention strategies should be
used?
Yes
What is causing the problem?
Is there a problem?
Is the intervention working?
Tilly, W. D. (2002). Best practices in school psychology as a problem solving enterprise. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology IV (Vol. 1, pp. 21-36). Washington, DC: National Association of School Psychologists.
No
No
How is Implementation Quality Addressed?
• The infant/toddler IGDI website provides supports:• information• training• Implementation• data handling (entry, charting)• reporting (child and program levels)• making intervention decisions (MOD)
How is Implementation Quality Addressed (Continued)?• Programs can monitoring their own implementation quality
in two ways:• Monitoring the quality of IGDI data collection within a program and
managing issues• Tracking the fidelity of intervention provided by home visitors and
parents in the home
Program Level Reports
Data Collection Indicators
Two Fidelity Indicators:
1. Certified Assessors2. Percentage Outliers
What Should Be Done?
Is it Being Done?
How is Progress Monitoring Data Used to Track a Child’s Response to Intervention?• A child’s individual growth trajectory interrupted at a point in time is examined for evidence of change in level and slope• A positive effect of intervention is signaled by a change in level, change in slope, or both
• A none effect of intervention is signaled by no change in level or slope
Is the Intervention Working – Child Level?
Change in Slope
Slopes Before and
After
Is It Working To Improve Children’s Outcomes?
• Program’s can monitor:• Child-level progress over time• Program level progress over time
Is it Working to Improve Children’s Outcomes? – Program Level
References
• Buzhardt, J., Greenwood, C. R., Walker, D., Anderson, R., Howard, W. J., & Carta, J. J. (2011). Effects of web-based support on Early Head Start home visitors’ use of evidence-based intervention decision making and growth in children’s expressive communication. NHSA Dialog: A Research-to-Practice Journal for the Early Childhood Field, 14(3), 121-146.
• Buzhardt, J., Walker, D., Greenwood, C. R., & Carta, J. J. (2011). A study of an online tool to support evidence-based practices with infants and toddlers. NHSA Dialog: A Research-to-Practice Journal for the Early Childhood Field, 14(3), 151-156.
• Greenwood, C. R., Buzhardt, J., Walker, D., Howard, W. J., & Anderson, R. (2011). Program-level influences on the measurement of early communication for infants and toddlers in Early Head Start. Journal of Early Intervention, 33(2 ), 110-134.
• Greenwood, C. R., Walker, D., & Buzhardt, J. (2010). The Early Communication Indicator (ECI) for Infants and Toddlers: Early Head Start Growth Norms from Two States. Journal of Early Intervention, 32(5), 310-334.
• Greenwood, C. R., Walker, D., Buzhardt, J., Howard, W. J., McCune, L., & Anderson, R. (2013). Evidence of a continuum in foundational expressive communication skills. Early Childhood Research Quarterly, 28, 540-554.
• Greenwood, C. R., Walker, D., Buzhardt, J., McCune, L., & Howard, W. J. 2013). Advancing the construct validity of the Early Communication Indicator (ECI) for infants and toddlers: Equivalence of growth trajectories across two Early Head Start samples. Early Childhood Research Quarterly, 28, 743-758.
Intentional Individualization: Using IGDI 2.0 progress monitoring data to inform instruction
Alisha Wackerle-Hollman Ph.D.Division of Early Childhood Conference
October 18th, 2013
“OUR BEST TEACHERS TODAY ARE USING REAL TIME DATA
IN WAYS THAT WOULD HAVE BEEN UNIMAGINABLE
JUST FIVE YEARS AGO. THEY NEED TO KNOW HOW WELL
THEIR STUDENTS ARE PERFORMING. THEY WANT TO KNOW EXACTLY
WHAT THEY NEED TO DO TO TEACH AND HOW TO TEACH IT.
(DUNCAN, 2009)
Individual Growth and Development Indicators (IGDIs 2.0)- Design Principles
• General Outcome Measure Approach– Treatment Independent ( Slavin & Madden, 2011)
• Construct-Aligned and Validated for Specific Purposes– Validity as an argument (Kane, 2013)
• Psychometrically Robust– Rasch Modeling
• Potential to match test material to student ability “window”• Increased opportunities for sensitivity to growth
Individual Growth and Development Indicators (IGDIs 2.0)- Content & Tasks• Four domains of early language and
literacy– Oral Language– Phonological Awareness– Alphabet Knowledge– Comprehension
• Five IGDI 2.0 tasks: Picture Naming, Rhyming, First Sounds, Sound Identification, and Which One Does Not Belong?
Individual Growth and Development Indicators (IGDIs 2.0)- Content & Tasks
Progress Monitoring: Making Decision
• 30 item sets selected from a performance range characteristic of Tier 2/Tier 3 candidacy.
• Designed to be used every three weeks– Efficiency in data-based decision making and practical
utility (Jenkins & Terjeson, 2011)– However, K-8 data suggests frequency may need to
be MORE often (Christ , Zopluoglu, Monaghen & Van Norman, 2012)
• Addressing ResponsivenessThe balancing act- the year before kindergarten
• Waiting for a skill to emerge• Waiting to make instructional changes• Waiting for robust trends in performance
Individual Growth and Development Indicators (IGDIs 2.0)- Content & Tasks
Progress-Monitoring: Making Decision• Evaluation of Progress
– Slope-based standards based on relative goals (Jenkins & Terjeson, 2011; Jenkins, Graff, Miglioretti, 2009)
– Goals established in reference to seasonal cut-points
• Intervention Individualization– Connecting the construct of interest to
meaningful and evidence-based intervention– Improving individualization through additional
data sources• Combining GOM indicators with Mastery
Monitoring information to best serve the student.
• Error Analysis
Instructional Changes• How do we examine student responses to use
data to make modifications?
Baseline Week 3 Week 6 Week 9 Week 120.25
0.35
0.45
0.55
0.65
0.75
0.85
0.95
1.05
1.15
Progress Monitoring –Treatment First Sounds
Ras
ch S
core
Instructional Changes
Baseline Week 3 Week 6 Week 9 Week 12-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
Progress Monitoring: Sound Identification
Ras
ch S
core
Intervention Considerations• What duration of intervention is
sufficient?– Constraints of Classroom dosage– 4 weeks as a bottom-line minimum to see
effectiveness? (Christ et al., 2012)– 8 weeks minimum for appropriate instructional
changes? (Ardoin, Christ, Morena, Cormier & Klingbeil, 2012; Shapiro, 2013)
• Intervention Effectiveness Factors– Quality of intervention – Quality of Fidelity of intervention administration– Dosage
Implementation Quality• Administrator Training
– Bi-annual fidelity checks– Emphasis on Standardization
• Scheduling and Real-Time Data-Based Decision Making– Behavioral Momentum– Zone of Proximal Development
• Chicken or the Egg?– Quality of intervention– Quality of assessment tool in sensitivity to change
Improving Outcomes: Certain Success?
• Reaching students with comprehensive efforts for intervention– Across academic and social settings– Increased opportunities to respond
• Monitoring progress frequently and with fidelity• Utilizing opportunities for data-based decision
making– Movement between Tier-level intervention– Maintaining Intervention long-enough to substantiate
potential changes.• Reflecting on screening data
– Are we changing the RTI triangle proportions?
Black Box of Individualization
Individualization To consider or
treat individually; particularize
Modified to suit wishes or needs of particular individual
Individualized Education Program/Individualized Family Service Plan Child’s present levels of
functioning/achievement How disability affects involvement in
general ed curriculum How disability affects child’s
participation in activities Goals that designed to meet child’s
needs resulting from disability
Systems ApproachIndividualization in context of
linked system, using CBA
Individualization and progress monitoring embedded in assessment and in daily intervention activities
Linked SystemSystem in which program
goals, assessment procedures, curricula/intervention, and program evaluation are related in content and in process
Program Goals & Philosophy
Linked System Approach
Collecting Information
Summarizing Information
Monitoring Curricular Approach• Weekly
• Quarterly• Annual
• Observations• Direct tests• Report
• Child goals• Family outcome
• Activity-based Intervention
Assessment IFSP
Evaluation Intervention
Individualization of Instruction4 Primary Steps
1. Get to know each child’s interests, needs, and abilities
2. Create opportunities for learning that build on children’s interests
Pretti-Frontczak & Bricker, 2004
Individualized Instruction3. Scaffold children’s learningthrough supportive interactions
4. Monitor children’s progresstoward achieving important goalsPretti-Frontczak & Bricker, 2004
Individualizing InstructionBe knowledgeable about
learners
Create learning opportunities embedded in daily routines, activities, experiences and draw into instructional interaction
Individualizing Instruction Implement a planned and structured
approach for curriculum content
Make thoughtful decisions about kind and amount of support for children
Monitor success of instruction to make sound decisions to support learning and development
Boat, Dinnebeil, & Baie.( 2010) Individualizing Instruction in Preschool Classrooms, Dimensions of Early Chilldhood (38)1, 3-10.
Activity-Based Intervention
Foundation of effective individualizationChild-initiated transactionsChildren’s goals/objectives embedded
in routine, planned or child-initiated activities
Logically occurring antecedents and consequences
Functional skills learned that can be generalized across settings and people
Advantages of CBA
o Measures functional skillso Comprehensive—across domainso Assessment/intervention in a
natural environmento Can be adapted and modified
Advantages of CBA
Items can be generalized across settings
Curriculum with steps to followItems translate into IFSP goals &
objectivesFamily involvementResearch base
AEPS Item
AEPS Goal
CRITERION Child performs the following play skills:
• Catches ball or similar object (3.1)• Kicks ball or similar object (3.2)• Throws ball or similar object at target (3.3)• Rolls ball at target (3.4)Child must score a 2 in Objective 3.1 through Objective 3.4 in order to score a 2 on Goal 3
Goal 3 Catches, kicks, throws, and rolls ball or similar object
PROGRESS MONITORING Collecting on-going data
Group and individual activitiesAssessment activity plans
Making sound decisions based on data
Progress Monitoring At the heart of accurate individualization and effective intervention
Scoring Option: 2
Child Consistently performs skill as specified in the criterion
Performs the skill independently Behavior is a functional part of the child’s
daily routine Child uses the skill across time, materials,
settings, and people
Progress Monitoring
•One child, one area
•One child, multiple areas
•Several children, one area
•Several children, multiple areas
Three to SixData Recording
Form
Group Data Collectionasks 4-wd P play walk hand reques
tJoseLydiaAmieNinaMateoArnie
Fine Motor AreaFine Motor skills are those that involve the movement and use of the hands. These skills include grasping, releasing, and using the index finger and thumb.
1. Does your child bring both hands to the middle of the body at the same time? (A1)
2. When playing with toys or objects, does your child bang the toys or objects together at midline when one toy or object is in EACH hand? (A2)
3. Does your child hold a hand-size object, such as a block or a small bowl, with either hand using the end of the thumb, the index and the second finger? The object is held by the fingers and is not resting in the palm. (A3)
4. Does your child pick up pea-size objects such as Cheerios or raisins with with either hand using the thumb and the index finger without resting the arm or hand on the table? (A4)
date
Link
Things were done very differently on the farm when I was your age, Kenny.
Individualizing InstructionProviding meaningful
learning opportunities to all children
On-goingmonitoring progress towards goals
Individualization Embedded in systems approach
Precise individualized goals Identified during CBA Embedded in on-going activities, routines Progress monitored weekly, monthly… Data based formative and summative
evaluation
Discussion How do we know when individualization
is being done well? What type of information would be
helpful to programs in answering that question? What types of indicators would be
helpful? Do different definitions of individualization
lead to different types of indicators? What types of research are necessary to
move us ahead?
Hope you’ve enjoyed the peek inside!