hawai’i january, 2008

Hawai’iHawai’iJanuary, 2008January, 2008

Progress toward Measuring Progress toward Measuring Goals in Early Intervention: Goals in Early Intervention:

What’s New from What What’s New from What CountsCounts

Kathy HebbelerKathy HebbelerECO at SRI InternationalECO at SRI International

Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2

ObjectivesObjectives

Review why data are being collected Describe national trends Identify and address challenges to

good data Discuss some preliminary data from

Hawai’i


Why are we doing this?


Keeping our eye on the prize:

High quality services for children and

families that will lead to good outcomes.


High Quality Data on OutcomesHigh Quality Data on Outcomes

Data are a piece of a system that helps to achieve overarching goals for children and families

Data yield Findings that can be interpreted as

having a particular meaning that should lead to specific actions to improve the system.

Prof’l Development•Preservice•Inservice

System for Producing Good Child and System for Producing Good Child and Family OutcomesFamily Outcomes

Good Federal policies and programs

Good State policies and programs

High quality services and supports for children 0-5 and their families

Good outcomes for children and families

Good Local policies and programs

Adequate funding

Strong Leadership


The Vision: Using Data as a Tool for The Vision: Using Data as a Tool for Program ImprovementProgram Improvement

Hawai’i will have quality data available on an ongoing basis about multiple components of the system Goals for children and families Services provided Personnel (types, qualifications, etc.) Etc.


Driving Force for Data on Child Goals Driving Force for Data on Child Goals Comes from the Federal LevelComes from the Federal Level

Government Performance and Results Government Performance and Results

Act (GPRA)Act (GPRA)

Program Assessment Rating Tool (PART)Program Assessment Rating Tool (PART)

Individuals with Disabilities Education Individuals with Disabilities Education

Act (IDEA)Act (IDEA)


Requires goals and indicators be

established for IDEA Indicators and data collection further

along for school age population than

for EC Previously, for early childhood data

had been collected on:

• Number of children served (Part C)

• Settings (both Part C and 619)

Government Performance and Results Government Performance and Results Act (GPRA) passed in 1993Act (GPRA) passed in 1993


130 programs examined in 2002; 50%

programs had no performance data

Programs looking at inputs, not results

Part C and Section 619

No long-term child outcome goals or data

Need to develop a strategy to collect

annual performance data in a timely

manner

OSEP: PART evaluation results (2002)OSEP: PART evaluation results (2002)


Federal Funding for Early InterventionFederal Funding for Early Intervention

Total U.S. Hawai’i

2004 $444,362,700 $2,177,738

2005 $440,808,096 $2,160,317

2006 $436,399,920 $2,138,714


SEC. 616. <<NOTE: 20 USC 1416.>> MONITORING, TECHNICAL ASSISTANCE, AND ENFORCEMENT. ``(a) Federal and State Monitoring.-…..…..``(2) Focused monitoring.--The primary focus of Federal and State monitoring activities described in paragraph (1) shall be on-- ``(A) improving educational results and functional outcomes for all children with disabilities;

Individuals with Disabilities Education ActIndividuals with Disabilities Education Act


Where are we now:

Federal reporting requirements


OSEP Reporting Requirements: OSEP Reporting Requirements: the Goals the Goals

Positive social emotional skills (including positive social relationships)

Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])

Use of appropriate behaviors to meet their needs


OSEP Reporting CategoriesOSEP Reporting Categories

Percentage of children who:

a. Did not improve functioning

b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers

c. Improved functioning to a level nearer to same-aged peers but did not reach it

d. Improved functioning to reach a level comparable to same-aged peers

e. Maintained functioning at a level comparable to same-aged peers

3 outcomes x 5 “measures” = 15 numbers


Reporting ScheduleReporting Schedule

Reported February 2007 Entry information: Age

expected? Yes, No One time requirement Reported for children

entering between July 1, 2005, and June 30, 2006

Due February 2008 Data in reporting

categories at exit for all children who have been in the program for at least 6 months

Must be reported for the year beginning July 1, 2006

Repeat with next year’s data in 2009, etc.


AlsoAlso

States are required to Make public data reported to OSEP Analyze state data by program (i.e.,

compute a through e for each program) Make public the data by program

Concept of Develomental Trajectory

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Typical development Atypical or delayed development

Illustration of 5 Possible Develomental Trajectories

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Maintained functioning comparable to age peersAchieved functioning comparable to age peersMoved nearer functioning comparable to age peersMade progress; no change in trajectoryDid not make progress

Change over 3 time points

0

5

10

15

20

25

30

35

40

45

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Goal Ratings and Developmental Trajectories

Age in Months


Point of clarificationPoint of clarification

“Why are we comparing children with delays and disabilities to typically

developing children?”


Point of clarificationPoint of clarification

Process is NOT about comparing groups of children – it IS about asking how close children are to being able to do what is expected at their age

Early learning guidelines Kindergarten and access to the

general curriculum

Source: National Early Intervention Longitudinal Study

Disability and Special Education Status in Kindergarten of Former EI Participants

32%

10%

58%No IEP, No Disability

No IEP, Disability

IEP


Where are we now:

State decisions and activities


To respond to federal

reporting requirements

To meet provider/teacher, local and/or state need for outcome

information and

to respond to federal reporting

requirements

Purpose

WHY?


State approaches State approaches Most states have embraced outcomes

measurement and are collecting outcomes data for their own purposes.

Many states are building bigger systems than needed to produce the federal data.

Go to www.the-eco-center.org for more information about what other states are doing


How are states collecting child How are states collecting child outcomes/goal data? outcomes/goal data?

Possible state approaches to collection of child data Child Outcomes Summary Form (COSF)

[= Early Intervention Child Goals Summary Form in HI]

Publisher’s online assessment system Single assessment statewide Other approaches


State approaches to measurement State approaches to measurement for Part C child outcomesfor Part C child outcomes

40 states using the ECO Child Outcomes 40 states using the ECO Child Outcomes Summary Form (COSF)Summary Form (COSF)

8 states using 1 assessment tool 8 states using 1 assessment tool statewidestatewide

3 states using on-line assessment 3 states using on-line assessment systems with the capacity to report OSEP systems with the capacity to report OSEP data reportsdata reports

5 states using other unique approaches5 states using other unique approaches


Variations across states in CGSF Variations across states in CGSF implementationimplementation

Some states started early (HI); some did not start until mid to late 2007

Some states completing at IFSP; others at a separate meeting

Some states including parents in the discussion; some are not


Where states are nowWhere states are now

First data on 5 categories due to OSEP February 3

Many states do not have data on many children yet

Many states focusing on improving the process of collecting the data


What do we know so far:

Positive impacts of the goals rating process


Positive impacts reported by statesPositive impacts reported by states

Increases focus on functional outcomes on IFSPs

Easier to write functional outcomes on IFSP

Facilitates communication with parents


Benefits of discussing the 3 goalsBenefits of discussing the 3 goals

“Requires us to talk & think in terms of functional behaviors, not test items

Incorporates the parents as active and knowledgeable participants

Looks at all settings and situations Bridges the gap between assessment

tools and real life.”From presentation by Sandi Harrington, Norfolk, VA Infant Development Program, at the OSEP EC Meeting, December 2007


BenefitsBenefits

“Is more meaningful to families Prepares the family for setting IFSP

outcomes – thinking about the skills they want their child to have to function in their daily family life

Guides us towards discipline-free contextualized goals.”

From presentation by Sandi Harrington, Norfolk, VA Infant Development Program, at the OSEP EC Meeting, December 2007


Benefits of INCLUDING familiesBenefits of INCLUDING families

“Determining child progress requires we use the family’s expertise and knowledge of their child across setting and situations

Our discussion becomes more inclusive with the family as an equal source of information for assessment purposes.”

From presentation by Sandi Harrington, Norfolk Infant Development Program, at the OSEP EC Meeting, December 2007



“One of the biggest shifts in practice, for many systems, was the move to compare their children in Part C to their same age peers.

Looking to children in the frame of same age peers allows us to have authentic, honest discussions with families about their child’s strengths and needs.”




“We need to be comfortable with reporting strengths AND areas of delay, while being family friendly.”



What do we know so far:

Challenges to getting good information


Need for good dataNeed for good data

Encompasses all three levels: federal, state, local

Depends on how well local programs are implementing procedures


What we are learning nationallyWhat we are learning nationally

The process of training for child outcomes data collection has uncovered other areas of significant need related to professional development.


Essential Knowledge for Completing Essential Knowledge for Completing the Child Goals Summary Form the Child Goals Summary Form

Between them, team members must:

1. Know about the child’s functioning across settings and situations

2. Understand age-expected child development

3. Understand the content of the three child outcomes

4. Know how to use the rating scale

5. Understand age expectations for child functioning within the child’s culture


Important pointImportant point

It is not necessary that all team members be knowledgeable in all 5 areas

Especially, no expectation that parents understand the rating scale or typical child development

But the professionals have to!


Providers need to know more about:Providers need to know more about:

Assessment How to gather assessment data to reflect

functioning across settings and situations, especially how to gather child functioning information from families

Understanding the results of the assessment Sharing assessment results sensitively and

honestly with families


Providers need to know more about:Providers need to know more about:

Functional outcomes What are they? How do they differ from outcomes organized

around domains? What do they mean for how professionals from

different disciplines operate as a team? Typical child development

What are the functional expectations for children at different ages with regard to each of the 3 goal statements?


Is this process too subjective to produce Is this process too subjective to produce good data?good data?

Best practices in assessment requires looking at multiples sources of information

Assessment as a tool vs. assessment as a process

Research on judgment-based assessment indicates it is as good or better than traditional assessment


What is informed opinion?What is informed opinion?

Clinical judgment (informed opinion) – knowledgeable perceptions of caregivers and professionals about the elusive and subtle capabilities of children in different settings


Clinical judgment provides good data Clinical judgment provides good data when….when….

Operational definition of child characteristics to be judged

Structured format for quantifying characteristics

Information from multiple setting and individuals

Training in methods that structure and quantify characteristics

Decision making based on consensus From Bagnato, Smith-Jones, Matesa & McKeating-

Esterle, 2006


Ratings clarificationRatings clarification

Highest category (Completely, 7) = Child functions in an age appropriate manner across settings and situations

Next highest (6) – Child functions in an age appropriate manner but there is a significant concern about some aspect of the child’s functioning



Somewhat (5) = Child shows a MIX of age appropriate and not age appropriate behaviors across settings and situations

Between emerging and somewhat (4) – Child shows some age appropriate behavior but rarely



Emerging (3) – No age appropriate behavior yet. Shows immediate foundational skills in some to all settings and situations

Between not yet and emerging (2) – No age appropriate behavior yet. Rarely uses immediate foundational skills (but does show some).



Not yet (1) – No age appropriate behavior yet. No immediate foundational skills yet.


Should the rating be whatever parents Should the rating be whatever parents want?want?

No, the rating is a team consensus Need to think about what and how

parents are being involved in this process

Have the assessment results been thoroughly explained?

Like so much in EI, the rating requires a partnership


Thinking about relating to adults, relating to other children, and (for those older than 18 months) following rules related to groups or interacting with others.

Δ How does the child relate to his/her parent(s)?

Δ How does the child relate to other relatives or extended family and close family friends (e.g., grandparents, aunts, extended kin, etc.)? Do these interactions with people differ depending on the setting the child is in with these people? Δ How does the child interact with familiar caregivers (e.g., child care providers, babysitters)?

Δ How does the child relate to strangers? At first? After a while? In different settings and using different approaches?

Δ How does the child interact with/respond to people in community settings (e.g., park, library, church, grocery store, with neighbors on walks, at the bus stop, in restaurants, at playgroups or outings, etc.)?

….

ECO Discussion Prompts: Child has positive social relationships (see ECO Tools)


Obtaining good dataObtaining good data

Threats to good data Local providers do not understand the

procedures Local providers do not follow the procedures And others…..

Process requires good training procedures Initial Ongoing


Many steps to ensuring quality dataMany steps to ensuring quality data

Before

Information sharing

Good data collection/Training

Good data system

During

Ongoing supervision

Feedback

Refresher training

AfterMonitoring

Validity analyses


Initial Data from Hawai’i


These data are very, very preliminary.

EIS First IFSP Ratings (n=1774)

0%

20%

40%

60%

80%

1 2 3 4 5 6 7

Child Goals Ratings

Pe

rce

nta

ge

of

Ch

ildre

n

SE Skills

Acq Use

Appr Action


EIS Average Ratings at Initial IFSPEIS Average Ratings at Initial IFSP

Social-Emotional

Skills

Acquire and Use

Knowledge and Skills

Appropriate Action

EIS

(N=1774)5.9 5.3 5.1

Healthy Start First IFSP Ratings (n =1311)

0%

20%

40%

60%

80%

1 2 3 4 5 6 7

Child Goals Ratings

Pe

rce

nta

ge

of

Ch

ildre

n

SE Skills

Acq Use

Appr Action


HS Average Ratings at Initial IFSPHS Average Ratings at Initial IFSP

Social-Emotional

Skills

Acquire and Use

Appropriate Action

HS

(N=1311) 6.4 6.4 6.4

Goal 1: SE Skills for HS and EIS at First IFSP

0%

20%

40%

60%

80%

1 2 3 4 5 6 7

Child Goals Rating at First IFSP

Pe

rce

nta

ge

of

Ch

ildre

n

Healthy Start (n=1311)

EIS (n=1774)

Goal 2: Acquire and Use Knowledge and Skills for HS and EIS at First IFSP

0%

20%

40%

60%

80%

1 2 3 4 5 6 7


Pe

rce

nta

ge

of

Ch

ildre

n


EIS (n=1777)

Goal 3: Appropriate Action to Meet Needs for HS and EIS at First IFSP

0%

20%

40%

60%

80%

1 2 3 4 5 6 7


Pe

rce

nta

ge

of

Ch

ildre

n


EIS (n=1774)

HS Rating for Social Skills at Initial IFSP by Island

0.00

0.20

0.40

0.60

0.80

1.00

1 2 3 4 5 6 7

Rating

%

Hawaii (n=344) Kauai (n=78) Maui (n=151) Oahu (n=736)

EIS Knowledge and Skills at Entry and Exit (N=115)

0

10

20

30

40

50

60

1 2 3 4 5 6 7

# o

f C

hil

dre

n

Entry Exit

HS S oc ial S kills at E ntry and E x it (N=60)

0

20

40

60

80

1 2 3 4 5 6 7

R a ting

# o

f C

hil

dre

n

Initial

E x it

EIS Entry and Exit Rating for Knowledge and Skills (N=115)

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7

Entry rating

# o

f ch

ild

ren

3 4 5 6 7

Healthy S tart E ntry and E xit R ating for Appropriate Ac tion (N=81)

0102030405060

1 2 3 4 5 6 7E ntry R a ting s

# of

chi

ldre

n

1 2 3 4 5 6 7

EIS: 6 Month Review Ratings by Entry Rating for Appropriate Action (N=691)

0

20

40

60

80

100

1 2 3 4 5 6 7

Entry

# o

f ch

ild

ren

1 2 3 4 5 6 7

Appropriate Action

Review Rating

Initial_3B 1 2 3 4 5 6 7 total

1 1 4 2 7

2 1 1 5 6 9 3 1 26

3 2 15 14 27 19 6 83

4 4 4 21 39 28 12 108

5 1 12 14 71 86 48 232

6 1 3 21 48 63 136

7 2 18 23 56 99

Review Total 2 13 38 60 185 207 186 691

Another State: OSEP Categories for Another State: OSEP Categories for Goal 1 by Exit StatusGoal 1 by Exit Status

OSEP Categ

ALL Exite

d

a 1%

b 5%

c 39%

d 44%

e 12%

N= 893

Another State: OSEP Categories for Another State: OSEP Categories for Outcome 1 by Exit StatusOutcome 1 by Exit Status

OSEP Cate

gExited

Before 3Exited at 3

ALL Exite

d

a 1% 1% 1%

b 2% 6% 5%

c 19% 45% 39%

d 57% 39% 44%

e 22% 9% 12%

N= 231 662 893

Another State: OSEP Categories for Another State: OSEP Categories for Outcome 1 by Exit StatusOutcome 1 by Exit Status

OSEP Cate

gComplete

d IFSPExited

Before 3Exited at 3

ALL Exite

d

a 0% 1% 1% 1%

b 1% 2% 6% 5%

c 12% 19% 45% 39%

d 65% 57% 39% 44%

e 22% 22% 9% 12%

N= 156 231 662 893


Questions to askQuestions to ask

Do the data make sense? Am I surprised? Do I believe the data?

Believe some of the data? All of the data?

If the data are reasonable (or when they become reasonable), what might they tell us?


ValidityValidity

Validity refers to the use of the information Does evidence and theory support the

interpretation of the data for the proposed use?

Or Are you justified in reaching the conclusion

you are reaching based on the data? Standards for Educational and Psychological Testing (1999) by

American Educational Research Association, American Psychological Association, National Council on Measurement in Education


How will/might these data be used?How will/might these data be used?

Federal level Overall funding decisions (accountability) Resource allocation (e.g., what kind of TA to fund?) Decisions about effectiveness of program in individual

states State level

Program effectiveness?? Program improvement??

Local level Program improvement??


What has ECO learned after 4+

years?


ECO Message: Strong commitmentECO Message: Strong commitment

States are committed to building good systems to collect data on how children are progressing Variations in how data are being collected Variations in how states plan to use the

information

Common thread: Widespread recogniton of the importance of the data


ECO Message: Need to build state ECO Message: Need to build state capacitycapacity

Implement oversight procedures around data quality

Examine data for validity Analyze and interpret data for program

improvement Develop messages for policy-makers,

public, media, families from the data


ECO Message: Need to build ECO Message: Need to build provider capacityprovider capacity

Assessment Functional outcomes Typical child development


ECO Message: Need for better ECO Message: Need for better early childhood assessment toolsearly childhood assessment tools

Designed around the 3 functional outcomes for all children

Designed to capture child functioning in a variety of setting and situations

Designed to be used in accountability and program evaluation

Current tools are antiquated: Need to incorporate latest research,

recommended practices, psychometrics


ECO Message: Need for more ECO Message: Need for more resources to ensure quality dataresources to ensure quality data

National resources to support and coordinate across states: Training needs Analysis and use of data

Support for states to continue to develop and validate their systems

Investment in research to examine how outcomes data collection being carried out (impact on quality, local practice, etc.)

Investment in new assessment tools


Change can be difficult….


For more information….


http://hawaii.gov/health/family-child-health/eis/whatcounts.html


www.the-eco-center.org

hawai’i january, 2008

Documents