Supporting States in Building a Child Outcomes Measurement

System

Prepared for the ECO Advisers MeetingFebruary 2011

Two Frameworks

• Child Outcomes Measurement System

• Family Experiences and Outcomes Measurement System

2Early Childhood Outcomes Center

Child Outcomes Measurement System

• The set of components a state needs to have in place to make full use of child outcomes data.

• NOT just a data system or a data collection method.

3Early Childhood Outcomes Center

Purpose of the Framework

• Provide a common language for ECO and other TA providers to use in discussing COMSs with states.

• Provide a organizing structure of categorizing resources and state examples related to implementation of a COMS.

• Serve as the organizing structure for the state self assessment

4Early Childhood Outcomes Center

Framework and Self-Assessment

• FRAMEWORK– Set of components and quality indicators– Provides the structure for the self-assessment

• SELF-ASSESSMENT – Scale that provides criteria for levels of

implementation within each quality indicator– Rating assigned based on level of

implementation within each indicator

5Early Childhood Outcomes Center

Process for Framework Development

• Built off what we had learned from ECO work with states

• Literature review• Repeated discussion and

review internally and with 7 Partner States

6Early Childhood Outcomes Center

Framework Partner States

State Part C 619

California X

Colorado X X

Delaware X X

Maine X X

Minnesota X X

New York X

Ohio X X

7Early Childhood Outcomes Center

8

COMS Framework Components

Data Collection and Trans-

mission

Analysis Reporting Using Data

Purpose

Evaluation

Cross-system Coordination

Quality Indicator

• Provides additional detail as to what constitutes quality implementation of the component.

• 18 quality indicators across the 7 components

9Early Childhood Outcomes Center

10Early Childhood Outcomes Center

Purpose

Data Collection and Transmission

Analysis

Reporting

Using Data

Evaluation

Cross-System Coordination

Components Quality Indicators

Purpose

1. State has articulated purpose(s) of COMS.

Data Collection and Transmission

2. Data collection procedures are carried out efficiently and effectively.

3. Providers, supervisors, and others involved in data collection have the required knowledge, skills, and commitment.

4. State's method for entering, transmitting, and storing data is effective and efficient.

Analysis

5. State identifies accountability and program improvement questions related to child outcomes.

6. Local programs identify accountability and program improvement questions related to child outcomes.

7. State agency analyzes data in a timely manner.

8. Local programs analyze data in a timely manner.

9. State agency ensures completeness and accuracy of data.

Reporting

10. State agency interprets, reports, and communicates information related to child outcomes.

11. Local programs interpret, report, and communicate information related to child outcomes.

Using Data

12. State agency makes regular use of information on child outcomes to improve programs.

13. Local programs makes regular use of information on child outcomes to improve programs.

Evaluation14. State evaluates its COMS regularly.

Cross-system Coordination

15. Part C and 619 coordinate child outcomes measurement.

16. Child outcomes measurement is integrated across early childhood (EC) programs statewide.

17. Child outcomes measurement is aligned with state’s early learning guidelines/standards.

18. State has a longitudinal data system to link child outcomes data from EC program participation to K–12 data.

Elements

a. State has…b. State has…c. State agency..d. Representative..e. State agency…f. State ……g. State provides…h. State has..

3 Quality Indicators for Data Collection and Transmission

• Data collection procedures are carried out efficiently and effectively.

• Providers, supervisors, and others involved in data collection have the required knowledge, skills, and commitment.

• State's method for entering, transmitting, and storing data is effective and efficient.

11Early Childhood Outcomes Center

The Self Assessment

12Early Childhood Outcomes Center

Purpose of the Self Assessment

• Assist states in setting priorities for improving their COMS

• Provide information to assist states in advocating for resources for systems development

13

• Provide guidance to states on what constitutes a high quality child outcomes measurement system.

Early Childhood Outcomes Center

• Each QI has multiple elements.

• Evidence for the extent of implementation for each element is provided.

• Each element is rated as• NY = Not Yet or Don’t know• IP = In Process, or• IF = Fully Implemented

• The QI is given a rating based on the ratings of the elements.

14Early Childhood Outcomes Center

• Live link from the element • Describes the element• Describes what “fully

implemented looks like”• Provides examples of what

“In process” might look like• Provides examples of how

states are addressing the element

• Provides additional resources related to the element

15Early Childhood Outcomes Center

Back-up for Each Element

The Scale for the Quality Indicators

Early Childhood Outcomes Center 16

Implementation of Elements

Quality Indicator

Score

All elements are fully implemented 7

Nearly all elements are fully implemented and the rest are in process

6

Most of the elements are fully implemented and the rest are in process.

5

At least one element is fully implemented and the rest are in process

4

All of the elements are in process 3

Some of the elements are in process 2

None of the elements are yet in process 1

17Early Childhood Outcomes Center

• Information can be entered online

• Will be stored for state by ECO

• Score from the QI page will automatically transfer to the overall score page

• Online scoring will show multiple marking points (i.e., change over time)

18Early Childhood Outcomes Center

Web-based Scoring?

Technology Support

• Online capability – Store the evidence (text)– Transfer QI rating to the score sheet and generate

bars– Also, allows ECO to use the aggregate state data to

track where states are overall in building COMS

• Downloadable version with many of the same features (Except ECO access to data)

19Early Childhood Outcomes Center

Working with the Self Assessment

20Early Childhood Outcomes Center

Possible Process

1. State identifies the highest priority QI(s).

2. Stakeholders convened– Priority QIs reviewed and rated– Plan developed to address elements not yet

fully implemented.– Progress reviewed with stakeholders at

regular intervals.

3. State identifies next set of QI(s), etc.

21Early Childhood Outcomes Center

Our questions for you

1. General impressions?– Content: Comprehensive? Appropriate?– Format/layout: User friendly? Understandable?– Length?– Scoring: Reasonable guideline?

2. Investment of resources– Continue development?

• Pick new partners?– Pursue the web version? Heavily? Lightly?– Pursue the downloadable Word version?– How important is it that ECO track where states are in building

their systems?

22Early Childhood Outcomes Center

For more information

• For updates to the framework and the self-assessment and resources to support the quality indicators:

www.the-eco-center.org

23Early Childhood Outcomes Center