Issues in Selecting Issues in Selecting Assessments for Measuring Assessments for Measuring
Outcomes for Young Outcomes for Young ChildrenChildren
Issues in Selecting Issues in Selecting Assessments for Measuring Assessments for Measuring
Outcomes for Young Outcomes for Young ChildrenChildren
Dale Walker & Kristie Pretti-Frontczak
ECO Center and Kent State University
Presentation at OSEP Early Childhood Conference Washington, DC, December 2005
2
Why Assessment?Why Assessment?
Gather information about skills and capabilities to make decisions about practice
To determine eligibility for services To determine if a child is benefiting from
services or if changes need to be made To measure development over time To document outcomes
3
Purpose of Assessments –Purpose of Assessments – It’s all It’s all about the question(s) you want to about the question(s) you want to answeranswer
Screening – Is there a suspected delay? Does the child need further assessment?
Eligibility determination – Is the child eligible for specialized services?
Program planning – What content should be taught? How should content be taught?
Progress monitoring – Are children making desired progress?
Program evaluation/Accountability – Is the program achieving it intended outcomes and/or meeting required outcomes?
4
Assessment Options Assessment Options
Norm-Referenced Criterion-Referenced Curriculum-Based Direct Observation Progress Monitoring Parent or Professional Report
Any combination of assessments…
5
Norm-Referenced Pros/ConsNorm-Referenced Pros/Cons Provides
information on development in relation to others
Already used for eligibility in many states
Diagnosis of developmental delay
Standardized procedures
Do not inform intervention Information removed from
context of child’s routines Usually not developed or
validated with children with disabilities
Do not meet many recommended practice standards
May be difficult to administer or require specialized training
6
Norm-Referenced Assessment Norm-Referenced Assessment TableTable
Table consists of a review of 18 norm-referenced assessments
Information regarding each assessment is provided including: Publisher information Areas of development assessed Test norms provided Scores produced Age range covered
http://fpsrv.dl.kent.edu/ecis/Web/Research/OSEP/NRT.pdf
7
Criterion-Referenced Criterion-Referenced Pros/ConsPros/Cons
Measure child’s performance of specific objectives
Direct link between assessment and intervention
Provides information on children’s strengths and emerging skills
Helps teams plan and meet individual children’s needs
Meets recommended assessment practice standards
Measures intra-child progress
May be used to measure program effectiveness
Requires agreement on criteria and standards
Criteria must be clear and appropriate
Usually does not show performance compared to other children
Do not have standard administration procedures
May not move child toward important goals
Scores may not reflect increasing proficiency toward outcomes
8
Curriculum-Based Pros/ConsCurriculum-Based Pros/Cons Provides link between
assessment and curriculum
Expectations based upon the curriculum and instruction
Can be used to plan intervention
Measure child’s current status on curriculum
Evaluate program effects Often team based Meets DEC and NAEYC
recommended standards Represents picture of the
child’s performance
May not have established reliability and validity
May not have procedures for comparing child to a normal distribution
Generally linked to a specific curriculum
Often composed of milestones that may not be in order of importance
9
Curriculum-Based Assessment Curriculum-Based Assessment Rating Rubric Rating Rubric
Evaluates the quality of CBAs for use with young children
Composed of 17 quality elements Used to guide teams in selecting
appropriate CBAs
http://fpsrv.dl.kent.edu/ecis/Web/Research/OSEP/CBArubric.pdf
10
Sample of CBA RubricSample of CBA Rubric
Element Unsatisfactory (0) Basic (1) Satisfactory (2) Excellent (3)
Adaptable for Special Needs
No consideration of special needs
Limited consideration of special needs through the assessment process and instrument does not allow for additional accommodations or modifications for special needs
Upfront considerations for special needs are not comprehensive, but assessment allows for some accommodations and/or modifications for special needs
Considers and provides specific strategies and procedures for accommodating and/or modifying the assessment for special needs
Aligns with Federal/State/Agency Standards and/or Outcomes
Does not align with Federal/State/Agency Standards and/or Outcomes
Aligns with less than half of the big ideas or concepts from Federal/State/Agency Standards and/or Outcomes
Aligns with more than half of the big ideas or concepts from Federal/State/Agency Standards and/or Outcomes
Aligns with a clear majority or all of the big ideas or concepts from Federal/State/Agency Standards and/or Outcomes
11
Progress Monitoring Pros/ConsProgress Monitoring Pros/Cons Used to monitor ongoing
progress toward important outcomes over time
Compare to children of similar ages over time
Repeatable measures for monitoring progress
Standardized administration Standards for technical
adequacy Efficient to administer May also be used as a
screening tool
Indicators of progress may be viewed as not being comprehensive
Not used for eligibility determination
May not provide specific skills to teach but indicators of important skills
12
Parent & Professional Report Parent & Professional Report Pros/ConsPros/Cons High social validity Provides diverse
perspective Important for
informing intervention, program, IFSP/IEP
Parents and professionals know the child, the environments in which they interact
Collaboration requires time and effort to establish
May not be reliable across time
Does not permit comparison across children
May include personal bias
13
Using Multiple Sources of Data or Using Multiple Sources of Data or Single Source to Measure Single Source to Measure Outcomes?Outcomes?
Pros and Cons Recommended practices Need to summarize information
generated Ways data can be used beyond
reporting OSEP outcomes
14
Using Data Beyond OSEP Using Data Beyond OSEP ReportingReporting Good assessment data can be used to….
Reveal patterns regarding children’s strengths and emerging skills
Develop functional and meaningful IFSPs/IEPs Inform program staff and families about strengths
and weaknesses Guide the development of intervention Monitor children’s progress to inform intervention
efforts Enhance collaboration Inform providers, programs, districts/parishes,
regions, and states regarding important trends
15
Ongoing work and Ongoing work and challenges…challenges…
Existing assessment tools were not developed to measure the three outcomes
ECO’s response: “Cross-walking” or mapping frequently used assessments to the outcomes
Work with publishers and state staff to develop guidance for how to use assessment results to generate OSEP-requested data
16
Work with Publishers and Work with Publishers and DevelopersDevelopers
Finalizing crosswalks Alignment with OSEP outcomes How to determine what is “typical” performance Age-anchored benchmarks to measures How scores can be summarized using the ECO
Summary Form Possible recalibration of scores in a way that
maintains the integrity of different assessments Pilot studies with GSEG and interested states Data summary report forms that assist users
with alignment of information from assessment to OSEP outcomes
17
Example of Developing a Example of Developing a Validated CrosswalkValidated Crosswalk
First align On the face of it – which items appear to
align/match which outcomes? Second validate
Do experts agree? Check for internal consistency
Third examine the sensitivity of the assessment in measuring child change
http://fpsrv.dl.kent.edu/ecis/Web/Research/OSEP/Steps.pdf
18
Example of Interpreting the Example of Interpreting the evidenceevidence
Standard scores Residual Change Scores Goal Attainment Scaling Number of objectives
achieved/Percent objectives achieved Rate of Growth Item Response Theory Proportional Change Index Stoplight model
19
Interpreting the AEPS for Interpreting the AEPS for AccountabilityAccountability First administration (near entry)
Is the child above or below a cut off score?
If above – considered to be developing typically If below – development is suspect
Which level of the AEPS was administered?
Child is less than three and Level I is used Child is less then three and Level II is used Child is older than three and Level I is used Child is older than three and Level II is used
20
Interpreting the AEPS for Interpreting the AEPS for AccountabilityAccountability
Second administration (near exit) Use cut off scores again Examine which level was used Look for
changes in area percent scores changes in scoring notes changes in which level was administered
21
Sample Cutoff ScoresSample Cutoff Scores
Level Age Intervals (months) Cutoff Score
Birth to three 25-30 50
31-36 60
Three to six 37-42 20
43-48 30
49-54 40
22
Questions? Questions?
For More Information see: For More Information see: http://www.the-ECO-center.orghttp://www.the-ECO-center.org
For More Information see: For More Information see: http://www.the-ECO-center.orghttp://www.the-ECO-center.org