collecting and reporting follow-up data or what’s behind table 5?

Table 5 and Follow-upTable 5 and Follow-up

Collecting and Reporting Follow-up DataOr What’s Behind Table 5?

American Institutes for ResearchFebruary 2005

Table 5 and Follow-upTable 5 and Follow-up2

Table 5: OutcomesTable 5: Outcomes

Earned a High School Diploma or GED Entered post-secondary School Entered Employment Retained Employment


Table 5: OutcomesTable 5: Outcomes States submit six (6) data points for

each of the four outcomes:

11 2 2 3 3 4 4 5 5 66

Core Follow-up Outcome

Measures

# Participants With Main or Secondary

Goal

# Participants Included in

Survey

# Participants Responding to Survey or

Used for Data Matching

Response Rate or % Available for Match

# Participants Achieving Outcome

% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)


Table 5: Column (B)Table 5: Column (B)

All participants who had the outcome as a goal and who exited during the program year.


Measures


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)


Table 5: Column (C)Table 5: Column (C)

The number of participants who were surveyed to assess whether they had achieved the outcome.

Note: C should be equal to B unless sampling was used, in which case C would be less than B. For data matching C = [BLANK]


Measures


Goal

# Participants Included in Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)


Table 5: Column (D)Table 5: Column (D)

Of those surveyed or data matched, the number who responded or were used for data match.

Note: D should be equal to or less than B and C each.


Measures


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)


Table 5: Column (E)Table 5: Column (E)

Percent of respondents or participants successfully matched; column D divided by

column B.

E = D/B * 100Should be > 50%, but likely not 100%


Measures


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)


Table 5: Column (F)Table 5: Column (F)

Number of participants achieving outcome. Note: F must be equal to or less than D.


Measures


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)


Table 5: Column (G)Table 5: Column (G)

Percent achieving outcome. G= F/D * 100


Measures


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)


Table 5: ExampleTable 5: Example


Measures


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)

Entered Employment

1,000 1,000 800 80% 200 25%


Real Data – PY 2002–03 Real Data – PY 2002–03 ExamplesExamples

State


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)

State 1 1,056 0 972 0% 356 37%

What type of follow-up was

used?



State


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)

State 1 1,056 0 972 0% 356 37%

Why is this number less

than column B?



State


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)

State 1 1,056 0 972 0% 356 37%

What data should be

reported here?



State


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)

State 1 1,056 0 972 0% 356 37%

Is this percent computed correctly?


Exercise QuestionsExercise QuestionsFor states 2–6: What method of follow-upmethod of follow-up did each

state use? Which states, if any, filled in the

table correctlycorrectly? Which did not? Do you see any errorserrors in the table? Do you have confidenceconfidence in these

data? Why or why not?


Exercise ExampleExercise Example

State


Goal


Survey





% Achieving Outcome

(A) (B) (C) (D) (E) (F) (G)

State 1 1,056 0 972 0% 356 37%

State 2 3,325 NOT AVAIL. 0% 1,062 32%

State 3 8,644 0 0 0% 4,067 47%

State 4 31,363 0 31,363 100% 21,779 69%

State 5 9,633 3,853 1,156 30% 543 13%

State 6 224 224 164 73% 96 59%


Top Errors for Table 5 Top Errors for Table 5 Missing Data

Put “0” for # surveyed in Column (C) Put “0” for # responded or matched in

Column (D) Unreasonably high response rates

Many 100% Unreasonably high data matching Many 100% (DOL goal for matching = 90%)

Unreasonable matching of # who had goal to # who responded

Column (D) = Column (B)


Top Errors for Table 5 Top Errors for Table 5 (cont’d.)(cont’d.)

Incorrectly computed percentages: Response rates calculated as D/C instead

of D/B (B and C should be the same unless using random sampling, however)

Number achieving outcome computed as F/B instead of F/D


20

02

–03

2

00

2–0

3

Data

Colle

ctio

n

Data

Colle

ctio

n

Meth

ods

Meth

ods


Data Matching MethodologyData Matching Methodology Produces better quality data:

More student coverage Better data validity Better response rate Matches Title I—better for OMB common measures

28 states use it for employment measures—we need more states!

What are the reasons for not data matching? How can we promote use of this methodology?


What’s Behind the Data? What’s Behind the Data? State responsibility to ensure accuracy

and quality of follow-up measures Requires monitoring of local program

procedures Key steps:

Identify students—goal setting and tracking Uniform data collection Training Reporting


Percent of Students with Percent of Students with GoalsGoals Measure # with

GoalTotal #

Students% with Goal

Entered Employment—Average ALL states 6,904 52,548 10%*

State 1 417 69,284 1%

State 2 2,624 138,184 2%

State 3 1,609 70,893 2%

Entered post-secondary—Average ALL states 2,919 52,548 6%

State 4 159 33,062 0%

State 5 333 31,574 1%

State 6 14,523 565,311 3%

High School Completion—Average ALL states 7,207 52,548 20%**

State 7 7,192 114,008 6%

State 8 2,768 32,492 9%

State 9 33,118 387,710 9%

* 38% Unemployed (Table 6)** 18% Enrolled in ASE


Quality Control: SurveysQuality Control: Surveys Identify students

Set student goals appropriately By exit quarter for employment measures

Data Collection Quarterly collection for employment State survey instrument Sufficient resources (staff, time, funds) Sampling method (if applicable)


Quality Control: Surveys Quality Control: Surveys (cont’d.)(cont’d.)

Training Trained staff on survey procedures Improving response rate

Reporting Quarterly reporting Database to link outcomes to programs

and students


Quality Control: Data Quality Control: Data MatchingMatching Procedure for collecting and validating

Social Security numbers Report all with goal, prior to eliminating

invalid or missing numbers (Table 5, Col. D) Data have exit quarter and employment

records matched after correct exit quarter

Only students with goal matched and/or reported—should not have 100% with outcome


Discussion: Issues and Discussion: Issues and StrategiesStrategies

Improving reporting and collecting Resources Training on Monitoring

collecting and reporting follow-up data or what’s behind table 5?

Documents