session #65 draft results of quality assurance program data from award year 2006-07 david rhodes and...

Session #65

Draft Results of Quality Assurance Program Data from

Award Year 2006-07

David Rhodes and Anne Tuccillo

2

Goals• Share draft results of program-

wide analysis of 2006-07 data

• Illustrate additional ways to analyze ISIR data

3

Background• Quality Assurance (QA) Program

• Participants granted regulatory flexibility to decide which ISIR data they verify

• ISIR Analysis Tool

4

The 2006-07 Analysis• 146 Quality Assurance Program

institutions• Each school drew a random

sample of at least 350 applicants• Each school verified the ISIR

information for the entire random sample of applicants

• We analyzed 68,077 records

5

Previous data collections

2005-06 2004-05

Population

ISIR Records subject to school verification

Random sample of all applicants

Number of schools

140 133

6

See a pattern?• Alternating focus

– Random sample of all applicants– Institutionally verified applicants

• This year (2007-08)– All QA Program institutions– ISIRS subject to institutional

verification

7

Is analysis over time appropriate?

• For individual schools, yes with care

• For program-wide analyses the answer “should” be no, but…..

8

When Looking Across Years Keep in Mind:

• The two different populations• Changes to institutional verification

criteria between years• Other differences between award

years• Percents and averages not counts

9

Why we “shouldn’t” • QA schools supplying data differ

slightly from year to year (146 ≠140 ≠133)

• Unmeasured differences in institutional verification across years

10

Key areas of analysis• Description of population• Critical ISIR fields• How changes affect aid eligibility• Improper payments in the Pell

Grant program

11

Characteristics of Aid Applicants at QA Program Schools 2006-07

62%

44%

40%

58%

0% 25% 50% 75% 100%

Dependent

Selected for School Verification

Selected for CPS Verification

Eligible for Pell

12

Dependent and independent record counts over time

42,455

94,866

24,310

25,622

42,297

14,785

0

25,000

50,000

75,000

100,000

125,000

150,000

2006-07 2005-06 2004-05

Dependent Independent

13

Dependent and independent percentages over time

62% 69% 62%

38% 31% 38%

0%

25%

50%

75%

100%

2006-07 2005-06 2004-05

Dependent Independent

14

Percentage of dependent applicants with changes to the most commonly changed ISIR Fields: 2006-07

22%

23%

26%

29%

29%

0% 25% 50% 75% 100%

Parent Tax Filing Status

Father I ncome from Work

Mother I ncome from Work

Parent Total fromWorksheet B

Parent AGI

15

Percentage of independent applicants with changes to the most commonly changed ISIR Fields: 2006-07

11%

12%

14%

18%

19%

0% 25% 50% 75% 100%

Student Total from Worksheet C

Student Tax Filing Status

Student Total from Worksheet B

Student AGI

Student Income from Work

16

Changes to critical ISIR fields among dependent students over time

29% 29%33%27%30% 32%

0%

25%

50%

75%

100%

Parent AGI Parent Total fromWorksheet B

2006-07 2005-06 2004-05

17

Changes to critical fields among independent students over time

18% 14%25%

15%19% 14%

0%

25%

50%

75%

100%

Student AGI Student Total fromWorksheet B

2006-07 2005-06 2004-05

18

Dependent records: percent of ISIR fields experiencing an EFC change - 2006-07

87%

89%

88%

90%

89%

0% 25% 50% 75% 100%

Parent Tax Filing Status

Father I ncome from Work

Mother I ncome from Work

Parent Total from Worksheet B

Parent AGI

19

Independent records: Percent of ISIR fields experiencing an EFC change - 2006-07

71%

62%

68%

68%

66%

0% 25% 50% 75% 100%

Student Total fromWorksheet C

Student Tax Filing Status

Student Total fromWorksheet B

Student AGI

Student Income from Work

20

Comparison of changes to EFC among dependent students with a change to the indicated ISIR field

89% 90%88% 90%91% 93%

0%

25%

50%

75%

100%

Parent AGI Parent Total fromWorksheet B

2006-07 2005-06 2004-05

21

Comparison of changes to EFC among independent students with a change to the indicated ISIR field

66% 68%67% 66%71% 72%

0%

25%

50%

75%

100%

Student AGI Student Total fromWorksheet B

2006-07 2005-06 2004-05

22

Percentage of dependent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07

43%

42%

46%

38%

49%

0% 25% 50% 75% 100%

Parent Tax Filing Status

Father I ncome from Work

Mother I ncome from Work

Parent Total from Worksheet B

Parent AGI

23

Percentage of independent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07

35%

31%

34%

38%

36%

0% 25% 50% 75% 100%

Student Total fromWorksheet C

Student Tax Filing Status

Student Total fromWorksheet B

Student AGI

Student Income from Work

24

Comparison of change to a Pell Grant among dependent students with a change to the indicated field

49%38%

58% 54%42%

35%

0%

25%

50%

75%

100%

Parent AGI Parent Total fromWorksheet B

2006-07 2005-06 2004-05

25

Comparison of change to Pell Grants among independent students with a change to the indicated field

38% 34%48% 45%

37% 34%

0%

25%

50%

75%

100%

Student AGI Student Total fromWorksheet B

2006-07 2005-06 2004-05

26

Average EFC change among dependent records with a change to the indicated ISIR field: 2006-07

1,039

913

645

819

648

0 400 800 1,200

Parent Tax Filing Status

Father I ncome from Work

Mother I ncome from Work

Parent Total from Worksheet B

Parent AGI

27

Average EFC change among independent records with a change to the indicated ISIR field: 2006-07

470

490

387

186

100

0 500 1,000 1,500

Student Total from Worksheet C

Student Tax Filing Status

Student Total from Worksheet B

Student AGI

Student I ncome from Work

28

Average EFC change among dependent students with a change to the indicated field over time

648819

980

1,214

0

500

1,000

1,500

Parent AGI Parent Total fromWorksheet B

2006-07 2005-06

29

Average EFC change among independent students with a change to the indicated field over time

103

387371

591

0

500

1,000

1,500

Student AGI Student Total fromWorksheet B

2006-07 2005-06

30

Average Pell change for dependent records with a change to the indicated ISIR field: 2006-07

-$344

-$305

-$257

-$314

-$270

-$600 -$400 -$200 $0

Parent Tax Filing Status

Father I ncome from Work

Mother I ncome from Work

Parent Total from Worksheet B

Parent AGI

31

Average Pell change for independent records with a change to the indicated ISIR field: 2006-07

-$368

-$347

-$257

-$122

-$102

-$600 -$400 -$200 $0

Student Total from Worksheet C

Student Tax Filing Status

Student Total from Worksheet B

Student AGI

Student I ncome from Work

32

Average Pell Grant change for dependent records with a change to the indicated field over time

-$270-$314

-$469-$504

-$600

-$400

-$200

$0

Parent AGIParent Total from

Worksheet B

2006-07 2005-06

33

Average Pell Grant change for independent records with a change to the indicated field over time

-$122

-$257

-$344

-$459

-$600

-$400

-$200

$0

Student AGIStudent Total from

Worksheet B

2006-07 2005-06

34

Potential improper payments in Pell Grants prior to verification at QA Schools: 2006-07

6.7% potential under-payments

9.1% potential over-payments

84.2% Pell Grant dollars not at risk

prior to verification

35

Improper Payments in Pell Grants before and after school verification

-9.1%

6.7%

-2.4%-0.7%

2.9%

-3.6%

-10.0%

-5.0%

0.0%

5.0%

10.0%

Potentialunder-awards

Perc

en

t o

f P

ell D

ollars

Before verification After school verification

Potenial over-awards

Net potenial over-awards

36

Improper Payments in Pell Grants before and after CPS verification

-9.1%

6.7%

-2.4%

2.4%4.1%

-1.7%

-10.0%

-5.0%

0.0%

5.0%

10.0%

Potentialunder-awards

Perc

en

t o

f P

ell D

ollars

Before verification After CPS verification

Potenial over-awards

Net potenial under-awards

37

Lessons Learned• While verification is crucial in some

cases, many ISIR records do not experience a meaningful change from their initial transaction value

• QA school verification procedures target larger EFC and Pell changes

• QA school verification reduces improper payment risks in Pell

38

What Do the Findings Mean for Schools and FSA?

Given that relatively few ISIR records experience a change that affects eligibility for need-based aid, verification efforts should strive to focus on the records that matter.

39

Two Ways to Focus• Look for the records that matter• Look for the records that don’t

40

Suggested Additional Analysis

Look for ways to reduce the number of school verified records that experience no or only trivial changes to aid eligibility after verification

41

Additional Analysis• ISIR Analysis Tool reports• Exporting data from the Tool

42

ISIR Analysis Tool• Key Filters

–EFC change < 400 & EFC change > -400

–EFC change > 400 –EFC change < -400–Institutional verification criteria

43

ISIR Analysis Tool• Key Reports

–Sample summary–Field Increment (EFC or AGI)

• Drill down to refine understanding

44

New Analytic “Recipe”

See handout

45

Exporting Data

See handout

Institutional Profiles

Comparison of Program-wide and school specific

data

47

2006-07 Institutional Profile

Example of a 2006-2007 Institutional Profile

48

Profile Differs from Year to Year

• 2005-06 provided data of improper payments PREVENTED by institutional verification

• 2006-07 provides data on estimated levels of POTENTIAL improper payments in a schools applicant population

49

Contact Information

We appreciate your feedback and comments. We can be reached at:

Email: david.rhodes@ed.gov anne.tuccillo@ed.gov