session #65 draft results of quality assurance program data from award year 2006-07 david rhodes and...
TRANSCRIPT
Session #65
Draft Results of Quality Assurance Program Data from
Award Year 2006-07
David Rhodes and Anne Tuccillo
2
Goals• Share draft results of program-
wide analysis of 2006-07 data
• Illustrate additional ways to analyze ISIR data
3
Background• Quality Assurance (QA) Program
• Participants granted regulatory flexibility to decide which ISIR data they verify
• ISIR Analysis Tool
4
The 2006-07 Analysis• 146 Quality Assurance Program
institutions• Each school drew a random
sample of at least 350 applicants• Each school verified the ISIR
information for the entire random sample of applicants
• We analyzed 68,077 records
5
Previous data collections
2005-06 2004-05
Population
ISIR Records subject to school verification
Random sample of all applicants
Number of schools
140 133
6
See a pattern?• Alternating focus
– Random sample of all applicants– Institutionally verified applicants
• This year (2007-08)– All QA Program institutions– ISIRS subject to institutional
verification
7
Is analysis over time appropriate?
• For individual schools, yes with care
• For program-wide analyses the answer “should” be no, but…..
8
When Looking Across Years Keep in Mind:
• The two different populations• Changes to institutional verification
criteria between years• Other differences between award
years• Percents and averages not counts
9
Why we “shouldn’t” • QA schools supplying data differ
slightly from year to year (146 ≠140 ≠133)
• Unmeasured differences in institutional verification across years
10
Key areas of analysis• Description of population• Critical ISIR fields• How changes affect aid eligibility• Improper payments in the Pell
Grant program
11
Characteristics of Aid Applicants at QA Program Schools 2006-07
62%
44%
40%
58%
0% 25% 50% 75% 100%
Dependent
Selected for School Verification
Selected for CPS Verification
Eligible for Pell
12
Dependent and independent record counts over time
42,455
94,866
24,310
25,622
42,297
14,785
0
25,000
50,000
75,000
100,000
125,000
150,000
2006-07 2005-06 2004-05
Dependent Independent
13
Dependent and independent percentages over time
62% 69% 62%
38% 31% 38%
0%
25%
50%
75%
100%
2006-07 2005-06 2004-05
Dependent Independent
14
Percentage of dependent applicants with changes to the most commonly changed ISIR Fields: 2006-07
22%
23%
26%
29%
29%
0% 25% 50% 75% 100%
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total fromWorksheet B
Parent AGI
15
Percentage of independent applicants with changes to the most commonly changed ISIR Fields: 2006-07
11%
12%
14%
18%
19%
0% 25% 50% 75% 100%
Student Total from Worksheet C
Student Tax Filing Status
Student Total from Worksheet B
Student AGI
Student Income from Work
16
Changes to critical ISIR fields among dependent students over time
29% 29%33%27%30% 32%
0%
25%
50%
75%
100%
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06 2004-05
17
Changes to critical fields among independent students over time
18% 14%25%
15%19% 14%
0%
25%
50%
75%
100%
Student AGI Student Total fromWorksheet B
2006-07 2005-06 2004-05
18
Dependent records: percent of ISIR fields experiencing an EFC change - 2006-07
87%
89%
88%
90%
89%
0% 25% 50% 75% 100%
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
19
Independent records: Percent of ISIR fields experiencing an EFC change - 2006-07
71%
62%
68%
68%
66%
0% 25% 50% 75% 100%
Student Total fromWorksheet C
Student Tax Filing Status
Student Total fromWorksheet B
Student AGI
Student Income from Work
20
Comparison of changes to EFC among dependent students with a change to the indicated ISIR field
89% 90%88% 90%91% 93%
0%
25%
50%
75%
100%
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06 2004-05
21
Comparison of changes to EFC among independent students with a change to the indicated ISIR field
66% 68%67% 66%71% 72%
0%
25%
50%
75%
100%
Student AGI Student Total fromWorksheet B
2006-07 2005-06 2004-05
22
Percentage of dependent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07
43%
42%
46%
38%
49%
0% 25% 50% 75% 100%
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
23
Percentage of independent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07
35%
31%
34%
38%
36%
0% 25% 50% 75% 100%
Student Total fromWorksheet C
Student Tax Filing Status
Student Total fromWorksheet B
Student AGI
Student Income from Work
24
Comparison of change to a Pell Grant among dependent students with a change to the indicated field
49%38%
58% 54%42%
35%
0%
25%
50%
75%
100%
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06 2004-05
25
Comparison of change to Pell Grants among independent students with a change to the indicated field
38% 34%48% 45%
37% 34%
0%
25%
50%
75%
100%
Student AGI Student Total fromWorksheet B
2006-07 2005-06 2004-05
26
Average EFC change among dependent records with a change to the indicated ISIR field: 2006-07
1,039
913
645
819
648
0 400 800 1,200
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
27
Average EFC change among independent records with a change to the indicated ISIR field: 2006-07
470
490
387
186
100
0 500 1,000 1,500
Student Total from Worksheet C
Student Tax Filing Status
Student Total from Worksheet B
Student AGI
Student I ncome from Work
28
Average EFC change among dependent students with a change to the indicated field over time
648819
980
1,214
0
500
1,000
1,500
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06
29
Average EFC change among independent students with a change to the indicated field over time
103
387371
591
0
500
1,000
1,500
Student AGI Student Total fromWorksheet B
2006-07 2005-06
30
Average Pell change for dependent records with a change to the indicated ISIR field: 2006-07
-$344
-$305
-$257
-$314
-$270
-$600 -$400 -$200 $0
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
31
Average Pell change for independent records with a change to the indicated ISIR field: 2006-07
-$368
-$347
-$257
-$122
-$102
-$600 -$400 -$200 $0
Student Total from Worksheet C
Student Tax Filing Status
Student Total from Worksheet B
Student AGI
Student I ncome from Work
32
Average Pell Grant change for dependent records with a change to the indicated field over time
-$270-$314
-$469-$504
-$600
-$400
-$200
$0
Parent AGIParent Total from
Worksheet B
2006-07 2005-06
33
Average Pell Grant change for independent records with a change to the indicated field over time
-$122
-$257
-$344
-$459
-$600
-$400
-$200
$0
Student AGIStudent Total from
Worksheet B
2006-07 2005-06
34
Potential improper payments in Pell Grants prior to verification at QA Schools: 2006-07
6.7% potential under-payments
9.1% potential over-payments
84.2% Pell Grant dollars not at risk
prior to verification
35
Improper Payments in Pell Grants before and after school verification
-9.1%
6.7%
-2.4%-0.7%
2.9%
-3.6%
-10.0%
-5.0%
0.0%
5.0%
10.0%
Potentialunder-awards
Perc
en
t o
f P
ell D
ollars
Before verification After school verification
Potenial over-awards
Net potenial over-awards
36
Improper Payments in Pell Grants before and after CPS verification
-9.1%
6.7%
-2.4%
2.4%4.1%
-1.7%
-10.0%
-5.0%
0.0%
5.0%
10.0%
Potentialunder-awards
Perc
en
t o
f P
ell D
ollars
Before verification After CPS verification
Potenial over-awards
Net potenial under-awards
37
Lessons Learned• While verification is crucial in some
cases, many ISIR records do not experience a meaningful change from their initial transaction value
• QA school verification procedures target larger EFC and Pell changes
• QA school verification reduces improper payment risks in Pell
38
What Do the Findings Mean for Schools and FSA?
Given that relatively few ISIR records experience a change that affects eligibility for need-based aid, verification efforts should strive to focus on the records that matter.
39
Two Ways to Focus• Look for the records that matter• Look for the records that don’t
40
Suggested Additional Analysis
Look for ways to reduce the number of school verified records that experience no or only trivial changes to aid eligibility after verification
41
Additional Analysis• ISIR Analysis Tool reports• Exporting data from the Tool
42
ISIR Analysis Tool• Key Filters
–EFC change < 400 & EFC change > -400
–EFC change > 400 –EFC change < -400–Institutional verification criteria
43
ISIR Analysis Tool• Key Reports
–Sample summary–Field Increment (EFC or AGI)
• Drill down to refine understanding
44
New Analytic “Recipe”
See handout
45
Exporting Data
See handout
Institutional Profiles
Comparison of Program-wide and school specific
data
47
2006-07 Institutional Profile
Example of a 2006-2007 Institutional Profile
48
Profile Differs from Year to Year
• 2005-06 provided data of improper payments PREVENTED by institutional verification
• 2006-07 provides data on estimated levels of POTENTIAL improper payments in a schools applicant population
49
Contact Information
We appreciate your feedback and comments. We can be reached at: