improving postsecondary data and data systems...

IMPROVING POSTSECONDARY DATA AND DATA SYSTEMS TO SUPPORT FEDERAL AND STATE POLICYMAKING

© 2015 Bill & Melinda Gates Foundation

June 2016

Jennifer EngleSenior Program Officer, DataPostsecondary Success

WHY ARE WE WORKING TO IMPROVEPOSTSECONDARY DATA?

Data and Information: Support a national data infrastructure that enables consistent collection andreporting of the key performance metrics for all students in all institutions that are essential forpromoting the change needed to reform the higher education system.

Finance and Financial Aid: Advance postsecondary finance and financial aid approaches aligned tothe goal of increasing participation, progression and completion for low-income and at-risk students.

College Readiness: Replace remedial education models that have been proven not to work withevidence-backed approaches.

Innovation and Scale: Support the development and oversight of programs that shorten time andcredits to credential and better fit with career and family obligations.

© Bill & Melinda Gates Foundation |

BMGF POSTSECONDARY SUCCESS ADVOCACY PRIORITIES

© Bill & Melinda Gates Foundation | 4

TRILLION DOLLAR QUESTION

DOES(ACCESS x COMPLETION) / COST

= VALUE

FOR STUDENTS & SOCIETY?

Source: Adapted from Institute for Higher Education Policy

In an era of escalating college costs, what do students and the public invest in postsecondary education and what do they get in return?

Due to an incomplete and disconnected postsecondary data infrastructure, we have onlypartial or no answers to basic questions such as:

How many non-traditional students attend college and do they successfully completecredentials? (This includes low-income, adult, and first-generation students as well asstudents who transfer and/or attend college part-time.)

Do students who do not graduate transfer to other colleges and earn degrees, or do theydrop out altogether?

How much debt are students accumulating in college, and can they repay their loans?

Are students obtaining employment in their field after college, and what do they earn?

How much are students learning in college, and how are they contributing to society?


CAN WE ADEQUATELY ANSWER QUESTIONS ABOUT POSTSECONDARY OUTCOMES AND VALUE? NO.

Source: Adapted from Institute for Higher Education Policy


DO BETTER DATA REALLY LEAD TO BETTER OUTCOMES? YES.

DATAWhen low-income

STUDENTS get targeted and timely information

about college prices and completion rates…

When COLLEGES use “real-time” data to identify

students at risk of not completing their

degrees…

When POLICYMAKERSuse data to set

meaningful performance targets for colleges…

ACTIONThey choose to attend colleges that offer more academic and financial

resources…

They can deploy “just in time” resources to help students get back on

track…

They can enact systemwide policies and reforms to remove major barriers to completion…

RESULTS

And they earn degrees at higher rates.

And boost retention and graduation rates by

double digits.

And measurably increase the number of students

earning degrees.

HOW ARE WE WORKING TO IMPROVEPOSTSECONDARY DATA?

There is clear evidence that better data leadto better outcomes in higher education.

There are two major barriers to obtainingbetter data at scale: data quality (e.g.metrics) and data infrastructure (e.g.systems).

Our work to date includes developing arobust metrics framework synthesizingadvances in data quality in the field.

Wide adoption of the framework requiresmajor improvements in data systems.Options range from incremental to all-in, butideal state to achieve our objectives is anational student-level data system.

PS DATA STRATEGY THEORY OF CHANGE

DATA CREATION

METRICS ADOPTION

SYSTEM QUALITY

DATA USE

Continuous Feedback Loop

Create & Test

Adopt for Limited Use

Collect for Wide Use


PS METRICS FRAMEWORK


DOZENS OF DATA INITIATIVES YIELDED NEW & IMPROVED METRICS DEMONSTRATING DEMAND & USE CASES


REVIEW OF DATA INITIATIVES, DASHBOARDS, FUNDING FORMULAS REVEALED FIELD CONVERGENCE AROUND KEY METRICS

Source: Gates Foundation (2016). Answering the Call.

http://postsecondary.gatesfoundation.org/wp-content/uploads/2016/02/AnsweringtheCall.pdf


STATES PARTICIPATING IN MANY OF THESE INITIATIVES, KY IN AT LEAST A DOZEN

Source: Institute for Higher Education Policy,http://www.ihep.org/postsecdata/mapping-data-landscape/resources/states-crosswalk

1. Aspen Prize2. Achieving the Dream3. Access to Success (past)4. Complete College America5. Consortium for Student Retention Data Exchange6. National Collegiate Athletic Association7. National Community College Benchmarking Project8. Student Achievement Measure9. Southern Regional Education Board10. Voluntary Framework of Accountability11. Voluntary Institutional Measures Project12. Voluntary Standards of Accountability

http://www.ihep.org/postsecdata/mapping-data-landscape/resources/states-crosswalk


DEVELOPING & DISSEMINATING A KEY PERFORMANCE METRICS FRAMEWORK FOR WIDE SCALE FIELD ADOPTION

ACCESS PROGRESSION COMPLETION COST POST-COLLEGE OUTCOMES

PERFORMANCE

▪ Enrollment ▪ Credit Accumulation▪ Credit Completion▪ Gateway Course Completion▪ Program of Study Selection ▪ Retention▪ Persistence

▪ Transfer▪ Graduation▪ Success▪ Completers

▪ Net Price▪ Unmet Need▪ Cumulative Debt

▪ Employment▪ Earnings▪ Loan Repayment▪ Graduate Education ▪ Learning Outcomes

EFFICIENCY▪ Expenditures per Student

▪ Cost of Uncompleted Credits ▪ Gateway Completion Costs▪ Change in Revenue from Change in Retention

▪ Time/Credits to Credential▪ Costs of Excess Credits▪ Completions per Student

▪ Student Share of Cost ▪ Expenditures per Completion

▪ Earnings Threshold

EQUITY

▪ Enrollment by at least Preparation, Income, Age, Race/Ethnicity

▪ Progression Performance at least by Preparation, Income, Age, Race/Ethnicity

▪Completion Performance and Efficiency by at least Preparation, Income, Age, Race/Ethnicity

▪ Net Price and UnmetNeed by at least Income▪ Debt by at least Income, Age, Race/Ethnicity, Completion Status

▪ Outcomes Performance and Efficiency by at least Income, Age, Race/Ethnicity, Completion Status

Key Student Characteristics

▪ Enrollment Status ▪ Economic Status▪ Attendance Pattern ▪ Race/Ethnicity▪ Degree-Seeking Status ▪ Age▪ Program of Study ▪ Gender▪ Academic Preparation ▪ First-Generation Status

Key Institutional Characteristics

▪ Sector ▪ Selectivity▪ Level ▪ Diversity▪ Degree/Program Mix ▪ MSI Status▪ Size ▪ Nontraditional Populations▪ Resources ▪ Modality

Source: Gates Foundation (2016). Answering the Call.



METRICS FRAMEWORK DESIGN PRINCIPLES

Counting All Students

Most initiatives began collecting data precisely because they could not track the outcomes of non-traditional students– such as part-time, underprepared, transfer, and low-income students – in existing national datasets like IPEDS. Assuch, the framework definitions reflect this progress in the field, and pushes the field further forward withrecommendations such as using 12-month instead of fall cohorts to capture the more than 1/3 of students who startafter the fall term, particularly in the community college and for-profit sectors.

Counting All Outcomes

Many initiatives track a more robust set of student outcomes, including transfer and completion at subsequentinstitutions. The framework reflects this progress in the field, but distinguishes between success rates (graduation orupward transfer from initial institution) and persistence rates (graduation, transfer, or still enrolled at initial orsubsequent institution) to encourage colleges and universities to use student persistence rates to set stretch goals forimproving their institutional success rates. Research shows that students who complete their programs are muchmore likely to do so at their initial institution.

Costs Count While most initiatives include many of the access, progression, and completion metrics in the framework, fewerinitiatives include cost and efficiency metrics. Although available data remain limited to construct these metrics, it wasimportant to include them in version 1 of the framework to signal the need to consider how resources can be moreefficiently allocated to improve student outcomes in this era of scarce public resources.

ConsideringPost-College

Outcomes

While most institutions cannot yet fully access data about their students’ post-college outcomes (as these arecollected and reported by state and federal agencies), it was important to signal to institutions that they should usecurrently available data, appropriately contextualized, to understand whether students are earning credentials thatimprove their economic and life chances.


STATES’ COLLECTION OF METRICS FRAMEWORKState Agency AL AK AZ AR CA

CCCCA

CSUCA UC CO CT FL GA HI ID IL IN KS KY … Total

Enrollment ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 57

Credit Accumulation ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 52

Credit Completion Ratio ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 51

Gateway Course Completion ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 32

Retention/Persistence Rate ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 55

Transfer Rate ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 52

Graduation Rate ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 41

Completers/Completions per Student ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 54

Net Price ♦ ♦ ♦ 8

Cumulative Debt ♦ ♦ ♦ ♦ ♦ 12

Employment /Earnings ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 38

Loan Repayment ♦ 4

Time to Credential ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 33

Credits to Credential ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 34

Total 6 13 5 10 7 7 10 10 11 5 12 13 11 9 12 7 11

Source: Armstrong, J. & Zaback, K. (2016). Assessing and improving state postsecondary data systems.

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/state_postsecondary_data_systems.pdf


STATES’ COLLECTION OF FRAMEWORK DISAGGREGATES

State Agency AL AK AZ AR CA CCC

CA CSU

CA UC CO CT FL

BOGFL

DOE GA HI ID IL IN KS KY … Total

Age ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 58

Gender ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 57

Race/Ethnicity ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 57

Economic Status (Pell) ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 37

Enrollment Status (First-Time, Transfer) ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 45

Degree-Seeking Status ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 56

Attendance Intensity (Full-, Part-Time) ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 53

Program/Major ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 55

Academic Preparation (Placement Scores) ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 30

Source: Armstrong, J. & Zaback, K. (2016). Assessing and improving state postsecondary data systems. Armstrong, J. & Whitfield, C. (2016). Strong Foundations.


http://www.sheeo.org/sites/default/files/publications/SHEEO_StrongFoundations2016_FINAL.pdf

“Answering the Call: Institutions and States Lead the Way Toward Better Measures of PostsecondaryPerformance” released by BMGF in February 2016 supported by a multi-media digital outreach campaign atwww.higheredfacts.org, including blogs, articles, videos, and social media on BMGF and other channelssuch as the Washington Post and Money.

“Toward Convergence: A Technical Guide for the Postsecondary Metrics Framework” released by IHEP inMay 2016 in multiple digital formats, including as an interactive guidebook on their resource-rich websitesupporting their Postsecondary Data Collaborative community.


WHERE TO FIND METRICS FRAMEWORK RESOURCES


http://www.higheredfacts.org/

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/ihep_toward_convergence_med.pdf

http://www.ihep.org/postsecdata/mapping-data-landscape/metrics-framework-technical-guide

http://www.ihep.org/postsecdata

Aligning measurement to the framework across the portfolio, ladderinginstitutional metrics with intervention, capacity, network, and strategy metricsand incorporating into investments as appropriate.

Framework in use with 100 institutional partners, supporting implementationand evaluation of their work and providing technical specifications and usecases for broader field collection of metrics.

Advocacy for framework adoption underway, including use by major datapartners and vendors (1-3 years), and federal and state campaigns to includemetrics in existing or improved data systems (3-5 years).


HOW ARE WE USING THE METRICS FRAMEWORK?


EXAMPLES OF METRICS USED IN STATE OBF FORMULAS

Access • Enrollment by student population

Progression & Completion

• Credentials conferred by student population and degree field

• Credentials conferred per student• Graduation and transfer rates by student

population• Course completion (including remedial) and

credit accumulation

Cost & Efficiency • Expenditures per completion• Changes in tuition and fees• Expenditures per student

Post-College Outcomes

• Employment, earnings, or further education• Scores on standardized learning assessments

Source: Rorison, J., Voight, M., and Engle, J. (2016). Employing Postsecondary Data for Effective State Finance Policymaking. Davies, L. (2014). State ‘Shared Responsibility’ Policies for Improved Outcomes: Lessons Learned

https://www.luminafoundation.org/files/resources/employing-postsecondary-data-1.pdf

http://hcmstrategists.com/wp-content/uploads/2014/04/HCM-State-Shared-Responsibility-RADD-2.0.pdf


SIMPLE “VALUE-ADDED” OBF MODEL FROM CCA

Source: Complete College America (2012). Value-Added Funding.

https://dl.dropboxusercontent.com/u/28697036/Value-Added%20Funding%20Do%20This.pdf

PS DATA INFRASTRUCTURE


STRENGTHENING POSTSECONDARY DATA INFRASTRUCTURE TO COLLECT AND REPORT QUALITY DATA FOR ALL STUDENTS IN ALL INSTITUTIONS

Current State Ideal State


STATE OF KY POSTSECONDARY DATA SYSTEMS

Source: SHEEO (2016). The State of State Postsecondary Data Systems.http://www.sheeo.org/sites/default/files/publications/SHEEO_StrongFoundations2016_FINAL.pdf

29 State Postsecondary Agencies Have Access to K12 and Workforce Data, Including Kentucky

Institutions Included in KY Postsecondary Data System:

• Two-Year Public Institutions• Four-Year Public Institutions• Private, Non-Profit Institutions




PS DATA INFRASTRUCTURE WORKING GROUP PAPERS IDENTIFIED IMPROVEMENTS NEEDED ACROSS PUBLIC AND PRIVATE OPTIONS…

1. Assess and improve institutional IR capacity: Association for Institutional Research

2. Assess and improve state longitudinal data systems: State Higher Education Executive Officers

3. Foster state-to-state data exchanges: Western Interchange Commission for Higher Education

4. Improve IPEDS: Institute for Higher Education Policy

5. Leverage and/or expand Federal Student Aid data: American Institutes for Research

6. Create a federal student unit record data system (SURDS): New America

7. Leverage/link to federal and/or state wage data: Workforce Data Quality Campaign

8. Leverage/link to other federal data: Center for American Progress

9. Leverage/link to NSC data: National Student Clearinghouse

10. Best practices in data privacy/security: Educause

Papers published May 2016 by the Institute for Higher Education Policy

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/institutional_research_capacity.pdf


http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/fostering_state-to-state_data_exchanges.pdf

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/putting_the_integrated_back_into_ipeds.pdf

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/using_and_improving_fsa_data_systems.pdf

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/building_a_student-level_data_system.pdf

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/leveraging_employment_data_0.pdf

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/linking_federal_data_systems.pdf

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/national_student_clearinghouse.pdf

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/information_security_and_privacy.pdf


IHEP convening an advisory group to work with a design firm to create technical specifications andrequirements for three options for a national postsecondary student-level data system: a federal system, afederated state exchange, and a private system.

The advisory group includes select authors from the infrastructure group series as well as respectedtechnical experts with relevant experience in education and beyond hailing from academe, state andfederal government, private industry, and foundations.

IHEP scoped the project, which starts this summer, as follows:• Three-phase process: 60-day scoping phase, 60-day working phase, 60-day refinement phase• Four key deliverables:

- Systems Architecture- Implementation Roadmap- Organizational Capacity and Operational Requirements- Estimates of Implementation and On-Going Costs and Time

AND NATIONAL POSTSECONDARY STUDENT-LEVEL DATA SYSTEM DESIGN PROJECT TO DETAIL TECHNICAL IMPLEMENTATION PLANS

© Bill & Melinda Gates Foundation |26

www.ihep.org/postsecdata

Conduct and commission timely and thoroughresearch and analysis on postsecondary dataand infrastructure (e.g. definitions, design).

Convene and support a coalition network of 35+organizations that collectively advance apostsecondary data advocacy agenda.

Build field convergence through advocacy andcommunications for proposals to improvepostsecondary data and systems.

Provide nonpartisan, expert technical assistanceto policymakers and staff to implement actions toimprove postsecondary data and infrastructure.

Solicit and share stories of data champions anduse cases with demonstrated impact on studentsuccess, leveraging the PostsecData community.

http://www.ihep.org/postsecdata

improving postsecondary data and data systems...

Documents