data driven decision-making in south dakotawere conducted by sheeo agency staff in seven states...

DATA DRIVEN DECISION-MAKING IN SOUTH DAKOTA

JOHN ARMSTRONG

EFFECTIVE USE OF STATE DATA SYSTEMS

SD

© 2016 by the State Higher Education Executive Officers Association (SHEEO) 2

This paper is based on research funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the author(s) and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.


TABLE OF CONTENTS

INTRODUCTION ...................................................................................................................................... 3

THE IMPORTANCE OF DATA FOR THE EXECUTIVE .............................................................................. 4

THE IMPETUS FOR SDBOR DASHBOARDS ........................................................................................... 6

REACTIONS TO SDBOR DASHBOARDS ................................................................................................11

DISPELLING MYTHS WITH DATA ......................................................................................................... 12

CONCLUSION......................................................................................................................................... 13


INTRODUCTIONAnalysis of student-level data to inform policy and promote student success is a core function of executive higher education agencies. Postsecondary data systems have expanded their collection of data elements for use by policymakers, institutional staff, and the general public. State coordinating and governing boards use these data systems for strategic planning, to allocate funding, establish performance metrics, evaluate academic programs, and inform students and their families. The State Higher Education Executive Officers association (SHEEO), as part of a project funded by the Bill & Melinda Gates Foundation (BMGF), surveyed state coordinating and governing boards on their collection and use of postsecondary student-level data. Following this, SHEEO identified seven states whose survey responses indicated an exemplary use of data in specific subject areas. In-person interviews were conducted by SHEEO agency staff in seven states selected for follow-up.

In 2015, SHEEO visited the South Dakota Board of Regents (SDBOR) to discuss the presentation of data to external stakeholders, its unique focus on Native American students, and the relationship between SDBOR leadership and its research staff. The Board of Regents governs the system of public higher education in the state. It has constitutional authority over South Dakota’s six public universities as well as two special schools 1 (see Figure 1). While each university in South Dakota has its own institutional research department, SDBOR staff members serve as administrators for campus data systems and have access to all of the underlying data. Regents Information Systems (RIS) contain unit record information on college readiness, enrollment, completions, degree trends, transfer outcomes, employment outcomes of graduates, and a variety of course-level data, among many other data elements.2

FIGURE 1: LOCATION OF SOUTH DAKOTA BOARD OF REGENTS CAMPUSES AND ADMINISTRATION

SDBOR has generated an annual Fact Book using the information institutions have submitted to RIS since 1985.3 In recent years, SDBOR has also generated a series of publicly available dashboards using RIS data and nationally available data sets. Many of South Dakota’s responses to SHEEO’s survey on data systems referenced the dashboards, which are continually updated to reflect the most recent RIS data. The dashboards produce a number of positive outcomes for SDBOR and serve a wide variety of higher education audiences. They result from close collaboration between agency leadership and research staff.

1. Special schools include the South Dakota School for the Deaf and the South Dakota School for the Blind and Visually Impaired.

2. John Armstrong and Christina Whitfield, The State of State Postsecondary Data Systems: Strong Foundations 2016. Boulder CO. State Higher Education Executive Officers association. 51 (May 2016) Available at http://www.sheeo.org/resources/publications/strong-foundations-2016

3. Available here: https://www.sdbor.edu/mediapubs/factbook/Pages/default.aspx

http://www.sheeo.org/resources/publications/strong-foundations-2016

http://www.sheeo.org/resources/publications/strong-foundations-2016

https://www.sdbor.edu/mediapubs/factbook/Pages/default.aspx


THE IMPORTANCE OF DATA FOR THE EXECUTIVE

As higher education agencies collect data for planning, program review and finance, the need for agency leadership to thoroughly understand the data and its implications is paramount. Business intelligence methods are an increasingly important tool for higher education executives. Higher education executives can look to public management information system (PMIS) theory from the business community for ideas on improving the use of data to drive decisions. PMIS best practices include collaboration among government agencies, and the intentional recruitment and retention of information system staff.4 SDBOR’s executive leadership has shared data linkages with other agencies, used publicly available data where available, and prioritized the hiring of talented research analysts to inform its decision-making process. In this respect, they have incorporated PMIS theory recommendations, which is a public-sector application of business intelligence techniques. Business intelligence has an increasingly important role in executive higher education agencies.

Throughout the course of the interview with SDBOR staff, it was clear that communication among the research staff of the Board of Regents is closely aligned with the vision of the senior leadership team. At the time of the visit, SDBOR was led by Dr. Jack Warner, currently an associate professor of higher education at Johnson Wales University. Warner served as the state higher education executive officer (SHEEO) of the Rhode Island Board of Governors for Higher Education and as vice chancellor of the Massachusetts Board of Higher Education before leading the SDBOR as executive director and chief executive officer.

In Warner’s remarks, he noted that he has invested in institutional research throughout his leadership tenures. He thinks it is vital for agency leadership to understand, as deeply as they can, the data on students attending higher education institutions in their state. This desire for understanding prompted his investment in research and analysis staff for the agency. After assembling the necessary information, leaders can choose which key metrics to focus on in order to move the needle and improve student success. This practice is consistent with literature on effective policy analysis.5

All coordinating and governing boards across the country have access to basic enrollment statistics and, in most cases, demographic information about their students. Warner advocates for starting with these commonly available measures to paint a picture of the overall higher education landscape in the state before deciding which metrics and areas of inquiry will require further research and what questions are needed to generate new and critical information. “It’s up to the leader to craft the questions.” From there, the agency must translate the questions into usable data that informs policy, guides decision-makers and influences an agenda that improves student outcomes across the state.

To that end, the staff who researched the questions Warner asked worked closely with him and other key stakeholders throughout their research process. Members of the research team are on SDBOR’s senior staff and regularly meet with agency leadership to report on progress, adjust lines of inquiry, and translate information into decision points.

4. Barry Bozeman and Stuart Bretschneider, “Public Management Information Systems: Theory and Prescription,” Public Administration Review 46. pp 475-487 (1986).

5. Eugene Bardach, A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem Solving. Third Edition. CQ Press, Washington, D.C. (2009).


One example of a report that followed this data to question policy is SDBOR’s report on Native American students attending South Dakota universities, entitled, Like two different worlds.6 Based on the basic demographic data that was available to Warner and SDBOR when he started, he developed the question of how to improve outcomes for these students. More information was needed to ascertain the nature of the gap in access between Native American and white students. He directed the research team to explore the issue further before coming up with a series of policy recommendations aimed at improving the student experience for Native Americans.

The report identified a series of barriers that Native American students face when pursuing a postsecondary degree or certificate. These barriers include stark poverty, a lack of mentorship, fear of culture shock and leaving family, and a lack of quality information about postsecondary education, among others. To address these barriers, the report recommended improved outreach to Native American high school students—particularly those living on reservations—that is family-centered and contains a mentoring approach in the admission and scholarship process for postsecondary education. It also recommends expanding scholarship and financial aid opportunities for these students and improving the cultural footprint on South Dakota campuses to improve retention.

Warner described the report as providing SDBOR with a deeper understanding of the problem and influencing SDBOR’s strategy for addressing it. As a result of this report and the grant that funded it, SDBOR was able to put two staff in the field who were specifically focused on these students and placed an academic advisor on each of the campuses that work with low-income students and provide summer bridge programs for Native American and other students. The continued collaboration between Warner and the research team influenced the report’s process, its recommendations, and the actions implemented to address the problem after the report’s release.

The timeline for using data to inform research, which then informs policies and agendas, is a model for state coordinating and governing boards. Additionally, it is critical for research staff to understand agency leadership’s vision for higher education, and which questions are of critical importance. Likewise, agency leadership benefits from investing in talented research staff who are able to provide critical information on state higher education. Regular communication between these two groups is essential for successfully implementing a data driven decision-making agenda.

6. Available here: https://www.sdbor.edu/mediapubs/documents/americanindiancollegegoingstudybor1213.pdf

https://www.sdbor.edu/mediapubs/documents/americanindiancollegegoingstudybor1213.pdf


THE IMPETUS FOR SDBOR DASHBOARDS

The method of using information to articulate strategic priorities and drive institutional change, enabled by collaboration among leadership and research staff, is further assisted by having transparent, detailed information for a wide variety of audiences. In addition to interviewing Warner, SHEEO staff spoke with Dr. Paul Turman, the system vice president for academic affairs at SDBOR and Dr. Daniel Palmer, the assistant vice president for institutional research and planning. Palmer first had the idea for the dashboards after continually releasing reports and white papers only to be asked additional questions and given requests for analysis by specific subpopulations. By putting the data in the hands of the stakeholders, structured in a way they can easily access the information and explore drill-down topics, many potential questions by Board of Regents members can be answered on demand.

For Turman, this is contrasted with previous reporting processes at SDBOR. Previous reports submitted to regents were static tables, charts and figures that were continually updated, with additional detail being processed by follow-up reports in response to requests from the regents. The dashboards allow users to manipulate elements based on their specific questions, and explore the information at the high school district-level (for data related to college readiness) and the postsecondary institution level to see differences not only for subpopulations but across secondary and postsecondary schools.

Both Turman and Palmer described the process of having many anticipated questions and follow-ups available to be explored in the dashboards as an efficiency gain for their research department. When, in the past, Board of Regents members requested additional breakouts or district-level data, it often meant that research staff would have to gather additional information and create a new report in order to address their questions. Many of these inquiries are now able to be answered in a centralized location.

Palmer described the process of taking the unified data from all SDBOR campuses and putting it into an online dashboard as a much faster process than anticipated. The amount of time spent synthesizing the RIS information and developing layouts for dashboards has offset the time that was previously spent doing additional follow-ups based on regents’ or legislators’ questions. SDBOR has also had a commitment to make the RIS data transparent and useable for a wide variety of audiences. There was a need for institutional stakeholders, state policymakers and the general public to have better access to information on postsecondary enrollment, completions, course outcomes and information about South Dakota higher education. The dashboards enable this access and transparency.


EXAMPLES OF SDBOR DASHBOARDSEach of the sixteen dashboards available on SDBOR’s website7 has a research question that informs the user about what the underlying data can answer. Examples of lead-in questions to the dashboards include: “How well do first-year students from South Dakota high schools perform during their first year of college; what are the college-going rates of South Dakota’s high school graduates, and where do they enroll; and how does South Dakota stack up against other states with respect to the cost of a public university education?” The high school transition dashboard, for example, contains school district-level enrollment figures for each South Dakota university as well as ACT score averages and remedial course enrollment rates (see Figures 2 and 3).

FIGURE 2: HIGH SCHOOL TRANSITION DASHBOARD

7. www.sdbor.edu/dashboards

SDBOR High School Transition Report

0% 20% 40% 60% 80%

BH

SU

AberdeenCentral High

School

All Other HighSchools

DSU


School


NSU


School


SDSM

T


School


SDSU


School


USD


School


9.0%(242)

7.6%(7)

6.1%(164)

2.2%(2)

58.7%(54)

5.7%(153)

9.0%(242)

2.2%(2)

44.5%(1,202)

17.4%(16)

25.8%(698)

12.0%(11)

Enrolled by University

Abe

rdee

n C

entr

al H

igh

Scho

olA

ll O

ther

Hig

h Sc

hool

s

98.9%Did Meet

97.8%Did Meet

Admission Reqs Met

92Grad Year 2014

17.5%39.6%33.7%

19.6%29.3%26.1%

Composite English Reading Math Science

0

10

20

23.022.5 22.1 21.923.423.3 22.9

21.223.222.5

ACT Mean Scores and Benchmark Attainmentx

(ACT Mean Scores)

(English) (Reading) (Math) (Science)

10%20%30%40%50%60%70%80%90%

80.4% 82.6%

69.6%72.8%

50.0%

62.9%

44.8%45.7%

(ACT Benchmark Attainment)

SDOS RecipientsRegents Scholars

Took AP Course(s)

StudentsHS / County Aberdeen Central High School

All Other HS High Schools

Aberdeen Central High School

All Other High Schools


Academic Distinctions

University All

SDBOR High School Transition Report

BH

SU


School


DSU


School


NSU


School


SDSM

T


School


SDSU


School


USD


School


Enrolled by University

Abe

rdee

n C

entr

al H

igh

Scho

olA

ll O

ther

Hig

h Sc

hool

s

Admission Reqs Met

92Grad Year 2014

Composite English Reading Math Science23.222.5

ACT Mean Scores and Benchmark Attainmentx

(ACT Mean Scores)

(English) (Reading) (Math) (Science)

44.8%45.7%

(ACT Benchmark Attainment)

SDOS RecipientsRegents Scholars

Took AP Course(s)

StudentsHS / County Aberdeen Central High School

All Other HS High Schools

Academic Distinctions

University All

http://www.sdbor.edu/dashboards


FIGURE 3: REMEDIAL ENROLLMENT AND FIRST-YEAR RETENTION RATES

The High School Transition report uses institutional and secondary district-level data that analyze unique students in RIS. However, other dashboards present South Dakota in context with other states across the country using nationally available data sets. To answer a question about whether South Dakota is a net importer of first-time college students, Palmer pulled data from the Integrated Postsecondary Education Data System (IPEDS). While this data is available publicly, SDBOR is one of the few examples of state-level postsecondary agencies presenting national information alongside other state-specific data elements. The net migration dashboard contains information on the student exchange rates of South Dakota and all other states. While the dashboard defaults to view information about South Dakota, it contains similar information for all other states. State agency personnel from other states could find the information useful (see Figure 4).

SDBOR High School Transition ReportGrad Year 2014

0% 20% 40% 60% 80%

No Rem. Placement

Rem. English

Rem. Math

Rem. Eng. AND Math

72.0%57.6%

15.4%15.2%

39.1%23.1%

12.0%10.5%

Placement into Remedial Coursework

0 5 10 15 20 25Average 1st Year Completed Credit

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Ave

rage

1st

Yea

r G

PA

None

Math or English

Math and English

None

Math or English

Math and English

First Year Outcomes (GPA, Credits) by Remediation Groupx(Circle size indicates number of students)


All Other High Schools 89.6%Retained

85.9%Retained14.1%

Retention into 2nd Semester in SDBOR System

92StudentsHS / County Aberdeen Central High School





Aberdeen Central High SchoolUniversity All


FIGURE 4: NET MIGRATION DASHBOARD

Palmer also uses nationally available data sets to provide users with context for workforce indicators in the dashboards, including labor force participation and projected demand for specific occupations. In response to Board of Regent inquiries about graduate salaries, Palmer built a dashboard that displays state comparisons of wages by occupation. The dashboard uses publicly available data from the Bureau of Labor Statistics (see Figure 5).

SDBOR Freshmen Migration Dashboard

2786

5787

-27505

1181

1310

-7607

8278

1718817

1717

-5219-17438

-12435

15060

-4992

-9601

-2136

-1522

-1273

8088

4602

4097

4989

5988

5304

2629

72026278

3459 4190

7652

4148

26183759

4217

1223

33136513

1729

7871

3151

882

609

708

109

618

-26

1225

Private 2-yr

Private 4-yr

Public 2-yr

Public 4-yr

Select a MeasureNet Migration Balance

Year (for map)2014

2000 2002 2004 2006 2008 2010 2012 2014

670583

-73

389

9571226

-5-3

1191

176

-29

-190 -56-58 -36

238

-68

-123

-51-34

-7-97-59

1197

21

State-by-State Exchange Datafor Selected State

Sector Trendsfor Selected State

Click map to select a state SectorAll

2014

MN

IA

Foreign

CA

CO

WY

WI

IL

TXFL

270

750

40

84

66

46

26

70

18

41

2000

2002

2004

2006

2008

2010

2012

2014

1225

-100

1311

830

490

570

129

-68

Annual Trendsfor Selected State


FIGURE 5: OCCUPATIONAL WAGES DASHBOARD

By framing each dashboard with a research question and using all relevant data sources available to the Board of Regents, the research team has constructed a transparent, detailed and user-friendly interface that is available to a wide variety of audiences.

SDBOR Occupational Wages Dashboard Select a MeasureAverage Annual Pay

Show Bars as:States

Show Occupations by:By occupational categoryBy specific occupation

Select an Occupation/StateAll OccupationsArchitecture & EngineeringArts, Design, Entert., Sports, & MediaBuilding & Grounds Cleaning & Maint.Business & Financial OperationsCommunity & Social ServiceComputer & MathematicalConst. & ExtractionEducation, Training, & LibraryFarming, Fishing, & ForestryFood Preparation & Serving RelatedHealthcare Practitioners & Tech.Healthcare SupportInstallation, Maint., & RepairLegalLife, Physical, & Social ScienceManagementOffice & Administrative Support

Year2015

Show/Hide US Total LineShow

Show/Hide All Occupations LineShow

Split by Typical Education (use with "Show Bars as Occupations-Specific")No

Please ensure all filters have a selection.

If they do, but you are still seeing this message, you mayneed to select a new geography or a different combinationof filters, since the Bureau of Labor Statistics does notprovide all estimates for all geographies.

- Chart Options -

0K 10K 20K 30K 40K 50K 60K 70K 80K 90K

District of ColumbiaMassachusetts

New YorkConnecticut

AlaskaCalifornia

New JerseyMaryland

WashingtonVirginia

ColoradoRhode Island

DelawareIllinois

MinnesotaNew Hampshire

US TotalOregonHawaii

TexasPennsylvania

MichiganVermont

WyomingNorth Dakota

GeorgiaArizona

OhioNorth Carolina

UtahWisconsin

MissouriMaine

New MexicoKansasFloridaNevada

NebraskaIndiana

AlabamaIowa

OklahomaTennesseeKentucky

IdahoLouisianaMontana

South CarolinaWest VirginiaSouth Dakota

ArkansasMississippi

$40,880

$42,800

$46,060

$50,300

$40,580$40,620

$38,820

$42,860

$48,320

$56,280

$43,640

$50,780

$38,540

$42,070$42,630

$42,930

$43,260

$43,930

$46,560

$45,660$45,850

$45,420

$54,630

$40,810

$48,100

$40,810

$49,970$49,740

$57,030

$46,550

$54,950$55,260

$37,620

$80,150

$44,750

$41,300

$39,100

$41,840$41,820

$54,010

$59,010

$47,740

$55,760

$41,920

$44,130

$46,310

$48,710

$45,310

$44,170

$43,170

$51,670$51,180

Average Annual Pay in 2015


REACTIONS TO SDBOR DASHBOARDS

The primary audience for the SDBOR dashboards is the Board of Regents. In regular reports to the regents, SDBOR research staff present information about student success and finance indicators. After presenting topics of importance to the regents, including content from the dashboards, this information becomes available to SDBOR for interacting with the state legislature.

Legislators and regents have reacted positively to these new tools, according to Warner. In particular, when policymakers were presented with national-level data—even data that was unflattering for South Dakota—the information gave them confidence in the strategic plan outlined by the Board of Regents to improve student outcomes. "Legislators will invest in confidence…That’s what we try to do—use data to build credibility."

In addition, the dashboards have resulted in greater internal confidence in the data submitted to the RIS. If policymakers are the primary audience for the dashboards, then institutional staff are a secondary audience. According to Palmer and Turman, institutions have reacted positively to seeing information from their campuses helpfully presented alongside their peer institutions, and they have confidence that the information is being correctly presented.

Finally, the information in the dashboards is available for use by the general public, including prospective students, parents and educational researchers. Palmer has presented his dashboards to national audiences and invited his peers in other states to use the information presented by SDBOR.


DISPELLING MYTHS WITH DATA

The dashboards have been a way to easily combat a persistent and false narrative about higher education in South Dakota. According to SDBOR staff, there was a perception among state legislators that South Dakota was experiencing “brain drain,” the departure of high school graduates who seek their college education out of state and don’t return as adults. One of the key takeaways from the nationally available data sets displayed in the dashboard was that South Dakota is actually a net importer of students. Displaying this information alongside information collected from the institutions allowed regents and other policymakers to see data that dispelled the myth of South Dakota’s “brain drain.”

Warner credits the dashboards with changing the perceptions of regents and legislators about student migration. According to Warner, the issue has been present in the minds of policymakers for some time. In the late 2000s, South Dakota lowered its out-of-state tuition rates in an effort to reduce the number of students who migrated. Local media has linked to the migration dashboards with the phrase, "Actually, we're in the black,” implying surprise that South Dakota is currently a net importer of students.8

Alongside the information on student migration, the dashboards also present labor market outcomes of South Dakota college graduates. “Legislators seem to be most interested in students who start in South Dakota or get a degree from us. Are they taking jobs in South Dakota?” Using data from RIS and the National Student Clearinghouse, SDBOR determined that 60.2% of graduates from South Dakota institutions were present in the South Dakota labor force one year after graduation.9 As of June 2016, South Dakota has the lowest unemployment rate in the nation.

8. http://madvilletimes.com/2014/11/south-dakota-enjoys-decade-long-trend-of-increased-college-freshman-imports

9. This figure does not include those who are enrolled in graduate school, employed by the federal government or military, or are self-employed.

http://madvilletimes.com/2014/11/south-dakota-enjoys-decade-long-trend-of-increased-college-freshman


CONCLUSION

South Dakota’s response to SHEEO’s survey on state data systems served as the impetus for gathering more information on the Board of Regents’ use of postsecondary student unit record data. In response to the question, “How has the student unit record system provided the greatest value to your state?” they responded, “It would be difficult to understate the advantages of a unified data system such as ours. It provides us the ability to quickly and reliably extract comparable data for all of our campuses, which in turn leads to high internal and external confidence in our analyses.” By publicly displaying much of the available information in the unified RIS database, SDBOR allows the wealth of information collected to be put to use by policymakers, institutional staff, researchers and the general public.

South Dakota is not the only state to publish interactive tools that allow for the general public to drill down into specific student success indicators. However, their dashboards, coupled with the close relationship between research staff and agency leadership, serve as an example to the higher education research community of the value of data transparency. Transparent and easily understandable figures and charts enable internal and external confidence in the data (allowing for data to be updated if mistakes are discovered) and reduce burden on research staff by anticipating questions that could be answered with summary-level data.

We encourage the higher education research community to adopt the “question first” model of conducting research, to invest in quality, trained research staff who closely collaborate with agency leadership to articulate agendas, and to make information publicly accessible to a broad audience.

STATE HIGHER EDUCATION EXECUTIVE OFFICERS

3035 CENTER GREEN DRIVE, SUITE 100, BOULDER, COLORADO, 80301 303.541.1600 • SHEEO.org

http://www.sheeo.org

data driven decision-making in south dakotawere conducted by sheeo agency staff in seven states...

Documents