outcomes & equity assessment › uw-s3-cdn › wp...english proficient economically...

Outcomes & Equity Assessment

PHASE FOUR: PROCESS IMPROVEMENT

The Office of Community College Research and Leadership (OCCRL) was established in 1989 at the University of Illinois at Urbana-Champaign. OCCRL is affiliated with the Department of Education Policy, Organization and Leadership in the College of Education. Our mission is to use research and evaluation methods to improve policies, programs, and practices to enhance community college education and transition to college for diverse learners at the state, national, and international levels. Projects of this office are supported by the Illinois Community College Board (ICCB) and the Illinois State Board of Education (ISBE), along with other state, federal, and private and not-for-profit organizations. The contents of publications do not necessarily represent the positions or policies of our sponsors or the University of Illinois. Comments or inquiries about our publications are welcome and should be directed to [email protected].

This module is part of a series of publications associated with the Pathways to Results initiative that is funded by a grant from the Illinois Community College Board (ICCB Grant Agreement Number 2014-07331).

Acknowledgement:

Staff of the Office of Community College Research and Leadership (OCCRL) thank the Illinois Community College Board for their persistent and gracious support of the Pathways to Results (PTR) initiative. We also express our thanks to the PTR team leaders and team members throughout the state who have piloted and implemented PTR since its inception. Finally, we thank Dana Hagerstrom, owner of Design House for her creative contributions to the design, production and editorial work on this publication.

Suggested Citation:

Taylor, J., Castro, E., Swanson, J., Harmon, T., Kristovich, S., Jones, A., & Kudaligama, V. (2015). Outcomes and equity assessment. (Rev. ed.). Champaign, IL: Office of Community College Research and Leadership, University of Illinois at Urbana-Champaign.

Copyright 2015 Board of Trustees, University of Illinois

PATHWAYS TO RESULTS: OUTCOMES & EQUITY ASSESSMENT

PHASE FOUR: PROCESS IMPROVEMENT 1

Pathways to Results (PTR) is an outcomes-focused, equity-guided process to improve pathways and policies that support student transition to and through postsecondary education and employment. PTR focuses on addressing equity gaps between diverse learner groups and continuously improving processes critical to student success, including retention, completion of postsecondary credentials, and transition to employment.

The PTR process is most effective when it begins with a strong collaboration of team members and partners focusing on the critical problems that get in the way of student success. These problems are identified when the teams use student-level data to identify outcome and equity gaps in results between racial, gender, low-income and other underserved groups. Major processes are assessed to understand how contributing factors create the identified problems and impede student success. Implementation and evaluation plans are designed to create solutions that improve the quality of pathways immediately and over time. PTR gives teams the opportunity to continuously improve pathways and produce ever-more equitable student outcomes. When PTR is implemented fully, the opportunity to improve pathways never ends.

An overarching goal and benefit of the PTR process is that it provides teams with the opportunity to continuously improve pathways and produce ever-more equitable student outcomes.

“

”

PATHWAYS TO RESULTS: OUTCOMES & EQUITY ASSESSMENT 1

Overview

In Outcomes and Equity Assessment, the process of identifying, selecting, and analyzing disaggregated student outcomes data is introduced using PTR’s definition of equity. Equitable student outcomes are a fundamental component of PTR and discussion throughout this process provides the foundation for understanding the improvement of student outcomes and equity.

This process begins with a readiness audit to gauge the level and type of support that teams need to carry out the activities. It engages team members in a discussion of PTR’s definition of equity and assists them in the selection of outcome data that can reveal gaps in student access and performance. Team members are asked to think about equity gaps and consider how student outcomes are different for various groups. Secondary and postsecondary institutional researchers play an important role in collecting and helping the team to analyze data to inform the PTR process.

Teams engage in analyzing student outcomes data and identify gaps, develop a problem description and goals that guide future work and identify processes to examine.

“A valuable aspect of the process was collecting and reviewing the data to truly understand who the students are that enroll in our programs and perhaps more importantly, who they are not.”

— PTR Team Member, Kaskaskia College Partnership


Purpose and Goals

The primary purpose of Outcomes and Equity Assessment is to determine, through data and information collection and analysis, where equity gaps exist to guide the PTR process.

The goals of this process are to:1. identify, collect, and interpret disaggregated student outcomes data.2. understand and use PTR equity concepts when analyzing student data.3. identify and begin to understand equity gaps.

Outcomes and Equity

An emphasis on equity is explicit in this process through the collection and analysis of disaggregated student data. To disaggregate student data means to break the data into segments of the student population based on categories such as race, ethnicity or socioeconomic status. The Outcomes and Equity Templates assist the teams to identify inequities among student groups and generate a conversation to address them.

What is Equity in PTR?

The PTR process supports improving student outcomes with a focus on equity. This means that PTR is committed to ensuring students persist and graduate at equitable rates at the secondary and postsecondary levels. An exploration of disaggregated student outcomes enables the identification of problems that impact student groups.

Applying an equity lens does not mean treating all students or individuals in the same way. Equity in this context means investing resources and designing programs to address the needs of students with different experiences and educational backgrounds. Targeted investment and program design should produce equal outcomes—the ultimate goal of an equity-minded process. Thus, the equity lens relies on exploring outcomes disaggregated by the following student characteristics:

• socioeconomic status (SES) • gender • age • other characteristics determined relevant by the team

Outputs

Materials available to support this process: • The Outcomes and Equity Templates with data disaggregated by student subgroups.

For instructions on how to use the templates see Appendix D, E and F. The elec-tronic templates are online at http://occrl.illinois.edu/projects/pathways/phases/2.

• Team Worksheet and Contribution to the Charter (See Appendix H): A summary of findings, problem description, goals, and an initial list of related processes.


Steps at a Glance

STEP 1: Outcomes and equity selection. All team members participate in two facilitated activities to a) identify and select outcomes data to collect, and b) develop an understanding of equity. The Team Readiness Tool (See Appendix B) helps the team identify the needs for technical, conceptual, and advisory assistance.

STEP 2: Data collection and sharing. Secondary and postsecondary institutional research staff collect and share data with team members using PTR reader-friendly outcomes templates.

STEP 3: Data analysis and interpretation. Team members participate in a facilitated activity to review and interpret disaggregated student outcomes data from an equity perspective. The team reaches consensus on the focus of the PTR project. The team also discusses and plans to collect qualitative data from student subgroups to bring clarity to the team’s quantitative data findings through focus groups, surveys, or interviews.

Step 1:Outcomes and Equity Selection

Step 2:Data Collection

and Sharing

Step 3:Data Analysis and

Interpretation


Who Should be Involved?

Outcomes and Equity Assessment is intentionally designed so all team members individually and collectively analyze and interpret data. Secondary and postsecondary institutional researchers may play significant roles in helping the team to identify and collect data and to assist the team members to process the data and decide on the focus of the PTR project.

Detailed Steps

Step 1: Outcomes and equity selection.

The primary function of Step 1 is to help the team determine what student outcomes data to collect and engage team members in an interactive activity that introduces PTR’s definition of equity. A primary tool associated with this step is the Team Readiness Tool (See Appendix B) that helps the team determine assistance needed for the work of Outcomes and Equity Assessment. The team leader can complete this tool individually or complete it with input from other team members.

Two types of data are needed for the PTR process: student characteristics and student outcomes. Teams need to explore the relationships between student characteristics and student outcomes, and they need to consider how student characteristics influence student outcomes.

Student characteristics – Attributes about a student usually do not change, such as gender, race/ethnicity and ability/disability. These attributes can be used to identify sub-groups of students whose outcomes can be compared. For example, a team can compare males to females on the grades received in an important course.

Student outcomes – Results that are impacted by a program, change or intervention, such as grades, cumulative grade point average (GPA), enrollment status, credits attempted and credits earned, certificate and degree attainment, etc.

When data are analyzed, team members compare outcomes for different sub-groups based on the student characteristics. Table 1 and Graph 1 illustrate this comparative analysis by using data adapted from one PTR team (using the pseudonym ‘PTR Partnership’). In this example the student outcome, fall to spring retention, is disaggregated by student characteristics. This team chose to use the “special populations” categories as defined by federal Perkins legislation. Special populations metrics are important when assessing programs that receive federal Perkins funding, but may not be used when assessing all pathways. PTR teams compare columns of student characteristics against rows of student outcomes.


Table 1. Fall to Spring Retention – Example from Practice

Fall to Spring

RetentionTotal

Students with

Disabilities

Limited English

Proficient

Economically Disadvantaged

Single Parents

Displaced Homemakers

Non-traditional

# of Students Retained 127 25 30 76 25 12 35

# of Total Students 194 32 46 150 36 17 78

Retention Rate 65% 78% 65% 51% 69% 71% 45%

Graph 1. Fall to Spring Retention – Example from Practice

78%

65%

51%

69% 71%

45%

0%

20%

40%

60%

80%

100%

Student withDisabilities

Limited EnglishProficient

EconomicallyDisadvantaged

Single Parents DisplacedHomemakers

NontraditionalStudents with


Analyzing Data in PTR

To determine what data to collect and analyze, teams identify questions that enable the group to delve into important relationships between student characteristics and student outcomes. A process of data analysis that is useful to PTR follows. It begins by analyzing data at the program level and then comparing program level data to institutional level data. Teams create a research question to guide this work, which is illustrated in a question posed by the PTR Partnership: “For the Production Management Program, how do minority students compare to White students in courses completed vs. attempted?” Note that this question includes both student characteristics and student outcomes as follows:

Student Characteristic: Students’ race/ethnicity Student Outcome: Course completed vs. course attempted

To answer the research question, the team defined the outcome as course completion vs. course attempted for students in the first semester of the program for a course identified as critical for the pathway; the team then collected these data on students in the Production Management program. Then, the outcomes data were disaggregated by race/ethnicity, socioeconomic status, gender, age, and other student characteristics identified as relevant by the team. Disaggregating data allows team members to recognize gaps between sub-groups. Table 2 displays the data that the PTR Partnership collected for the student outcomes disaggregated by race/ethnicity. These results reveal an equity gap that requires additional analysis.

Table 2. Program Snapshot – Example from Practice

First Semester Course

Completion vs. Course Attempted

Total African American

American Indian or

Alaska Native

Asian American

Hispanic/ Latino(a) White

Native Hawaiian or Other Pacific

Islander

Two or More Races

# of Students completing course 127 28 6 11 12 53 6 8

# of Students attempted course 194 43 10 15 18 72 10 18

% Course Completion to Attempted

65% 65% 60% 73% 67% 74% 60% 44%

In this example, the total proportion of students who completed course vs. number of students who attempted the selected course in the Production Management program is 65%. In this example, the team chose to disaggregate their data by students’ race/ethnicity. This is a critical step in outcomes assessment for any program or pathway. Analysis of these data show large differences based on race/ethnicity sub-groups. For example, the data show that a smaller percentage of African American, American Indian or Alaska Native, and Hispanic/Latino(a) student sub-groups who attempt the course have successfully completed the course, compared to the White sub-group. Because there are more White


students in the program, the total percentage of 65% mostly reflects the White sub-group’s result, which is relatively high at 74%. However, a more nuanced analysis that disaggregates the outcome by race/ethnic sub-group points to an equity gap within the PTR Partnership. This type of comparative analysis, where student outcomes are analyzed for different sub-groups, is fundamental to PTR.

Equity is also measured comparatively by examining disaggregated program level data to disaggregated institutional level data. This allows the team to compare student outcomes at the program level to the institutional level. It reveals results relative to critical questions about student success in the pathway or program of study. Consider the examples of program level data in Table 2 above and institutional level data in Table 3 below. This comparison is also illustrated in Graph 2.

Table 3. Institutional Snapshot - Example from PracticeFirst Semester

Course Completion vs. Course Attempted

Total African American

American Indian or

Alaska Native

Asian American

Hispanic/ Latino(a) White

Native Hawaiian or Other Pacific

Islander

Two or More Races

# of students who completed course 2875 552 17 275 360 1575 16 34

# of students who attempted course 3780 750 25 325 524 2008 23 45

% Course Completion to Attempted

76% 74% 68% 85% 69% 78% 70% 76%

Graph 2. Comparison of Program Level and Institutional Level Data – Example from Practice

0%

20%

40%

60%

80%

100%

Total Black orAfricanAmerican

AmericanIndian orAlaskaNative

Asian Hispanic White NativeHawaiian or

OtherPacificIslander

Two orMore Races

Program Level Institutional Level


This comparative analysis suggests a number of differences between program level and institutional level outcomes. For both institutional and program data, White students perform better than African Americans, American Indian or Alaska Natives, Hispanic/Latino(a) students, Native Hawaiian or Other Pacific Islander students and students declaring Two or More Races. The performance of Hispanic/Latino(a) students is similar at the program level (67%) and institutional level (69%), but the performance of African American students is lower at the program level (70%) than institutional level (74%). Asian American students perform better than White students at the program level and institutional level, but there is nearly a 12% disparity between institutional and program level data for Asian Americans.

When disaggregating student outcomes data, PTR teams should be cautious when interpreting data based small numbers of students in a single cohort. For example, in Table 2 there are only ten American Indian or Alaska Native students. Similarly, course enrollment of Asian American, Hispanic/Latino(a), and Native Hawaiian or Other Pacific Islander students is low compared to African American and White students. To achieve larger student numbers, one approach is to look at multiple student cohorts in the pathway over a period of time. For example, the PTR team could collect outcomes data for students who enrolled in the program between FY2010-2014. By increasing the number of students in the analysis, the results can increase the confidence and integrity of the data and help the team determine important student outcome trends. However, teams should also exercise caution in aggregating data if unique circumstances (for example, a one-time training program that enrolled a large group of students in a single course) can cause temporary fluctuations in student outcomes.

Considering equity at the program and institutional levels is an important endeavor but it is only the beginning. It may also be important to compare data representing the program level to data representing the larger context of the district, the community and region, the state and the nation. These exercises provide additional insights into how the programs of study are working toward or against educational equity. However, parity between student groups does not necessarily mean equity because the needs of certain sub-populations of students could be substantially greater than others; it is an indicator of difference between sub-groups that deserves further study. PTR teams need to consider results with a deep understanding of the program, institutional, and larger contexts.

Student Outcomes Selection Tool and Activity

Now that the team has examined examples from the PTR Partnership and has developed a basic understanding of equity in the PTR context, this experience should be used to determine data the teams wants to collect for the PTR project.

Before teams decide what data to collect, they should develop inquiry questions that include student outcomes and student characteristics. Teams should reference the Outcomes Menu used in Engagement and Commitment and the Student Outcomes and Selection Worksheet (see Appendix C) of this module.


A. Start with the broad goals of the PTR project (i.e., to increase student success in the pathway).

Generate questions from the goals that compare characteristics to outcomes. For example: Are there socioeconomic differences in retention in the identified pathway or process? (The characteristic here is socioeconomic status and the outcome is retention.)

B. Use the Outcomes Menu to select outcomes to use to address the inquiry questions. Focus on a few outcomes listed in the Outcomes Menu or additional outcomes developed by the team. Note: the inquiry questions can change and evolve by the end of this exercise.

C. Place the inquiry questions on a white board or flip chart (so all can see). Using two different colored markers, underline the characteristics with one color and the outcomes with another. The team should refine the outcome definitions and student characteristics to determine what data to collect. PTR teams should collect data on the following student characteristics for each outcome: • race/ethnicity • socioeconomic status • gender • age • other characteristics determined relevant by the team

D. Identify definitions for the outcomes selected. Team members should think about how they use the outcomes in their current work. It is important to get both secondary and postsecondary perspectives. For example, using the previous inquiry question, Are there socioeconomic differences in retention in the identified pathway, the team should define the outcome of retention in ways that make sense at both levels.

Below are some examples of measures of retention: • fall-to-spring retention • fall-to-fall retention

Teams also need to determine the student sub-groups to be included in the analysis. For example, if fall-to-fall retention is the defined outcome, the team could collect data on different sub-groups of students by enrollment status (full-time, part-time) by academic year. In other words, they could calculate retention for these sub-groups in academic year 2011-12, academic year 2012-13, and academic year 2013-14.

E. Use the Student Outcomes and Selection Worksheet (see Appendix C) to write down the final list of inquiry questions, student outcomes, definitions for the outcomes selected, and student characteristics.

F. Determine how to get the data, what resources are available, and what further conversations about outcomes are needed.


Step 2: Data collection and sharing.

The second step is the collection and sharing of data. The Outcomes and Equity Templates are provided to teams and can be modified based on the outcome variables selected during Step 1. Electronic copies of all templates (as Excel files) and detailed directions for completing the templates are provided on the OCCRL website at http://occrl.illinois.edu/projects/pathways/ptr-phase-two-outcomes-and-equity-assessment/. These electronic templates are designed so that all percentage calculations are automatically completed based on the data entered. In addition, charts and graphs that provide visual representations of the data are also automatically generated. See Appendix D, E and F for detailed instructions for the templates. Once the templates are finished, data and graphs are sent to all team members to individually review. Team members complete the Data Review Worksheet (See Appendix G) prior to gathering as a full team in Step 3.

Step 3: Data analysis and interpretation.

The goal of this step is to engage team members in interpreting data previously collected by institutional researchers and reviewed individually by team members. This activity provides team members with an opportunity to contribute to the analysis and interpretation of the data.

The team meets to review the completed Data Review Worksheet (see Appendix G) and discuss the primary findings. The group then makes suggestions for the “Bullet list of findings from data analysis and interpretation” section of the Team Worksheet and Contribution to Charter (see Appendix H). Based on the list of findings, the team then develops a problem description. In this process, the team reviews the problem statement identified in the PTR Charter document and reflects on potential changes that are needed. This problem description can be conceptualized as a narrative of the relevant findings from the data analysis. At this time, the team should also consider their lingering questions about identified equity gaps or even particularly strong outcomes among certain student groups. Some of these questions may be answered, at least in part, by additional data collection in the form of engaging students with interviews, focus groups, or surveys. This deeper look using qualitative data is a useful bridge to Process Assessment.

The team may not be able to focus on all of the findings and can decide which findings are most compelling and worthy of pursuit for the PTR project. The team leader should then set reasonable goals for the PTR project related to the problem description.

Reflection Questions

1. What value did the team find in the analysis of disaggregated student outcomes?

2. How was the team’s approach to data collection and analysis similar to or different from previous partnership efforts or other established institutional practices?

3. What did the team learn about outcomes and equity?


References & Resources

EquityAchieving The Dream. (2011). Retrieved from http://www.achievingthedream.org/

Hurtado, S. (2007). Linking diversity with the educational and civic missions of higher education. Review of Higher Education, 30(2), 185–196.

The Stanford Center for the Study of Poverty and Inequality. (2011). Retrieved from http://www.stanford.edu/group/scspi/index.html

U.S. Department of Labor, & U.S. Bureau of Labor Statistics. (2009). Highlights of women’s earnings in 2008. Retrieved from http://www.bls.gov/cps/cpswom2008.pdf

Witham, K., Malcom-Piqueux, L., Bensimon, E. M., & Dowd, A. C. (2015). America’s unmet promise: The imperative for equity in higher education. Washington, DC: Association of American Colleges & Universities. Retrieved from http://www.aacu.org/publications-research/publications/americas-unmet-promise-imperative-equity-higher-education

Outcomes Assessment

Bauman, G. L., Bustillos, L. T., Bensimon, E. M., Brown, M. C., & Bartee, R. D. (2005). Achieving equitable educational outcomes with all students: The institution’s roles and responsibilities. Washington, DC: Association of American Colleges and Universities. Retrieved from http://www.aacu.org/inclusive_excellence/documents/bauman_et_al.pdf

Finley, A., & McNair, T. (2013). Assessing underserved students’ engagement in high-impact practices. Washington, DC: Association of American Colleges and Universities. http://www.aacu.org/sites/default/files/files/assessinghips/AssessingHIPS_TGGrantReport.pdf

Reyna, R. (2010). Complete to compete: Common college completion metrics. Washington, DC: Center for Best Practices, National Governor’s Association. Retrieved from http://highered.colorado.gov/Publications/General/StrategicPlanning/Meetings/Resources/Access/Access_100713_Compete_to_Complete_RyanReyna_June2010.pdf

Data UsageDowd, A. C. (2005). Data don’t drive: Building a practitioners-driven culture of

inquiry to assess community college performance. Indianapolis, IN: Lumina Foundation. Retrieved from https://folio.iupui.edu/bitstream/handle/10244/261/datadontdrive2005.pdf?sequence=1

Jenkins, D. (2011). Redesigning community colleges for completion: Lessons from research on high-performance organizations. [Working Paper 24]. New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/Publication.asp?UID=844

Urban Institute. (2008). The case for evidence-based policy: Beyond ideology, politics, and guesswork. Retrieved from http://www.urban.org/UploadedPDF/901189_evidencebased.pdf

Center for Minority Serving Institutions (2013). Using educational data to increase learning, retention, and degree attainment at Minority Serving Institutions (MSIs). Penn Graduate School of Education: Retrieved from http://www.gse.upenn.edu/pdf/cmsi/using_educational_data.pdf.


Appendix AWhy Equity?

PTR is concerned with equity for several reasons. Below is a list of intellectual and empirical rationales for the emphasis on equity in PTR:

• Disparities in academic outcomes for underrepresented students continue to persist. At nearly all points in the educational pipeline, racial and ethnic minorities, low socioeconomic students, students with disabilities, and other underrepresented students underperform compared to their counterparts.

• The student demographics of the U.S. are rapidly changing. The U.S. Census Bureau predicts that no single racial or ethnic group will constitute more than 50% of the population by 2050. Given these demographic changes it will be impossible for states or institutions to reach their completion goals without eliminating achievement gaps.

• Employment opportunities are becoming increasingly diversified and the demands of workforce are changing requiring all students to engage and succeed in diverse workplace and community environments.

• Diversity improves student outcomes, offers students from different backgrounds the opportunity to interact and learn from one another, and contributes to the cultivation of students’ interest in the public good, poverty issues, and other democratic sensibilities.

• Education in the U.S. has long been understood as an opportunity equalizer, a function of the ‘American Dream.’ In order for this dream to persist, students from the most marginalized corners of this country must be given equal educational opportunity so as not to reproduce existing social inequities.

• Institutions often focus without success on so-called “deficits” (e.g. poor motivation or underpreparedness) among minority populations as the primary source of achievement gaps. In contrast, PTR’s equity lens focuses on equity as an institutional responsibility to use data and resources to ensure students achieve equitable outcomes, even from unequal starting points.

Reflection Questions:

1. Why is equity important to the PTR team?

2. How do members of the PTR team think about equity, and how do they think equity applies to the PTR process?

3. What does the PTR team need to learn about diversity, equity and student outcomes?


Appendix BTeam Readiness Tool

This tool helps the team identify their data and information sources, resources, and readiness. This identification helps make data collection happen and determines technical assistance they need to engage in this process. The team leader and institutional researchers should complete this tool, including team members as appropriate.

Questions for Secondary Institutions:

1. In what ways do faculty and staff use data to help guide their decisions?

2. What level of access does the team have to enrollment data by student characteristics (e.g., race, gender)?

3. Who has access to the person or office that prepares data for your school or district? Is this person part of your PTR team? If not, can they assist your team?

4. Who has access to secondary district data (is there a person on the team who can enter/extract secondary data)?

5. Does the pathway your team is looking at have dual credit course offerings? If so, what access does your team have to that enrollment data (secondary or postsecondary)?

6. How confident are you that you can obtain enrollment data for this pathway?

7. What other data are available at the secondary level?

8. What are the barriers associated with the collection of secondary data for PTR?


Questions for Postsecondary and Adult Education Organizations:

1. In what ways do the faculty and staff use data to help guide their decisions?

2. Does the community college have an institutional research office? If so, are they available to provide data for the team? If not, what unit(s) provide data and information for decision-making about your program? (Who do you go to when you need data?)

3. Does your organization compile an annual profile that summarizes student characteristics? (Sometimes these are called “who we are” reports, environmental scanning, college or unit profiles, college fact sheets). If so, do you have access to this information?

6. How confident are you that you can obtain enrollment data for your selected pathway?

7. What are the barriers associated with the collection of postsecondary and adult education data for PTR?

Other Questions:

1. To what extent do you sense PTR team members are comfortable reading, analyzing, and interpreting data?

2. What types of resources might be helpful to assist your team during this process?


Appendix CStudent Outcomes and Selection Worksheet

Teams should use this worksheet to enumerate multiple inquiry questions they would like to answer with their data. The team should determine one or more student outcome(s) which would help answer these questions. The Outcomes Menu from Engagement and Commitment provides a list of student outcomes for the identified pathway and provides a starting point for student outcomes. Then, teams define the outcome selected and student characteristics/categories to be collected that correspond to the inquiry question.

Inquiry Question Student Outcome Outcome Definition

Student Characteristics and Categories


Appendix DOutcomes and Equity Template Instructions

Template Set OneDemographics by Sub-Group

INSTRUCTIONS FOR PTR TEMPLATE SETS

Three groups of Outcomes and Equity Template Sets (Excel Worksheets) have been created by OCCRL to report enrollments and outcomes by sub-groups. The electronic templates are online at http://http://occrl.illinois.edu/projects/pathways/ptr-phase-two-outcomes-and-equity-assessment/.

Each set of templates contains individual worksheets for the following sub-groups. Work-sheets are uniquely named to indicate the sub-group analyzed in each sheet.

•• Race and ethnicity (categories recognized by the Department of Education in 2007)•• Gender•• Socioeconomic status•• Age•• Special Populations (if applicable)

TEMPLATE SET ONE: Demographics by Sub-Group

Template Set One shows the demographic characteristics of students in local high schools, the community college, and the pathway broken out by sub-group. The purpose of these templates is to help PTR teams understand who is enrolling and accessing the identified pathway that the team has decided to improve.

Template Set One includes five templates. The templates are structured to allow teams to observe sub-groups of students who are enrolled in high schools, the community college(s), and the identified pathway that the PTR team has identified for its project. The five catego-ries of demographic characteristics included in the templates are Race/Ethnicity, Special Populations, Gender, Socioeconomic Status, and Age. Teams may add additional categories based on their interests, and some of these categories can include English Language Learn-ers, part-time or full-time enrolled, part-time or full-time employed, first-generation, etc.

Also, PTR teams may modify the demographic characteristics categories based on the needs of their project. For example, some PTR teams might define Socioeconomic Status (SES) based on secondary students’ eligibility for free lunch or a postsecondary student’s eligibil-ity for Pell. Others may define SES based on parental income and educational levels. Avail-ability of data is often a primary determinant of the outcomes that PTR teams are able to use for this analysis.

Step 1

The PTR team should determine the cohort(s) to analyze in Template Set One. A team might identify all students who first enrolled as majors in a program in fall 2013 as a cohort, or


the team might narrow this cohort to look at only those students who had also enrolled in a related pathway at the secondary level. This analysis is likely to involve small numbers of enrollees, but it may be valuable to understand outcomes of students who matriculate from the secondary to the postsecondary level in a particular pathway. On the other hand, it may be useful to conduct analysis with larger numbers of students, so the PTR team may decide to combine all students who first enrolled as majors in a program in fall 2012, fall 2013, and fall 2014. The bottom line is that there is no one right way to conduct this process. What the PTR team should do depends on what it is trying to learn and how it wants to im-prove the selected program, while recognizing circumstances that are unique to their team.

More specifically, on each worksheet in Template Set One (with the exception of the “Age” characteristic) there are four types of cohorts: 1) the Postsecondary Pathway Student Cohort; 2) the Secondary Pathway Student Cohort; 3) the college group; and 4) the high school group. The Postsecondary and Secondary Pathway Student Cohorts should be de-fined by the teams as described in the previous paragraph. The college group is defined as all students in the community college. The high school group uses enrollment numbers from the area high school, these enrollments can be acquired from numerous sources including from the EFE regional director, from the high schools and from the state boards of educa-tion.

Finally, the PTR team can create a comparison group to which sub-group analysis can be applied and compared to analyze the student composition of the identified cohorts. One example of a comparison group is residents of the community college district. Conclu-sions about these comparisons need to be made very carefully, particularly when the cohort groups are small and sub-group numbers fluctuate substantially from one year to the next. This is one reason that it is useful to look at more than one cohort so that enrollment pat-terns by sub-groups can be identified.

Step 2

Use each template and fill-in the “Count of Students” cells (the blue highlighted cells) based on high school enrollment data, college enrollment data, program enrollment data, and/or any other data the team decides to collect. The percentages (in the red highlighted cells) are automatically calculated based on student counts. Therefore, it is important not to edit the red highlighted cells or the equations will be deleted.

Step 3

The graphs are automatically populated based on the data inserted into the templates. Once tables are complete, PTR team leaders can share the templates with team members and ask for their analysis. The tables and graphs can also be copied and pasted into a word docu-ment to send to team members. The decision about format may depend on team member familiarity and comfort-level using Excel software. Team members who have experience with Excel may find it advantageous to use this format because it provides the opportunity to modify data to simulate the impact on specified enrollment and outcomes of interest to the group.


Appendix EOutcomes and Equity Template Instructions

Template Set TwoOutcomes by Sub-Group


Three groups of Outcomes and Equity Template Sets (Excel Worksheets) have been cre-ated by OCCRL to report program enrollments and outcomes by sub-groups. The electronic templates are online at http://http://occrl.illinois.edu/projects/pathways/ptr-phase-two-outcomes-and-equity-assessment/.



TEMPLATE SET TWO: Outcomes by Sub-Group

Template Set Two shows outcomes for students in a secondary and/or postsecondary path-way. The student groups (sometimes called cohorts) may be at the secondary level, the postsecondary level, or start in secondary and continue to the postsecondary level. The pur-pose of these outcomes templates is to illustrate how PTR teams can identify which student sub-groups should be included in the project.

Template Set Two includes several outcomes-oriented templates, beginning with the fol-lowing four:

•• Fall-to-spring retention

•• Course completion

•• Certificate completion•• Degree completion

Teams should use these templates as examples to create new outcomes templates that align with the program and problem that their team has identified. These templates are structured to allow teams to study outcomes for sub-groups of students who are enrolled in the iden-tified pathway. The templates show data disaggregated by Race/Ethnicity, Special Popula-tions, Gender, Socioeconomic Status, and Age. Teams may adjust the templates to analyze other characteristics, such as first-generation status, English language learners, or special education status, by copying the templates to new worksheets and editing them to fit the PTR team’s project.


Step 1

The PTR team should determine the cohort(s) to analyze. For example, a team might iden-tify all students who first enrolled as majors in a postsecondary pathway in fall 2011 as a cohort. If enrollment figures for one semester are too low for analysis, a PTR team may de-cide to combine all the students who first enrolled in a postsecondary pathway in fall 2011, fall 2012, and fall 2013. Exactly what the PTR team should do depends on what it is trying to learn and how it wants to improve the selected pathway. Interpreting results based on small numbers can be problematic in this kind of analysis. In particular, teams should be cautious about generalizing results from a single small cohort. If combining groups is pos-sible to create a larger group, including looking at two, three or more cohorts then teams may feel more confident about concluding that patterns exist in the data than if they look at one cohort that has a small number of students. If combining multiple periods of time, teams should also consider whether unusual spikes or drops were seen in any single period. The team may want to look at the patterns in three cohorts that have been aggregated and in the three cohorts without disaggregating. Both of these studies may be informative to PTR teams.

Step 2

PTR teams need to select relevant outcomes measures from the PTR Outcomes Menu (see Engagement & Commitment Module), or select another outcome that relevant to the proj-ect. The four outcomes that have templates that automatically create tables and graphs are:

•• Fall-to-spring retention

•• Course completion

•• Certificate completion•• Degree completion

The Single Outcome Templates are made to record a single outcome measure, such as fall-to- spring retention, disaggregated by demographic characteristics. The purpose of these templates is to see the sub-group’s performance (by demographic characteristic) for select-ed outcome(s). This analysis should be considered a baseline analysis as it provides a snap-shot that is relatively limited, but it is a way to begin to determine whether patterns exist in student performance on outcomes pertaining to the pathway for specific sub-groups.

With the Course Completion Template, the team should choose one course that it believes is critically important to the identified pathway. Performance in this course is vital to stu-dents’ performance and matriculation. Examples could be a developmental (math, reading or writing) course that is key to entry into the program, a gatekeeper course (e.g., the first re-quired core CTE course or general education core requirement), a capstone course, or other similar course identified by the team. Most important, the chosen course should be one that is known to be critical to student progression through the pathway and to achieving a suc-cessful completion in the future.

With the Certificate Completion Template, the team will measure the rate of completion of students enrolled in a selected pathway that offers certificate options. The team will iden-tify a cohort of students (see Step 1) and track them for a specified amount of time using selected outcomes or performance measures. For some teams this may including Perkins Core Indicators and Performance Measures.


The Degree Completion Template is similar to the Certificate Completion Template. This template will measure the rate of Associate Degree attainment in the selected pathway at the end of a given time period, for the cohort of students identified by the team (see Step 1). The time period considered can be tied, if applicable, to the Perkins Core Indicators and Performance Measures.

With the Fall-to-Spring Retention Template, the team will measure the retention rate of students enrolled in selected program of study. The template is designed to follow a cohort of students over the specified period of time (see Step 1). The template can be changed to track retention over a year (Fall to Fall) or for other periods of time that are logical to the PTR team and to the problem it is attempting to solve with the PTR process.

Step 3

The outcomes templates have been populated with numbers to provide an example to help the PTR team understand how to interpret data. Then, use the templates by filling in the ap-propriate cells (the blue highlighted cells and the purple highlighted cells in the templates for special populations) based on the data your team has collected from an identified stu-dent group/cohort (as explained above). The percentages and totals (in the red highlighted cells) are automatically calculated and a graphic is also automatically prepared to display the results. It is important not to edit the red highlighted cells or the equations that create the results and automatically prepare the graphics will be deleted.

Step 4

As noted, the graphs are automatically populated based on the data inserted into the tem-plates. Once tables are complete, PTR team leaders can share their templates with team members and ask them for their analysis. (PTR teams may also want to enter data at a team meeting, so that the group understands where the numbers come from and how the spreadsheets can be used.) The decision about format may depend on team member famil-iarity and comfort-level using Excel software. Team members who have experience with Excel may find it advantageous to use this format because it provides the opportunity to modify data and see how outcomes of interest to the PTR team are affected. It is important for the cohorts to be creative! The results will be used for improvement purposes only.


Appendix FOutcomes and Equity Template Instructions

Template Set ThreeLongitudinal Enrollment by Sub-Group


Three groups of Outcomes and Equity Template Sets (Excel Worksheets) have been cre-ated by OCCRL to report pathway enrollments and outcomes by sub-groups. The electronic templates are online at http://http://occrl.illinois.edu/projects/pathways/ptr-phase-two-outcomes-and-equity-assessment/.



TEMPLATE THREE: Longitudinal Enrollment by Sub-Group

These templates are for longitudinal analysis of enrollment at the secondary and postsecondary levels, disaggregated by race and ethnicity sub-groups. They are more relevant for PTR Teams working with enrollment figures high enough to draw robust infer-ences. (If smaller numbers of students are enrolled, Template Set One is a more appropri-ate tool.) These templates allow for data to be tracked for a target group of students over time, for example, from secondary to postsecondary and from one year to the next. The templates also allow for comparisons to other groups that are determined by the PTR team to be relevant to the project.

Step 1

The first data table on each of the templates only looks at cohorts of students enrolled at a given point of time. The team has the flexibility to choose the point of time for the first table. The worksheet in Template Three tracks four types of cohorts: 1) the Postsecondary Pathway Student Cohort; 2) the Secondary Pathway Student Cohort; 3) the college group; and 4) the high school group. The Postsecondary and Secondary Pathway Student Cohorts should be defined by the teams as described above in the directions for Template Set One, Step 1. The college group is defined as all students in the community college. The high school group uses enrollment numbers from the area high school, and these enrollments can be acquired from numerous sources including from the EFE regional director, from the


high schools and from the state boards of education.

The remaining data tables are longitudinal and record head counts of students for each year, disaggregated by race & ethnicity, special populations, gender, socioeconomic status, and age sub-groups. The time period would optimally correspond to academic years. Teams enter data for each of the sub-groups. The table of Total Students Per Year is automatically generated.

Step 2

In each table (except Total Students Per Year) fill-in the “Count of Students” cells (the blue highlighted cells) based on high school enrollment data, college enrollment data, and path-way enrollment data. The percentages (in the red highlighted cells) and all information in the table for Total Students Per Year are automatically calculated based on sub-group student counts entered by teams. Therefore, it is important not to edit the red highlighted cells or the equations will be deleted. In the longitudinal analysis, teams can also select the length of time period considered, based on data availability.

Step 3

The graphs are automatically populated based on the data inserted into the templates. Once tables are complete, PTR team leaders can share the templates with team members and ask for their analysis. The tables and graphs can also be copied and pasted into a word docu-ment to send to team members. The decision about format may depend on team member familiarity and comfort-level using Excel software. Team members who have experience with Excel may find it advantageous to use this format because it provides the opportunity to modify data to simulate the impact on specified enrollment and outcomes of interest to the group.


Appendix GData Review Worksheet

Instructions: Based on your individual review of the data, answer the following questions to help guide your interpretation. This worksheet should be completed by individual team members before the team meeting in Step 3 of the Outcomes and Equity Assessment process.

1. What do you see as areas of strength? What outcomes have improved over the past few years?

2. Based on the disaggregated data, what outcome gaps have closed?

3. What are some areas for growth? What outcomes have declined over the past few years?

4. Based on the disaggregated data, what outcome gaps have been growing?

5. What aspects of these outcome data were unexpected or surprising? Why?

6. Is there anything that is unclear or confusing about the data?

7. How do aggregate and disaggregate program level outcome data compare to aggregate and disaggregate institutional level outcome data?


8. What are the limitations of these data?

9. After reviewing these data, what additional questions do you have?

10. Could some of these questions be answered by gathering perspectives from current or past students? If so, how might you build collection of these data through focus groups, surveys, or one-on-one interviews into your next steps?

11. Are there course sections, individuals, or cohorts that appear to perform better than their peers in the same demographic groups? If so, how can you learn about their strengths or positive experiences in a way that can inform your next steps?

12. Now that you have interpreted the data, speculate why you think student outcomes are equitable or inequitable. If applicable, speculate on why you think there are increases or decreases in student outcomes over time. Also if applicable, speculate on why you think student outcomes in the pathway are better or worse than student outcomes at the institutional level.

13. As you move into process assessment, how can your findings be clarified by exploring student-level process data in terms of participation in the pathway? For example, examining course-by-course student success or participation in learning communities, orientation, supplemental instruction, advising, etc.


Appendix HTeam Worksheet and Contribution to Charter

This Worksheet includes six main components and is part two of the Charter. These six components include:

1. Brief summary of activities2. Attachment that includes data collected (soft copy and hard copy)3. Bullet list of findings from data analysis and interpretation4. Problem description5. Team PTR goals6. Preliminary list of processes to examine in the next phase

The Team Leader should distribute this worksheet during the team meeting in Step 3 so team members can use this to guide their thinking and to guide the development of the team’s contribution.

Brief Summary of Activities: In two or three paragraphs, briefly describe the team’s activities. What did the team do, who was involved, and in what sequence did the teams’ activities occur?

Attachment of Data: Attach a copy (submit a soft copy and hard copy) of all data collected. This is the data distributed to team members and the data discussed during the Step 3 meeting.

Bullet List of Findings: This list represents the team’s analysis and interpretation of findings that emerges from individual review and the team meeting during Step 3. A bullet list is suggested so findings are represented in a readable format and are easily understood by diverse audiences. It is critical that teams include both nuanced findings (e.g., differences in outcomes according to student characteristics) and summative findings (trends or patterns across student outcomes and student characteristics) in this bullet list.

Problem Description: In two or three paragraphs, identify the primary problems identified by the team. This problem description can be conceptualized as the narrative version of the relevant findings from the data analysis and interpretation. This problem description will guide the remainder of the teams PTR work.

Team PTR Goals: Now that the team has described the problem, identify measurable goals that relate to the findings and problem description based on student outcomes. Identify outcomes based on student characteristics and the extent to which outcomes are equitable. List the student outcome, the existing status of the outcome, and provide short-term and long-term goals (See Table 4). Outcomes can be both quantitative and qualitative. Teams should return to these goals as their PTR work progresses to reevaluate the feasibility of the goals in light of their work in the following phases. It is critical to note that these should all be student outcomes and not policies, practices, or processes that affect student outcomes. In other words, goals should not address how student outcomes will be improved. Two examples are included in Table 4.


Table 4. Student Outcomes and Selection Worksheet

Outcome Existing Status Short-Term Goal Long-Term Goal

Example:

Fall to Spring Retention Rate for ALL Students in the pathway

74.3%Increase retention rate by 3-5% in one academic year.

Increase retention rate by 8-10% in five academic years.

Example:

Fall to Spring Retention Rate for Hispanic/Latino(a) Students in the pathway

54.8%Increase retention rate by 4-7% in one academic year.

Increase retention rate by 15-20% in five years.

List of Processes: The Process Assessment component of PTR will begin with a structured examination of key institutional or organizational processes the team believes are influencing student results. To start this work, teams are asked to brainstorm a short list of these processes that they see reflected in the problem description and/or goals.


Notes


Office of Community College Research and Leadership

University of Illinois51 Gerty Drive, 129 CRCChampaign, IL [email protected]://occrl.illinois.edu (217) 244-9390

outcomes & equity assessment › uw-s3-cdn › wp...english proficient economically...

Documents