kathy booth, rp group john hetts, long beach city college andrew lamanque, rp group what role does...

Kathy Booth, RP Group

John Hetts, Long Beach City College

Andrew LaManque, RP Group

What Role Does Transcript Analysis Play in Multiple Measures Assessment?

Regional Research GroupsOctober 19, 2012

2

Session Outcomes1. Clarify the current status of common assessment

efforts in the state

2. Explain how one college used an analysis of transcript data to change its placement practices

3. Share preliminary findings from a study replicating this analysis at 20 colleges

What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

3

STATEWIDE EFFORTS ON COMMON ASSESSMENT

Building common systems for sharing assessment data


4

SSTF Recommendation 2.1

Community colleges will develop and implement a common centralized assessment for English reading and writing, mathematics, and ESL that can provide diagnostic information to inform curriculum development and student placement and that, over time, will be aligned with the K-12 Common Core State Standards (CCSS) and assessments.

What’s Completion Got to Do with It? – April 2012

Statewide Efforts on Common Assessment: the Task Force

Statewide Efforts on Common Assessment: Some History

5What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

AB743Centralized Assessment Bill signed by the Governor in 2011 but a permanent funding source has yet to be identified.

2012 TTAC Recommendations, April 2012Develop a universally available assessment system that provides access to assessment results, as well as K-12 and CCC transcript data, to inform local placement.

CCC Assessment Committee, June 2012Bring together an ad hoc workgroup to flesh out the details of the data warehouse and bring back the results to this committee. This should include the crosswalk as well.

Statewide Efforts on Common Assessment: TTAC


• The topic of centralized assessment has highlighted the importance of utilizing a variety of data points in the process of student assessment and predictive behaviors.

• Since an ongoing funding source has yet to be

identified, the Centralized Assessment Advisory Committee approved the plan to pursue the development of a centralized data warehouse

• The idea of predictive analytics offers additional possibilities for the area of student placement and support services.

Statewide Efforts on Common Assessment: TTAC


• The Data Warehouse Work Group proposes to link data in phases. Phase 1 will focus on data that is accessible to the project team, Cal-PASS, EAP, CCCApply and MIS.

• A key challenge with these data sets will be to link student records.

• The project team recognizes the value of transcript information among community colleges and will work to develop a plan for collecting college transcript information.

• K-12 transcript information is dependent upon K-12 institutions implementing an electronic transcript system; currently no K-12 institution has a system in place.


Source: Tim Calhoon

Statewide Efforts on Common Assessment

Q & A

What questions do you have about the statewide work on common assessment?

Use the chat box on the left side of your screen to type in questions.


10

HOW THINGS GOT STARTED

Looking closer at assessment and placement


LBCC’s ResearchSought local answers to 3 questions:

• What predicts how students assess and place into developmental courses?

• What predicts how students perform in those courses?

• How well are placement and performance aligned?


Our Research

• Five longitudinal cohorts tracking more than 7,000 high school graduates who attended LBCC from a large local unified school district

• Simplified presentation of utility of 11th grade California Standards Test (CST) scores and high school grades for predicting:

• How students are assessed and placed into the levels of our developmental skills sequences

• How students perform in those classes


13

CST ELA (z) Eng Grade (12)

GPA (other)0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4 1.34x

.00

.30**

Predicting Placement

Alignment in English

* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10

CST ELA (z) Eng Grade (12)

GPA (other)0.0

0.2

0.4

0.6

0.8

1.0

.17*

.37***

.88x

Predicting Performance

14

CST Math (z)

Math GPA

GPA (12) GPA (11)0.0

0.2

0.4

0.6

0.8

1.0

.75x

.20

.00.07

Predicting Placement

CST Math (z)

Math GPA

GPA (12) GPA (11)0.0

0.2

0.4

0.6

0.8

1.0

.20*.25**

.73x

.21*

Predicting Performance

Alignment in Math

* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10

15

The Take-Aways

•Standardized tests best predict standardized tests

•Classroom performance best predicts classroom performance

•More information tells you more about a student than less information.


Evidence-Based Change:Fall 2012, Promise Pathways966 students who completed matriculation by May 5 received

• Pilots predictive placement model in English & math using multi-method, evidence-based assessment of students’ preparation and readiness

• Prescriptive, full-time course load Emphasis on early completion of foundational skills

Taken together, what does this mean for our students?

• Elephant in the subway slides


English: Full Implementation of Placement and Prescriptive EnrollmentAlternative vs. Traditional Placement

English 1 (or higher) One level below Three levels below

Traditional 14% 38% 47%

Pathways 56% 17% 27%

5%

15%

25%

35%

45%

55%

65%

14%

38%

47%56%

17%

27%

Traditional Pathways


Math: Partial Implementation of Placement (but only partial prescriptive enrollment)

Transfer Level Intermediate Algebra (1)

Algebra (2) Pre-Algebra (3)

Arithmetic (4)

Traditional 9% 27% 42% 20% 2%

Pathways 31% 30% 23% 16% 1%

5%

15%

25%

35%

45%

9%

27%

42%

20%

2%

31% 30%

23%

16%

1%

Traditional Pathways


Placement is Not Enough: Full-Time Students

Total Full-time0

200

400

600

800

1000

1200

1400

1600

1800 1650

400

1514

966

Typical LUSD 2012-2013


20

What LBCC Gained By Using Multiple Measures

• Gives students earlier but appropriate access to transfer level courses

• Increases the number of students that attain meaningful educational outcomes & decreases time to do so

• Challenges conventional wisdom about students’ readiness for college

• Opens new avenues of discussion about research and kick-starts opportunities for experimentation and innovation

• Provides concrete steps that any college—like yours—could take to dramatically improve all students’ futures


Q & A

What questions do you have about Long Beach City College’s research and placement changes?

Use the chat box on the left side of your screen to type in questions.


22

A BRIEF OVERVIEW

The Student Transcript-Enhanced Placement Project (STEPS)


23

What Are We Trying to Do?• Examine the value of using transcripts as part of

the assessment process• Create predictive models using Cal-PASS data to

study students who had already taken community college courses

• The models analyze how well transcript data predicts the first English & math courses students take and how well they do in them

• Recruit colleges to do local analyses to generate campus-specific insights and trigger CCC/K-12 conversations


24

Related Work• Brown & Conely. (2007). Comparing state high school

assessments to standards for success in entry-level university courses.

• Brown & Niemi. (2007). Investigating the alignment of high school and community college assessments in California.

• Belfield & Crosta. (2012). Predicting Success in College: The Importance of Placement Tests and High School Transcripts.

• Willett, Hayward, & Dahlstrom. (2007). Leveraging the CST’s.• Wurtz. (2007). Relations among high school math performance

and college placements and achievement.• Fuenmayor, Hetts, & Rothstein. (2011). Assessing Assessment:

Evaluating Models of Assessment and Placement.• English Curriculum Alignment Project (ECAP) between

Grossmont College and Grossmont UHSD.


25

What Is Happening Now?• Missing data weakened the predictive models

(this highlights the value of strengthening data sharing among segments)

• Most colleges are still working on their analyses

• Bottom line: So far, high school performance is partially predictive of college performance


26

In Englishtests

predict tests,grades and courses

matter but vary by college

27

Cox & Snell pseudo R-square ~ 0.35

0

1

2

3

4

5

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HS GPA*

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce

Predictor Variable Category

Predicting College English Level

Strongest

Intermediate

Weakest


28

In math, thespecific

classes taken in high schoolpredict placement

most strongly

29


0

1

2

3

4

5

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HS GPA*

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce


Predicting College Math Level

Strongest

Intermediate

Weakest


30

In English,

gradespredict

grades

31


0

1

2

3

4

5

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HSGPA*

CollegeCourseLevel

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce


Predicting College English Success

Strongest

Intermediate

Weakest


32

In math, success

predictors vary by

college

33


0

1

2

3

4

5

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HSGPA*

CollegeCourseLevel

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce


Predicting College Math Success

Strongest

Intermediate

Weakest


34

What Happens Next?

• Due to challenges with missing data, additional techniques will be applied with participating colleges

• We are launching a larger study using Cal-PASS data that will help answer additional research questions about using transcripts for assessment


What Can You Do?Replicate this research at your campus

• http://www.rpgroup.org/projects/STEPS or http://bit.ly/RPSTEPS

Learn more about research on placement exams• “Where to Begin? The Evolving Role of Placement Exams for Stu

dents Starting College” – Jobs for the Future/Achieving the Dream http://bit.ly/AlternativeAssessment

Find out more about Long Beach’s efforts• http://www.rpgroup.org/resources/promising-pathways

Find out more about the previous Cal-PASS study• http://www.rpgroup.org/content/2009-rp-statewide-research-

award-winner-using-california-standards-test-identify-remediation


http://www.rpgroup.org/projects/STEPS

http://bit.ly/RPSTEPS

http://www.achievingthedream.org/resource/where_to_begin_the_evolving_role_of_placement_exams_for_students_starting_college

http://www.achievingthedream.org/resource/where_to_begin_the_evolving_role_of_placement_exams_for_students_starting_college

http://bit.ly/AlternativeAssessment

36

Find Out More

Terrence Willett, RP [email protected]

John Hetts, Long Beach City [email protected]

mailto:[email protected]

kathy booth, rp group john hetts, long beach city college andrew lamanque, rp group what role does...

Documents