Developing Reading Proficiency Using Accuplacer as a Tool for Research and Assessment

Jody WorleyDirector of Institutional Research and Assessment

Margaret E. LeeDean of Student Services, Metro Campus

Tulsa Community College

League for Innovation in the Community CollegeInnovations Conference 2004

San Francisco, CAMarch 3, 2004

New statewide requirement• In 1994: OSRHE required all higher

education institutions to require readingcompetency for incoming student admission

• The regents allowed for institutionaldiscretion to implement the requirement

Assessing college readingAssessing college readingcompetencycompetency

• Can our students read?• How do we know?

Prevailing Assumption:Prevailing Assumption:

• Student success does not depend solely ontaking developmental courses.

THEREFORE:

• Developmental courses should be optional.

Tulsa Community College’s approach:Tulsa Community College’s approach:

• Mandatory assessment of reading skill atcollege entry

• Mandatory advisement of developmentaloptions for students below college level

• Optional enrollment in developmentalreading courses

Time passed, things changed:Time passed, things changed:

• 1999: new student information systemchanged TCC’s enrollment procedure

• Increasing sense of institutional accountability• Increasing concern for student success

Links Between Assessment of Learning Outcomes,Planning and Budgeting

TCC Mission

TCC Strategic Vision

Assessment

Budgeting

Planning Accountability

GoverningBoards

AccreditingAgencies

• Entry Level

• General Education

• Discipline/Program Outcomes

• Support Areas

• Student Satisfaction

Linkage between the Institution’s Mission Statement andUse of Assessment Results

TCC’s Strategic Vision

Goal Statements:

Focused statementsfor student outcomesassessment efforts.

Intended

Outcomes / Objectives

Statements of what iscurrently being assessed -descriptions of what weintend for students to . . .

1. Know (cognitive)

2. Think (affective)

3. Do (behavioral)

Means ofAssessment

What are thecriteria formeeting theobjectives?

How will weknow if theobjectiveshave beenmet?

Results

Documentresultsfrom thevariousassessmentmeasures.

TCC’s Mission

Entry Level Assessment Committee’s Goals:Entry Level Assessment Committee’s Goals:

• Effectively apply the OSRHE readingcompetency requirement

• Allow maximum student autonomy

• Encourage reading skill development inways that improve both student attainment andperformance

Guiding PrinciplesGuiding Principles

• Identify and “stick to” your mission.• State your assumptions a priori• Listen to the data, but rely on your expertise

and experience.• Be open and willing to change directions

midstream, or at least consider alternatives.

New Hypothesis:New Hypothesis:

• Student success is directly connected totaking developmental courses.

THEREFORE:

• Developmental courses should be mandatoryand should precede college level work.

Proposal:Proposal:

• Restrict enrollment options for students whoneed reading development

• Test restrictions for effectiveness throughperformance data

• Evaluate the effectiveness of thedevelopmental program

Assessment Tool:Assessment Tool:

The College Board’s AccuplacerComputerized Placement Test

Successful placement program in mathematics:• Validation studies over 7 years• Placement program for:

– College level mathematics– Developmental mathematics

Assessment Tool:Assessment Tool:

The College Board’sAccuplacer CPT Reading Comprehension

• College-Level: > 80• Underprepared: 66 – 79

(1 developmental Reading Course)

• Seriously Deficient: < 66(2 developmental Reading Courses)

Initial Faculty Recommendations:Initial Faculty Recommendations:

• Restrict enrollment in certain courses tostudents with demonstrated college levelreading competency.

• Require reading skill development BEFOREenrollment in college courses

• Require qualifying cut score for Reading II• Measure improvement in Reading I and II• Consider mandatory development

The StudyThe Study

• Select the cohort

• Classify entering students by their developmentalpath

• Analyze academic attainment and performanceafter 3 years

Preliminary Results: AttainmentPreliminary Results: Attainment

• Students who developtheir skills earn morehours than those whoneed to develop butdon’t.

• Development makesa difference forstudent persistence.

3555613248N =

College ReadyNo DevelopmentDeveloped Skills

Tota

l Earn

ed H

ours

as

of S

um

mer

1998

20

18

16

14

12

10

8

6

Preliminary Results: PerformancePreliminary Results: Performance

3555613248N =

College ReadyNo Dev.Developed Skills

Pe

rfo

rma

nce

(q

pts

_2

/eh

rs_

2)

as

of

Su

mm

er

19

98

2.6

2.4

2.2

2.0

1.8

1.6

1.4

1.2

• Students who developtheir skills outperformthose who need todevelop but don’t.

• Development makes adifference for studentperformance.

What we learned:What we learned:

• Development matters!

• Student success is directly related to takingdevelopmental courses.

THEREFORE:• Development should be mandatory.

Conclusion #1:Conclusion #1:• Developmental courses improve student success.

THEREFORE:• Developmental courses should be mandatory.

Recommendation #1:Recommendation #1:• Block enrollment for reading proficiency as originally

planned in preliminary recommendations.

New Question:New Question:

• Should development be complete beforeenrolling in restricted courses?

• Faculty recommendation: YES

Developmental options:

102146 613

3555

0500

1000150020002500300035004000

Dev. O

nly

Concu

rren

tN

o Dev

.

Colle

ge L

evel

• Group 1: onlydevelopmental courses

• Group 2: developmentaland college level courses

• Group 3: nodevelopmental courses

• Group 4: entered atcollege reading level

Total N = 4416

Assessment Tool:Assessment Tool:

The College Board’sAccuplacer CPT Reading Comprehension

• College-Level: > 80• Underprepared: 66 – 79

(1 developmental Reading Course)

• Seriously Deficient: < 66(2 developmental Reading Courses)

Results: AttainmentResults: Attainment

• Students who developedskills while takingcollege level coursesearned more hours thanstudents who developedskills prior to taking anycollege level courses.

3555613146102N =

College Ready

No Development

Dev Concurrent

Dev Only

Tota

l Ear

ned

Hour

s as

of S

umm

er 1

998

30

25

20

15

10

5

0

Results: PerformanceResults: PerformanceReading development

with college levelcoursework:

• Outperform studentswho take onlydevelopmental courses.

• Outperform studentswho do not develop theirskills.

3555613146102N =

College Ready

No Development

Dev. Concurrent

Dev. Only

Perfo

rman

ce a

s of

Sum

mer

199

8

2.6

2.4

2.2

2.0

1.8

1.6

1.4

1.2

1.0

New question:New question:

Why do developmental students who take collegecourses outperform students who only take

developmental courses?

Hypothesis:The disparity in performance derives from

differences in student skill level.

Attainment and Deficiency LevelAttainment and Deficiency Level

355575716042N =College Ready

SD & rem concurrent

Up & rem concurrent

SD & rem in dev only

Up & rem. dev only

Tota

l Hou

rs a

s of

Sum

mer

199

8

30

20

10

0

• Seriously deficient studentsattained more hours thanunderprepared students whenthey took college levelcourses withdevelopmental courses.

• Seriously deficient studentswho developed skillsconcurrently also attainedmore hours than collegeready students.

Performance and Deficiency LevelPerformance and Deficiency Level• Contrary to our expectations, Level of

deficiency did not affectperformance among students whodeveloped their skills.

• As expected, underprepared studentsoutperformed students with seriousdeficiencies among students did notdevelop their reading skills.

Conclusion #2:Conclusion #2:• Differences in student success are NOT attributable

SOLELY to taking developmental courses.

• Enrollment in college level courses contributes tostudent success, IF development is taking place.

Recommendation #2:Recommendation #2:• Permit concurrent enrollment in college level

courses, even for seriously deficient students.

Conclusion #3:Conclusion #3:

Recommendation #3:Recommendation #3:

• In addition to taking developmental courses,student success is affected by other factors wehave not yet identified.

• Examine student experience within thedevelopmental reading program to identify otherkey factors related to student success.

• Pre- and post-test for reading skill in alldevelopmental reading courses.

Unanswered question:Unanswered question:

• Development improves student success….

BUT:

• Why do college-ready students still outperformstudents who take one developmental course?

Follow-up study:Follow-up study:

How much improvement in reading skill canstudents expect from participating indevelopmental reading courses?

Pre- Post-Testing on Placement TestsPre- Post-Testing on Placement Tests

Assumption: If developmental courses aredesigned to remove reading deficiencies, post-test scores on placement test should be higherthan pre-test scores (hopefully “college-level”).

Reading I Results: 2002-2003Reading I Results: 2002-2003N Avg.

Pre-test ScoreAvg.

Post-testScore

Sig.

Nelson-DennyComprehension

291 33.11(GE = 9.3)

37.07(GE = 9.7)

YES

Nelson-DennyTotal

292 64.94(GE = 9.1)

71.30(GE = 9.6)

YES

CPT ReadingComprehension

340 50.69 56.02 YES

Reading II Results: 2002-2003Reading II Results: 2002-2003N Avg.

Pre-test ScoreAvg.

Post-testScore

Sig.

Nelson-DennyComprehension

352 38.65(GE = 10.1)

44.03(GE = 11.5)

YES

Nelson-DennyTotal

351 77.19(GE = 10.1)

85.19(GE = 11.3)

YES

CPT ReadingComprehension

479 66.51 65.54 NO

What are these results telling us?What are these results telling us?• These results suggest that while the raw scores improve

on the post-test, the deficiencies that were identified onthe pre-test might still be present.

However,We know from the previous study that successful

completion of developmental courses correlates withsuccess in college-level courses.

ConclusionsConclusions• Something from the developmental course

experience contributes to college success, butthe tests do not measure whatever that is.

• Although the placement tests do not appearto adequately measure improvement, theyare effective for identifying students whoneed to develop skills and ensuring that theyget the help they need.

Still to be explored:Still to be explored:• Why do college ready students outperform

students who meet developmentrequirements?

• What college level courses do studentstake concurrent with developmental courses?

• Why do seriously deficient students attainmore credit hours than underpreparedstudents?

• How can we appropriately place students inour developmental reading program?

Action taken:Action taken:

• Entry level assessment committee shared informationwith developmental reading discipline review

• Faculty innovation grant

• Federal grant proposals

Decisions Based on EvidenceDecisions Based on Evidence

• Enrollment Practices based on StudentPerformance

• Ongoing collaboration between the Officeof Institutional Research and the Entry-Level Assessment Committee to studyplacement in mathematics and writing

Contact Information:Contact Information:

Jody Worleyjworley@tulsacc.edu

918 - 595 - 7925Director of Research & Assessment

Tulsa Community College6111 E. Skelly DriveTulsa, OK 74135

End of presentationEnd of presentation