“When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations on the assessment of online learners.”

Oosterhof, et al. (2008) Assessing Learners Online

Assessing Online Instruction at Carteret Community College

Donald Staub, Title III Director Carteret Community College

?

?

?

?

?

?

The Lineup

• DL at CCC

• The Need to Assess

• How We’re Doing It

• Conclusions & Solutions

• Discussion

DL @ CCC• DL is 10 years old

• 34 programs @ CCC

• ~1700 enrollment SP09

• Fall 2008• 348 total sections taught

• 270 seated sections

• 61 Internet

• 15 Hybrid

The Need to Assess

• SACS

• Common Sense• it’s the right thing to do

• we need some answers

“Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”

(CS 3.3.1)

COC Distance Education Policy Statementhttp://www.sacscoc.org/pdf/081705/distance education.pdf

DL @ CCC• 270 seated sections

• 73% success / 22% failed or withdrew (15% W)

• 61 internet sections• 65% success / 33% D, F, WD (16% W)

• 15 hybrid sections• 81% success / 18% D, F, WD (9% W)

• FA07 internet: 59 sections; 34% D, F, W

• FA06 internet: 48 sections; 34% D, F, W

How We’re Doing It

• QAP

• Grade Distributions

• ILLOs & PLLOs

• Same instructor

The QAP(Quality Assessment Plan)

http://ccctitle3.wordpress.com/

• Developed in-house (using models)

• Peer-reviewed by content specialist

• Stipends

• All current online courses have been reviewed

• All new online courses must be reviewed

http://ccctitle3.wordpress.com/

The QAP Score Card

The QAP Score Card: Learning

Grade Distributions(Fall 2008)

TOTAL ENROL A B C D F WD WD*

Seat 3643 0.31 0.26 0.14 0.72 0.04 0.07 0.06 0.09 0.26IN 1159 0.32 0.22 0.12 0.65 0.05 0.12 0.08 0.08 0.33HY 188 0.48 0.22 0.11 0.81 0.04 0.05 0.02 0.06 0.18

ILLOs

• Communication

• Computer Literacy

• Critical Thinking

• Humanities & Fine Arts

• Information Literacy

• Personal Growth & Responsibility

Computer Literacy - CIS 110Outcome: Students demonstrate proficiency in the use of computer technology including application software, system software, and internet browsers.

Assessment: Final Exam in CIS 110. This exam consists of a hands-on skills assignment for Word, Excel, Access and PowerPoint and a 25 question Multiple choice assignment with questions regarding computer hardware, system software, internet browsers and other related computer terminology.

Computer Literacy 5 Sections: CIS 110 3-WB; 2-IN

• Attrition: IN01: 35% IN02: 26% WB03: 17% WB04: 28% WB05N: 18%

• Cross-section comparison (i.e. one course, different instructors)

Student Name Word pts. (20%)

Excel pts. (15%)

PPT pts. (10%)

Access pts. (15%)

Integration pts. (15%)

Theory pts. (25%)

Final Score 75%

A 19 14 10 12 15 15 85 1B 17 14 10 10 13 21 85 1C 17 12 10 15 15 25 94 1D 10 7 0 0 0 12 29E 15 15 10 14 15 16 85 1F 19 15 10 15 15 23 97 1G 16 15 10 12 15 20 88 1H 19 15 10 15 15 24 98 1I 20 12 10 10 12 22 86 1J 20 12 10 13 15 25 95 1K 10 15 10 15 8 20 78 1L 19 15 10 15 15 24 98 1M 15 15 10 15 15 16 86 1N 14 11 10 12 15 24 86 1

Average 16.42857 13.35714 9.285714 12.35714 12.92308 20.23077 8582.14% 89.05% 92.86% 82.38% 86.15% 80.92%

% of students >=75 92.86%

Word pts. (20%)

Excel pts. (15%)

PowerPoint pts. (10%)

Access pts. (15%)

Integration pts. (15%)

Theory pts. (25%) Final Score

IN01 80 73.8 95.7 64.7 83 81 79IN02 91 95 100 79 94 81.9 89WB03 91 75.5 96 78 88.5 80.8 84.6WB04 92.5 89 100 85 92.7 77 80.9WB05 82 89 92.8 82 86 80.9 85

Computer Literacy - CIS 110Same instructor teaching one section of each

Spring 2008 online <75% seated <75%# of Ss @ 10% 24 16Final # of Ss 16 12Attrition 25% 25%Word (20) 15.75 9 17.8 5Excell (15) 12.3 5 12.2 5PPT (10) 6 16 6.3 11Access (15) 13.4 6 12.75 6Integration (15) 13.25 4 13.8 4Theory (25) 19.29 3 20.5 3Final (100) 73.3 6 81.2 6

Personal Growth & Responsibility ACA 115

Outcome: Students demonstrate the ability to understand and manage self effectively.

Points Understanding Self ~~ Grading Criteria PointsEarned

30Points

(6 pts. each)

1. Identify long term and short term educational goals: a. Two (2) Long Term: 1 year—5 years b. Three (3) Short Term: 1 week—16 weeksCompleted Goal Development Form must be submitted.

20Points

(5 pts. eachfor results + 5pts. total forexplanation)

2. Evaluate the results of the following inventories in relation to your educational goals: a. Career Inventory b. Learning Styles Inventory c. Personality InventoryResponses should be written in paragraph form using completesentences and appropriate grammar.

20Points

(4 pts. pervalue with

explanation)

3. Identify five values that shape your belief system. Explain why these values are important to you.

Responses should be written in paragraph form using completesentences and appropriate grammar.

Points Managing Self ~~ Grading Criteria PointsEarned

15Points

(3 pts. each)

Advisement Status Sheet1. Placement Test or SAT Scores2. Distance Learning Orientation3. Keyboarding Test4. Computer Lit eracy Test5. Advisor/Program Coordinator Information

Completed Advisement Status Sheet must be submitted.

15Points

(5 pts. each)

Two Semester Class Schedule a. Complete the curriculum check sheet for your Program of Study (Major) by checking courses currently enrolled in and indicating the grade for courses completed. College Preparation courses should be listed on a separate piece of paper and attached to the check sheet. b. Create a trial course schedule for Spring 2009. c. Create a trial course schedule for Summer 2009 or for Fall 2009.

Personal Growth & Responsibility ACA 115

Personal Growth & Responsibility ACA 115

Advisement Project

• In the fall of 2008, 79% of all ACA students (10 sections) scored 80 or above.

• For online sections (2 sections), 70% (7 out of 10) scored 80 or above.

Personal Growth & Responsibility PSY 150

1

Missing,inadequate ormarginallycompleted

2

Acceptable oradequately metrequirements

3

Exceptional orexceededrequirements

Pointsearned

Studentdeveloped anaction plan forpersonal growthor responsibility.

No action plan.Hint of actionplan withoutspecifics.Plan does notrelate to selfhelp book.Lacks timeline.

Action plan hasspecifics andrelates to selfhelp book.Action plan hastimeline.

Action planmeets acceptablestandards as wellas includesadditionalresearch beyondself help book.

Studentidentifiedobstacles whichcould preventgrowth orchange andidentified how toovercome saidobstacles.

Obstacles notidentified.Only oneobstacleidentified.Obstaclesidentified butlacks how toovercome them.

Student identifiesat least oneobstacle and planfor overcomingobstacle, butlacks depth.

Identifiedmultipleobstacles anddetermined howto overcomethem.

Studentanalyzed howhis or her lifewould beimproved by thischange.

Failed todemonstrateawareness ofhow life will beimproved byproposedchange.

Student mentionschange, butanalysis andexplanation arecursory.

Demonstratedawareness ofhow life will beimproved byproposed change.

Row 1 2+ 3/3Sec 04 7/10 (70%) 2/10 (20%)IN06 4/6 (66%) 1/6 (16%)IN07 4/6 (66%)

Row 2 2+ 3/3Sec 04 9/10 (90%) 5/10 (50%)IN06 1/6 (16%)IN07 5/6 (83%) 3/6 (50%)

Row 3 2+ 3/3Sec 04 7/10 (70%) 5/10 (50%)IN06 3/6 (66%)IN07 6/6 (66%) 2/6 (33%)

PLLOsSame instructor teaching one section of each (take a look at a common assignment from each section)

• ACA 115

• ACC 120

• BUS 110

• ECO 251

• EDU 119

ACA 115 -

Learning Styles Project

<75%Instructor 1 WB02 WB05 IN10

18% 41% 36%

Instuctor 2 WB01 IN096% 24%

Students identify and explain the strategy used to prepare for an exam.

ACC 120

BUS 110

ECO 250

EDU 119

4 test test items related to: working/communicating with parents, parental rights, and regulations dealing with parental contact/information.

Item # # incorrect– online

# incorrect- seated

4 1 021 0 022 2 223 1 1

n=20 n=12

Conclusions & Solutions

“The largest issue confronting online assessments is that of the dynamics that occur when the instructor and student do not share the same immediate space.”

Milam, et al. (2004). In, Developing and Implementing Assessment of Student Learning Outcomes

• Instructors who don’t treat online & seated equitably (e.g. additional support to seated sections)

• Using the observation instrument correctly. Some of this was a direct result of them using a document that was scanned and they had to print/copy to use. I believe some of the students had trouble with the process. (EDU 119)

• Math & the fax machine

“The seated students did however have the opportunity to submit their observation checks sheets for revision if needed.”

“Students will no longer have to print the inventory results. Instead, they will write a narrative explaining their results.”

“Students will no longer need to submit a curriculum check sheet (on-line students had difficulty accessing them from the website).”

Discussion

“In many instances, the assessment of an online course is not in alignment with the type of instruction that may have occurred in the course.”

Palloff & Pratt (2009) Assessing the Online Learner

Donald Staub Title III Director

Carteret Community Collegestaubd@carteret.edu

252- 222-6010

http://ccctitle3.wordpress.com/