when learners are at a distance, the careful design of assessments is particularly important,...
TRANSCRIPT
![Page 1: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/1.jpg)
“When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations on the assessment of online learners.”Oosterhof, et al. (2008) Assessing Learners Online
![Page 2: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/2.jpg)
Donald Staub, Title III Director Carteret Community College
?
?
?
?
?
?
Assessing Distance Learning: Are they learning what we
think they are?
![Page 3: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/3.jpg)
The Lineup
• DL @ CCC
• Why Worry about DL Assessment?
• How We’re Doing It
• The DL Program
• DL in Programs
• Barriers, Conclusions & Solutionsccctitle3.wordpress.com
![Page 4: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/4.jpg)
DL @ CCC• 33 programs @ CCC
• FA10 enrollment ~1800
• DL is 10+ years old
• FA10
• 336 total sections taught
• 243 seated sections (72%)
• 64 Internet (20%)
• 29 Hybrid (8%)
• 58% FT / 42% PT
![Page 5: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/5.jpg)
Why worry about DL assessment?
Reason #1: Student SuccessReason #2: Persistence(btw: success vs. retention vs. persistence)
@ the College-Level:
![Page 6: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/6.jpg)
@ the Course-Level:
Reason #3: Generalizability
(c.f. Oosterhof, et al, 2008)
“Generalizability is the degree to which learner performances that are observed generalize to performances that are not observed.” [i.e. learning]
![Page 7: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/7.jpg)
SACS
Reason #4
![Page 8: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/8.jpg)
“Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”
(CS 3.3.1)
COC Distance Education Policy Statementhttp://www.sacscoc.org/pdf/Distance%20and%20correspondence%20policy%20final.pdf
![Page 9: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/9.jpg)
The (former) Guidelines
![Page 10: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/10.jpg)
![Page 11: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/11.jpg)
2009-2012 Strategic Plan
We will create a culture where all institutional actions are focused on improving student learning and success and leading to student goal completion.
Q1: Is the distance education program reflected in the institution’s strategic (or master) plan?
PM 16—Success and Withdrawal Rates by Instructional modalities
![Page 12: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/12.jpg)
Q2: Is there evidence that outcomes for the program have been identified?
DL-AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.
DL-AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.
Other possibilities: Growth rate, Success rates, Infrastructure-related….
![Page 13: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/13.jpg)
AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.
Assessment: Quality Assessment Plan, Success/ Persistence Rates, Student Surveys/Evaluations.
AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.
Assessment: The number of internet/hybrid sections offered.
Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?
![Page 14: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/14.jpg)
The QAP(Quality Assessment Plan)
• Developed in-house (using models)
• Peer-reviewed by content specialist
• Stipends (T3)
• All current online courses have been reviewed
• All new online courses must be reviewed/ certified
http://web.carteret.edu/keoughp/TitleIII/FinalQAP.xls
![Page 15: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/15.jpg)
The QAP Score Card
![Page 16: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/16.jpg)
The QAP Score Card: Learning
![Page 17: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/17.jpg)
• 248 seated sections
• 72% success / 25% D, F, WD (14% W)
• 62 internet sections
• 68% success / 31% D, F, WD (16% W)
• 22 hybrid sections
• 75% success / 24% D, F, WD (9% W)
• FA08 internet: 61 sections; 67% success
• FA07 internet: 59 sections; 66% success
• FA06 internet: 48 sections; 66% success
Success - Fall 2009
![Page 18: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/18.jpg)
Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?
• Modality Comparison
• Grade Distributions
• ILLOs (Gen Ed outcomes)
• Program Reviews• “Evaluate Instructional Modalities”
![Page 19: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/19.jpg)
Modality Comparison
Institutional-level
![Page 20: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/20.jpg)
Modality Comparison
Gateway courses
![Page 21: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/21.jpg)
Modality Comparison
Program-level
![Page 22: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/22.jpg)
Grade Distributions
FA 08
ModalityTotal Enrol A B C
Success Rate D F WD WD*
TRAD 3643 0.31 0.26 0.14 0.72 0.04 0.07 0.06 0.09 0.26IN 1159 0.32 0.22 0.12 0.65 0.05 0.12 0.08 0.08 0.33HY 188 0.48 0.22 0.11 0.81 0.04 0.05 0.02 0.06 0.18
FA09
ModalityTotal Enrol A B C
Success Rate D F WD WD*
TRAD 3714 0.33 0.26 0.13 0.72 0.04 0.08 0.06 0.08 0.26IN 1362 0.31 0.24 0.12 0.68 0.05 0.1 0.07 0.09 0.32HY 256 0.41 0.18 0.14 0.74 0.04 0.1 0.08 0.03 0.24
![Page 23: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/23.jpg)
ILLOs (Gen Ed outcomes)
1. Communication
2. Computer Literacy
3. Critical Thinking
4. Humanities & Fine Arts
5. Information Literacy
6. Personal Growth & Responsibility
![Page 24: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/24.jpg)
Computer Literacy - CIS 110
Outcome: Students demonstrate proficiency in the use of computer technology including application software, system software, and internet browsers.
Assessment: Final Exam in CIS 110.
![Page 25: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/25.jpg)
Word pts. (20%)
Excel pts. (15%)
PowerPoint pts. (10%)
Access pts. (15%)
Integration pts. (15%)
Theory pts. (25%)
Final Score
IN01 80 73.8 95.7 64.7 83 81 79IN02 91 95 100 79 94 81.9 89WB03 91 75.5 96 78 88.5 80.8 84.6WB04 92.5 89 100 85 92.7 77 80.9WB05 82 89 92.8 82 86 80.9 85
Computer Literacy - CIS 110 Section-by-section analysis
![Page 26: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/26.jpg)
Computer Literacy - CIS 110Same instructor teaching one section of
eachSpring 2008 online <75% seated <75%# of Ss @ 10% 24 16Final # of Ss 16 12Attrition 25% 25%Word (20) 15.75 9 17.8 5Excell (15) 12.3 5 12.2 5PPT (10) 6 16 6.3 11Access (15) 13.4 6 12.75 6Integration (15) 13.25 4 13.8 4Theory (25) 19.29 3 20.5 3Final (100) 73.3 6 81.2 6
![Page 27: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/27.jpg)
Personal Growth & Responsibility ACA 115
Outcome: Students demonstrate the ability to understand and manage self effectively.
Results Spring 2009: 131 total projects submitted; 81% scored 80 or above.
Seated 115: 85% scored 80 or above Online 115: 81% scored 80 or above
Assessment: Advisement assignment in ACA 115 & 118 evaluated by rubric.
![Page 28: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/28.jpg)
Q4: Has the institution developed student learning competencies for the courses/programs offered by distance education? If these are the same competencies for courses/programs offered by “traditional” methodologies, is assessment identified for distance learning students separate from students taking courses by “traditional” methodologies?
Yes! … No?…maybe??
• PLLOs
• Course-Level Assessment
![Page 29: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/29.jpg)
PLLOs (SLOs)
Same instructor teaching one section of each (take a look at a common assignment from each
section)
• ACC 120
• BUS 110
• ECO 251/252
![Page 30: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/30.jpg)
ACC 120
![Page 31: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/31.jpg)
BUS 110
ECO 250
![Page 32: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/32.jpg)
Course-level assessment
• ART 111
• HIS 111
• RCP 114
![Page 33: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/33.jpg)
ART 111
Grade Evaluation ( Percentages and Points )
AssignmentCategory
Number ofAssignments
% of ea.towardFinalGrade
Total pointsgenerated by ea.
individualassignment
Total % ofCategory inFinal Grade
Total Pointsof Category
in Final grade
WeeklyQuizzes
14 2.5% ea. 80 35% 1,120
DiscussionBoard
16 2.5% ea. 80 40% 1,280
Mid TermEssay
1 10% ea. 320 10% 320
Final Exam 1 15% ea. 480 15% 480
Totals 32 100% 3200
![Page 34: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/34.jpg)
A - (100 - 90): - You posted 3 or more times and you completed the Assignment with a well thought through response. (The more you post MEANINGFUL responses over 3 the higher your grade. Though welcome and expected, please understand that a One or Two sentence "feel good" post such as (... Great response Michaela!!) or (... I love how you put the meaning and then went down and talked about them. it made it easy to understand!!) will count for very little towards your grade.
![Page 35: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/35.jpg)
HIS 111
Essays (online) vs. Research Paper (seated) … same rubric
![Page 36: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/36.jpg)
RCP 114
![Page 37: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/37.jpg)
Conclusions
“I got rid of [the discussion board]…it’s a pain in the ass.”
“I have to pull teeth to get them to post even when they get a score”
20-30% of entire grade is Db…“but it’s not really a discussion board… it’s more of an assignment tool”
![Page 38: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/38.jpg)
“Participation is reserved for those on the bubble.”
“There’s nothing like a good old test. It averages. Students can’t argue the grade. They’re either right or wrong.”
![Page 39: When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513bdbf55034646298b4816/html5/thumbnails/39.jpg)
Donald Staub Title III Director
Carteret Community [email protected]
252- 222-6010
http://ccctitle3.wordpress.com/