Part I: The 3 T’s of Online Assessment: Part I: The 3 T’s of Online Assessment: Tools, Techniques, and (Saving) TimeTools, Techniques, and (Saving) Time

Curtis J. Bonk, Professor, Indiana UniversityPresident, CourseShare

http://php.indiana.edu/~cjbonk; cjbonk@indiana.edu

Vanessa Paz Dennen, Assistant ProfessorFlorida State University

http://www.vanessadennen.com, vdennen@fsu.edu

Session ObjectivesSession Objectives

Detail online assessment techniques Discuss how to match learning activities with

learner assessments Examine instructor time and comfort issues Discuss ways to limit and detect cheating and

plagiarism Document online tools and resources for

assessment

Online Assessment Online Assessment TechniquesTechniques

(with some time-saving(with some time-saving

tips added in…)tips added in…)

Is this motivating? Is this motivating? How would you feel?How would you feel?

You take an online class.You take an online class. You read some Web pages.You read some Web pages. Maybe you watch some videos or hear some Maybe you watch some videos or hear some

audio clips.audio clips. Maybe you ponder some study review Maybe you ponder some study review

questions.questions. You take a multiple choice test online.You take a multiple choice test online. You receive an automated score on the test.You receive an automated score on the test. Class is over.Class is over.

How about this scenario?How about this scenario?

You take an online class.You take an online class. You “meet” your fellow students on the d-board.You “meet” your fellow students on the d-board. You read some materials. You find and share You read some materials. You find and share

some materials too.some materials too. You participate in some discussions of course You participate in some discussions of course

concepts.concepts. You take a multiple choice test.You take a multiple choice test. You receive an automated score on the test.You receive an automated score on the test. Class is over.Class is over.

Commentary on Scenario 1Commentary on Scenario 1

No interaction with peers. No interaction with peers. Students don’t feel “missed” if they don’t Students don’t feel “missed” if they don’t

participate.participate. Not clear why course is online (except Not clear why course is online (except

perhaps for media elements).perhaps for media elements). Potential for immediate feedback is nice -- Potential for immediate feedback is nice --

but assessment format is limiting.but assessment format is limiting.

Commentary on Scenario 2Commentary on Scenario 2

Interaction with peers is great. Serves as a Interaction with peers is great. Serves as a motivator.motivator.

Community is likely to develop.Community is likely to develop. Students will feel involved and important if Students will feel involved and important if

they share examples and resources.they share examples and resources. Assessment format may not be well Assessment format may not be well

aligned given the activities.aligned given the activities. Class lacks closure in a manner Class lacks closure in a manner

appropriate to the activities.appropriate to the activities.

Assessment and LearningAssessment and Learning

Course objectives, activities, and Course objectives, activities, and assessments should be in alignmentassessments should be in alignment This tends to be an issue in courses This tends to be an issue in courses

regardless of medium.regardless of medium. Example:Example:

In class students conduct a debateIn class students conduct a debate Students are tested on their ability to recall Students are tested on their ability to recall

factsfacts

Mis-aligned Online Learning Mis-aligned Online Learning and Assessmentand Assessment

A not-uncommon scenarioA not-uncommon scenario Discussion is used as a learning activityDiscussion is used as a learning activity Students are required to participateStudents are required to participate Participation is noted by how many messages were Participation is noted by how many messages were

composed by a studentcomposed by a student But does this method measure learning?But does this method measure learning?

Common Online Common Online Assessment ComplaintsAssessment Complaints

Instructor perspectiveInstructor perspective There’s too much to assess!There’s too much to assess! I don’t know what activities to assess!I don’t know what activities to assess! I don’t know if students really are ready for the I don’t know if students really are ready for the

test!test! How do I know the student actually did the How do I know the student actually did the

work/took the test?work/took the test?

Common Online Common Online Assessment ComplaintsAssessment Complaints

Student perspectiveStudent perspective If they’re supposed to discuss, why If they’re supposed to discuss, why

doesn’t that count as part of their grade?doesn’t that count as part of their grade? If they’re just supposed to do something, If they’re just supposed to do something,

why does quality matter?why does quality matter? I just got a number, no feedback.I just got a number, no feedback. I didn’t get participation feedback.I didn’t get participation feedback.

The Feedback IssueThe Feedback Issue

Students participating in online activities look for Students participating in online activities look for feedback to know:feedback to know: A. the instructor is reading their contributionsA. the instructor is reading their contributions B. their participation is valuedB. their participation is valued C. their participation is adequate, in terms of quality C. their participation is adequate, in terms of quality

and quantityand quantity Feedback need not be individualized to be Feedback need not be individualized to be

effectiveeffective Whole class commentary provided on a regular basis Whole class commentary provided on a regular basis

was found to be just as satisfactory from the student was found to be just as satisfactory from the student point of view (Dennen, 2001)point of view (Dennen, 2001)

The Assessment IssueThe Assessment Issue

Often, online activities go Often, online activities go unassessedunassessed ““Add-on” syndrome: Adding an Add-on” syndrome: Adding an

online activity to a previously online activity to a previously designed class because it designed class because it sounds like a good ideasounds like a good idea

The Assessment IssueThe Assessment Issue

Students are more likely to participate when Students are more likely to participate when then know there is impact on their gradethen know there is impact on their grade Direct impact: graded on participation Direct impact: graded on participation

(quality, quantity or both)(quality, quantity or both) Indirect impact: participation should Indirect impact: participation should

bolster performance on other assessmentsbolster performance on other assessments Students quickly become aware if an Students quickly become aware if an

online activity is not related to assessed online activity is not related to assessed learning objectiveslearning objectives

The Assessment Issue, Cont.The Assessment Issue, Cont.

Sometimes the wrong things are assessedSometimes the wrong things are assessed Examples:Examples:

Assessing students’ online moderation Assessing students’ online moderation skills when the course topic/learning skills when the course topic/learning objectives have nothing to do with online objectives have nothing to do with online moderationmoderation

Assessing quantity of participation, but Assessing quantity of participation, but not quality not quality • which, granted, is easier, but encourages which, granted, is easier, but encourages

sloppy message posting rather than thoughtful sloppy message posting rather than thoughtful learning dialogueslearning dialogues

Bloom’s Revised TaxonomyBloom’s Revised Taxonomy

A useful tool for checking alignmentA useful tool for checking alignment Also great to guide your course design!Also great to guide your course design!

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

To Find Out More…To Find Out More…

A TaxonomyA Taxonomy

For LearningFor Learning

Teaching andTeaching and

AssessingAssessing

By AndersonBy Anderson

And KrathwohlAnd Krathwohl

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

Assessment TechniquesAssessment Techniques

Options…Options… Formative or SummativeFormative or Summative Student-led Student-led (Self or Peer)(Self or Peer) or Teacher-led or Teacher-led Public or PrivatePublic or Private Process or ProductProcess or ProductOther issues to consider…Other issues to consider… ““Objective” or InterpretiveObjective” or Interpretive Rubric-based or WholisticRubric-based or Wholistic

Formative vs. Summative Formative vs. Summative AssessmentsAssessments

FormativeFormative Alleviate student Alleviate student

anxieties re: anxieties re: expectationsexpectations Seem especially high Seem especially high

in online classesin online classes

Encourage working Encourage working toward masterytoward mastery

Can be informalCan be informal

SummativeSummative Used for student’s Used for student’s

gradegrade Assumed to be “best” Assumed to be “best”

efforteffort

Example: Online Example: Online Formative AssessmentFormative Assessment

Paper draft discussion forumPaper draft discussion forum Start a discussion forum for papers-in-Start a discussion forum for papers-in-

progressprogress Have each student start a thread and post Have each student start a thread and post

elements of their papers as they complete elements of their papers as they complete them (e.g., topic, major claims, research them (e.g., topic, major claims, research sources)sources)• A schedule for each element is usefulA schedule for each element is useful

Monitor and provide feedbackMonitor and provide feedback

Student vs. Teacher LedStudent vs. Teacher Led

Teacher-ledTeacher-led ““Traditional” Traditional”

assessmentassessment Most often summativeMost often summative

Student-ledStudent-led Students may assess Students may assess

self or peersself or peers May be formative or May be formative or

summativesummative Can greatly relieve Can greatly relieve

instructor burdeninstructor burden Students reinforce Students reinforce

concepts through concepts through feedback processfeedback process

Examples: Online Self-Examples: Online Self-AssessmentAssessment

Self-tests: Use test tool to create self-tests Self-tests: Use test tool to create self-tests (multiple choice, true false)(multiple choice, true false) May wish to track student effortsMay wish to track student efforts Can incentivize use (essentially, use as a Can incentivize use (essentially, use as a

learning tool)learning tool) Reflection papers: Have students submit Reflection papers: Have students submit

brief, focused papers expressing the brief, focused papers expressing the strengths and assessments of their strengths and assessments of their assignment(s)assignment(s)

Example: Online Example: Online Peer Assessment (Formative)Peer Assessment (Formative)

Feedback groups: Assign students in Feedback groups: Assign students in groups to provide formative feedback on groups to provide formative feedback on projects and papersprojects and papers Often raises quality of assignmentsOften raises quality of assignments Need a structure with clear deadlinesNeed a structure with clear deadlines Need prompts and models to guide students Need prompts and models to guide students May wish to assess feedback May wish to assess feedback

process/contributionsprocess/contributions

Example: Online Peer Example: Online Peer Assessment (Summative)Assessment (Summative)

Conference Presentations: Have students Conference Presentations: Have students “present” their work and ask questions/provide “present” their work and ask questions/provide feedback to others.feedback to others. In d-board, have students attach papers to In d-board, have students attach papers to

messages; post a message with a synopsis; or attach messages; post a message with a synopsis; or attach a powerpoint presentationa powerpoint presentation

Each student/team should have their own threadEach student/team should have their own thread Feedback should occur during a defined period of Feedback should occur during a defined period of

time.time. May consider allowing students to rate assignments May consider allowing students to rate assignments

on certain dimensionson certain dimensions

Public vs. PrivatePublic vs. Private

PrivatePrivate Work is submitted to Work is submitted to

the teacher onlythe teacher only Entire burden of Entire burden of

feedback is on feedback is on teacherteacher

Important if assessing Important if assessing at fact levelat fact level

PublicPublic Peers can see each Peers can see each

others’ work (either in others’ work (either in process or process or completed)completed)

Peers may comment Peers may comment on each others’ workon each others’ work

Often increases Often increases quality of work quality of work submittedsubmitted

Process vs. ProductProcess vs. Product

ProductProduct The end deliverableThe end deliverable Look for polish, Look for polish,

accuracyaccuracy

ProcessProcess How the student got How the student got

therethere Look for Look for

thoughtfulness of thoughtfulness of approach, intentapproach, intent

Assessing ProcessAssessing Process

Easy to doEasy to do Many technology tools will archive student Many technology tools will archive student

work/interactionswork/interactions Students create a document trail in processStudents create a document trail in process

Helps students develop metacognitive Helps students develop metacognitive knowledgeknowledge Instructors structure/model/encourage productive Instructors structure/model/encourage productive

work processeswork processes Students learn how to manage their own work Students learn how to manage their own work

processesprocesses

Why Assess Process?Why Assess Process?

For the instructor …For the instructor … Provides formative feedback on course Provides formative feedback on course

(e.g., helps gather data about why students (e.g., helps gather data about why students have difficulty with product-oriented have difficulty with product-oriented assessments)assessments)

Clarifies who is doing most work in small Clarifies who is doing most work in small group assignmentsgroup assignments

Helps prevent cheatingHelps prevent cheating

Why Assess Process?Why Assess Process?

For the student …For the student … Typically improves the quality of their Typically improves the quality of their

productsproducts Helps them develop productive work Helps them develop productive work

processesprocesses Puts on a schedulePuts on a schedule Shows that you care about individual growthShows that you care about individual growth

Assessment Project CycleAssessment Project Cycle

From From Classroom Assessment TechniquesClassroom Assessment Techniques by Angelo & Cross (1993)by Angelo & Cross (1993)

Step 1: PlanStep 1: Plan Choose classChoose class Focus on assessable questionFocus on assessable question Design project to answer questionDesign project to answer question

Assessment Project Cycle [2]Assessment Project Cycle [2]

Step 2: ImplementStep 2: Implement Teach target lessonTeach target lesson Collect assessment dataCollect assessment data Analyze dataAnalyze data

Step 3:Step 3: Interpret resultsInterpret results Communicate resultsCommunicate results Evaluate assessment projectEvaluate assessment project

I. Term PapersI. Term Papers

How to do online:How to do online: Have students each start their own thread Have students each start their own thread

and post topic of interestand post topic of interest Peers and instructors give feedbackPeers and instructors give feedback Students post thesis statements, research Students post thesis statements, research

sources, etc., with iterations of feedbacksources, etc., with iterations of feedback Final paper is postedFinal paper is posted

Term Paper AssessmentsTerm Paper Assessments

Product: the paperProduct: the paper Process: quality and timeliness of Process: quality and timeliness of

student work from time when paper is student work from time when paper is assignedassigned

Process: quality and timeliness of Process: quality and timeliness of feedback provided to peersfeedback provided to peers

Process: responsiveness to feedback Process: responsiveness to feedback received from instructor and peersreceived from instructor and peers

II. Discussion AssignmentsII. Discussion Assignments

1. 1. Chain of thoughtChain of thought Have students develop a solution to a Have students develop a solution to a

problemproblem Have students indicate what led them to Have students indicate what led them to

a particular conclusion, method or a particular conclusion, method or approachapproach

Can be done in a discussion boardCan be done in a discussion board

Discussion AssignmentsDiscussion Assignments

2. 2. Theory to PracticeTheory to Practice Have students match up theories you are Have students match up theories you are

learning about to actual problemslearning about to actual problems Present students with problems and have Present students with problems and have

them explain what theories they would them explain what theories they would use to solve these problems and how they use to solve these problems and how they would approach itwould approach it

Debrief the assignmentDebrief the assignment

Discussion AssignmentDiscussion Assignment

3. 3. Synthesizer (i.e., wrapper)Synthesizer (i.e., wrapper) Have students take roles being the Have students take roles being the

weekly synthesizer of class discussionweekly synthesizer of class discussion Add a “meta” level in which students Add a “meta” level in which students

narrate their own experiences while narrate their own experiences while reading the weekly discussionreading the weekly discussion

Reflect on how life relates to discussionReflect on how life relates to discussion

III. Group ProjectsIII. Group Projects

Tools usedTools used Chat:Chat: brainstorming ideas, making group decisions, brainstorming ideas, making group decisions,

regular way to feel connected (should be archived)regular way to feel connected (should be archived)

Discussion board:Discussion board: commenting on drafts commenting on drafts

E-mail:E-mail: quick feedback quick feedback

File exchange:File exchange: sharing project files sharing project files

MS Word:MS Word: Track changes Track changes

Group Project AssessmentsGroup Project Assessments

Product: project files that are turned inProduct: project files that are turned in Process: online archive demonstratingProcess: online archive demonstrating

Who contributed whatWho contributed what Who provided peer feedbackWho provided peer feedback Who worked in a timely mannerWho worked in a timely manner How collaborative a group wasHow collaborative a group was

Process: peer ratingsProcess: peer ratings Process: interim instructor consultationsProcess: interim instructor consultations

Group Project AssignmentsGroup Project Assignments

1. Work Plans1. Work Plans Have students develop a plan of work Have students develop a plan of work

for their projectfor their project Make them outline topic, schedule, Make them outline topic, schedule,

resources needed, division of labor and resources needed, division of labor and anticipated form of final deliverablesanticipated form of final deliverables

At end of project, have students At end of project, have students evaluate how well they followed their evaluate how well they followed their own plan and how useful it wasown plan and how useful it was

Project AssignmentsProject Assignments

2. Research Trail2. Research Trail Have students document the steps Have students document the steps

they took in the research process and they took in the research process and the resultsthe results

Ask for a brief reflection on how Ask for a brief reflection on how effective their process was and what effective their process was and what they might change the next timethey might change the next time

Project AssignmentsProject Assignments

3. 3. Process PresentationsProcess Presentations Have students focus on their process as well Have students focus on their process as well

as their product in class presentationsas their product in class presentations To maintain focus, ask them to share 3 main To maintain focus, ask them to share 3 main

lessons learnedlessons learned Might ask for some process documents to be Might ask for some process documents to be

shared, like an early draftshared, like an early draft

Project AssignmentsProject Assignments

4. 4. Design JournalDesign Journal Have students maintain a journal of all Have students maintain a journal of all

ideas related to their projectideas related to their project Encourage sketches, lists, organizational Encourage sketches, lists, organizational

charts, etc.charts, etc. Require journals to be turned in with final Require journals to be turned in with final

projectsprojects

IV. Reflection AssignmentsIV. Reflection Assignments

Have students keep a weekly journal of Have students keep a weekly journal of their thoughts on readings and course their thoughts on readings and course content AND real-world related content AND real-world related instances that they noticedinstances that they noticed

May make these public, with each May make these public, with each student having their own discussion student having their own discussion threadthread

Making it HappenMaking it Happen

Learners need to see that process is valuable:Learners need to see that process is valuable: Model appropriate processesModel appropriate processes Provide students with scaffolding (guide Provide students with scaffolding (guide

sheets) to structure their processessheets) to structure their processes Give students feedback on their processGive students feedback on their process Require students to reflect on their processesRequire students to reflect on their processes Grade students on processGrade students on process

Vanessa’s TopVanessa’s TopTime-Saving TipsTime-Saving Tips

Before you assign it, ask yourself “can I reasonably Before you assign it, ask yourself “can I reasonably assess it?”assess it?”

Rely on students/peers for providing some feedbackRely on students/peers for providing some feedback Let students know what to expect up frontLet students know what to expect up front Choose the right tool for the jobChoose the right tool for the job

Get to know the editing and commenting features of your Get to know the editing and commenting features of your favorite programsfavorite programs

Use rubrics!Use rubrics! I make them in Word, and then while grading I highlight I make them in Word, and then while grading I highlight

or bold the section that applies to the student’s projects. or bold the section that applies to the student’s projects. Add a few comments at the bottom = speedy grading!Add a few comments at the bottom = speedy grading!