1 june 10-15, 2012 growing community; growing possibilities switching to on-line evaluations for...
TRANSCRIPT
1
June 10-15, 2012
Growing Community; Growing Possibilities
Switching to on-line Switching to on-line evaluations for courses evaluations for courses at UC Berkeleyat UC Berkeley
Daphne Ogle, Lead Design, UC BerkeleyAaron Zeckoski, Lead Developer,
Unicon
22012 Jasig Sakai Conference
SpeakersSpeakers Daphne Ogle Berkeley Design Lead
OAE since 2010, Fluid Project, Sakai since 2004
Aaron Zeckoski Unicon Development Lead
Evaluation System original lead (2006), Evals software since 2001, Sakai since 2004
32012 Jasig Sakai Conference
Unicon ProfileUnicon Profile
42012 Jasig Sakai Conference
Unicon technology & domain Unicon technology & domain expertiseexpertise
52012 Jasig Sakai Conference
Unicon service capabilities / offeringsUnicon service capabilities / offerings
62012 Jasig Sakai Conference
Intro to UC Berkeley on-line course evaluation project
Demo: Updates to CLE Evaluation System for Spring pilot
Future work What we learned Questions & Answers
Today’s topicsToday’s topics
72012 Jasig Sakai Conference
Introduction to online Introduction to online evaluations project at UC evaluations project at UC
BerkeleyBerkeley
Overall project initiative
82012 Jasig Sakai Conference
Online evaluations for coursesOnline evaluations for courses New shared questions
◦ campus-wide◦ instructional format-based (14 formats)◦ piloted first
On-line◦ pilot CLE Evaluation System◦ development to meet Berkeley needs & overall
system improvement◦ Berkeley & Unicon collaboration
92012 Jasig Sakai Conference
Goals of initiativeGoals of initiative Rapid access to useful information to
help faculty improve their courses and their teaching effectiveness
Improvements to the quality & integrity of data that the campus uses to understand and recognize teaching contributions
21st century tools for staff to perform their jobs efficiently and free up time for direct service to students and faculty
High quality information for students about courses and instructors at Berkeley
Cynthia Schrager, Assistant Vice Provost
10
2012 Jasig Sakai Conference
Project time lineProject time line
Summer 2010
Spring 2011
Summer 2011
Fall 2011
Spring 2012
Summer 2012
Fall 2012
Research new questions & instructional formats
Evaluation systems comparative analysis
Design, analysis & development CLE eval sys
(in UCB branch)
Campus buy-in
Fall 2010
Pilot new questions & instructional formats
Pilot CLE eval sys
Assess pilot
Design & development (intend to move to trunk)
Larger 2nd pilot
11
2012 Jasig Sakai Conference
Spring Pilot
Introduction to online Introduction to online evaluations project at UC evaluations project at UC
BerkeleyBerkeley
12
2012 Jasig Sakai Conference
Goals of spring pilotGoals of spring pilot Establish baseline response rate Assess CLE evaluation system
◦ technical longterm sustainability◦ user experience
Build knowledge of CLE evaluation system Build knowledge of infrastructure needs
13
2012 Jasig Sakai Conference
Spring pilot success criteriaSpring pilot success criteria Establish baseline response rate Collect data on function & experience User experience meets or exceeds previous Transparency across campus Elicit higher quality response Gain understanding to inform Fall design &
development effort
14
2012 Jasig Sakai Conference
Spring pilot makeupSpring pilot makeup 8 departments 61 courses (primary & secondary) 2300+ enrollments
15
2012 Jasig Sakai Conference
Spring pilotSpring pilotDemo
16
2012 Jasig Sakai Conference
Spring pilot developmentSpring pilot development Significant time understanding system Overall model & process to follow
◦ “uber” template◦ self-service
Focus on student & instructor experience System areas of focus
◦ Dashboard◦ Evaluation presentation◦ Administration page◦ Notifications
17
2012 Jasig Sakai Conference
Dashboard - studentDashboard - student
18
2012 Jasig Sakai Conference
Dashboard - administratorDashboard - administrator
19
2012 Jasig Sakai Conference
EvaluationEvaluation
20
2012 Jasig Sakai Conference
AdministrationAdministration
21
2012 Jasig Sakai Conference
NotificationsNotifications
22
2012 Jasig Sakai Conference
Bulk import / export configurationsBulk import / export configurations
23
2012 Jasig Sakai Conference
Live DemoLive Demo
http://berkeley-eval.unicon.net/portal
24
2012 Jasig Sakai Conference
Lessons learnedLessons learnedFood for thought
25
2012 Jasig Sakai Conference
Lessons learnedLessons learned Configurability + flexibility = Complexity
◦ documentation not always up to date◦ allow significant time to “play around”
Unfinished functionality◦ email notifications◦ options in evaluation creation
Global tool◦ acts different than other tools◦ where will it be exposed?
26
2012 Jasig Sakai Conference
Lessons learned - cont’dLessons learned - cont’d Allow a lot of time for QA
◦ see 1st bullet: lots of variables◦ QA’ing up to the last minute
Clean data required◦ setup if not using group provider◦ site roles NOT always evaluation roles
Communicate!◦ be transparent◦ remind users it’s a pilot
27
2012 Jasig Sakai Conference
Lessons learned - cont’dLessons learned - cont’d Learn from other implementors
◦ document models, workflows & implications◦ community project?
Culture & legal requirements vary across campus◦ viewing respondents◦ roles, permissions, viewing each others
evaluations
28
2012 Jasig Sakai Conference
Future plansFuture plansEvaluation system at Berkeley
29
2012 Jasig Sakai Conference
What’s next What’s next (1 of 2)(1 of 2)?? Feedback sessions with users on Spring pilot Needs analysis around results Move our work back into trunk Configuration option to hide respondents page UX improvements
◦ results page◦ multiple instructors (Berkeley-centric)◦ add respondent link to dashboard◦ make display options clear in context◦ configuration page (information architecture)
30
2012 Jasig Sakai Conference
What else is next?What else is next? Investigation
◦ support for instructional formats (Berkeley-centric)◦ give department admins access to results◦ Evalsys tool added to My Workspace automatically◦ group provider & SIS integration◦ partial OAE integration
31
2012 Jasig Sakai Conference
ResultsResults(early idea)(early idea)
32
2012 Jasig Sakai Conference
Results(early idea)Results(early idea)
33
2012 Jasig Sakai Conference
Sneak peak: dashboardSneak peak: dashboard
34
2012 Jasig Sakai Conference
Sneak peak: scale creationSneak peak: scale creation
35
2012 Jasig Sakai Conference
Sneak peak: scale creationSneak peak: scale creation
36
2012 Jasig Sakai Conference
ResourcesResources Berkeley Confluence
◦ https://confluence.media.berkeley.edu/confluence/display/EVAL/Home
Sakai Confluence◦ https://confluence.sakaiproject.org/display/EVALSYS/Home
Sakai Jira◦ https://jira.sakaiproject.org/browse/EVALSYS
37
2012 Jasig Sakai Conference
Thank you!Thank you! Questions, comments, discussion...