1

HSU LMS Canvas Evaluation Report

June 14, 2016

Executive Summary During Spring 2016, the LMS Working Group, composed of Eamon Daly, Jim Graham (chair), Yoon G.

Kim, Cyril Oberlander, Anne Paulet, Kim Vincent-Layton, and Jeanne Wielgus, conducted an evaluation

of Canvas as an alternative to Moodle as the Learning Management System (LMS) for HSU. The Working

Group conducted a series of “world cafes” and hands-on sessions to obtain feedback from faculty,

students and staff about the use of Canvas. The group also had 54 faculty who offered 61 courses to

1,622 students using Canvas while maintaining an Issues List of concerns which the committee worked

on addressing. At the end of the semester, Institutional Research (IR) distributed a feedback form to all

faculty and students regarding the importance of features and the level of satisfaction for Moodle and

Canvas. Based on the evaluation of both LMSs and considering issues of student learning and teaching

effectiveness along with improvement of long term institutional goals, the LMS Working Group

concludes that a transition to Canvas will best enable HSU to achieve its goals of improved teaching and

learning and controlling costs.

This report was prepared by the LMS Working Group with support from a large number of individuals

including Lisa Castellino, Alex Hwu, Anna Kircher, Matt Koelling, Sabre Stacey, Cassandra Tex, Tim Payer,

and the numerous faculty, staff, and students who helped with the evaluation.

Introduction HSU has been using Moodle for 10 years and some issues have arisen with it. Some members of HSU,

including faculty, were interested in looking at other Learning Management Systems. While there were a

variety of systems available, Canvas was markedly different than the others and has been well received

by academic institutions. Additionally, its usage is increasing. Based on this, HSU spent Spring 2016

semester evaluating the Instructure.com hosted instance of Canvas from faculty, student, staff,

financial, and technical perspectives. Below is a summary of this information with additional details

provided in appendices or on the web site at www.humboldt.edu/lms.

Process/Methodology Our intent in this task was to follow a process that was well defined, inclusive of stakeholders, and

visible to everyone involved. Originally, the LMS Working Group planned on evaluating a variety of

alternatives for the LMS at HSU. However, the Provost at that time decided, in the interest of time, that

we only consider Canvas because it had already been identified as the most likely alternative to Moodle

by Alex Hwu and Anna Kircher. It was also decided that the Provost would make the final decision on

which LMS HSU would be using. Based on this, the LMS Working Group defined areas of investigation--

learning and teaching effectiveness, concerns, institutional value, and comparison--with the goal of

providing this report to the Provost by the end of the 2016 spring semester.

2

Learning and teaching effectiveness was investigated by initiating a spring evaluation of Canvas in

classes and a series of feedback and demonstration sessions. The evaluation included 54 faculty, 61

courses, and 1,622 students using Canvas as their LMS. There were 12 hands-on demonstrations and

focus groups with faculty, staff, and students. We also created online discussion groups for faculty, staff,

and students. Issues that users encountered were recorded on issues lists. Simultaneously, we began

collecting issues for Moodle. At the end of the semester, IR administered a feedback form to all HSU

faculty and students.

Technical issues were investigated primarily through using the software and working with

Instructure.com. We consulted with Cassandra Tex from the Student Disability Resource Center who

provided feedback and an accessibility report on both Canvas and Moodle. Her report is included in this

paper and is based on previous, and relatively thorough, evaluations by her colleagues, though it should

be noted that these are not comparable evaluations. See reports at the LMS Web Site. Finances were

developed based on figures provided by Instructure.com and Information Technology Services (ITS).

Throughout the evaluation process we communicated with faculty, staff, and students through email

and with a publically-available web site (www.humboldt.edu/lms).

3

Learning and Teaching Effectiveness

LMS Survey Summary A Campus LMS survey was administered by the Office of Institutional Research & Planning, and was sent

to all faculty (n=568) and all students (n=9,054) registered in state instruction for Spring 2016, including

students registered through Extended Education. The initial deployment was May 4, 2016, with

reminders sent out on May 11, 2016. The survey was sent through QuestionPro via email. The faculty

survey had a 16% (n= 103) participation rate. The student survey had a 4% (n=379) participation rate.

Two summary graphs are below, and details are available on the LMS Evaluation web site. In the charts

below, satisfaction was ranked from Not Satisfied (1) to Very Satisfied (5) with a similar scale for

importance with (1) being Not Important and (5) being Very Important. In the student survey, the lowest

average Canvas satisfaction score was 3.7 (Access the LMS from a mobile device (Canvas)) which was

greater than the highest average Moodle score of 3.5. In the faculty survey, the lowest average Canvas

satisfaction score was 3.7 (Creating appropriate quizzes for the content you are teaching) which was

greater than the highest average Moodle score of 3.6. Results show a clear preference for Canvas by

both faculty and students.

2

2.5

3

3.5

4

4.5

5

3 3 . 5 4 4 . 5 5

SA

TIS

FA

CT

ION

IMPORTANCE

STUDENT RESULTS

Canvas

Moodle

4

LMS Survey Results On the following two pages are the results and graphs supplied by IR.

2

2.5

3

3.5

4

4.5

5

3 3 . 5 4 4 . 5 5

SA

TIS

FA

CT

ION

IMPORTANCE

FACULTY RESULTS

Canvas

Moodle

5

6

7

Industry Trends Below is a chart showing the trends in market share for LMSs in North American Institutions. The chart

shows that Blackboard continues to be the leader, largely by acquiring other organizations. Canvas is

currently increasing more than other LMSs.

A recent update by Edutechnica (the source below for the 2013-2015 data) for Spring 2016, notes: “It is

still the case that the majority of institutions that do pilot Canvas end up switching to it.” These 2016

updates, made in March, can be located at http://edutechnica.com/2016/03/20/lms-data-spring-2016-

updates/ though full data is only available for a fee.

Concerns

Common Issues for Canvas & Moodle Regardless of which LMS is chosen, there will be opportunities for working with faculty and students to

maximize utilization of the LMS as a tool for student engagement and success. There are issues with the

gradebook in both platforms; however, the HSU faculty survey indicates that they are much more

satisfied with the Canvas gradebook. Neither platform will meet 100% of students’ accessibility needs;

however, there are workarounds or plans to address those issues on both platforms.

8

Summary of Issues for Canvas The issues discussed in this section represent the most commonly reported ones or those that have a

systemic rather than individual repercussion if not addressed. These issues do not by any means

represent the totality of issues reported by users - faculty, staff, and students; however, resolved issues

have been excluded and can be found on the HSU LMS website if further review is needed.

Rubrics and Grading: several faculty noted that the Canvas rubric does not allow moving criteria. Canvas

Gradebook: it appears to have issues when there are more than 2,000 entries. Quiz Editor: Feedback on

functionality of the quiz editor has been quite diverse. The issues that the LMS group received

consistently were: when designing a quiz, the ability to format HTML code is limited; quiz banks with

over 25 questions require instructors to manually enable review of correct answers post-quiz; when

students re-attempt a quiz, previously scored correct answers are not saved. Some members of the

Math Department have reported issues that are exclusive to their needs; those issues can be found on

the HSU LMS website.

Regarding overall functionality: Canvas lacks the ability to be able to receive anonymous feedback from

students while also allowing the faculty member to know generically who has provided feedback.

Attachments sent with emails through Canvas are not consistently received.

Summary of Issues for Moodle Moodle has been the LMS for HSU for over 10 years. It is hosted on our servers and managed on

campus, providing us with unique insights that we do not have with Canvas. This also presents us with

issues which we would not experience with a hosted solution. The issues below represent the most

commonly and consistently reported ones.

Gradebook: Issues related to the functionality of the gradebook module are the most numerous on the

list of known issues. Among the most recently noted: the gradebook does not work well with adding

comments/feedback directly to the assignment (however, a workaround is available). Users get logged

out while entering grades and entries get lost, requiring the user to re-enter previous entries. Moodle

has three different aggregation configurations (ways the grades can be calculated). Faculty must choose

the proper aggregation when the course is created in order for the gradebook to calculate assignment

groups and total grades properly. There is a possible workaround under consideration: one aggregation

configuration for GradeBook.

LMS Down-time: Because Moodle is a software package run on the HSU servers, maintenance

downtimes are required at regular intervals. Presently ITS standard maintenance schedule for all servers

is the third Sunday of each month between 5:00am and 11:00am. Depending on what maintenance is

needed, Moodle may be unavailable during these maintenance windows. To accommodate version

upgrades and other unusually large maintenance requirements, ITS has also been scheduling multi-day

pre-semester maintenance with advanced notification to campus users.

Google Calendar Integration: This feature has been reported to be unreliable (a fix maybe available with

future releases).

9

Course Templates: Students have expressed dislike for the variety of Moodle layout templates available

for the professor to select because this creates inconsistency between classes.

Accessibility The following was provided by Cassandra Tex, HSU’s Assistive Technology Specialist in the Student

Disability Resource Center. The full report is available at the LMS web site.

There are accessibility issues with Canvas as noted in the testing and evaluation done by my

colleagues from across the country. There are numerous “supports with exceptions” items in

the Moodle report from SSB Bart Group indicating that there could be some functional

accessibility issues with Moodle as well. Comparable documents do not exist that can easily be

compared to see which product has a higher level of functional accessibility. The idea when

evaluating products is to determine the products that meet your functional needs and then

evaluate for accessibility. All things being equal, if two products meet your functional needs,

you select the product that has the higher level of functional accessibility. Given the

information and documents we have at this point in time, we cannot fully evaluate which

product has the higher level of functional accessibility.

With either product (because we know neither product is 100% accessible) there will need to be

equally effective accessibility planning. This should be done well in advance of a student (or

employee) need so that we are not scrambling for ideas on how to accommodate someone.

Intellectual Property The following section is from Instructure’s user agreement which all users, faculty and students, must

agree to in order to use Canvas. There is some concern that this gives Instructure the right to use faculty

and student created material within Canvas but not limited to HSU’s instance of Canvas. We discussed

this with Instructure and they did not want to modify the agreement. Anna Kircher felt the contract

with Instructure could limit use to just within HSU’s instance of Canvas so we did not pursue the issue

further. Terms of Use: https://www.instructure.com/policies/terms-of-use

4. Ownership.

4.1 Instructure Properties. Except with respect to Your Content and User Content, you

agree that Instructure and its suppliers own all rights, title and interest in the Instructure

Properties. Instructure’s name and other related graphics, logos, service marks and trade

names used on or in connection with the Instructure Properties are the trademarks of

Instructure and may not be used without permission in connection with any third-party

products or services. Other trademarks, service marks and trade names that may appear

on or in the Instructure Properties are the property of their respective owners.

4.2 Your Content. Except with respect to Your Content, you agree that you have no

right or title in or to any Content that appears on or in the Instructure Properties.

Instructure does not claim ownership of Your Content. However, you grant Instructure a

10

fully paid, royalty-free, perpetual, irrevocable, worldwide, royalty-free, non-exclusive and

fully sublicensable right (including any moral rights) and license to use, license, distribute,

reproduce, modify, adapt, publicly perform, and publicly display, Your Content (in whole

or in part) for the purposes of operating and providing the Instructure Properties. When

you as a User post or publish Your Content on or in the Instructure Properties, you

represent that you have the authority to grant the aforementioned license to Instructure.

Please remember that other Users may search for, see, use, modify, and reproduce any of

Your Content that you submit to any area of the Instructure Properties that is generally

available to all Users. You warrant that the holder of any worldwide intellectual property

right, including moral rights, in Your Content, has completely and effectively waived all

such rights and validly and irrevocably granted to you the right to grant the license stated

above.

Institutional Value

Value Analysis HSU, as with most academic institutions, is under pressure to control costs and improve outcomes. An

evaluation of the Learning Management System is crucial to assessing the design and impact of the tool

most faculty use in their teaching and learning environments.

During the LMS evaluation, a survey of HSU students showed that students were more satisfied with

features and functionality from Canvas than they were from Moodle. In particular, the three most

important features and functions of an LMS from the student’s perspective--easily find current material,

see grades, and know when assignments are due--all had higher satisfaction scores with Canvas than

with Moodle. Based on our current Moodle use in Spring 2016, 1.34 million sessions and 8.12 million

page views, we can predict that the level of satisfaction increase of 1 point translates to a significant

increase in the effectiveness of the learning environment, and likely, an increased level of engagement

and learning.

The survey results of HSU faculty comparing the functionality and importance of features in an LMS

similarly show greater satisfaction with Canvas. Faculty were more satisfied with features and

functionality from Canvas. In fact, the lowest average Canvas satisfaction score was 3.7 which was

greater than the highest average Moodle score of 3.6. This translates to a much more effective tool for

faculty. Canvas aligns more with HSU Faculty value of intuitive interface and ease of grading, two of the

most important features of an LMS. Reducing the time faculty spend working with an LMS increases

their capacity to work with students and on various university initiatives. If faculty have an LMS they

find easy to use, they are more likely to experiment with a variety of the features thereby improving

teaching and learning opportunities. In order to improve learning outcomes and student retention, we

see the student and faculty survey results as strong evidence that migration to Canvas provides

opportunities to improve our ability to meet student and faculty needs, and to innovate and meet future

needs.

11

What is clear from the Canvas-Moodle evaluation by the LMS Working Group is that both platforms

have their strengths and weaknesses. If Canvas is selected, it is an opportunity to incorporate some

thoughtful design by faculty working together during the migration to a more recently developed cloud

based software. If continuation of Moodle is selected, we must see this as an opportunity to develop

shared effective strategies and branding, as well as develop our open source software and ITS support

for the campus LMS.

While there is a one-time cost to migrate to Canvas, the financial analysis below indicates that it will cost

an estimated $35,000 less annually than Moodle. When this is combined with the savings in faculty and

student time due to ease of use and improved interaction, this suggests Canvas will provide a greater

ability to control costs and improve learning outcomes.

12

Finances Below is a simplified analysis of the annual costs associated with staying with Moodle vs. moving to

Canvas. These costs do not reflect that the LMS Coordinator will need to be focused on supporting the

ongoing use of Moodle during the transition year. Support for Canvas during the transition year will

need to be provided by the Instructional Designers in CEEE, limiting the scope of work they can

accomplish on other initiatives. The costs for Canvas assume we are using the 24/7 support. After initial

usage of Canvas support, some universities have transitioned to supporting Canvas themselves or to

more limited Canvas support.

One Time Costs

Canvas Startup 24,000

Programmer ½ time salary/benefits year one

transition 27,558 (17,593)

Administrative Support Coordinator I 34,152 (15,368)

Total One Time Costs and Benefits $118,671

Annual Recurring Costs for an LMS

CANVAS MOODLE

People Salary (Benefits) Salary (Benefits)

LMS Support Specialist 0* 62,000 (37,185)

LMS Coordinator 50,000 (26,000) 50,000 (26,000)

Programmer 27,558 (17,593)** 55,116 (35,186)

Students 24,000 (240) 48,000 (480)

Total Salary and Benefits $145,391 $313,967

Additional Costs

License 87,217 0

Instructure’s 24/7 Premier User Support 17,443 0

Tier 1 Support 28,798 0

Recurring cost for server 0 2,500

Additional storage space for courses*** 4,000

Total Additional Costs $137,458 2,500

Total Costs $282,849 $316,466

* The Moodle Specialist position would not be refilled if we move to Canvas.

** The Programmer position is a permanent bargaining unit position; the full costs for this position, $55,116

(35,186), continue regardless of the work assigned. If campus moves to Canvas, this position will be partially

assigned to non-LMS work, but the need for base budget for the position will continue. Because we are not hosting

Canvas, we expect the programmer support to be about ½ the cost for Moodle.

*** We may need additional funds to cover space costs as these figures reflect ½ gigabyte per class and our classes

average 1 gigabyte. This would be approximately $4,000 annually.

13

Comparison

SWOT

Analysis

If we go with Canvas from Instructure If we stay with HSU hosted Moodle

Strengths Ease of learning and use

Modern user interface

Better mobile support

Canvas is down for only several hours at night for updates

California Community College System has moved to Canvas

SpeedGrader feature allows quick commenting and grading of documents.

Calendar that integrates with Google calendar

24/7 support option available

We have existing Moodle installation and expertise

There are functionalities available in Moodle that are missing in Canvas (detailed quizzes, anonymous questionnaire, book tool, glossary, lessons, workshops)

Don’t have to pay license fees

Allows submission of programming scripts

Weaknesses We lose control of the implementation of the LMS as Canvas is a hosted solution

Quiz editor does not meet the needs of some of the specific equation handling used by some members of the Mathematics department faculty

Faculty and ITS will have a significant amount of work to move courses from Moodle to Canvas

A substantial number of students and faculty will need to use two LMSs for a year

Includes functional accessibility issues

Does not allow programing scripts to be submitted for assignments

We need more personnel and computer hardware to maintain Moodle

Our server infrastructure is currently unstable and has resulted in Moodle being unavailable repeatedly during the semester. Dedicated servers have been added and this should stabilize Moodle.

Moodle is down for several days at a time for upgrades, twice per year

Mobile app is not enabled

Includes functional accessibility issues

Moodle does not save grades as you are entering them

Moodle does not notify before logging you out, at which point unsaved data is lost.

Can connect to Google Calendar but it is not reliable

Onsite support

Opportunities Canvas has less flexibility which may increase consistency across classes

Reduces HSU’s staffing needs

Change in LMS is a faculty development and course redesign opportunity

Moodle provides faculty with more options to customize their classes

Supporting open source community

Threats Instructure could increase the costs

Risk to intellectual property and confidentiality

Long-term financial viability of Instructure

Limited redundancy in support staff

Difficult to recruit and retain IT staff

14

LMS Usage Statistics This section provides the statistics that are available from Canvas and Moodle which we felt were

relevant to selecting an LMS for HSU. As the figures and charts indicate, Canvas and Moodle provide

different statistics on usage. We have added additional information to allow more of an apples to

apples comparison. Please note that the data looks at Spring 2016 only.

Canvas

Below are the statistics that are available to administrators within the Canvas solution.

Canvas Spring 2016

Total Courses 61

Total Faculty 54

Total Students 1,622

Figure 1: Activity by date. This graph displays the number of page views by date in Canvas.

Figure 2: This graph displays the number of page views for each specific Canvas resource during Spring

2016.

15

Moodle

The following data was extracted from the HSU Moodle instance.

Moodle Spring 2016

Total Courses 1,090

Total Faculty 726

Total Students 8,323

Figure 3: Activity by date for Spring 2016. This graph displays the number of page views by date in

Moodle.

16

Figure 4: This graph displays the number of page views for each specific Moodle category during Spring

2016.

Below are usage statistics for Moodle in the spring of 2016.

Operating System

Total number of Sessions1: 1,339,135 Windows 49.64% of Sessions

Page views2: 8,176,849 Macintosh 30.89% of Sessions

Pages/Session3: 6.11 iOS 11.85% of Sessions

Avg. Session Duration4: 00:07:35 Android 5.64% of Sessions

Bounce Rate5: 12.89% Chrome OS 1.63% of Sessions

Other6 0.35% of Sessions

Total space usage: 5.6 TB (includes 5 semesters plus ongoing courses)

1 A session is the period of time a user is actively engaged with Moodle.

2 Total number of pages viewed. Repeated views of a single page are counted.

3 Average number of pages viewed during a session. Repeated views of a single page are counted.

4 Average length of a session. Format is Hr:Mn:Sc.

5 The percentage of visits in which the person left the site from the entrance page without interacting.

6 Includes those users whose language, country, etc. is not specified in the browser.

17

Moodle Mobile (17% of overall accesses)

Tablets or phones using a web browser (rather than through an app).

iOS 64.73% of Sessions

Android 29.90% of Sessions

Windows 4.87% of Sessions

Other8 0.5% of Sessions

There are additional graphs in the Moodle 2016 spreadsheet. You will need to download the file and open it in

Excel to view the graphs.

Conclusion Based on the evaluation of both LMSs and considering issues of student learning and teaching

effectiveness along with improvement of long term institutional goals, the LMS Working Group

concludes that a transition to Canvas will best enable HSU to achieve its goals of improved teaching and

learning as well as controlling costs.

Implications Some users will have unmet needs with either Canvas or Moodle; and both LMSs require active

maintenance of third party plugins or applications such as clickers, Turnitin, etc. Each LMS platform has

at least the following identified issues:

Canvas Quiz editor does not meet the needs of some of the specific equation handling used by some

members of the Mathematics department faculty

Canvas has been responsive to some issues identified, however, not all, so we need to ensure

our needs are met

HSU migration plan will take about one year (fall 2016 through summer 2017)

Migration requires faculty effort to move courses to Canvas

Moodle HSU needs to plan the management of Moodle, and to capture and address issues from users

There is a need for Moodle collaboration, including continuing to work with partners in San

Francisco to improve Moodle

Need to update to Moodle 3.0

Complications of multiple gradebook options means we may need to simplify the gradebook to

one option

For teaching and learning to be more effective, the LMS must be managed and supported well as a

technology and service. Developing a support framework that identifies issues and collaboratively solves

them is fundamental to improve student success; similarly, for faculty effectively to employ any LMS,

the incorporation of instructional designers and faculty development are essential.