building moocs: scalable course development & delivery

Building MOOCs: Scalable Course Development &

Delivery

Learning E-ssentials 2013

Ken CurrieOpus Learning

12th December 2013

1

This Talk

• Three main themes:– Why Design is critical, but not necessary complex– Why an Engineering approach is the only way to

build and deliver– The implications of Delivery, Support and

Assessment• Fourth theme?

– Implementation– MOOC specifics

2

Useful Quote

• “Large scale online learning could provide continuity for learners throughout their lives between different education systems.” – Association of Learning Technology Draft Green

Paper 2013.

3

Useless Quote

• “Enterprise technology for authoring, publishing and distributing is still largely in an immature state.”– Steve Barden, Lead Consultant, LINE

Communications, Inside Learning Technologies & Skills Magazine - November 2013

4

MOOCs or Programmes?

• Some of you may be here to hear about MOOCs– The story doesn’t really change, though there are

some delivery and operational extras• An example: Swim Coaching

– http://ties.courseworker.com– University of Iceland offering– Exploitation of the output of an EU LdV Project

5

History of Opus

• A spinout from CAPDM.com– Developing and delivering Masters programmes

for over 20 years• Opus delivers highly interactive courses,

delivered via platforms such as Moodle– With tools to manage and monitor learning with

or without tutor support. • An SQA Approved Centre

– A wholly online College

6

Show of hands

• How many people here build and deliver online distance education?

• What scale? Is it a challenge?• Opus has 99 SQA SCQF Level 7/8 courses

– All content full• Edinburgh Napier University’s Global Online

– 18 masters courses @ 20 credits– All content full

• Sound online pedagogy?7

Case: Heriot-Watt University

• Edinburgh Business School Online MBA– Europe’s largest DL MBA; the world's most international– 14,000 successful MBA graduates from 180 countries– Twice awarded the Queen's Award for Exports– Students from over 50% of Fortune 100 companies– 4,400 study hours; English, Spanish, Chinese, Arabic. – 800 different deliverables for 37 released courses in print and online.– 20 million words of English; 8,000 multi-lingual vector diagrams, etc.– 14 year unbroken record of revision controlled single source XML

– Has earned over £150 million– Now on its third VLE – but not a single character has had to change to

accommodate these moves!8

MOOCs vs Programmes

• Not a lot of difference?– Opus aims for tutor-lite courses (so high quality)– MOOCs take this to the limit

• Content full– Socially constructed content? No.– Opus delivers all the content a student needs– Complete, interactive, interesting!

• But develop social learners

9

Theme 1: the Design Challenge

• To build, deliver and manage– 99 HN courses for Opus– 18 masters level courses for Edinburgh Napier– … + others!

• To manage > 100 academics/teachers/authors• To manage very large domains of content• To deliver to many platforms

– Moodle, Blackboard, ePub, tablet/iPad, PDF, …

10

Strategic , not Tactical

• These challenges must be planned for– A strategy

• Many (large) organisations have no such strategy for this scale of development– Does yours? Most universities don’t, for example

• Structure & Content Management is critical– Communicate a consistent and understood

message to all parties– Minimise the entropy

11

Three Linked Parties

• Authors, engineers, students– Three links in a chain

• Many authors write great works– Few write good distance learning materials– Learning materials have to be fit for purpose

• Numerous authors– They must all hear the same message– They must all understand it

12

Talk in simple terms - components

13

Standard programme design

• Standardise course design within programmes

14

Simple message

• This structure carries a message that is:– Familiar: all components understood– Understood: can be communicated safely– Scoped: easy to put a scale to components– Quality Assured: all content must comply

structurally, in scope, academically, pedagogically

15

The Three Links

• Communicating the message– Authors work to a common template– Students consume a consistent product

• Know how to use it• Appreciate the clarity?• Keep retention rates up?

– Engineers, such as CAPDM, can build the product• Cost effectively• Consistently

16

SQA: the Start Point

• SQA provide ‘arrangement’ documents for all– Frameworks (e.g. HND Business) and – Units (e.g. Business Accounting)

• Unit specifications of:– Outcomes– Knowledge and/or Skills– Evidence Requirements– + Assessment Guidelines

• Universities have similar, perhaps less rigorous, guidelines 17

Theme 2: Engineering

• These courses are complex (in a structural sense)– Content-full, pedagogically rich?– Highly integrated and linked– Delivered to multiple platforms– Highly functional and interactive– Rich navigation and search

• Too complex to build by hand– The typical ‘bricolage’ approach cannot work

18

Standards

• All course materials are designed for their learning potential– Completely agnostic wrt technology– Single source masters of all content– Standards based: XML, MP3/4, etc.

• Structural descriptions of the assembly of this content through …– An XML manifest

• Focus on learning19

Publishing Goals

• To go from the single source master to full delivery– On all chosen platforms (e.g. Moodle)– In about 5 minutes– All published forms (xHTML, PDF, …)– Automatically

• Delivery to be– Complete and fully integrated– Fully functional

20

Where’s the middle bit?

• Where’s the instructional design, the delays, the faffing about with the VLE?– Eliminated!– This is all in the engineering, so fully automated– Styles, structures, etc., are all parameterised– Consistency of interpretations and delivery

• The Authors/Module Leaders can be in control– … in theory!

21

Semantic Mark-up

• Word, HTML and many other formats are ‘dumb’– XML can embody rich metadata and semantics– Though “File | Save As” XML is also dumb

• Our job? – To capture all learning content in a rich mark-up– To apply a consistent and rich semantic

interpretation to that content for delivery

22

Empowering Authors

• Authors write what they need– If the mention a ‘Forum’ then we guarantee its

presence and a link to it• It is our job to interpret their needs and to

engineer a solution– You will never hear “we can’t do that as the

platform doesn’t offer that”– A challenge at times

• We interpret good learning content23

Flexibility

• Single source masters with an ‘engineered’ delivery provides flexibility– Changes made in one place– Style(s) added during publishing– New platforms added non-disruptively– Multiple instances easily supported– Freed from specific technologies

• Smart, agile content– Standards in action

24

Theme 3: Delivery to Assessment

• Assessment can take many forms– Formative, e.g. through topic quizzes– Summative, e.g. exams and assignments– Ipsative, e.g. through comparison of self with

earlier marks• Students build up a Reflective Log, over time

– A longitudinal measure (Digital Work Book)– Continually improve it– Shared with the tutor (peers potentially)

25

Assessments & the SQA

• Why are we trying to ‘change’ things?• Units are stand alone

– Can be delivered in a very flexible manner– But very clear pointers as to how assessment

should be carried out• No longer appropriate to base an assessment

of competence on a final three hour written examination.

26

HND Business: Unit by Unit

• HND Business: each student has to– Sit 6 formal examinations– Deliver 6 documented portfolio exercises– Complete 22 other assessment reports

• total word count of 34,500

• An HE Diploma?– 12 modules each with final examinations or

assignment coursework

27

Need: Learning Analytics

• MOOCs are gathering global analytics– Measure everything but the individual

• Opus is looking to move towards developing the ‘Digital Self’– How you model yourself as an individual, and

access your data– How you are measured at all times– The DWB is our first step in recognising the

potential of ‘always-on’ learning

28

A Response: a Digital Work Book

• Longitudinal assessment– Augments examination and assignment

• Part of our ‘Social Learner’ aim– No faith in social learning (students creating the

learning content)– Great belief in breeding social learners– Many levels from ‘none’ to

• Interact with content for own ends (e.g. DWB)• Comment locally with others (e.g. Scotsman)• Interact and discuss with all peers (e.g. Forums)

29

Interacting with Content Objects

Learner interacting with content, leaving traces

Student interacts with content

Makes notes, responds to quiz, send messages, etc

Capture notesStore in MLE

Print off record of learning The Digital Workbook

30

Social Networks

• Opus is looking to build 000s of small groups– MOOCs are looking for a group of 000s

• Our social networks are small scale but highly linked (see Anderson 2003)

• Our goal: 5,000 students, but– 100 centres with 50 students– Model the individual, or a small group

31

How do students interact when learning

Student

Environment

Content Tutor

Student/Student

Content/ContentTutor/Tutor

Environment/Environment

Student/Tutor

Tutor/Environment

Student/Environment

Student/Content

Content/Environment

Tutor/Content

Based on diagram inAnderson 2003 32

Big Data?

• The analytics of the individual are of more interest to us– Not specifically looking at group behaviour– Individuals, in depth and across courses

• Digital Self profiles may be limited– E.g. me to Facebook and Linked-In (no pedometer

around my waist!)– How do we enhance profiles?

33

The Role of the Learning Materials

• Authors are encouraged to develop clear ‘concept maps’ for students to follow– Learning paths must go through key ‘concept

gateways’– Evidence of competent/critical thinking gathered

• The core is learning content that– Is interactive– Measures its own consumption– Builds ‘Digital Self’ data

• How to make it more intelligent? 34

Student Learning Aims

• Our aims for our learners– To see how they change over time– To teach these learners how to learn

• Their interface– Our content, systems and support– Must be smart– Back to the interpretation of the semantically rich

XML base

35

Analytics: Some Conclusions

• Analytics and tracking, via the DWB, etc., appears to– Provide the potential of personalised learning

within mechanical presentation of study– Permit formative assessment in the content,

driving tracking and learning progress– Provide feedback on Tutor effectiveness and the

effectiveness of the pedagogy underpinning the content

36

Theme 4: Implementation Details

• Note– No mention of Authoring Tools or proprietary

technologies– No real mention of e-learning– But massive exploitation of the technology of

delivery• Standards imply

– Technology agnosticism– Development outside of deliver environment(s)– Delivery into that/these environment(s)

37

A complete System

• Our final systems integrate– All assignment statements, requests and

submissions– Online tutor support and a Help Desk– Payment clearance and immediate access to study

materials• Our MOOCs include

– 100% automation– Social interaction– Auto generation of a Certificate of Completion

38

Back to Opus Learning

• Opus is a wholly online college– SQA Approved– Offering HNC/Ds and other courses– With its own students and tutors

• Opus is also a B2B provider of– Ready packaged courses and programmes– A full own-brand ‘College in a Box’– Consultancy for aspiring SQA centres

• Opus /CAPDM is a custom publisher & developer of ODL 39

Strategic?

40

Quote Revisited: Mature?

• “Enterprise technology for authoring, publishing and distributing is still largely in an immature state.”– Steve Barden, Lead Consultant, LINE

Communications, Inside Learning Technologies & Skills Magazine - November 2013

41

What people think we do

42

What we think we do

43

The End

Ken CurrieOpus Learning

kwc@capdm.com

44

The Digital Workbook is

• A portfolio embedded in the content

• A dashboard for learning • Rich seam of data and

behaviours for mining• Ongoing formative

assessment, and potential summative assessment

• A pdf for the student45

Case study HND Business Unit by Unit Assessment

In a typical instance of the HND Business each student has to:– Sit 6 formal examinations– Deliver 6 documented portfolio exercises , and – 22 other assessments - reports with a total word

count of 34,500.

A university Dip HE might only have 12 modules with 12 final examinations or courseworks

46

Managing Learning with Technology

• At Opus we track student behaviour. • We use 2 levels of tracking

– Moodle Based (the VLE) and – Content Based.

47

Tracking through the learning content

• Opus content is massively connected to the students workbook

• Every reflection, exercise, quiz, short response and so on are logged within the learning content and in the workbook

• The tutor can view, track and compare workbooks

48

DWB Examples – Source Page

49

50

51

DWB Dashboard –Managing groups of students

52

The Commercial Imperative

We wish to – drive down cost of assessment– Drive up the quality and efficiency of assessment

within the context of a well resourced learner.

We have to manage costs, and use of human resource

53

Assessor/Tutor Activity Model in Distance Learning

1. Monitor VLE activity, identify any candidate with lack of progress and contact course leader for action.

2. Monitor Digital Workbook (if used) and identify any outliers in progress, contact course leader with details.

3. Respond to direct tickets from candidate, making use of FAQs for responses, if new response is required, copy response to course leader.

4. Responding to assessments; remediation advice

54

Learning Centre Cost Model

• Based on the previous models– Time per tutor/unit/student can be estimated– Payment to Tutor/assessor is known– Viability of the Distance Course can be calculated.

• We need to know these numbers and to watch them

55

Potential Cost Model

Possible costs/charges per studentawarding body charges = £250;tutoring = £1000; tech + online content licence = £300; agent fees = £250Centre overheads per student = £500

Total cost per student = £2300

56

Potential Centre Surplus

• In UK Charge to student £3500-£4500• In India Charge to student £1500-£2500

We cannot serve one of largest markets in the world, with this model.

We have to be able to manage costs.

57

Learning from MOOCS

• Massive size of cohort• Highly programmed week by week• Clearly signposted weekly targets• Using the student body as the main tutor

toolset• Building a community of activists and lurkers• Managing and controlling costs• Generate revenue by conversions

58

Learning from MOOCS

• Do not do assessments – Until students join full programme

• Keep the granularity of the course fairly small – not full degrees

• Set up an honour system – make the students police themselves.

59

Applying all these lessons

• Manage the cohort size• Make sure investment reflects assessment and

tutor support provided – the added value of a non-free course

• Integrate assessments across units; manage the assessment overload

• Make assessments social, encourage group help. Track this activity!

60

Opus StrategySmall Private Open Courses

• Organise cohorts in groups of 10s, not groups of 10k’s or 100k’s

• Localise tutors and assessors• Build communities including assessors• Track activity and behaviour at keyboard level and at

cognitive level• Record activity – perhaps award a badge for partially

completed activity• Guide the learners towards completion, intervene as

necessary61

Creating the Social Learner

• Our DWB is part of a social learner strategy– Rich interaction spaces– Personal interaction with the content (DWB)– Interaction on specific issues (traversing the web)– Social learner (Forums, Facebook, etc.)

• Manage forum and virtual space activity

62

The Future

• The DWB is sharable, including – with the Tutor – with other students, groups

• In the future it will– support many content types– integrate with other portfolio components

• The DWB will be our main assessment toolset• The DWB will be our main behaviour manager

63

Conclusion

• Do not just set exams and give badges– Track and Assess– Intervene and support– Audit and learn lessons

• Build a community for the whole qualification• Record learning and make available for future

generations• Continually improve learning content.

64