measuring the business impact of learning: lagging indicators to predictive analytics

@leolearning @WatershedLRS

Measuring the Business Impact

of Learning Lagging indicators to predictive analytics

Gareth JonesPiers Lea

Chief Strategy Officer: LEO

Mike Rustici

CEO: Watershed Principal Consultant: LEO

www.leolearning.com www.watershedlrs.com

http://www.leolearning.com/

http://www.watershedlrs.com/


Points we will cover today

• Current state of play

• Why learning analytics matter

• The components of a business impact strategy

• Case Studies- how others have done them

• A strategic framework


Where are you on the measurement journey?


State of PlayPiers Lea


Respondents from over 350 companies

in various sectors, from government

to automotive, aviation, defence

and technology

LEO and Watershed Research

I want to use analytics to improve learning

I believe it is possible to demonstrate learning’s impact

@leolearning

I feel executive pressure to measure learning’s impact

@leolearning

The biggest challenge of measuring the impact of

learning in my organisation is...


L&D skills gaps … what the research is telling us

@leolearning @WatershedLRSGlobal Human Capital Trends 2014: Engaging the 21st-century workforce - A report by Deloitte Consulting LLP and Bersin by Deloitte


84% of those using analytics

actively review their programs

against organizational goals

“Making an Impact - How L&D leaders can

demonstrate value” Towards Maturity and

LEO Learning, June 2016

Some are now red hot … we need to increase the impact!


L&D skills gaps … the analytics capability curve


Big Picture – Why Learning

Analytics Matter Mike


“Scalable learning is

the new reason for

large organizations to

exist”

John HagelFounder

Deloitte Center for the Edge Innovation


“Without data

you’re just

another person

with an opinion”

W Edward DemingManagement Systems Legend


“L&D is responsible for improving our

most important asset, our people. If we

can improve L&D, we can improve the

world”

MeI said that

Understanding learning analytics: Complexity

measurement

The simple act of

tracking things and

recording values. Can

be passively or

actively collected.

evaluation

The process of trying to

make meaning from the

data measured.

Descriptive analytics.

Does the data mean

something “good” or

“bad”?

advanced

evaluation

When data sets get large

enough we can use

advanced evaluation

techniques to discover

powerful insights. Data

mining, AI, machine

learning, etc.

predictive and

prescriptive

Make predictions and

decisions based on data

and advanced

algorithms.

Recommendations

engines are the best

example in learning.


Learning

ExperienceLearner

Learning

Program

Understanding learning analytics: Categories

Understand an overall

learning program. Is this

initiative helping to meet

business objectives.

Understand a learner or group

of learners. Ensure

organizational readiness and

compliance.

Understand more about a

specific learning activity.

Maximize effectiveness and

spot problems.

@leolearning @WatershedLRS© Watershed Systems, Inc 2017


all data

Learning is so much bigger than the

LMS. Track informal and experiential

activities

xAPI and the LRS present a transformative

technology for learning analytics. Seamlessly

bring all your data together in one platform and

perform real-time analytics tailor made for L&D

business impact

Include data about behaviors and

performance in your learning

analytics real time

Create, view and distribute reports

seamlessly in real-time

xAPI & data analytics engine


Which category would you like to measure most?


Key components of a business impact strategyGareth


Royal Institution


The Top 10 components


1: Partner with the business

and co-create

Components of a business-impact strategy



and co-create

• 2: Define business drivers




and co-create


• 3. Map and get

stakeholders on board




and co-create


• 3. Map and get


• 4: Define chain of evidence/existing KPIs




and co-create


• 3. Map and get



• 5: Define data sources and types




and co-create


• 3. Map and get





6. Define technical integration

steps and tools



and co-create


• 3. Map and get






steps and tools

7. Run a pilot and refine the reporting

processes and procedures



and co-create


• 3. Map and get






steps and tools



8. Document the business

case and strategy



and co-create


• 3. Map and get






steps and tools




case and strategy

9. Define the change programme



and co-create


• 3. Map and get






steps and tools




case and strategy

9. Define the change programme

10. Measure your measurement strategy


Case Studies Mike & Piers

@leolearning @WatershedLRSSource: Getting Buy-In From the Boss. Andy Webb, Applied


Comprehension vs. Application

Source: Getting Buy-In From the Boss. Andy Webb, Applied


Visibility Through Dashboard, Mapping, Sorting

Source: Getting Buy-In From the Boss. Andy Webb, Applied

@leolearning @WatershedLRSSource: Getting Buy-In From the Boss. Andy Webb, Applied


Ford of Europe

• Reduced time to competence from 36

to 13 months

• 93% of key messages understood by

learners

• Significant increase in car sales: 2.4

cars/annum per head on average*

* Control groups of more than 1,000 people


0

1

2

3

4

5

6

7

8

79% 100%

Average sales per person

Achievement is also

important – those who

passed Fiesta e-

learning with 100%

sold 2 more cars than

those who passed

with 79%

e-learning generates incremental sales


Code Blue

When a patient's heart stops

and needs to be resuscitated.


Key Metrics

Time to

first drug

Time to

defibrillation

Time to

chest


Learning Ecosystem

Mock Code Blue/ In

person ObservationMobile Simulation/

Defibrillator App

LMS Training Lab

Learning Analytics

Platform (Watershed)


Does the amount of training activity over time have an impact on Code Blue KPIs?

Time to

defibrillation

Time to

chest

Time to first

drug

Arrival

times

Time hands off

chest

Training

Activities

Jan 2015 Feb 2015 Mar 2015Apr

2015

May

2015Jun

2015

Tra

inin

g

Activitie

s

Code B

lue

KP

Is

Training Effectiveness


Creating A StrategyPiers


What degree of pressure does your senior

management place on measuring the business

impact of learning?

Close

OUTPUT:Final documents and

lessons learned

A. Finalise all

deliverables

B. Post project

review

C. Next steps

D. Formal close

Change

OUTPUT:Change programme and

presentation

A. Document

enabling

organizational

changes

B. Stakeholder roles

and

responsibilities

C. Change

management

implementation

plan

D. Presentation of

strategy

Create

OUTPUT:Strategy and

Business case

A. Establish costs

B. Define benefits

C. Document

business case

D. Document

strategy

Design

OUTPUT:Measurement

framework

A. Run workshop

with key

stakeholders

B. Create operating

model to measure

business impact

(people, process,

content,

technology)

C. Refine

measurement

methodology and

framework

D. Initial pilot

(optional)

A. Desk research and plan

stakeholder workshops

B. Document chain of

evidence

C. High level data types

(e.g. business KPIs)

D. High level data sources

(learning/business

systems)

E. Technical integration

requirements

F. Dashboard

specification &

prototyping

Inquire

OUTPUT:Prototype measurement

framework

Measuring the business impact of learning: Process for creating a documented strategy

A. Establish business

drivers

B. Prioritize critical

capabilities

C. Map stakeholders

D. Refine inquiry

method

E. Agree output

F. Create plan and

reporting protocols

Initiate

OUTPUT:The plan and schedule

© Copyright LEO Learning 2017


What percentage of your L&D budget is

committed to measuring the business impact of

learning solutions?


?


Look out for our follow up:1. Top 10 components measuring business Impact

2. 5 Steps to get started with Learning Analytics eBook

Ready to get started?

measuring the business impact of learning: lagging indicators to predictive analytics

Data & Analytics