Welcome…

…and thank you for waking up!

Metrics that Matter®

Learning Analytics Demo

Danny BrownKA Analytics Symposium 2014

San Francisco, CA

Powered by KnowledgeAdvisors

Top quotes from Day 1…

From 2013

Talent Analytics can more than double corporate productivity gains. - McKinsey

Top performing companies are 3x more likely to be sophisticated in analytics. - - MIT Sloan – IBM

What lever is your business trying to move? What needle are you trying to move? - Motorola

HR people like to look for green and CEO’s look for red. - William Sullivan

So, what is Learning Analytics anyway?

Agenda

I. MTM Technology

� Learning Analytics Demo

II. Tomorrow Morning

� Talent Analytics Suite Overview

� My esteemed colleague –Tasneem Sanwarwalla

� Same time & room

AUDIENCE POLL

INSTRUCTORS?

DEVELOPERS?

OTHERS IN THECROWD?

• To gather information related to learning measurement

• To learn what questions MTM can answer

• To see a demonstration of MTM

• Anything else?

Why are you here this early in the morning…

� How are we performing?� Which programs are driving results?� Who are our best suppliers?� Where are we misaligned?� Do we have any scrap learning?� Are we improving employee

performance?Just to name a few…

What questions will MTM answer?

Evals/Surveys

Business Metrics

� Employees� Events

360s

� Competencies� Projects

Tests

� Assessment� Certification

Integration

Talent Systems

Automate

data collection

Data Collection

Analytics

CRM TMS Finance

ERP HRIS AccountingLMS HRIS TMS

� Role-Based Reporting� Filtering & Automation� Dashboards� Executive Summaries

The Big Picture

Talent Analytics SUITES

Onboarding EDITION

Learning EDITION

Leadership EDITION

Engagement EDITION

Capabilities EDITION

Performance EDITION

� 1st Year Retention

� Speed to Performance

� Expertise & Skills

� Workforce Performance

� Manager Capability

� Strategic Vision & Execution

� Bench Strength

� Talent for Future Success

� Retention of Talent

� Workforce Productivity

� Goal Achievement

� Organizational Alignment

LearningEdition

� Expertise & Skills

� Workforce Performance

Decision Support

� Which courses should we revise or retire?

� How can we reduce scrap learning?

� Where should we focus to increase impact & ROI?

Learning Analytics

Actions

Key Metrics

Visualization

Data Collection

� Learning Dashboard

� Course Effectiveness Report

� Instructor Performance Report

� Mgr/VP/CLO Summary

� Quality of Learning

� Application to Job

� Performance Improvement

� Net Promoter

� Class Evaluations

� On-the-Job Evaluations

� Social Learning Feedback

� Test Scores

� Increase quality of programs

� Align programs to business

� Reduce scrap learning

� Improve manager support

It all begins with Smart Sheets

13

Data Process FlowLMS Enrollment• Class information

• Learner demographics

On-the-Job Evaluation• Impact, results indicators

• Post-class support input

Manager Feedback

• Impact and results

indicators and validation

• Support reinforcement

Reports & Dashboards

Evaluations/Tests/360s

• Satisfaction indicators

• Impact, results predictors

Instructor Evaluation• Feedback on content,

environment, & students

Informal PortalFeedback

Learning Analytics Example

Profile:Large Business Services CompanyMulti-Million $ Annual L&D Investment

Key Metrics:

� Program Effectiveness

� L&D Operation Efficiency

� Linkage to Business Outcomes

Solution:

� Automated post-class &on-the-job evaluations

� Monthly imports of business metrics

� Operational & executive reporting

Results:

� Strengthened partnership with business

� High ranking on Training Top 125

U

Victor1558

BrightonNewer facilitator, recently transferred to training from the field

U

TravisCurriculum Designer, leading the effort to revamp the new hire program

Victor1558

U

OliviaVP, L&D organization,

Needs to be more strategic with executives

Victor1558

U

Learning Edition

� Evaluations

� Reports

� Business Results

� Dashboards

DEMONSTRATION

Sunrise Session:Talent Analytics Suite Overview

7:30 – 8:15 am • Cathedral Hill

Tasneem SanwarwallaSolutions EngineeringKnowledgeAdvisors312-676-4460

tsanwarwalla@knowledgeadvisors.com

Let’s do this again tomorrow morning shall we…

Once again thank you!

Danny BrownDir., Solutions Engineering

+1 321.474.9220

dbrown@knowledgeadvisors.com

Contact me to:

� Discuss your situation

� Take a deeper dive

� Receive these slides

APPENDIX

MTM Sample Reports

Quarterly Executive Reporting

Dashboards

Weekly / Monthly Reporting

• Understand organizational trends• Guide alignment discussions• Monitor business learning impact

• Inform investment allocations• Monitor efficiency, effectiveness and

alignment to business outcomes

• Identify ‘localized’ areas for improvement

• Assess impacts of changes• Monitor course/ instructor

performance

• Monitor facilitation• Identify changes before

next class

Decisions Supported

© 2013 Copyright KnowledgeAdvisors

Class Reporting

Automated Reporting - Support Decisions

Actionable Insights for Executives

Alignment with the Business

The range of reports and tools available in Metrics that Matter® has given us timely, valuable insights into our programs and has allowed us to focus our energies on analyzing and interpreting the results.

Benchmark Performance

The fact that we can show how we are performing against like businesses is very powerful. That is something we never had before.

Minimize Scrap Learning

“In a 12 month period, our business alignment, courseware, environment, instructor and job impact scores all increased 5+%, while the number of responses received monthly increased 30% in the same period.”

Predictive Impact & ROIPost Event Follow Up Manager

My Data Benchmark My Data Benchmark My Data Benchmark

Linkage to Business Outcomes

“Thanks to KnowledgeAdvisors and Metrics that Matter, for the first time in P&G’s history, a training team was able to report ROI hard measures, which resulted in additional funding for 3 managers at a time when P&G was cutting back on staff.”

Automate Class Reports & Alerts

Improve Vendor & Instructor Performance

4.64

4.42

4.35

4.22

4.17

3.90

1 2 3 4 5

Leadership and Professional Development(N=195)

IT and Systems (N=2,457)

Technical Skills (N=873)

New Hire Institute (N=866)

Sales (N=320)

Corporate University (N=45,065)

Highest/Lowest Rated Learning ProvidersAVG=4.02

4.96

4.95

4.95

4.08

4.05

3.95

1 2 3 4 5

Litics, Anna (N=26)

Patel, Cal (N=22)

Ta, Dave (N=30)

Bholi, Ran (N=56)

Richard, Nina (N=37)

Orland, Martin (N=217)

Highest/Lowest Rated InstructorsAVG=4.53

“Metrics that Matter helps us understand the level of quality and effectiveness related to our programs, not just operational numbers. Most importantly, it helps us demonstrate improvement in the learner experience.”

Revise Courses with Qualitative Analysis

Do you know…

50%

38%

33%

18%

10%

Typical Company

MTM Client Averagewithin 6 Months

MTM Client Average

MTM Top 25%

MTM Top 5%

Scrap Learning