Five Keys for Success in Outcomes Assessment

Presented by:

Jeremy C. Lundberg, MSSW

Derek T. Dietze, MA, FACEHP, CCMEP

Christopher Viereck, PhD, CCMEP

EthosCE LMSEnterprise-level learning management system.

2012 Brandon Hall Award-Winner

Designed for CME and ACCME PARS compliance.

Integrated assessment, evaluation, and certificate.

Personal “My CME” transcript.

RSS Management

Automates and streamlines “back office” administration.

Powerful 24/7 reporting engine.

Common Model in CME

Meeting Logistics

Online Courses EvaluationsAssessments

A Strategic Model for CME

HR Management System

3rd Party Applications

Learning Management System

External Education Partners

Association Management System

Question 1. What are your challenges with outcomes assessment?(Check all that apply)

1. I don’t know where to start

2. I don’t understand the differences in outcomes levels

3. I can’t get the level of outcomes data I want

4. I have trouble linking gaps to learning objectives, to outcomes

5. I don’t know how to meet commercial supporter expectations

1. Plan Ahead…Way Ahead

• Create a detailed descriptive outcomes assessment plan at the earliest stages of grant request or activity development

• Develop assessment questions in concert with content, not as an afterthought

• Will you have enough resources available to create assessment questions, implement the plan, and analyze/report the results?

• Plan with the end in mind

• Examples

2. Measure to YourEducational Design

• What do you expect to change in your learners?

• In what domains do these expected changes fall?

• Knowledge/awareness, confidence, competence, performance, patient outcomes

• Accurately represent the Moore level(s)

• For higher outcomes levels, distinguish between subjective and objective assessment

2. Measure to YourEducational Design (cont.)

• If your learning objectives are focused on knowledge and competence change, why assess performance change or patient outcomes?

• When you measure to your design you have: • Enhanced credibility• More effective use of resources

Question 2. Please rate your confidence in optimally aligning outcomes questions with

learning objectives and gaps.

1. Not confident at all

2. Not very confident

3. Somewhat confident

4. Very confident

5. Extremely confident

3. Align, Align, Align

• Bring all of the following into alignment:• Gaps• Learning objectives• Desired results• Instructional design• Content• Format• Outcomes questions

3. Align, Align, Align (cont.)

• Demonstrate alignment in your planning documents

• Examples:

• Each learning objective is linked to a specific gap

• Each outcomes assessment question is linked to a specific learning objective

• When your alignment is good: much greater opportunities for positive results

Question 3. When you promise “Level 5 Outcomes Results”, what does this really mean

to you?

1. Chart pull data on participant performance

2. Performance changes self-reported by participants

3. Planned performance changes

4. Patient outcomes reported by a participant

5. I have absolutely no idea!

4. Keep the PlanSimple and Specific

• Best option: Describe what you are going to measure, how you are going measure it, and when you plan to deliver on your promises.

• Be sure to include:• Outcomes objectives and overall methods

• Activity-specific outcomes details

• Your approach to statistical analysis

• What is in the results reports, and when they will be delivered

• How you will share results?

4. Keep the PlanSimple and Specific (cont.)

• Your outcomes plan is the implementation guide

• Complex, overly ambiguous or ambitious plans work against you:

• Ambiguous plans create misunderstandings

• Falling short of original plans hurts your long-term credibility

• Reality check: Do you have the internal and external resources?

5. Use Multiple Types of Assessment Questions

• Don’t put all your eggs in one basket by using only one question type.

• Consider using more than one question type to address your learning objectives.

• Example: Use a combination of pre/post multiple-choice, case-based or agreement scale questions.

• Mandatory: Align the assessment questions with the gaps and learning objectives.

Five Keys for Success in Outcomes Assessment

• Plan Ahead…Way Ahead

• Measure to Your Educational Design

• Align, Align, Align

• Keep the Plan Simple and Specific

• Use Multiple Types of Assessment Questions

Leverage the Value of Your Outcomes Results

• Summarize your results succinctly

• Validate your needs assessment

• Determine the degree to which your learning objectives were met

• Relative effectiveness of different question types

• Identify effective versus less effective content

• Inform future content development and improvements

Leverage the Value of Your Outcomes Results (cont.)

• Obtain more financial support: insert results in an upcoming grant request

• Assist commercial supporters in selling the value of CME within their organization

• Justify department budgets

• Allocate resources more efficiently

• Share results in clinical and CME community

Tools for Success

Image of EthosCE report screen

Collaborative partnerships

Question & Answer

Thank YouJeremy C. Lundberg, MSSW

(267) 234-7401jlundberg@dlc-solutions.com

www.EthosCE.comderek.dietze@improvecme.com

christopher.viereck@improvecme.com