The Deming Wheel

• From the graphic point of view, P.D.C.A. is represented by a moving circle

• The movement stands for dynamism and continuity of the application process.

From the graphic point of view, P.D.C.A. is represented by a moving circle called the Deming wheel.The movement stands for dynamism and continuity of the application process.P D C A CYCLE

http://www.iwolm.com/en/the-pdca-method-or-deming-wheel-for-your-improvement/11

Theoretical Frameworks Utilization Focused (Patton)

• Evaluation of effectiveness by potential users• Who might use• What will be used• Intention to use

12

Evaluation Techniques

Logic Models

“A graphic representation of a program that describes the program’s essential components and expected accomplishments and conveys the logical relationship between these components and their outcoms.” (Conrad et al. 1999)

Logic models guide evaluation

http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

14

How the Logic Model Guides Evaluation

•Provides the program description• Aids in matching evaluation to the program• Identifies what and when to measure

• Are you interested in process and/or outcomes?

• Keeps focus on key, important information• What do we really need to know??• Where should limited evaluation resources be used?

15

OUTPUTS

Basic Logic Model

INPUTS OUTCOMES

ParticipantsActivities

16

Input

Products (teaching components)

InputsInvestment/Resources

Time

People

Money

Materials

17

Output

Activities ParticipationWhat is done Who receives

Training/teaching Counts

Products (teaching components) Description

Develop resources Satisfaction

Form collaborations/partnerships

Assessment tools and assessment

18

Outcomes

Primary Secondary Tertiary Immediate

(Short term)Behavior Results

(Long-term) Knowledge Teaching Adaptation

Skills Practice Standardization

Abilities Improved practices

Changes in attitudes

Assessments (tools and assessment)

19

When to evaluate?

• Before the experience/event/class• Mini-assessments within (quiz, exercise, etc)• Post-test only• Pre-test and Post-test• Retrospective Pretest and Post-test• Pre-test, Post-test, and follow-up• Intermediate testing

• Can be combined with pre and post test designs

20

Evaluation Design

• Experimental• Uses random assignment, control

• Quasi-experimental• Groups from “natural” characteristics

• Males to females

• Class A to Class B

• Non-experimental • Compares before and after

21

Types of Assessment

• Qualitative• Open-ended

• Quantitative• Constrained Choice

• Mixed Method• Using both quantitative and qualitative techniques

to collect data

22

Mixed Methods Evaluation

• Enhances both formative and summative evaluation

• Triangulation from combining both types of evaluation• Merging• Explanatory • Exploratory

23

Mixed Methods Evaluation - Merging

• Triangulation from combining both types of evaluation• Merging

Interpretation

Qualitative Quantitative

24

Mixed Methods Evaluation - Explanatory

• Triangulation from combining both types of evaluation

• Explanatory • Quantitative, then Qualitative, followed by interpretation

Quantitative QualitativeIdentify Issues Interpretation

25

Mixed Methods Evaluation - Exploratory

• Triangulation from combining both types of evaluation

• Exploratory

Qualitative QuantitativeDesign Measure Interpretation

26

Mixed Methods Evaluation

• Triangulation from combining both types of evaluation• Merging

• Explanatory • Quantitative, then Qualitative, followed by interpretation

• Exploratory

Interpretation

Quantitative

Qualitative

QualitativeIdentify Issues

Quantitative

Interpretation

Qualitative QuantitativeDesign Measure Interpretation

27

Selecting the Evaluation Method

• Participation records• Self-report • Achievement (knowledge) tests• Interviews• Focus groups• Direct observations • Medical record reviews• Product count

28

Example

29

Example

30

Examples

31

1.1 The percent of adults aged 65 and older in the United States is currently approximately ________ and will increase to ________ by the middle of the century.a. 10% 25% b. 12% 20% c. 12% 15% *d. 13% 20% e. 13% 25%

1.2 Persons reaching age 65 have an average life-expectancy of an additional _____ years.a. 5 b. 9 c. 10 d. 15 *e. 19

Examples:

32

Example Faculty Scholars Program Evaluation - Focus Group Questions

Group Leader (minimal input only, to encourage participation, and probe for responses): _______________________________________

Note Takers (should not speak or interact with participants, not necessary to link participants to responses except for question 1:

_________________________ ___________________________

1. Introductory questions: Pass out index cards and ask to have these two questions written. After collecting cards, go around the room and ask for answers. Note numbers of persons who find any specific topic useful, even if not the “most useful.” If a card notes that “Advanced Directives” was really useful, ask how many others also found this material pertinent. It will be interesting to all to learn how implementation will be achieved.

What did you find the most useful topic in the program this year?

How have you or how do you plan to implement what you have learned?

2. Transition questions

Are there topic areas that should be added to the program? Deleted? More or less emphasis?

3. Key questions

What are your overall thoughts about the program?

Communication: PROBE: Communication about the program? (e-mail, phone conversations, reminders, etc.)

How did you like/use: the WIKI, jump drives, hard copies? Do you have suggestions for the Newsletter guideline?

Content: PROBE: in-person sessions, self-study materials, conference?

Organization: PROBE: Smaller work teams? Group work? Self-directed projects?

How was the interaction with faculty scholars from other disciplines?

5. Ending questions

If there was one thing you would suggest . . . 33

Evaluation: Quality Improvement vs. Research

• Institution-specific• Reporting to stakeholders is not research

• Funders• Collaborators

• UAB Institutional Review Board defines research as any presentation in a local or national forum• IRB approval is needed

34

American Evaluation Association

• Founded 1986• http://www.eval.org/p/cm/ld/fid=1

35

Evaluation Strategies for

Educational Interventions

November 1, 2013 GEC Faculty Scholars Program

Patricia Sawyer, PhD