European Lean Educator Conference 2017 (ELEC 2017)

Nijmegen, 7th – 10th November 2017

Measuring Lean

Competencies – An

Approach for

Quantifying the

Learning Outcome

of a Simulation

Game

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 2

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 3

Introduction Technology Centre PULS (Production and

Logistics Systems)

4 professors with

doctoral students

and administration

personnel

Intensive

cooperation with

industry and science

State of the art and

best practice centre

for Lean Production

and Lean Logistics

15.11.2017 Landshut University of Applied Sciences, Germany 4

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 5

Problems evaluating Competence Transfer of Lean

Simulation Games

Simulation games

in Lean education

Few empirical

analysis of learning

effectiveness

Subjective

measurement of

competency

development

15.11.2017 Landshut University of Applied Sciences, Germany 6

Source: own representation based on (Schenk und Wojanowski 2004; Beckmann 2004)

Research question: Does the participation in a Lean simulation game

increase statistically significant the Lean competency of the participants?

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 7

Measuring the Effectiveness of Simulation Games

15.11.2017 Landshut University of Applied Sciences, Germany 8

Source: own representation based on (North und Reinhardt 2005)

Measuring the Effectiveness of Simulation Games

Competency Transfer Table

15.11.2017 Landshut University of Applied Sciences, Germany 9

Source: own representation according to (Tisch et al. 2014)

Measuring the Effectiveness of Simulation Games

Approach

1. Identification of competencies and respective activities

2. Creation of a scenario, where it is possible to carry out the activities

repeatedly

3. Communication of the starting scenario and the scenario targets to

the participants

15.11.2017 Landshut University of Applied Sciences, Germany 10

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 11

Lean Simulation Game “From Push to Pull”

Competencies:

design of a waste-free internal

and external logistics system

with the use of Lean Logistics

methods

design of a waste-free

workplace with focus on high

quality and maximum

productivity

design of a customer oriented

and holistic production

system, based on the

principles of Lean Production15.11.2017 Landshut University of Applied Sciences, Germany 12

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 13

Concept of Measuring the Learning Outcome of a Lean

Simulation Game

15.11.2017 Landshut University of Applied Sciences, Germany 14

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 15

Application and Results of the Measuring Concept

Pre- and Post-Test

15.11.2017 Landshut University of Applied Sciences, Germany 16

Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7

pre-test 52% 47% 49% 56% 57% 46% 44%

post-test 57% 66% 60% 60% 66% 57% 62%

30%

35%

40%

45%

50%

55%

60%

65%

70%

perc

en

tage

of

corr

ect

an

sw

ers

Results of the pre- and post-test

pre-test post-test

Application and Results of the Measuring Concept

Push and Pull Systems

Correlation of evaluation results of different game roles

Immersion Avg. results 1 (push) Avg. results 2 (pull)

1 = time measurement 4,21 5,53

2 = quality 3,93 5,80

3 = logistics 4,14 6,14

4 = secondary assembly 4,33 4,74

5 = main assembly 3,77 6,32

r = - 0,33 0,13

15.11.2017 Landshut University of Applied Sciences, Germany 17

No statistically significant correlation between game role and learning

outcome

Application and Results of the Measuring Concept

Interviews

15.11.2017 Landshut University of Applied Sciences, Germany 18

Correlation of evaluation results of different game roles

# Interviewee 1 2 3 4 5 6 7 8 ØØ in

%

Result logistics task (max. 4) 4 1 2 4 3 3 2 4 2,9 72 %

Result case study (max. 5) 5 2 3 3 4 3 1 4 3,1 63 %

Self-evaluation about attained

new competencies(1 = not the case; 5 = fully applies)

4 3 5 4 5 5 4 4 4,3

Would you prefer a more realistic

gaming environment?(1 = not the case; 5 = fully applies)

4 2 1 3 2 1 2 1 2,0

Correlation of self-evaluation to

objective resultsr = 0,418

Agenda

1. Introduction Technology Centre PULS

2. Problems evaluating Competence Transfer of Lean Simulation

Games

3. Measuring the Effectiveness of Simulation Games

4. Lean Simulation Game “From Push to Pull”

5. Concept of Measuring the Learning Outcome of a Lean Simulation

Game

6. Application and Results of the Measuring Concept

7. Conclusion

15.11.2017 Landshut University of Applied Sciences, Germany 19

Conclusion

Competency transfer table successful

Growth in knowledge easily determinable

Observation of actions during game rounds difficult

Interviews necessary for observation of actions

Recommendation:

parallel design of evaluation concept and simulation game

15.11.2017 Landshut University of Applied Sciences, Germany 20

Technology Centre PULS

(Production and Logistics Systems)

Bräuhausgasse 33 ∙ D-84130 Dingolfing

www.tz-puls.de

Stefan Blöchl, M.Eng.

Research Assistant and

doctoral candidate on the

field of Industry 4.0 in

production logistics

Mathias Michalicki, M.Eng.

Research Assistant and

doctoral candidate on the

field of Controlling for Lean