the effect of efqm self-assessments on action research performance: lessons on how to improve the...
TRANSCRIPT
The effect of EFQM self-assessments on action research performance: Lessons on how to improve the efficiency and effectiveness of
research on New Public Management
Petter Øgland, Department of Informatics, University of Oslo
NEON, 25th-27th November 2014
Plan for presentation• Motivation (3 minutes)
– Problem: Action research is important but risky design for studying NPM– Possible solution: Use TQM for controlling and improving action research
• EFQM model and double-loop learning (7 minutes)– Hypothesis 1: EFQM improves EFFICIENCY of action research– Hypothesis 2: EFQM improves EFFECTIVENESS of action research– Hypothesis 3: EFQM improves LEARNING about improving action research
• Testing the hypotheses in a NTAX action research study (7 minutes)– The NTAX action research is slow and costly– EFQM succeeds in diagnosing and suggesting treatments– Discussion: Controlling and improving AR within AR traditions based on
interpretivism, post-positivism and critical theory• Conclusions (3 minutes)
– Critical theory allows EFQM to align with action research– Action researchers should use NPM logic for improving AR performance
New Public Management (NPM): Managerial logic vs. practitioner logic
How can practitioner logic be understood in the context of an economic/managerial need to improve cost-efficiency?
Action research is a way of building knowledge by designing and evaluating social interventions (Clark, 1972)
But it is a research design with high risk for failure (Simonsen, 2007)
What if we applied the logic of NPM on action research?
1. Define the production process: Producing change and scientific knowledge
2. Improve the production process: Reducing costs and cycle time in AR process by use of TQM methods
Using the European Foundation for Quality Management (EFQM) model
Theoretical framework (Argyris & Schön, 1978)
1. EFFICIENCY of action research (strategy)
3. LEARNING HOW TO DESIGNSingle-loop and double-loop learning
2. EFFECTIVENESS of action research (goals)
Use TQM-model (EFQM) to evaluate consequence of action strategy
Hypothesis 1:EFQM improvesEFFICIENCY of action research (strategy)
Use TQM-model (EFQM) to reflect on governing variables
Hypothesis 2: EFQM improvesEFFECTIVENESS of action research (goals)
Use TQM-model (EFQM) to improve AR performance
Hypothesis 3: EFQM improves LEARNING HOW TO DESIGN Single-loop and double-loop learning
Case: Trying to improve action research at NTAX by use of EFQM
Operational hypothesis: Applying EFQM self-assessments will result in improved efficiency, effectiveness and learning on how to do Action Research.
Diagnosis:NPM strategy of outsourcing IT services, managing through contracts, and controlling through ISO 9001 audits. No audit process.
Treatment plan:Design audit process through use of action research approach. NTAX wants AR to be externally funded.
Execution and evaluation:Eleven months of trying to get funded, but no success so far.
Learning:NTAX research has to focus on AR method as no action is allowed.
Using the EFQM model to diagnose the action research project
Diagnosis AR:AR strategy of proposing to design audit process through action research fails because NTAX will not allow AR without external funding.
0
10
20
30
40
50
60
70
80
Leadership Strategy People Partnership Processes
EFQM 2013 points (enablers)
0
10
20
30
40
50
60
70
80
Customer People Society Business
EFQM 2013 points (results)
Using the EFQM model to plan treatments for action research project
Treatment AR:EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues.
0
10
20
30
40
50
60
70
80
Leadership Strategy People Partnership Processes
EFQM 2013 points (enablers)
0
10
20
30
40
50
60
70
80
Customer People Society Business
EFQM 2013 points (results)
Using the EFQM model to plan treatments for action research project
Treatment AR:EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues.
0
10
20
30
40
50
60
70
80
Leadership Strategy People Partnership Processes
EFQM 2013 points (enablers)
0
10
20
30
40
50
60
70
80
Customer People Society Business
EFQM 2013 points (results)
Sub-criteria 4a, 4b, 4c, 4d, 4e
Using the EFQM model to plan treatments for action research project
Treatment AR:EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues.
0
10
20
30
40
50
60
70
80
Leadership Strategy People Partnership Processes
EFQM 2013 points (enablers)
0
10
20
30
40
50
60
70
80
Customer People Society Business
EFQM 2013 points (results)
Sub-criteria 1c, 2a, 2c, 3b, 3c, 4a, 5b, 5c, 5d, 5e
Using the EFQM model to evaluate the action research project
Evaluation AR:Partnership with UiO improved by collaborating on grant application, but no measurable effect on customer (NTAX) because of poorly calibrated measurement methods.
0
10
20
30
40
50
60
70
80
Leadership Strategy People Partnership Processes
EFQM 2013 points (enablers) EFMQ 2014
0
10
20
30
40
50
60
70
80
Customer People Society Business
EFQM 2013 points (results) EFQM 2014
Using the EFQM model to reflect on the action research project
Validity and reliability:When comparing with EFQM results with earlier experiments, the observed change from 2013 to 2014 is not significant.
0
100
200
300
400
500
600
700
2000
2002
2004
2006
2008
2010
2012
2014
EFQM total AVG = 284.7UCL = 602.9 LCL = 0.0
Learning AR:Use feedback from rejection for writing new grant application.
Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project• Diagnosis AR performance• Plan for improving AR
performance• Execute plan and record
what happens• Evaluate new AR
performance level• Reflect on causes for
change in AR performance
Use of EFQM model• EFQM self-assessment• Use EFQM theory about
causes and effects• New EFQM assessment of
enablers• New EFQM assessment of
results• Observed change was not
statistically significant
Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project• Diagnosis AR performance• Plan for improving AR
performance• Execute plan and record
what happens• Evaluate new AR
performance level• Reflect on causes for
change in AR performance
Use of EFQM model• EFQM self-assessment• Use EFQM theory about
causes and effects• New EFQM assessment of
enablers• New EFQM assessment of
results• Observed change was not
statistically significant
Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project• Diagnosis AR performance• Plan for improving AR
performance• Execute plan and record
what happens• Evaluate new AR
performance level• Reflect on causes for
change in AR performance
Use of EFQM model• EFQM self-assessment• Use EFQM theory about
causes and effects• New EFQM assessment of
enablers• New EFQM assessment of
results• Observed change was not
statistically significant
Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project• Diagnosis AR performance• Plan for improving AR
performance• Execute plan and record
what happens• Evaluate new AR
performance level• Reflect on causes for
change in AR performance
Use of EFQM model• EFQM self-assessment• Use EFQM theory about
causes and effects• New EFQM assessment of
enablers• New EFQM assessment of
results• Observed change was not
statistically significant
Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project• Diagnosis AR performance• Plan for improving AR
performance• Execute plan and record
what happens• Evaluate new AR
performance level• Reflect on causes for
change in AR performance
Use of EFQM model• EFQM self-assessment• Use EFQM theory about
causes and effects• New EFQM assessment of
enablers• New EFQM assessment of
results• Observed change was not
statistically significant
Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project• Diagnosis AR performance• Plan for improving AR
performance• Execute plan and record
what happens• Evaluate new AR
performance level• Reflect on causes for
change in AR performance
Use of EFQM model• EFQM self-assessment• Use EFQM theory about
causes and effects• New EFQM assessment of
enablers• New EFQM assessment of
results• Observed change was not
statistically significant
Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project• Diagnosis AR performance• Plan for improving AR
performance• Execute plan and record
what happens• Evaluate new AR
performance level• Reflect on causes for
change in AR performance
Use of EFQM model• EFQM self-assessment• Use EFQM theory about
causes and effects• New EFQM assessment of
enablers• New EFQM assessment of
results• Observed change was not
statistically significant
H1, H2 & H3: Confirmed by qualitative dataH1 & H2: Not confirmed by quantitative data
Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
RICH DESCRIPTIONS are of limited use if they do not result in better treatments
Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
RICH DESCRIPTIONS are of limited use if they do not result in better treatments
EFQM suggests that TREATMENTS should be evaluated on objective terms
Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING, but not as a goal in itself
RICH DESCRIPTIONS are of limited use if they do not result in better treatments
EFQM suggests that TREATMENTS should be evaluated on objective terms
Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS
Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS
The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context
Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS
The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context
An interpretive approach is needed for DIAGNOSING and REFLECTING on the outcomes of the TREATMENTS within the local context
Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS, but not as a goal in itself
The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context
An interpretive approach is needed for DIAGNOSING and REFLECTING on the outcomes of the TREATMENTS within the local context
Discussion: Related research - critical AR approach
Herbert Simon was part of the progressive “research and reform” movement in Chicago in the 1930s
Discussion: Related research - critical AR approach
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)
Discussion: Related research - critical AR approach
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical theory (Elster, 1982)
Discussion: Related research - critical AR approach
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical theory (Elster, 1982)
Game theory as a foundation for social theory in general (Gintis, 2009)
Discussion: Related research - critical AR approach
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical theory (Elster, 1982)
Game theory as a foundation for social theory in general (Gintis, 2009)
√
Conclusion: Contributions to theoryH1: EFQM improves AR efficiencyThe action strategy is improved through SPC and related TQM methods
Conclusion: Contributions to theoryH1: EFQM improves AR efficiencyThe action strategy is improved through SPC and related TQM methods
H2: EFQM improves AR effectivenessThe governing variables are updated due to causal relationships in EFQM model
Conclusion: Contributions to theoryH1: EFQM improves AR efficiencyThe action strategy is improved through SPC and related TQM methods
H2: EFQM improves AR effectivenessThe governing variables are updated due to causal relationships in EFQM model
H3: EFQM improves AR learningBy conceptualising the social situation as a game it can be analysed in a rational manner (“empirical economics”)
Conclusion: Contributions to practice
Political consciousnessNPM logic is compatible with practitioner logic when viewing the conflict between the two groups through the use of game theory
Conclusion: Contributions to practice
Political consciousnessNPM logic is compatible with practitioner logic when viewing the conflict between the two groups through the use of game theory
Conflict resolutionAction researchers should use NPM logic for assessing and improving their own strategic performance within the game
Summary of presentation• Contributions to theory and implications for practice
– Critical theory allows EFQM to align with action research– Action researchers should use NPM logic for improving AR performance
• Why was critical theory important in this context?– The NTAX action research process needs to be improved– EFQM is successfully used for improving the NTAX action research process– Discussion: Comparison of EFQM approach for improving AR with literature on AR
improvement based interpretivism, post-positivism and critical theory• Why did we believe the EFQM model would work?
– Hypothesis 1: EFQM improves EFFICIENCY of action research– Hypothesis 2: EFQM improves EFFECTIVENESS of action research– Hypothesis 3: EFQM improves LEARNING about improving action research
• Why was this interesting?– Problem: Action research is important but risky design for studying NPM– Possible solution: Use TQM for controlling and improving action research