“smart” or not so smart? benedict wauters. the traditional view… to increase performance...
TRANSCRIPT
“SMART” or not so smart?Benedict Wauters
2
The traditional view…
• To increase performance managers are told they need to motivate their people as follows: Set stretch goals: targets
• on outputs to stimulate improving internal processes: under direct influence allows to provide quick feed-back
• after outputs covered, also include outcome targets to stimulate to work across boundaries
however, long time lags, many influences
Next, celebrate achievement (usually not failure)• Leaders lead the celebration and ensure renewed focus
R. Behn, Public Administration Review, 2003
SMART?
3
• specific: target a specific area for improvement;• measurable: quantify or at least suggest an
indicator of progress;• assignable: specify who will do it;• realistic: state what results can be realistically
achieved, given available resources;• time-related: specify when the results can be
achieved.Source: Doran, 1981
SMART
SMART?
4
What does the scientific evidence tell us?
5
Benedict Wauters
SMART or not: are simple management recipes useful to improve performance in a complex world? A critical reflection based on the experience of the Flemish ESF Agency.
Paper submitted for the conference on “Prestaties van organisaties in de publieke sector: van wegen naar gewicht verliezen” / “Performance of public sector organisations: from weighing to
losing weight.”
Politicologenetmaal Gent on 30 and 31 mei 2013
Further reading
The trouble with targets and motivation
1. Goals setting matter for performance
2. What we measure gets done, including by cheating, gaming etc.
3. A picture says more than a 1000 words (or some numbers)
4. Goals motivate, but what really counts in a complex world is the type of motivation
6
1. Goals matter for performance
• Challenging, quantified goals lead to more focus, energy, persistence, smarter work and hence… performance!
• …but then there also needs to be committment for the goal:because one believes one can achieve it (self-efficacy):
helped by role models, training, … and leaders who communicate trust and help reflect about HOW to get things done
because one believes it is important: helped by a broader vision and public committment to it from a supportive leader, participation in goal setting, explaining the rationale of the goal
7
8
1. Goals matter for performance
• …and positive feed-back is crucial: Explain what people did right, which strengthens self-efficacy
again… …in combination with knowing what are obstacles and how to
deal with them
• ….but if the limits of what one is able (capacities) and allowed (environment, e.g. work overload) to do are reached, performance drops again “The assignment of ambitious goals without any guidance on
ways to attain them often lead to stress, pressures on personal time, burnout, and in some instances unethical behavior. It is both foolish and immoral for organizations to assign stretch goals, and then fail to give employees the means to succeed...” Seijts et al (2005)
9
Don’t set challenging goals if you do not know how to reach them
1. Goals matter for performance
11
1. Goals matter for performance
• Two (very important) caveats: the theory operates only at the individual task level :
• Not possible to determine at an aggregate level the constraints from the environment nor the capacities of all people for all tasks
in many (probably most) cases learning goals rather than performance goals are required!
• eg. not “ensure x% more participants with a job after 6 months”, but “search and try 5 strategies to…”
• also challenging and quantitative!
12
Target setting at the aggregate level…
1. Goals matter for performance
13
In many cases however, HOW to achieve a goal is all but clear and you need to set a learning goal
1. Goals matter for performance
14
1. Goals matter for performance
• Learning goals –when? “tasks for which minimal prior learning or performance routines
exist, or tasks where strategies that were once effective suddenly cease to be so, relocate the purpose of or benefit of goal setting from one of primarily motivation to that of knowledge acquisition, environmental scanning, and seeking feed-back”
“Meta-cognition is particularly necessary in environments with minimal structure or guidance”
• Why? Because with a performance goals “Yes, I can” (self-efficacy) quickly becomes, in the above context, “No, I can’t”, which leads to putting in less effort and taking less risk (needed here!) than for a learning goal
• The issue then becomes how you can determine in practice (rather than in a controlled experiment) if a learning goal is required? Perhaps most of the time?
16Andy Neely et al. How to avoid the problems of target setting, 2010, PMO symposium
*Source: Andy Neely is widely recognised as one of the world's leading authorities on organisational performance measurement and management. He has authored over 100 books and articles, including "Measuring Business Performance", published by the Economist and "The Performance Prism", published by the Financial Times. He has won numerous awards for his research and chairs the Performance Measurement Association, an international network for those interested in the performance measurement and management.
• A. Neely also affirms that what is not problematic are overall (beneficial) goals: “Close as you can”: pursuit of perfection without actually assuming you
can achieve this e.g. no accidents, zero defects, getting nearer to the best (benchmark),… Indeed, it can never make sense to have as a target “x% of employees have an accident”
“Far as you can”: when there is no concept of “perfection” to get close to, but rather a concept of going further and further e.g. increasing the number of website visitors
2. What gets measured, gets done…
“There is some evidence that targets and such “carrots and sticks” work, particularly if the desired outcome is focussed and measurable, as in the case of hospital waiting times…
…The two assumptions underlying such governance structures don’t hold for public service delivery, however: measurement error is an inherent problem, as is the resultant potential for undesired as well as desired responses, and the evidence bears this out.”
17UK ESRC, 2010
18
2. What gets measured, gets done…
19
2. What gets measured, gets done…
• Question: do you think the medical staff and teachers are to blame?
20
2. What gets measured, gets done…
• Measurement problems are to blame … Impossible to measure all important dimensions of
performance with numbers (but possible to quantify many things that do not really matter)
The same indicators can have many different interpretations (eg. “a job”, any job?) and hide crucial difference for sub-groups
Descriptive indicators say little about the “why” of the number (cause-effect)
Data on performance sometimes has to come from the same organisation/person that is being judged
21
2. What gets measured, gets done…• …and are less pronounced for some types
of business than others…
22
Pollit, Logic and limits of performance measurement. EGPA conference in Bergen of september 2012Pollit, Integrating financial and performance management OECD Journal on budgeting 2001
Producer services (tangible, standard)(e.g. issuing driver’s licences, building roads, mail service, routine tax collection): repetitive and standardised tasks
Procedural services (non-tangible ideal)(e.g. policy advice, coordination activities, regulation, diplomacy): less standardised / routine than producer services, many “other” influences
Craft services (e.g. forestry park rangers, detective work, medicine): we will know if things got better but cannot easily know/understand how
Coping organisations (non-tangible, individually tailored)(counselling e.g. for mental health): some routine aspects but mostly adjusting to individual, personal needs and contexts
EASILY MEASURABLE OUTCOMES
EA
SIL
Y M
EA
SU
RA
BLE
O
UT
PU
TS
/ U
NIT
CO
ST
yes
no
yes noLess issues: clear what good performance is, easy to measure
Requires more learning oriented use of performance info
23
2. What gets measured, gets done…
• …and lead to behavioral issues:
unconscious:
• blind for unintended consequences and what is not
explicitly measured
• We tend to ignore qualitative info and focus on
quantitative
• Risk for goals or strategies (that were succesfull in
the past) to become ‘sacred’
24
2. What gets measured, gets done…
conscious
• cheating: tampering with data or reporting
• gaming:
Ratchet effects: not wanting to surpass a target because
the next one will be based on it
Threshold effects: mediocrity institutialised at
organisational level
Output distortion / goal displacement: creaming, parking,
selective interpretation of indicator (eg a “job”), shifting
the burden (exclude weaker students from exams)
25
An example of conscious unwanted behaviour…
2. What gets measured, gets done…
• … the latter especially if there is salience…due to reputational risk, job security, earned
autonomy, budgets……and reward (incl. money)
26
“…never answered the question what has to happen if output targets are not achieved… In practice we see very few sanctions, but in theory sanctions are advocated and the threat of sanctions is always in the air. This perverse incentive leads to manipulation of data…and gaming… No campaign to promote civil service values can compensate for that…. Everything we knew about planning in socialist states but now limited to the public sector of a market economy”. OECD, 2011
“An ethical organizational culture can reign in the harmful effects of goal setting, but at the same time, the use of goals can influence organizational culture. Specifically, the use of goal setting… creates a focus on ends rather than means…. goal setting impedes ethical decision making by making it harder for employees to recognize ethical issues and easier for them to rationalize unethical behavior.”
27
Ordonez et al, Harvard Business School (2009)
28
2. What gets measured, gets done…• Chris Pollit also reports two more downstream consequences
Logic of escalation: • Because of measurement problems, we will have a tendency to have an
ever growing arsenal of measures • These represent a temptation to be used to link to incentives• Struggles then become more frequent and fierce on what how and what to
measure• This leads to the rise of an expert community (statisticans, consultants,
researchers, ….) especially in politically salient and complex contexts (such as education and health care), less in simpler services such as issuing drivers licences or in more consensual /trusting cultures
Performance paradox:• Performance indicators tend to wear out : they become less and less
discriminatory between good and poor performance Due to perverse learning (how to cheat and game) Due to actually provoking a convergence of performance (down or upwards)
Pollit, Logic and limits of performance measurement. EGPA conference in Bergen of september 2012
For info
2. What gets measured, gets done…• Many of the studies that report problems with performance
management come from the US and the UK: Some countries (eg Nordic ones) are less comfortable with playing
blaming and shaming games but see performance data rather as one element among others in an ongoing dialogue between different levels of management, embedded in a more stable, mutually trusting organisational relation and a consensus oriented culture
Some cultures have different attitudes towards the acceptance of gaming and cheating
Well-equipped, well-trained civil servants can make better use of performance data than under-resourced ones
The more actors (ministers, top officials, technocrats, operational staff, legislatures, mass media, citizens) involved, the more interpretations and the higher the chance of conflict
• Nevertheless, all problems have been reported in all contexts
29Pollit, Logic and limits of performance measurement. EGPA conference in Bergen of september 2012
For info
2. What gets measured, gets done…
• Some routes to resolve the problems? extensive testing and revising indicators, with clarity about
limitations (!) keeping measures unknown or unpredictable e.g. unanounced
inspections, vague standards, changing indicators (however, if there is little stability, there is little chance to learn)
multiple indicators AND methods (eg. strategic stories like in Sensemaker, Most Significant Change) concerning outcomes AND process, in combination with “soft judgement”
allowing “dirty” data but not “bad” data accountability (towards external stakeholders) NOT for meeting
targets but for showing one posed the right questions, looked for answers, took action based on the answers
30
31
3. A picture tells more than a thousand words (or a number)
• Mentally visualising the future :Outcomes:
• visualising succes leads to feeling “good” but this relaxes people rather than energise them
Process to achieve outcomes:• brings information as well as a pre-taster of our emotional
reaction • better planning, higher ambition, less fear lead to better
outcome• But necessary to ensure outcome is clear first as trade-offs
between various possible outcomes and processes delay decison-making and committment
A picture says more than a thousands words, or a few numbers…
3. A picture tells more than a thousand words (or a number)
32
4. Understanding motivation
33
Do it for external
punishment /reward
Don’t do it or on automatic pilot / zombie mode
Do it forglory, pride,
avoiding shame and
fear
Do it because your recognise it as important
Do it because it fits with who
you are
Do it because you find it enjoyable as such
4. Understanding motivation
• If we have so a strong, natural tendency “to seek out challenges, novelty and opportunities to learn”, what went wrong, where?
• It started in our education: The name of the game was NOT learning but performing:
mistakes were punished, correct answers rewarded If you don’t have the right answer, keep silent Knowledge was always something the teacher had, not the
student (authority figure with all the answers –ring a bell?) If we operated like that as 2 year olds, we would never have
learned to walk
35
* P. Senge, It’s the learning: the real lesson of the quality movement, The journal for quality and participation, 1999
For info
SMART or STUPID?• SMART:
regarding specific tasks, in coherence with higher level more abstract goals (specific)…
support the use of mental simulation (e.g in the form of stories) focusing on the process of achieving an outcome, itself mentally simulated and framed as a “growth” oriented goal (measurable)…
at the level of individual staff (assignable) not of the organization or a unit…
ensuring there is enough thought put into considering the complexity of a task as well as constraints, both in terms of what people are able and allowed to do, and hence deciding on appropriately challenging learning or performance objectives linked to strategies that help achieve them (realistic)…
within a specific time constraint relevant to the task (time bound)• SMART should be accompanied with positive feed-back
36
37
SMART or STUPID?
• SMART and “smart” feed-back: Positive, constructive feed-back (offering information about what
was done well)… relying on soft judgment of (potentially “dirty” but not “bad”) data
collected with a variety of methods (accompanied by an understanding of their limitations)…
should be used by those that can make use of it, to guarantee a focus on removing obstacles - rather than on being fixated on the degree of shortfall- towards a goal.
• This kind of feed-back is closely linked with accountability being delivered by “the way people give an account of what they have done and why, rather than describing in a more limited way if they have hit a target or not”.
38
SMART or STUPID?
• STUPID: high level, uniform quantitative targets (specific)… within a standard (mostly one year) period, regardless of the
task (timed)… completely divorced from any discussion regarding constraints
or complexity (unrealistic)… driven by external pressure (pressurized)… blissfully ignorant of any strategies that could actually help
realize the targets (ignorant)… hence decreasing autonomous and intrinsic motivation
(demotivation) and ensuing well-being and performance.
• STUPID also tends to go hand in hand with negative feed-back (not hitting the target)
Use of performance information
1. Next to “motivate”…
2. To budget/plan
3. To control
4. To promote
5. To learn
6. To evaluate
7. …ultimately, to improve
39
Bob Simons is the Charles M. Williams Professor of Business Administration at Harvard Business School. Over the last 30 years, Simons has taught accounting, management control, and strategy implementation courses in both the Harvard MBA and Executive Education Programs.
Robert D. Behn, Lecturer in Public Policy, focuses his research, teaching, and thinking on the leadership challenge of improving the performance of public agencies. He is the faculty chair of the School's executive program, Driving Government Performance: Leadership Strategies that Produce Results and conducts custom-designed executive education programs for government jurisdictions and public agencies.
• The Netherlands and wider in the OECD: “When looking at…the Netherlands, it seems fair to
say that performance information has been used by line ministries extensively for compliance and legitimisation…”
“The 2011 OECD Performance Budgeting Survey shows that line ministries most commonly use PI to increase spending in their negotiations with central budget authorities”
“Experience in the Netherlands and other OECD demonstrates that the effects of PBB on budgetary deliberations by parliaments* have been nearly absent”
41M. De Jong et al., OECD Journal on Budgeting, 2013
*after finance/budget minister sends it to Parliament
42
Budgeting/planning -1• From budget/planning to improvement? Ensure
readyness! we need to estimate (predictable) demand AND we need to
estimate our required capacity in terms of people and resources (in diverse parts of an organisation that need to coordinate) to meet this demand
Then add reserve capacity for the uncertain demand How? Based on how we have been doing before, given system
conditions (stock-taking AFTER the fact) Otherwise, we will NOT have the required capacities in place
and will fail to satisfactorily meet demand Budgeting therefore is an exercise in ensuring the organisation
is as ready as possible as, naturally, performance will be lower for unprepared organisations than for prepared ones
However, this clearly has nothing to do with setting targets
43
Budgeting / planning -2
At the macro level in government?• usually political priorities determine envelopes at macro level
(should we invest in roads or in education?) • performance budgeting created the expectation that overall
government performance can be improved by reducing/increasing budgets if poor/good performance in terms of “output”
• but how would government know if past output for the money is sufficient or could be improved?
better to know if service provision relative to its purpose (in the eyes of the customer) is under control and if there is waste
money is not always the answer e.g. lack of leadership in rethinking the system comes first
• only evaluation can determine which services deliver more/less outcomes and why
• government at macro level should engage its public service providers on this!
R. Behn, Public Administration Review, 2003M. De Jong et al., OECD Journal on Budgeting, 2013
Budgeting/planning -3
Demand plan: planned (in/decrease of predictable demand)
Capacity requirements + free capacity maintained (in % used of available people and facilities)
Factoring in expected process improvements
Direct operational, maintenance AND improvement costs
= OPEXCapital expenditure
(replacement/expansion of existing facilities for current business)
=CAPEXIndirect operational cost=OPEX
Strategic initiatives
STRATEX(can be
bothexpenses
orcapitalisa
tion)
PLANS
BUDGET
Operational plans (with a one year time horizon) specify how much demand is expected to beaddressed by which parts of the entity.
45
Structural Funds programming• Also for the programmes in Structural Funds, it makes sense to
budget/plan outputs: How much output do we think the budget can buy?
• Otherwise, project promotors / beneficiaries could be very disappointed if they find out there is no money for them
• Or expectations of how much the money would deliver could be seriously wrong For known, well-understood, repeat services:
• We should know how much it cost in the past, which is a good estimate if there is no reason to think it will be more/less in the future
For complex projects (innovation, solutions development):• Output = project • Budget = maximum budget a project can have
But it is a mistake to turn this into YES/NO targets, with reward and punishment attached!
• As to “results” (outcomes): only evaluation can tell us what we really achieved (after the fact) hence it makes no sense to set “targets” and incentivise them on results
46
• “…ensure that measures used for planning and budgeting purposes are not confused with measures used for improvement and development”.
Andy Neely et al. How to avoid the problems of target setting, 2010, PMO symposium
47
Learning and evaluation-1
• From learning and evaluation to improving? What is (not) working AND why* To get better understanding in general (enlightenment) for those
involved (new insights, new ideas, changed perceptions)* Not so easy:
• we observe only what we measure, which may not matter or show random correlations
• different people understand the same data differently and draw different lessons from it
• measurement systems (incl. as used in evaluation) reflect what decision-makers expect to see
• however, real learning often is triggered by the unexpected How can you design a measurement system to detect what you
don’t expect?
R. Behn, Public Administration Review, 2003*”programme learning” as inM. De Jong et al., OECD Journal on Budgeting, 2013
48
Learning and evaluation-2
• to detect the unexpected formal measurement systems must: cast a wide net (variety of measurement covering the entire system from end-to
end, internal and external) avoid excessive aggregation
• formal systems are not enough. There also needs to be informal “measurement”
this is the idea behind “management/data collection by walking around” where every story that people tell managers are the ultimate in disaggregation
• deviance is not always obvious as in the case of a clear failure: to detect deviances and understand if they are worth further investigation, service
providers must understand the internal and external environment (be sensitised) furthermore, to learn from failure, there cannot be an environment that focuses
primarily on assigning blame as people will then try to hide the deviant data
• after a significant deviance is detected, a learning strategy has to be deployed that probes for causes and implications:
will require expert knowledge and other sources of information beyond the measurement system itself
measurement itself is more likely to suggest topics for investigation than to directly impart key operational lessons
R. Behn, Public Administration Review, 2003*M. De Jong et al., OECD Journal on Budgeting, 2013
Aggregate common indicators cannot catch much
X visits of teachers to parents
X kids put more effort into homework
+X kids show higher attendance at school
X kids have better
start in the labour market
Actual intervention
Aggregate common indicators cannot catch much
EC states: Participants refer to persons benefiting directly from an ESF investment and who can be identified and asked for their characteristics, and for whom specific expenditure is earmarked. Other beneficiaries should not be counted as participants.
X participants(parents)
X participants in employment 6 months after
leaving
�participants gaining a qualification upon leaving+ �participants in employment
upon leaving
X visits of teachers to parents
X kids put more effort into homework
+X kids show higher attendance at school
X kids have better
start in the labour market
Actual intervention
Common indicators
51
Learning and evaluation-3
Evaluation is a specific way to “measure”:• Can be used to answer the question how well are we doing (‘narrow’
accountability) Typically compare with others, or with the past Danger of simplistic comparisons (ignoring context) with others, unclear
objectives and/or inadequate “programmatic” structure (multiple constraints, inadequate resources, unreasonable timetables)
Requires looking at outcomes (relative to input), taking account of context (what is due to other factors than our action)
• Can also relate to a “needs” assessment (what is happening/changing at citizen level) irrespective of some particular public action (key for relevance), sometimes part of “ex ante” evaluation
• As with any other measurement, in evaluation, “deviance” from what we expect is only a starting point to probe further so we can learn to help us improve
R. Behn, Public Administration Review, 2003M. De Jong et al., OECD Journal on Budgeting, 2013
52
Learning and evaluation-4
Targets offer little scope for detecting interesting deviations (that
helps us probe further):
• Hit them / Did not hit them
“experience shows that most operational performance indicators
deliver little, if any value in terms of actionable insights unless
they are charted… in time-series format”* e.g.
• Statistical process charts
• Sensemaker
*Andy Neely et al. How to avoid the problems of target setting, 2010, PMO symposium
53
Control-1
• Control to improve?
Establish standards, in principle on the process however, there are
some considerations:
• sometimes its is not easy/possible to observe (complicated) processes (e.g.
a manager) and we would rather measure output
• or, monitoring the process , however tightly, does not mean the output will
necessary be satisfactory (e.g. we can observe a researcher, reading,
writing, discussing etc. but whether /how this leads to a good piece of
research in the end is not so certain) so we want to measure the output
anyway
• or it costs a great deal less to monitor the output rather than the process
R. Behn, Public Administration Review, 2003
R. Simons, Performance measurement and control systems, ch. 4,2000
54
Control-2• if one wants to constrain experimentation/innovation in the process
as much as possible by tightly controlling the process (e.g. where safety is an issue or where tinkering in only one part of the process can have serious consequences for downstream parts etc.), one cannot wait for the output to control
• when neither process nor output can easily be controlled, we need to resort to “input control”, usually in the form of people through the recuitment process (e.g. highly capable and integer individuals
but...*• “standards” are derived from production environments where they
are needed for interoperability / integration E.g. tolerances in terms of how much front-window opening of a car can
deviate from norm, before fitting the glass becomes a problem!
• in most services, the “standard” is continuously set by the users, with variation to be absorbed by staff in a co-creating process
if not, it may be best to automate with ICT (eg license plates)
*John Seddon
55
Promote-1• To promote:
Convince others (politicians, journalists, citizens, ….) we are doing good job
• Performance info can be used to “legitimise”*: rationalise, justify, validate courses of (past/present/future) actions and decisions (incl. on budgets)
• It can be used to “reassure”* that government is doing what it is supposed to with taxpayers’s money (also called “transparency”)
• It can be used to “show compliance”* with regulations regarding performance management
• It can, by the above, unlock extra resources (e.g. earned autonomy, extra budgets, attract dedicated people,…) and hence improvement
Requires easily understood measures that the particular stakeholders care about
• Eg for DMV (dpt. for motor vehicles) more important to show how much you have to wait to get a check (compared to waiting perhaps for other services), than to show accident rates due to malfunctioning cars
R. Behn, Public Administration Review, 2003*M. De Jong et al., OECD Journal on Budgeting, 2013
Promote-2
56
Using performance information for “promotion”?
Trouble in measurement paradise?
57
Purpose Bias in terms of use
Budgeting/ planning
None: we want the information to be as accurate as possible to be as well-prepared as possible.
Learning None: we want the information to be as accurate as possible to learn
Motivation Managers want to “stretch” their expectations for staff, which leads staff to understate current performance. Which detracts from accuracy.
Control We want to know if standards are being met. This conflicts with the desire to motivate people to “stretch” as well as the desire to have an accurate picture of real (not within tolerance) performance levels.
Evaluation People have an interest to meet targets, if they are salient, in any way possible (also undesired ways). This detracts from the “accuracy” requirement. On the other hand, many factors beyond staff’s control contribute to / detract from the observed performance and should be adjusted for to accurately evaluate them.
Promotion The organisation has an interest in setting low expectations, that can surely be met, to maintain credibility. Conflicts again with accuracy.
R. Simons, Performance measurement and control systems, ch. 4,2000
For info
58
This is where the use for budgeting/ planning happens
Here we use it for control, evaluation, learning, promotion
For info