designing policy experimentation
TRANSCRIPT
![Page 1: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/1.jpg)
Designing policy experimentationHowtoiden*fy,design,runandlearnfrominnova*vepolicyinterven*ons
Christian BasonChief Executive, Danish Design Centre
![Page 2: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/2.jpg)
![Page 3: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/3.jpg)
![Page 4: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/4.jpg)
![Page 5: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/5.jpg)
![Page 6: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/6.jpg)
Policy Press 2010 Gower Ashgate 2014 Policy Press 2017
![Page 7: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/7.jpg)
PART IA model for policy experimentation 9.30 Welcome Embracing double-edged complexity Towards a new conception of the role of governmentA holistic approach to policy experiments From challenges and opportunities to impactHorizon scanning Sensing the next policy challengeCo-designing hypotheses to test How to build an experiment portfolioCo-producing by experimentation Prototyping, programming and scaling using design approachesLearning from experiments Measuring outcomes10.20 Q&A
10.30 End session
![Page 8: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/8.jpg)
![Page 9: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/9.jpg)
![Page 10: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/10.jpg)
Public policy: A design problem “How can you make sensible policy or strategy in a nondeterministic, evolutionary, highly complex world, that is, a world where the most desirable outcomes are unknown but there may be many possible acceptable outcomes, where change is characterized by both path dependence and unpredictability, and where there are many diverse components, interactions, and feedback among components and multiple dimensions to each problem? This is the design problem with respect to public policy.”
Carlsson (2004:36)
![Page 11: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/11.jpg)
![Page 12: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/12.jpg)
When government policy fails…
• United States: Obamacare digital platform
• Denmark: Runaway applications for solar energy scheme
• Germany: Voluntary Technical Year
• Singapore: Relationship programmes
![Page 13: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/13.jpg)
Embracing double-edged complexity
![Page 14: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/14.jpg)
”[Incomplexsettings]insteadofattemptingtoimposeacourseofaction,leadersmustpatientlyallowthepathforwardtorevealitself.Theyneedtoprobefirst,thensense,andthenrespond.” DavidSnowden
![Page 15: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/15.jpg)
”The state has not just fixed markets, but actively created them”. Marianna Mazzucato
![Page 16: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/16.jpg)
”[We] suggest institutional changes that shift innovation policy towards a more experimental conception of the role of the state in facilitating entrepreneurship, and thereby innovation”. Hasan Bakshi
![Page 17: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/17.jpg)
“This country needs, and unless I mistake its temper, the country demands bold persistent experimentation. It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something.”
Franklin Delano Roosevelt
![Page 18: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/18.jpg)
“An appalling piece of political stupidity.”
Louis Howe, adviser to Franklin Delano Roosevelt
![Page 19: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/19.jpg)
A policy experimentation model
![Page 20: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/20.jpg)
Horizon scanningSensing the next policy challenge
![Page 21: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/21.jpg)
A policy experimentation model
![Page 22: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/22.jpg)
Horizon scanning
What?
• Sensing coming trends and developments with potential policy or organisational consequence
• Establishing insight, foresight and scenarios to visualize plausible futures
Why?
• Creating awareness of context factors of importance to the organisation
• Preparedness, resilience in view of possible disruptions
• Basis for policy planning and action
Key questions?
• Which political, economic, environmental, societal and technological factors should we care about?
• How could these driving forces influence us in the future?
• What should we do now to shape our future in a desirable direction?
![Page 23: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/23.jpg)
Horizon scanning
Cases
• The Singaporean Government: 2050 foresight strategy
• Policy Horizons Canada: IMPACT - a serious foresigt board game for public servants
• OECD: Schooling for tomorrow
• Danish Design Center: Scenarios Healthcare Denmark 2050
• UAE: Museum of government futures
• Dubai Future Foundation
![Page 24: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/24.jpg)
Co-designDesigning policy with people, not for them
![Page 25: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/25.jpg)
A policy experimentation model
![Page 26: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/26.jpg)
Co-design What?
• Exploring problems from end user perspective
• Co-creating new ideas with users and stakeholders
• Prototyping and testing early ideas “in the lab”
Why?
• To build an early validation of fit and function of a policy idea
• Create basis for redesign and ultimately for decision-making
Key questions?
• Who are the end users?
• How might this policy intervention work for them?
• Which other aspects do we need to take into account?
![Page 27: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/27.jpg)
Source: MindLab
![Page 28: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/28.jpg)
![Page 29: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/29.jpg)
Source: MindLab
![Page 30: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/30.jpg)
Co-design
Cases:
The lab movement
![Page 31: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/31.jpg)
Co-design
What we do
EXPLORING THE
PROBLEM SPACE
GENERATING ALTERNATIVE
SCENARIOS
ENACTING NEW
PRACTICES
Ethnographic research
Prioritization
Concept Development
Ideation
Visualisation
Prototyping
User testing
RealisationPattern
recognition
![Page 32: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/32.jpg)
![Page 33: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/33.jpg)
Co-producing by experimentationEstablishing hypotheses of change for realizing interventions
![Page 34: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/34.jpg)
A policy experimentation model
![Page 35: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/35.jpg)
Co-producing by experimentation
What?
• Organising and implementing policy through collaborative networks
• Leveraging all relevant resources to produce policy outcomes
• Establishing the hypotheses of change to experiment with policy by co-production
• Ensuring rigorous collection of qualitative and quantitative
Why?
• To be explicit about which actions and factors we expect will create intended change
• Raise awareness about critical success factors
• To know what to measure to track changes, including unintended consequences
Key questions?
• Based on our co-design process, which hypothesis is it now we are testing?
• What inputs, activities and outputs do we expect to realize?
• What would outcomes look like, if we are successful?
![Page 36: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/36.jpg)
Co-producing by experimentation
What we do
View all policy interventions as essentially experimental
Realise co-production at three scales
• Prototype: High on experimentingKey questions: How does the intervention work? Who does it work for (who benefits)?
• Program: High on learning Key questions: How can we learn from this now that the design is being realized?
• Scale: High on sharingKey questions: How can we share our insights and tools? Which actors can embed activities to go to scale? How can we reach more people/businesses?
![Page 37: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/37.jpg)
• Finland PMO: Government experimentation programme and funding platform for citizen-led experiments
• UK CO: Government Digital Services
• UAE: Dubai Future Accelerators Programme
Co-producing by experimentation
Cases
![Page 38: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/38.jpg)
EFFEKTER
AKTIVITETER
PRODUKTER LEVERANCER
OUTPUT
AKTIVITET AKTIVITET AKTIVITET AKTIVITET
AKTIVITET
CRITICAL SUCCESS FACTORS
INPUT ACTIVITIES OUTPUTS OUTCOMES LONG-TERM
OUTCOMES SHORT-TERM
CRITICAL SUCCESS FACTORS CRITICAL SUCCESS FACTORS CRITICAL SUCCESS FACTORS
Hypotheses of change frameworkProblem: … Hypotheses: …
![Page 39: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/39.jpg)
Measuring outcomesDocumenting, learning and improving performance
![Page 40: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/40.jpg)
A policy experimentation model
![Page 41: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/41.jpg)
“If you don’t measure outcomes, you cannot tell the difference between success and failure. That means you might be rewarding failure.”
Ray Rist, former senior advisor, World Bank
![Page 42: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/42.jpg)
Outcome measurement
What?
• Establishing a systematic set of methodologies to document inputs, activities, outputs, and short- and long term outcomes of interventions
• Establishing key perfomance indicators: Best indications of what success could look like
• Collecting data systematically
Why?
• Using data to document for accountability and transparency
• Drive continuous learning, and increase organisational performance
• Produce stronger outcomes
Key questions?
• Do our hypotheses hold?
• Are we achieving the positive change and outcomes we intended?
• What are unintended consequences - what should we adjust?
![Page 43: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/43.jpg)
The evidence ladder
No knowledge about outcomes
Proven outcomes
Documented outcomes
Apparent outcomes
No documentation
“Sunshine stories”
Systematic documentation eg surveys, rigorous case studies, A/B testing
Research-based documentation eg randomized controlled trials, meta studies
![Page 44: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/44.jpg)
Different approaches to measuring outcomes Case: iTeams report from Nesta and Bloomberg
![Page 45: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/45.jpg)
Measuring outcomes
What do we do?
Outcome measurement system
Measures outcomes systematically around three overall strategic objectives:
• Contribution to business growth (economic value)
• Contribution to branding of Danish design (economic value)
• Contribution to societal impacts (societal outcomes in education and sustainability)
This is done by assessing progress against a logic model of hypotheses of change effect chains.
Quantitatively: Surveys among businesses, media impact data, etc.
Qualitatively: Observation studies, interviews, design research
Enables cost benefit analysis: What is the return of investing in the Danish Design Centre?
User feedback surveys
• Net promoter scores, evaluation of projects et promotor score
• Measures loyalty from participants in seminars and events
Measuring design impact for business
• Comprehensive case research methodology
• Survey data
• National statistical data
![Page 46: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/46.jpg)
Measuring outcomes
How does it differ in experimental policy?
Traditional policy(operations)
Experimental policy (innovation)
Purpose Documentation, accountability, performance
Learning, adaptation, redesign (or termination)
Focus Optimizing the use of existing resources
Discovering additional resources to be leveraged
Data Mainly quantitative Quantitative and qualitative
Tools Statistics, surveys, other A/B test, RCT’s, cases, design research, future probes, etc.
Time horizon Long-term systematic measurements; on-going
Tailored on concrete prototype or programme design
Challenge Setting right KPIs - and meeting them!
Capturing causal elements of hypotheses of change
![Page 47: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/47.jpg)
EXPLORING THE
PROBLEM SPACE
GENERATING ALTERNATIVE SCENARIOS ENACTING
NEW PRACTICES
#1 Challenging assumptions
#2 Leveraging empathy
#3 Stewarding divergence
#4 Navigating
the unknown
#6 Insisting on
value- creation
#5 Making the
future concrete
Design engagement:An experimental mindset.
![Page 48: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/48.jpg)
Towards experimental government?• Which approaches do you use today from design to
measurement?
• How could systematic experimentation become part of the “new normal” of governing?
• What would be the benefits?
• Which challenges to you foresee?
![Page 49: Designing Policy Experimentation](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5874c32a1a28ab8f508b5c33/html5/thumbnails/49.jpg)
ddc.dk