how to build a winning experimentation program & team | optimizely anz webinar 8
TRANSCRIPT
Building a Winning Experimentation
Team December 2017
Managing Director, Australia & New Zealand
/danross9
Dan Ross
By the end of today’s session, you will have learned:
• Why experimentation is central to competing and innovating
• Areas to assess when building your experimentation capability
• How organisational culture helps scale an experimentation program
The world’s leading companies utiliseexperimentation to build a culture that fosters
innovation and agility
Optimization Is A Journey
2015 2017
“If you want to gain a competitive advantage, master the science of
conducting online tests. The returns you reap – in cost savings, new
revenue, and improved user experience – can be huge.”
“…experiment with everything” approach has
surprisingly large payoffs.”
“…rigorous online experiments should be standard
operating procedure.”
“Our success
at Amazon is a function of
how many experiments we do
per year, per month, per week, per day…”
J E F F B E Z O S
“Every decision is an experiment.”
M E . R I G H T N O W
Soon after Ron Johnson left Apple to become the CEO of J.C. Penney, in
2011, his team implemented a bold plan that eliminated coupons and
clearance racks, filled stores with branded boutiques, and used technology to
eliminate cashiers, cash registers, and checkout counters. Yet just 17
months after Johnson joined Penney, sales had plunged, losses had
soared, and Johnson had lost his job. The retailer then did an about-face.
Had J.C. Penney done thorough experiments on its CEO’s
proposed changes, the company might have discovered
that customers would probably reject them.Source: Harvard Business Review,
“The Discipline of Business Experimentation” December 2014
EVERY CHANGE IS AN EXPERIMENT...CONTROLLED OR NOT
Stefan Thomke ,
Harvard Business School
Fortune 1000
2012 2013 2014 2015 2016 2017
0
100
200
300
400
500
600
700
50
150
+56%
100
200
300
400
500
600
700
Fortune 1000 Culture of Experimentation Index
50
150
2012 2013 2014 2015 2016 2017
0
+56%
+756
%
1,000,000+E X P E R I M E N T S
CULTURE OF EXPERIMENTATION
0
10
100
1000
VE
LO
CIT
Y
MATURITY
10000
VE
LO
CIT
Y
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000
VE
LO
CIT
Y
EXPERIMENTATION
PROGRAM
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000
VE
LO
CIT
Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION
MATURITY
0
10
100
1000
10000
The key to experimentation is to have both the right tools (software) in combination with the right
people and processes
Experimentation Maturity Model
So how do you bring this concept to your organisation?
The first step is to understand what we see as the common attributes within these market leading
organisations.
These attributes when combined together represent a “Culture of Experimentation”
Experimentation is deeply
embedded into company
culture
Dedicated, customised technology
conducts experiments and surfaces
insights
Attributes of Mature “Culture of Experimentation”
Organisations
Enfranchised teams work on
independent optimisation
roadmaps
Central groups manage data
collection and analysis,
empowers individuals with
resources and process
Strategy and metrics for digital
performance directly linked to
company success and widely
circulated
Collective expertise creates virtuous
cycle of learning and best practices
These Culture of Experimentation Organisations would be considered “best-in-class” globally.
So if that is what the end state of building an experimentation program looks like, where do most
organisations stand today?
Most organisations would be considered Experimentation Hero Organisations
Goals and processes poorly
formalised, lack executive
sponsorship
Experimentation are focused
marketing and sales channels
only versus every business
department
Attributes of Early Stage “Experimentation Hero”
Organisations
Consumer intentions and digital
behavior poorly understood
Experimentation treated as series of
tactics to ‘improve conversion rate’ Skill sets and tools divided across
functional silos or not available at all
Lone individuals or small teams
working through design, production,
and analysis despite limited experience
The vast majority of organisations fall somewhere between Mature “Cultures of Experimentation”
and Early Stage “Experimentation Hero” Organisations
…but most exemplify more criteria from the 2nd set of attributes
Mature Cultures of
Experimentation
Early Stage Experimentation Organisations
Embedded into company
culture
Enfranchised teams
Empowered individuals
Dedicated technology
conducts experiments
Metrics linked to company
success
Collective expertise
Lack executive sponsorship
Customer digital behaviors
poorly understood
Experimentation seen only as
Conversion Rate optimisation
Focused solely on sales and
marketing funnels
Tools and skilled people in
silos
Limited expertise available
This is the called the Alignment Gap
The gap between these two sets of attributes represents the key opportunity to create and sustain
a competitive advantage in experimentation
Mature Cultures of
Experimentation
Early Stage Experimentation Organisations
This gap has numerous root causes for why an
organisation may be on one side or the other including
business model, organisational structure, technology
investments, and available trained staff.
Yet, there are measures that any organisation can take
to efficiently leapfrog these hurdles to the front of their
competitive group.
So how do you bridge The Alignment Gap?
To bridge the Alignment Gap your organisation must operationalise
your experimentation program
How
Operationalising
Experimentation
Identify Team and Support Their Growth
Embrace Experimentation Methodology
Understand Operational Metrics
Understand Success Factors
Getting Started Today
Identify Your Team
CORE EXPERIMENTATION TEAMSKILLSETS & TEAM ROLE
Executive Sponsor Project Manager Technical Lead Developer Content
Invest in your most valuable assets:
Identify a Team and Support Professional Growth
Certify Measure Socialise
You can also jumpstart your experimentation program or enhance
your existing team’s capabilities by utilising a best-in-class partner
Once you have a team in place we move to the next key concept
The practice of experimentation follows a defined set of repeatable actions called the
Experimentation Methodology
Ideation
Generate ideas for experiments
Experimentation Methodology
Planning
Design
individual
experiments
and
campaigns
Experimentation Methodology
Developmen
t
Build your experiment and launch it
Experimentation Methodology
Analysis Evaluate the results
of experiment
Experimentation Methodology
Experimentation Methodology
Analysis
Developmen
t
Planning
Ideation
The Experimentation Methodology is universal framework for any sized organisation, but you have
to map it for your business.
To learn more about mapping the Experimentation Methodology please visit the Optimizely Knowledge Base for Optimisation Methodology: Running your
Optimisation Program
Experimentation Maturity Model
Most team’s don’t effectively measure their of use of the Experimentation
Methodology.
This is a missed opportunity.
Structured measurement gives you leading indicators of performance called
Operational Metrics.
These Operational Metrics are the Success Factors for having a world-class
experimentation program
Experimentation Success
Factors
VelocityAgility
Efficiency
Quality
Throughpu
t
Operational Metrics
Impact Rate
Drag
Iteration=
=
=
=
Exp. Methodology Planning Development Ideation Analysis
Operational
MetricsThroughput Drag Impact Rate Iteration
Why it’s
Important
Large percentage of
experiments don’t win.
Improving the quantity of
experiments will ultimately
generate more success.
Knowing how long it takes to
both build an experiment and
how long it takes for each
stage of the experimentation
methodology is important for
measuring overall program
improvement.
To give your team the best
chance of creating a
winning experiment you
need to gather insightful
data to inform hypothesis
generation.
The process of aggregating,
interpreting, and sharing the
experiment results within the
organisation to inform the
next experiment.
Description
(What You Measure)
1.) Overall number of
experiments per property
2.) Number of experiments
per month and week for
each property.
1.) Number of hours spent
redeveloping experiments due
to QA or troubleshooting
issues compared to
‘productive’ time.
2.) Length in hours and days
of the entire experiment
production cycle.
1.) Percentage of
experiments generating
meaningful result (with a
statistically significant
winner or loser).
1.) Percentage of
experiments put into
production and iterated upon.
2.) Number of days from
current experiment to iterated
upon experiment.
Success Factors Velocity Efficiency Quality Agility
Description Quantity of experiments
being ran.
Rate at which tests move
through both Development
stage and the overall
Experimentation Methodology.
Average likelihood that a
given experiment will
produce business impact
(either positive or negative)
The quantity and speed the
experimentation program acts
on experiment results and
iterates upon them.
It is critical to create a habit of optimizing your experimentation program toward these Operational Metrics to ensure you are
continually measuring and improving on your Success Factors.
Now that we know the Experimentation Methodology, Operational Metrics, and Success Factors how do we get started
today?
The most successful experimentation programs result in business
transformation. Get on this journey today!
Executives
Prioritise and evangelise for experimentation within your business
Identify your team and invest in their success by giving them the time
availability, resources, and goals for the overall experimentation program that
align with business objectives
Executives and Practitioners
Begin mapping the Experimentation Methodology to your organisation
Practitioners
Learn the best practices recommended by Optimizely for starting your
experimentation program
Getting Started Today
