how to build a winning experimentation program & team | optimizely anz webinar 8

Building a Winning Experimentation

Team December 2017

Managing Director, Australia & New Zealand

dan.ross@optimizely.com

/danross9

Dan Ross

By the end of today’s session, you will have learned:

• Why experimentation is central to competing and innovating

• Areas to assess when building your experimentation capability

• How organisational culture helps scale an experimentation program

The world’s leading companies utiliseexperimentation to build a culture that fosters

innovation and agility

Optimization Is A Journey

2015 2017

“If you want to gain a competitive advantage, master the science of

conducting online tests. The returns you reap – in cost savings, new

revenue, and improved user experience – can be huge.”

“…experiment with everything” approach has

surprisingly large payoffs.”

“…rigorous online experiments should be standard

operating procedure.”

“Our success

at Amazon is a function of

how many experiments we do

per year, per month, per week, per day…”

J E F F B E Z O S

“Every decision is an experiment.”

M E . R I G H T N O W

Soon after Ron Johnson left Apple to become the CEO of J.C. Penney, in

2011, his team implemented a bold plan that eliminated coupons and

clearance racks, filled stores with branded boutiques, and used technology to

eliminate cashiers, cash registers, and checkout counters. Yet just 17

months after Johnson joined Penney, sales had plunged, losses had

soared, and Johnson had lost his job. The retailer then did an about-face.

Had J.C. Penney done thorough experiments on its CEO’s

proposed changes, the company might have discovered

that customers would probably reject them.Source: Harvard Business Review,

“The Discipline of Business Experimentation” December 2014

EVERY CHANGE IS AN EXPERIMENT...CONTROLLED OR NOT

Stefan Thomke ,

Harvard Business School

Fortune 1000

2012 2013 2014 2015 2016 2017

0

100

200

300

400

500

600

700

50

150

+56%

100

200

300

400

500

600

700

Fortune 1000 Culture of Experimentation Index

50

150

2012 2013 2014 2015 2016 2017

0

+56%

+756

%

1,000,000+E X P E R I M E N T S

CULTURE OF EXPERIMENTATION

0

10

100

1000

VE

LO

CIT

Y

MATURITY

10000

VE

LO

CIT

Y

EXPERIMENTATION

HERO

MATURITY

0

10

100

1000

10000

VE

LO

CIT

Y

EXPERIMENTATION

PROGRAM

EXPERIMENTATION

HERO

MATURITY

0

10

100

1000

10000

VE

LO

CIT

Y

EXPERIMENTATION

HERO

EXPERIMENTATION

PROGRAM

CULTURE OF

EXPERIMENTATION

MATURITY

0

10

100

1000

10000

The key to experimentation is to have both the right tools (software) in combination with the right

people and processes

Experimentation Maturity Model

So how do you bring this concept to your organisation?

The first step is to understand what we see as the common attributes within these market leading

organisations.

These attributes when combined together represent a “Culture of Experimentation”

Experimentation is deeply

embedded into company

culture

Dedicated, customised technology

conducts experiments and surfaces

insights

Attributes of Mature “Culture of Experimentation”

Organisations

Enfranchised teams work on

independent optimisation

roadmaps

Central groups manage data

collection and analysis,

empowers individuals with

resources and process

Strategy and metrics for digital

performance directly linked to

company success and widely

circulated

Collective expertise creates virtuous

cycle of learning and best practices

These Culture of Experimentation Organisations would be considered “best-in-class” globally.

So if that is what the end state of building an experimentation program looks like, where do most

organisations stand today?

Most organisations would be considered Experimentation Hero Organisations

Goals and processes poorly

formalised, lack executive

sponsorship

Experimentation are focused

marketing and sales channels

only versus every business

department

Attributes of Early Stage “Experimentation Hero”

Organisations

Consumer intentions and digital

behavior poorly understood

Experimentation treated as series of

tactics to ‘improve conversion rate’ Skill sets and tools divided across

functional silos or not available at all

Lone individuals or small teams

working through design, production,

and analysis despite limited experience

The vast majority of organisations fall somewhere between Mature “Cultures of Experimentation”

and Early Stage “Experimentation Hero” Organisations

…but most exemplify more criteria from the 2nd set of attributes

Mature Cultures of

Experimentation

Early Stage Experimentation Organisations

Embedded into company

culture

Enfranchised teams

Empowered individuals

Dedicated technology

conducts experiments

Metrics linked to company

success

Collective expertise

Lack executive sponsorship

Customer digital behaviors

poorly understood

Experimentation seen only as

Conversion Rate optimisation

Focused solely on sales and

marketing funnels

Tools and skilled people in

silos

Limited expertise available

This is the called the Alignment Gap

The gap between these two sets of attributes represents the key opportunity to create and sustain

a competitive advantage in experimentation

Mature Cultures of

Experimentation

Early Stage Experimentation Organisations

This gap has numerous root causes for why an

organisation may be on one side or the other including

business model, organisational structure, technology

investments, and available trained staff.

Yet, there are measures that any organisation can take

to efficiently leapfrog these hurdles to the front of their

competitive group.

So how do you bridge The Alignment Gap?

To bridge the Alignment Gap your organisation must operationalise

your experimentation program

How

Operationalising

Experimentation

Identify Team and Support Their Growth

Embrace Experimentation Methodology

Understand Operational Metrics

Understand Success Factors

Getting Started Today

Identify Your Team

CORE EXPERIMENTATION TEAMSKILLSETS & TEAM ROLE

Executive Sponsor Project Manager Technical Lead Developer Content

Invest in your most valuable assets:

Identify a Team and Support Professional Growth

Certify Measure Socialise

You can also jumpstart your experimentation program or enhance

your existing team’s capabilities by utilising a best-in-class partner

Once you have a team in place we move to the next key concept

The practice of experimentation follows a defined set of repeatable actions called the

Experimentation Methodology

Ideation

Generate ideas for experiments

Experimentation Methodology

Planning

Design

individual

experiments

and

campaigns

Experimentation Methodology

Developmen

t

Build your experiment and launch it

Experimentation Methodology

Analysis Evaluate the results

of experiment

Experimentation Methodology

Experimentation Methodology

Analysis

Developmen

t

Planning

Ideation

The Experimentation Methodology is universal framework for any sized organisation, but you have

to map it for your business.

To learn more about mapping the Experimentation Methodology please visit the Optimizely Knowledge Base for Optimisation Methodology: Running your

Optimisation Program

Experimentation Maturity Model

Most team’s don’t effectively measure their of use of the Experimentation

Methodology.

This is a missed opportunity.

Structured measurement gives you leading indicators of performance called

Operational Metrics.

These Operational Metrics are the Success Factors for having a world-class

experimentation program

Experimentation Success

Factors

VelocityAgility

Efficiency

Quality

Throughpu

t

Operational Metrics

Impact Rate

Drag

Iteration=

=

=

=

Exp. Methodology Planning Development Ideation Analysis

Operational

MetricsThroughput Drag Impact Rate Iteration

Why it’s

Important

Large percentage of

experiments don’t win.

Improving the quantity of

experiments will ultimately

generate more success.

Knowing how long it takes to

both build an experiment and

how long it takes for each

stage of the experimentation

methodology is important for

measuring overall program

improvement.

To give your team the best

chance of creating a

winning experiment you

need to gather insightful

data to inform hypothesis

generation.

The process of aggregating,

interpreting, and sharing the

experiment results within the

organisation to inform the

next experiment.

Description

(What You Measure)

1.) Overall number of

experiments per property

2.) Number of experiments

per month and week for

each property.

1.) Number of hours spent

redeveloping experiments due

to QA or troubleshooting

issues compared to

‘productive’ time.

2.) Length in hours and days

of the entire experiment

production cycle.

1.) Percentage of

experiments generating

meaningful result (with a

statistically significant

winner or loser).

1.) Percentage of

experiments put into

production and iterated upon.

2.) Number of days from

current experiment to iterated

upon experiment.

Success Factors Velocity Efficiency Quality Agility

Description Quantity of experiments

being ran.

Rate at which tests move

through both Development

stage and the overall

Experimentation Methodology.

Average likelihood that a

given experiment will

produce business impact

(either positive or negative)

The quantity and speed the

experimentation program acts

on experiment results and

iterates upon them.

It is critical to create a habit of optimizing your experimentation program toward these Operational Metrics to ensure you are

continually measuring and improving on your Success Factors.

Now that we know the Experimentation Methodology, Operational Metrics, and Success Factors how do we get started

today?

The most successful experimentation programs result in business

transformation. Get on this journey today!

Executives

Prioritise and evangelise for experimentation within your business

Identify your team and invest in their success by giving them the time

availability, resources, and goals for the overall experimentation program that

align with business objectives

Executives and Practitioners

Begin mapping the Experimentation Methodology to your organisation

Practitioners

Learn the best practices recommended by Optimizely for starting your

experimentation program

Getting Started Today

Thank You

Dan RossOptimizely ANZdan.ross@optimizely.com/danross9