presentation: philips
TRANSCRIPT
![Page 1: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/1.jpg)
Experiment Design Moving from ideas to business
Dylan EvansSenior Design Lead for Digital User ExperienceNovember 24th 2016
© Philips 2016
![Page 2: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/2.jpg)
2
Twitter @veluuria
Where I started…Moving from fields to offices
![Page 3: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/3.jpg)
There is no such thing
as a failed experiment...
…only experiments with unexpected outcomes.
Buckminster Fuller
![Page 4: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/4.jpg)
Why?
![Page 5: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/5.jpg)
On Experimentation
In a lot of ways building a company is like following the scientific method. You try a bunch of different hypotheses, and if you set up the experiments well, then you kind of learn what to do…
We invest in this huge testing framework. At any given point in time, there’s not just one version of Facebook running in the world. There’re probably tens of thousands of versions running because engineers here have the power to try out an idea and ship it to maybe 10,000 people or 100,000 people. And then they get a readout.
![Page 6: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/6.jpg)
As humans, we:
3. When successful, assume we know what made it successful
2. Inflate the impact of a success1. Overestimate the probability of success
![Page 7: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/7.jpg)
If we’re so biased,
how do we inform the
decision to invest?
![Page 8: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/8.jpg)
Validated learning…
… is a process by which we learn by experimenting with an idea and
measuring it to validate its effect.
![Page 9: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/9.jpg)
By using validated learning we can create the best business recommendation based on how it touches:
• the consumer• the business
![Page 10: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/10.jpg)
A point about learning…
Unconscious Incompetence
Conscious Incompetence
Conscious Competence
Unconscious Competence
Ignorance is bliss
I’m not doing it
right!
I can do it if I try
![Page 11: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/11.jpg)
To learn, we must experiment
![Page 12: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/12.jpg)
…is about finding what works best, and quickly.
Experimenting…
…creates the opportunity to
explore multiple directions,
choose the bestand refine it.
![Page 13: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/13.jpg)
Experimenting helps…
Discover fatal flaws early,
before all
time and money has been spent
![Page 14: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/14.jpg)
Reduce the risk of keeping things untested and…
Experimenting helps…
![Page 15: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/15.jpg)
(Highest Paid Person’s Opinion)
Experimenting helps…
Overcome opinion-led decision
making by HiPPOs
![Page 16: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/16.jpg)
How?
![Page 17: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/17.jpg)
Each experiment tests a falsifiable hypothesis.
Focus on speed for faster validated learning.
Experiment Design
![Page 18: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/18.jpg)
Whyweneedtoexperiment
• AShumans,we:• 1. Overestimateprobabilityofsuccess2. Inflatetheimpactofasuccess3. Whensuccessful,assumeweknowwhatmadeitsuccessful
• Changingenvironment
![Page 19: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/19.jpg)
Typical graph for development
Options for Success
Cost of change
Time Launch
Am
ou
nt
Planning
Costs increase, options decrease
![Page 20: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/20.jpg)
The first principle is that you must not fool yourself - and you are the easiest person to fool.
Richard Feynman
![Page 21: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/21.jpg)
Imagefrom
https://grassho
pperhe
rder.com
/assum
ption-vs-hypothe
sis-to
-the-de
ath/
![Page 22: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/22.jpg)
Imagefrom
https://grassho
pperhe
rder.com
/assum
ption-vs-hypothe
sis-to
-the-de
ath/
![Page 23: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/23.jpg)
hypothesisPronunciation: /hīˈpäTHəsəs/
NOUNA supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation
assumptionPronunciation: /əˈsəm(p)SH(ə)n/
NOUNA thing that is accepted as true or as certain to happen, without proof.
![Page 24: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/24.jpg)
Assumption 1
Hypothesis1
Hypothesis2
Hypothesis3
Hypothesis4
Experiment1
Experiment3
Experiment2
Break assumptions down in to hypotheses and experiments to gain coverage
![Page 25: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/25.jpg)
It’s child's play
Children iterate towards a solution through trial and error
![Page 26: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/26.jpg)
Yourjourney
![Page 27: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/27.jpg)
Incrementing vs. Iterating
Idea Business
![Page 28: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/28.jpg)
Specify hypothesis
Design experiment
& actions to be taken with output
Run test and take actions
The Experimental Design iterative learning loop
State what we believe to be true and required for the offer’s success
Choose a methodology define the measure and set a target and actions
Run test, capture the outcomes and follow actions to be taken
Prioritise!
If hypothesis is confirmed, we persevere and GO.
If falsified, we pivot and change the proposition.
What will kill the proposition first?
2
34
1
![Page 29: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/29.jpg)
Idea BusinessMany, many iterations
![Page 30: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/30.jpg)
Exp 1 Exp 2 Exp 3 Exp 4
Experiment along the journeyCheck if the performance matches the promise
Idea BusinessMany, many iterations
VideoAdvert(A/B)
Web Page(A/B)
PaperPrototype
![Page 31: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/31.jpg)
1 Placed Facebook ads 2 Clicked through to Landing Page 3 Sign-up
Sequence experiments to test the future experience
4 Recruited for beta product test
![Page 32: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/32.jpg)
Prototypes are the engine of experimental design
Making stuff deepens our thinking and drives ACTION
If you want to make something great, start making.
Tom Kelley, IDEO
Having stuff means we can share with people who will “get it” more easily
We can iterate and make better development decisions
1 2 3
‘‘
![Page 33: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/33.jpg)
The purpose of a Minimum Viable Proposition (MVP) is to maximize learning per euro we spend*.
* using customers and not testers
![Page 34: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/34.jpg)
Real“Test” products
Consumerperception
Our perception
MVP 1.0
Prototypes
€/real & LCM
1.0
Propositions on the market
MVP MVP
Learning tools
MVPs are prototypes that customers believe to be real
Proto A Proto B Proto C Proto DProto AProto C
![Page 35: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/35.jpg)
Experts can guide to the right experiment design
Low Fidelity - FASTER
Buy Use
High Fidelity - SLOWER
Wizard of Oz
ConciergeImposter Judo
Analog/Physical
Dry-Wallet High Hurdle
Video Trailer
Crowd-Funding
Pitch
Landing page
Qualitative interviews / groups
Survey
Web Prototype
Paper Mockup / POP
Keynote / InVision Prototype
Online Ads
TypeForm Landing
Native OS Prototype
Role Play
QuantitativeProduct Usage
tests
Technical / sensory testing
BASES / Concept test
Classic methodsRecent digital-enabled methods
Shop alongs
© Philips 2016
![Page 36: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/36.jpg)
Example journey
![Page 37: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/37.jpg)
Exp 1 Exp 2 Exp 3 Exp 4
Experiment along the journeyCheck if the performance matches the promise
Idea BusinessMany, many iterations
Video
![Page 38: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/38.jpg)
Concept video
![Page 39: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/39.jpg)
When we involve real customers, we need to engage them through storytelling
Addpic
![Page 40: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/40.jpg)
1. Create a story2. Make the video3. Fix the video
![Page 41: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/41.jpg)
![Page 42: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/42.jpg)
Toocomplex?
A simple structure is your friend:Confrontation/tensionEnlightenment/ChangeExplorationClosure } e.g. try a four
act structure
![Page 43: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/43.jpg)
Exp 1 Exp 2 Exp 3 Exp 4
Experiment along the journeyCheck if the performance matches the promise
Idea BusinessMany, many iterations
VideoAdvert(A/B)
Web Page(A/B)
![Page 44: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/44.jpg)
A/B testBig directionsIn the early stages, it’s not a small variable vs. small variable (aka optimisation)
We have to answer and find the big directions before we can optimise.
![Page 45: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/45.jpg)
Choose the best channel for your communication
http://www.growthtribe.io/blog/brass-method-your-ultimate-guide-to-prioritising-which-customer-acquisition-channel-to-test-first/
Facebook isn’t always the right channel to reach your audience.
![Page 46: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/46.jpg)
AnalyticsHow many & when?
- e.g. Google Analytics
- e.g. Advert analytics
What did they do?
- E.g. Hotjar
n.b. The numbers never match
![Page 47: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/47.jpg)
Exp 1 Exp 2 Exp 3 Exp 4
Experiment along the journeyCheck if the performance matches the promise
Idea BusinessMany, many iterations
VideoAdvert(A/B)
Web Page(A/B)
PaperPrototype
![Page 48: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/48.jpg)
1st consultskills
2nd consulthabit
3rd consultLast resort
basic instruction one remedy one Talcum powder (messy but it works)
basic instruction two remedy two
gauge skill level advanced instruction
introduction to new skill error strategy
terminology understanding confront habit
surprising instruction
Concierge MVPTeam identified an advice model which appears to very effective
![Page 49: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/49.jpg)
Paper Prototype
![Page 50: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/50.jpg)
GoodBad
While the one on the left looks nicer, it is not usable.Iterate through your sketches, throw lots away, keep the best stuff.
![Page 51: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/51.jpg)
How useful?Try http://popapp.in and see how quickly you can get a prototype running.
Use this to test navigation & meaning very early on. Use the results to iterate through a web app with full analytics. Use this to understand if a feature is worth building.
![Page 52: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/52.jpg)
Your Team
![Page 53: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/53.jpg)
Draw on expertise around you to:
Structure Effective Hypotheses
Identify the right Risks in the Idea - Elephant in the room
Ensure Stakeholders stay accountable to the Results
Create the ability to Run Quick, Iterative Experiments
Stop the experiment when you’ve collected the full data
Hire statistical knowledge (sample # you must use to find a significant result)
Run at same times - compare apples with apples
Create control variables across experiments
Integrate traditional testing where relevant
Build your individual expertise
Use clear assessment criteria (Desirability, Feasibility, Viability)
![Page 54: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/54.jpg)
A TEAMMARKETING PROFESSIONALSDESIGNERS & STRATEGISTSTECHNOLOGY SPECIALISTSDEVELOPERSSUBJECT MATTER EXPERTSSTAKEHOLDERS
![Page 55: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/55.jpg)
Closing
![Page 56: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/56.jpg)
After each experiment, there are always 4 possible outcomes
1. GO You are confident that your hypothesis is valid and have the evidence to show it
2. Confirm
3. Pivot
4. STOP
You are confident that your hypothesis is correct but you need more evidence to support it
You no longer believe that this offer is a right for you to pursue at this time.
You believe in the vision but you need to re-visit the means & path to get to the vision
![Page 57: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/57.jpg)
Experiment DesignKey Take Aways
![Page 58: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/58.jpg)
KPI driven
Simple
As close to reality as possible
Prioritized
Statistically significant
Allowed to fail!
Experiments must be:
![Page 59: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/59.jpg)
Build the whole journey
Speed is your ally. Perfection comes later
Your first experiment is a rehearsal
Your approach & mindset
![Page 60: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/60.jpg)
Fail Fast, learn fast.
![Page 61: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/61.jpg)
The only way to
proceed is to get
your feet wet
![Page 62: Presentation: Philips](https://reader036.vdocuments.mx/reader036/viewer/2022070515/5877478b1a28ab84388b4abf/html5/thumbnails/62.jpg)
Thank you