how to design with science and not destroy the magic
TRANSCRIPT
@mrjoe
How to design with scienceJanuary 8, 2015
Joe Leech
Waving einstein
How to design with science and not destroy the magic
@mrjoe
@mrjoe
[Newton] has destroyed all the poetry of the rainbow, by reducing it to the prismatic colours. – John Keats
@mrjoe
The product goes live and it's lost
it’s magic
The team / client
continually questioning your
choices
Being micromanaged by your boss / client
Ever heard these issues? Hands-up? I had the first one so badly for about a year, I called it the curse of Joe.
@mrjoe
It started with a need
6
http://www.cxpartners.co.uk/cxblog/the_myth_of_the_page_fold_evidence_from_user_testing/
It started with a need. I had heard the same thing again and again.
I wrote this, have you read it? Almost half a million of you have.
@mrjoe
Here’s the data, from an eyetracking study. It shows 2 designs, one to the left with stuff crammed above the fold. The one on the right where things are spaced out more. People explored more when there was less stuff crammed above the fold.
It showed me there was a need for evidence, reasoning and science to support the design process.
@mrjoe
Design is not so much a design issue as a power struggle. – Alan Cooper
Alan cooper, created visual basic amongst other things, teaches design leadership. Harsh words! So what do we do?
@mrjoe
Design is the process of choosing and organising code words, images and messages
into a form that communicates and influences its audience – @mrjoe + The UK Design
council
@mrjoe
Here’s some research I did many years ago for the trainline. What does this photo show?
This one photo saved countless time for thousands of people.
@mrjoe
2.
1.
It tells us two things. 1. When people travel they are often carrying loads of stuff 2. most people have a phone and are looking for the conformation email. They don’t print it out. We took the confirmation number and put it as the email subject line.
@mrjoe
I didn’t see them [the asterisks]. There’s nothing that explains what they mean.
How did you hear about us?*
How old are you? *
I’ve heard this a few times in user research. What does it mean? Not enough people to make it statistically significant but enough to change my approach to forms.
@mrjoe
How did you hear about us?
How old are you?Optional
We don’t mark mandatory, we mark optional. It’s enough to get me to change my design approach. Is there enough evidence to prove it from just a handful of users? No, it requires another set of supporting data. Which is why we need to use each approach in combination with another.
@mrjoe
hמp://www.slate.com/articles/technology/technology/2014/06/online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html?utm_content=bufferfa367&utm_medium=social&utm_source=twiמer.com&utm_campaign=buffer
http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg
In 2003 Mel Karmazin, a Viacom executive. Viacom are a huge media company where the bulk of their income is from advertising, visited the Google offices. He met Sergey and Larry who preceded to tell him all the ways they could measure and track advertising online. His one comment:
@mrjoe
You’re f---ing with the magic! – Mel Karmazin, Viacom executive
hמp://www.slate.com/articles/technology/technology/2014/06/online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html?utm_content=bufferfa367&utm_medium=social&utm_source=twiמer.com&utm_campaign=buffer
http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg
@mrjoe
http://justinmrao.com/lewis_rao_nearimpossibility.pdfOn the Near Impossibility of Measuring the
Returns to Advertising⇤
Randall A. LewisGoogle, Inc.
Justin M. RaoMicrosoft Research
April 23, 2013
Abstract
Classical theories of the firm assume access to reliable signals to measurethe causal impact of choice variables on profit. For advertising expenditurewe show, using twenty-five online field experiments (representing $2.8 mil-lion) with major U.S. retailers and brokerages, that this assumption typicallydoes not hold. Statistical evidence from the randomized trials is very weakbecause individual-level sales are incredibly volatile relative to the per capitacost of a campaign—a “small” impact on a noisy dependent variable can gen-erate positive returns. A concise statistical argument shows that the requiredsample size for an experiment to generate su�ciently informative confidenceintervals is typically in excess of ten million person-weeks. This also impliesthat heterogeneity bias (or model misspecification) unaccounted for by ob-servational methods only needs to explain a tiny fraction of the variation insales to severely bias estimates. The weak informational feedback means mostfirms cannot even approach profit maximization.
Keywords: advertising, field experiments, causal inference, electronic commerce,return on investment, informationJEL Codes: L10, M37, C93
⇤Previous versions circulated under the name “On the Near Impossibility of Measuring Adver-tising E↵ectiveness.” We especially thank David Reiley for his contributions to this work. He alsocame up with the title. Ned Augenblick, Arun Chandrasekhar, Garrett Johnson, Clara Lewis, R.Preston McAfee, Markus Mobius, Michael Schwarz, and Lars Lefgren gave us valuable feedbackas well. We also thank attendees at Brigham Young University’s Economics Seminar, the BeckerFriedman Institute Advances with Field Experiments Conference, and other venues where we havepresented this work. We also thank countless engineers, sales people, and product managers atYahoo, Inc. Much of this work was done when the authors were at Yahoo! Research, Santa Clara,CA.
This paper was published. The study was undertaken written when Lewis and Rao were at Yahoo.
What does this tell us? There is no proof this works. It’s built on faith.
@mrjoe
The 5 basic business models.
19
http://mrjoe.uk
Advertising Affiliate feesSelling Subscription
/ Licensing Market place
What does that mean for you. You need to understand some business basics. I wrote an article. more coming soon. These are the 5 basic income streams for organisations. Understand how they generate money and you have more science to base your designs on. Read more at http://mrjoe.uk
@mrjoe 22
5 years ago. I was a single man. At a point in life where it was becoming harder to meet new people. Someone suggested internet dating. I thought wow, ok, I can do this. My UX background is perfect for meeting the perfect person.
@mrjoe
http://blog.okcupid.com/index.php/the-4-big-myths-of-profile-pictures/
I went in search of data, OK Cupid has a blog. Full of great insights, like what is the perfect profile picture. I set about getting all UX on it, user researching my profile. At one point I emailed OK Cupid and asked if I could drop a google analytics tag my page. I learned a lot about how data can inform design. I also met my wife.
@mrjoe
Classic -Conversion (sales / visits)-Product page conversion (sales / visits to product page)-Funnel analysis-Basket abandonment-Bounce rate-Sales-Leads-Subscribers-Unique visitors-Returning visitors-Page views per visit -Visit to order ratio-Load time-Registrations-Visit / session length -Page views per visit-Time on page-Time on site-Form abandonment-Failed internal search-Referring pages / links-Geographic locations -Print page
Banners:-Click through rate-Impressions
Financial / sales / business -Average order value-Basket value-Profit margin-Average sales price-Cross sell-Gross margin-Category margin-Cost per lead-Customer acquisition cost-Lifetime customer value-Average customer value-Membership / subscription churn
RSS -Feedburner subscriptions-Shares on Google Reader
Call centre calls -Average call length-Support vs sales calls-Inbound vs outbound calls-Web generated calls (unique number on website)-Web fulfilled information calls
SEO -SEM keyword value-SEO positioning-Changes in SERP results/rankings-Top entry pages-Number of keywords triggering results for your site-Number of clicks to your site from keywords-Google trends-Inbound links (back link discovery)-Percentage share of each engine-Branded vs non-branded searches-Affiliate links-Affiliate fees
Social media -Facebook referrals-Incoming Twitter links-Facebook sends/shares/mentions-Facebook likes-Facebook fans-Facebook fan rates-Tweets-Retweets-@s on Twitter
-Twitter followers-Twitter follow rate-Google +1s-Bookmarks on Delicious-StumbleUpon thumbs up-StumpleUpon reviews-Diggs-Google BlogSearch links-Blog comments-Blog articles-Video views-Youtube favourites-Youtube channel subscriptions-Youtube channel comments-Youtube video reviews-Slideshare views-Forum mentions-Thread size-Online review mentions-Stars in reviews-Bit.ly / URL shorteners usage / clicks
Third party / benchmarks -Comscore -Hitwise-Alexa-Compete (US)
Email & campaigns -Email newsletter churn-Email sign-up-Emails sent-Emails bounced (bad address)-Email forwards-Email campaigns-Open rate-Delivery rate (sent - bounces)-Click through rate (CTR)-Email related to conversion / other metric-Unsubscribe rate
-SMS subscribers-Via print publication / 3rd party (unique URLs)
Internal search -Search no results-# Search 1 - 10 results -# Search 10 - 25 results-# Search over 50 results
IA: -GOMS technique-Time to content-Clicks to content -Back button clicks -Task completion-Errors
Testing metrics: -Words Recognition Rate-Reported expectations and performance-Facial reaction-Number of back presses-Gap satisfaction-Path's taken as a measure of scent-Work-flow matches mental model or not
Satisfaction measurements: -Net Promoter Score-ASQ PDF: After Scenario Questionnaire (3 Questions) -NASA-TLX : NASA's task load index is a measure of mental effort (5 Questions) -SMEQPDF: Subjective Mental Effort Questionnaire -UMEPDF : Usability Magnitude Estimation -SEQ PDF: Single Ease Question
A bewildering amount of metrics. Here’s some I collected over a couple of years. This can be dangerous, organisations can be tied to data, refusing to make a decision without it.
@mrjoe
I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that… … I won’t miss a design philosophy that lives or dies strictly by the sword of data. – Doug Bowmen
http://stopdesign.com/archive/2009/03/20/goodbye-google.html
This is taken from Doug Bowmen’s blog, after he left Google, he was their first visual designer. Pretty damning.
@mrjoe
Classic -Conversion (sales / visits)-Product page conversion (sales / visits to product page)-Funnel analysis-Basket abandonment-Bounce rate-Sales-Leads-Subscribers-Unique visitors-Returning visitors-Page views per visit -Visit to order ratio-Load time-Registrations-Visit / session length -Page views per visit-Time on page-Time on site-Form abandonment-Failed internal search-Referring pages / links-Geographic locations -Print page
Banners:-Click through rate-Impressions
Financial / sales / business -Average order value-Basket value-Profit margin-Average sales price-Cross sell-Gross margin-Category margin-Cost per lead-Customer acquisition cost-Lifetime customer value-Average customer value-Membership / subscription churn
RSS -Feedburner subscriptions-Shares on Google Reader
Call centre calls -Average call length-Support vs sales calls-Inbound vs outbound calls-Web generated calls (unique number on website)-Web fulfilled information calls
SEO -SEM keyword value-SEO positioning-Changes in SERP results/rankings-Top entry pages-Number of keywords triggering results for your site-Number of clicks to your site from keywords-Google trends-Inbound links (back link discovery)-Percentage share of each engine-Branded vs non-branded searches-Affiliate links-Affiliate fees
Social media -Facebook referrals-Incoming Twitter links-Facebook sends/shares/mentions-Facebook likes-Facebook fans-Facebook fan rates-Tweets-Retweets-@s on Twitter
-Twitter followers-Twitter follow rate-Google +1s-Bookmarks on Delicious-StumbleUpon thumbs up-StumpleUpon reviews-Diggs-Google BlogSearch links-Blog comments-Blog articles-Video views-Youtube favourites-Youtube channel subscriptions-Youtube channel comments-Youtube video reviews-Slideshare views-Forum mentions-Thread size-Online review mentions-Stars in reviews-Bit.ly / URL shorteners usage / clicks
Third party / benchmarks -Comscore -Hitwise-Alexa-Compete (US)
Email & campaigns -Email newsletter churn-Email sign-up-Emails sent-Emails bounced (bad address)-Email forwards-Email campaigns-Open rate-Delivery rate (sent - bounces)-Click through rate (CTR)-Email related to conversion / other metric-Unsubscribe rate
-SMS subscribers-Via print publication / 3rd party (unique URLs)
Internal search -Search no results-# Search 1 - 10 results -# Search 10 - 25 results-# Search over 50 results
IA: -GOMS technique-Time to content-Clicks to content -Back button clicks -Task completion-Errors
Testing metrics: -Words Recognition Rate-Reported expectations and performance-Facial reaction-Number of back presses-Gap satisfaction-Path's taken as a measure of scent-Work-flow matches mental model or not
Satisfaction measurements: -Net Promoter Score-ASQ PDF: After Scenario Questionnaire (3 Questions) -NASA-TLX : NASA's task load index is a measure of mental effort (5 Questions) -SMEQPDF: Subjective Mental Effort Questionnaire -UMEPDF : Usability Magnitude Estimation -SEQ PDF: Single Ease Question
Measure 2 to 3 things.
Each has: 1. Timescale
2. Benchmark 3. Reason to be
reported 4. Associated
action
http://mrjoe.uk/ux-and-roi-what-to-measure-and-what-to-expect/
Truth is it’s important to measure. It shows confidence in your design. Here’s some tips to get it right. See a talk by me on this here http://mrjoe.uk/ux-and-roi-what-to-measure-and-what-to-expect/
@mrjoe http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_squirrel/
The squirrel evolved bit by bit to fly. Small incremental improvements that allowed him to jump further, land better, and eventually to glide. The squirrel has reached a point where he isn’t going to get better at flying (called a local maximum).
@mrjoe
The soaring eagle. Elegantly drifting through the air with little effort. The eagle also evolves, little by little to its own maxim. It’s a superb flyer, soaring to 15,000 feet instead of making long leaps between trees. The eagle started from a better set of basics: front limbs to wings, reptile scales to feathers.
Good design gives a good starting point for evolution. MVT can get you to a maxim, but it can’t help you choose the best starting point to get to the highest maxim. A strong design leads to elegant flight. A weak design leads to hopping between branches. We need to combine MVT with craft. Read more: http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_
@mrjoe
Sore like an eagle don’t leap like a squirrelhttp://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_squirrel/
@mrjoe
James Vicary, in 1957, flashed up hungry eat popcorn, thirsty drink coke as single frames during a movie showing.
@mrjoe
58%A reported 58% increase in sales of popcorn. But it was later found out to be a hoax. He made it up to prop up his failing advertising business.
@mrjoe
Some examples, but there is no proof. http://www.badabingrecords.com/2011/03/the-subliminal-scares-a-short-history-of-an-american-obsession/ http://www.businessinsider.com/subliminal-ads-2011-5?op=1
@mrjoe
Strikingly, there was no significant effect.
http://www.bbc.co.uk/news/magazine-30878843
The media’s obsession. They were shown the same three-minute clip from the BBC/Kudos drama Spooks twice, but each time one group wore the blindfolds. The clip watched by the group with red blindfolds contained a 10-millisecond flash, every five seconds, of the word “Lipton", http://www.bbc.co.uk/news/magazine-30878843
So what can we do?
@mrjoe
There are resources out there that help with theory. This is from Ribot, alumni of my psychology for designers workshop. http://coglode.com/
Workshop: http://mrjoe.uk/psychology-design-ux-workshop/
@mrjoe PsychologyForDesigners.com
Or you can buy my book, I show you how to find a theory and apply it to design. How to use psychology to advocate design.
PsychologyForDesigners.com
@mrjoe
Title Text
38
A designer who doesn't understand psychology
is going to be no more successful
than an architect who doesn't
understand physics Image: http://victorenrich.com/archives/155
@mrjoe
@mrjoe
The biggest myth ever perpetuated in the design field is that good design sells itself. – Mike
Montiero
@mrjoe
'I feel the design, minimalist in style, reflects a modern design aesthetic’
- The DesignerWhat was wrong with that statement? Anytime you say I feel, it leaves the door open for the ‘Well I feel’. It becomes a battle of wills, there is no subjectivity to judge the design against.
@mrjoe http://www.gensleron.com/storage/post-images/cities/reinventingaclassicreuniontowerobservationdeckreopens/1.jpg
Which reminds me, I was once looking over a beautiful view, but I got annoyed at the poor information design of the support panel. My wife simply said, chill the f$ck out and enjoy the view, the view is the focus not the design of the information panel.
@mrjoe
Art: …works to be appreciated primarily for their beauty or
emotional power. – Merriam Webster Dictionary
@mrjoe
http://www.ise.bgu.ac.il/faculty/noam/papers/00_nt_ask_di_iwc.pdf
Art does support design. Pretty things are easier to use.