wrap up & experimentation · falls along the range of 0 to the max number in the distribution...
TRANSCRIPT
![Page 1: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/1.jpg)
Wrap up & Experimentation
CS147L Lecture 8Mike Krieger
Wednesday, November 25, 2009
![Page 2: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/2.jpg)
Intro
Wednesday, November 25, 2009
![Page 3: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/3.jpg)
Welcome back!
Wednesday, November 25, 2009
![Page 4: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/4.jpg)
By the end of today...
- Questions from implementations
- A few implementation loose ends
- A/B testing primer
- Google Analytics
Wednesday, November 25, 2009
![Page 5: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/5.jpg)
Questions?
Wednesday, November 25, 2009
![Page 6: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/6.jpg)
Loose ends
Wednesday, November 25, 2009
![Page 7: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/7.jpg)
Floaty bar
Wednesday, November 25, 2009
![Page 8: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/8.jpg)
Canonical implementation
- Gmail's mobile web app
Wednesday, November 25, 2009
![Page 9: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/9.jpg)
Gmail Demo
Wednesday, November 25, 2009
![Page 10: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/10.jpg)
Fitts Thumb
- Though hand input not quite the same as mouse, general principle applies:
- Minimize thumb-moving distance
- Make targets even larger than you think they need to be (thumbs are clumsy)
Wednesday, November 25, 2009
![Page 11: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/11.jpg)
Getting plugin
- Included with jQTouch under extensions/
- Copy jqt.floaty.js into your JS folder
Wednesday, November 25, 2009
![Page 12: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/12.jpg)
Integrating & customizing
Wednesday, November 25, 2009
![Page 13: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/13.jpg)
Initializing
Wednesday, November 25, 2009
![Page 14: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/14.jpg)
Styling
Wednesday, November 25, 2009
![Page 15: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/15.jpg)
Demo
- floaty.html
Wednesday, November 25, 2009
![Page 16: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/16.jpg)
Bottom bar
Wednesday, November 25, 2009
![Page 17: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/17.jpg)
UITabBarController on iPhone
Wednesday, November 25, 2009
![Page 18: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/18.jpg)
This won't work...
- Traditional approach: position: fixed at bottom:0
- Or, div with overflow:hidden and bottom bar with absolute position and bottom:0
Wednesday, November 25, 2009
![Page 19: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/19.jpg)
On the iPhone
- Scrolling scrolls entire page
- Floaty bar is probably the way to go...
- You could hack it up, but your users would have to learn to two-finger scroll for everything
Wednesday, November 25, 2009
![Page 20: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/20.jpg)
Full screening
Wednesday, November 25, 2009
![Page 21: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/21.jpg)
App mode
- Only engaged when users click home button (might want to prompt them, or do it before hand)
Wednesday, November 25, 2009
![Page 22: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/22.jpg)
App mode
Wednesday, November 25, 2009
![Page 23: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/23.jpg)
A/B Testing
Wednesday, November 25, 2009
![Page 24: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/24.jpg)
Why A/B test?
- You can have the best designers & great PMs...
- But nothing beats seeing what on earth people actually do
- Differences in usability, virality, and revenue
Wednesday, November 25, 2009
![Page 25: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/25.jpg)
Framework
- Selecting an experiment
- Choosing variations
- Selecting / sampling users
- Deploying & serving variations
- Measuring user behavior
- Analyzing results
Wednesday, November 25, 2009
![Page 26: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/26.jpg)
What makes a good experiment?
- Measuring user funnels through a sale
- Click-through rates for links
- Time spent / time until an action is taken
- Performance questions
- Email newsletters
Wednesday, November 25, 2009
![Page 27: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/27.jpg)
(continued)
- Minor tweaks to site design
- Flows with a clear goal
Wednesday, November 25, 2009
![Page 28: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/28.jpg)
In sum—
- a measurable user behavior that you believe will be modulated by tweaks in design
Wednesday, November 25, 2009
![Page 29: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/29.jpg)
What A/B testing won't tell you
- Is it aesthetically pleasing?
- Is it fun?
- Is it accessible?
- Is this even what my company should be doing?
Wednesday, November 25, 2009
![Page 30: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/30.jpg)
As Buxton would say
- A/B testing will help you get the design right, but can't help you get the right design in the first place
Wednesday, November 25, 2009
![Page 31: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/31.jpg)
Hillclimbing
controlled web experiments
Wednesday, November 25, 2009
![Page 32: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/32.jpg)
But really...
controlled web experiments
Wednesday, November 25, 2009
![Page 33: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/33.jpg)
Choosing Variations
- Think in terms of variables
- Spectrum of choices
- If time (and participant pool), look at interactions, too
Wednesday, November 25, 2009
![Page 34: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/34.jpg)
Examples
- Twitter homepage call to action
- Join the Conversation
- or
- Get Started
Wednesday, November 25, 2009
![Page 35: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/35.jpg)
Iteration
Wednesday, November 25, 2009
![Page 36: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/36.jpg)
Google Homepage
Wednesday, November 25, 2009
![Page 37: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/37.jpg)
...experiment?
- People actively want to join Google's A/B tests
- But can use interest/reactions as proxies for results in this case
Wednesday, November 25, 2009
![Page 38: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/38.jpg)
Selecting/sampling users
- Two general approaches
Wednesday, November 25, 2009
![Page 39: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/39.jpg)
The naive way
- Every time a user loads a page, they have a random chance of ending up in a bucket
Wednesday, November 25, 2009
![Page 40: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/40.jpg)
Why doesn't this work?
- Order effects
- Confuse the users, who want a consistent experience
- Random functions not so random
Wednesday, November 25, 2009
![Page 41: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/41.jpg)
Using a hashing function
- Suggested by Kohavi in his Web Experimentation paper
- How it's implemented at Meebo
Wednesday, November 25, 2009
![Page 42: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/42.jpg)
General idea
- Be as consistent as possible per user
- If we can, use User ID (across computers)
- If we can't, use a cookie (at least consistent at one computer)
- At the very worst, assign randomly
Wednesday, November 25, 2009
![Page 43: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/43.jpg)
MD5
- Hashing function; not great for encryption but fine for our purposes
- Problem: hashes will be long strings and we actually want a probability distribution
Wednesday, November 25, 2009
![Page 44: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/44.jpg)
Solution
- Hash the unique identifier plus the experiment name
- Get the hexadecimal digest of the resulting hash
- Convert to a decimal and see where it falls along the range of 0 to the Max number in the distribution
Wednesday, November 25, 2009
![Page 45: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/45.jpg)
In other words...
Wednesday, November 25, 2009
![Page 46: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/46.jpg)
(continued)
Wednesday, November 25, 2009
![Page 47: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/47.jpg)
Notes
- Will be evenly distributed from 0 to 1.0
- We can use this probability to bucket people
- Given the same input, will result in same number every time
Wednesday, November 25, 2009
![Page 48: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/48.jpg)
Deploying & serving variations
- For prototypes, much can be hard-coded
- For real production use, infrastructure can make life easier in the long run
Wednesday, November 25, 2009
![Page 49: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/49.jpg)
One easy way
Wednesday, November 25, 2009
![Page 50: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/50.jpg)
In the long term
- Build out front-end to turn experiments on/off or config file
Wednesday, November 25, 2009
![Page 51: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/51.jpg)
Overriding Javascript
- Problem: you already have most of your functions defined, but want your treatment to do something slightly different
Wednesday, November 25, 2009
![Page 52: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/52.jpg)
Encapsulation
Wednesday, November 25, 2009
![Page 53: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/53.jpg)
Overwriting
Wednesday, November 25, 2009
![Page 54: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/54.jpg)
Monkeypatching
- Idea: we want to do mostly the same thing, but do something before/afterwards that's slightly different, or modify the input
Wednesday, November 25, 2009
![Page 55: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/55.jpg)
How to
Wednesday, November 25, 2009
![Page 56: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/56.jpg)
Measuring behavior
- Are people actually doing something different?
- Using log lines, or writing straight to DB
Wednesday, November 25, 2009
![Page 57: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/57.jpg)
Normal Log lines10.32.109.7 - - [18/Nov/2009:22:36:32 -0800] "GET /courses/cs147/images/media.jpg HTTP/1.1" 200 191910 "http://hci.stanford.edu/courses/cs147/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"
Wednesday, November 25, 2009
![Page 58: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/58.jpg)
Tracking Log Lines10.32.109.7 - - [18/Nov/2009:22:36:32 -0800] "GET /track?condition=bluebutton&event=click&timebeforeclick=500 HTTP/1.1" 200 191910 "http://hci.stanford.edu/courses/cs147/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"
Wednesday, November 25, 2009
![Page 59: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/59.jpg)
How to process?
- In the small: use Python
- In the large: use Hadoop and Pig
Wednesday, November 25, 2009
![Page 60: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/60.jpg)
Pig, ultrabriefly
- (because I think this will be huge in a year or so)
Wednesday, November 25, 2009
![Page 61: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/61.jpg)
Pig
- SQL-like language built on top of Hadoop
- Makes writing Map/Reduce tasks really quick
- In use at Yahoo!, Twitter, Meebo, etc
Wednesday, November 25, 2009
![Page 62: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/62.jpg)
Pig sample code
Wednesday, November 25, 2009
![Page 63: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/63.jpg)
And the best part...
- Will compile & run for you over as many machines as necessary
Wednesday, November 25, 2009
![Page 64: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/64.jpg)
PHP Logging scripts
- Wouldn't work for production data
- Fine for any A/B tests or just logging / instrumentation you want to do
Wednesday, November 25, 2009
![Page 65: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/65.jpg)
DB Schema
Wednesday, November 25, 2009
![Page 66: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/66.jpg)
Logging events
Wednesday, November 25, 2009
![Page 67: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/67.jpg)
Example
Wednesday, November 25, 2009
![Page 68: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/68.jpg)
HTML
Wednesday, November 25, 2009
![Page 69: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/69.jpg)
JS
Wednesday, November 25, 2009
![Page 70: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/70.jpg)
Closing the loop
- Get the data out & aggregate
- Visualize!
Wednesday, November 25, 2009
![Page 71: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/71.jpg)
Reading data from SQLite
- report.php
Wednesday, November 25, 2009
![Page 72: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/72.jpg)
Basic code
Wednesday, November 25, 2009
![Page 73: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/73.jpg)
Reformat as data series
Wednesday, November 25, 2009
![Page 74: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/74.jpg)
Result[{"label":"blue","data":[[0,2]]},{"label":"red","data":[[1,12]]},{"label":"green","data":[[2,1]]}]
Wednesday, November 25, 2009
![Page 75: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/75.jpg)
Analysis options
- Excel
- Tableau
- R
- Javascript or Flash graphing/visualization libraries
Wednesday, November 25, 2009
![Page 76: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/76.jpg)
Briefly: flot
- jQuery plugin
- Super useful for basic graphing & charting needs
- Also handles time-series data well
- http://code.google.com/p/flot/
Wednesday, November 25, 2009
![Page 77: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/77.jpg)
From report.php to flot
Wednesday, November 25, 2009
![Page 78: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/78.jpg)
Demoflot.html
Wednesday, November 25, 2009
![Page 79: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/79.jpg)
Even better: Protovis
- Stanford Graphics lab project
- http://vis.stanford.edu/protovis/
Wednesday, November 25, 2009
![Page 80: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/80.jpg)
Significant change?
- Chi-Squared test
Wednesday, November 25, 2009
![Page 81: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/81.jpg)
Chi-Squared
- Idea: measure whether a particular distribution of measures deviates significantly from expected
Wednesday, November 25, 2009
![Page 82: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/82.jpg)
Null hypothesis
- Button color has no impact on click-through rate
Wednesday, November 25, 2009
![Page 83: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/83.jpg)
Testing
Where:O(i) is the observed frequency,E(i) is the expected frequencydegrees of freedom = (number of categories) - 1
Wednesday, November 25, 2009
![Page 84: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/84.jpg)
Sample data
Blue Red Green Total
267 267 266 800
Significant?
Wednesday, November 25, 2009
![Page 85: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/85.jpg)
Sample data
Blue Red Green Total
270 250 260 800
Significant?
Wednesday, November 25, 2009
![Page 86: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/86.jpg)
Sum the differences
Blue: (270-267)^2 / 267 = 0.03Red: (250-267)^2 / 267 = 1.08Green: (280-267)^2 / 267 = 0.63
0.03 + 1.08 + 0.63 = 1.74 = χ2
Wednesday, November 25, 2009
![Page 87: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/87.jpg)
Look it up in table
- (or use R/SPSS/something fancier)
- http://www2.lv.psu.edu/jxm57/irp/chisquar.html
Wednesday, November 25, 2009
![Page 88: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/88.jpg)
Significant?
p ~ 0.4 (we want 0.05)not significant
Wednesday, November 25, 2009
![Page 89: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/89.jpg)
Sample data
Blue Red Green Total
240 230 330 800
Significant?
Wednesday, November 25, 2009
![Page 90: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/90.jpg)
Sum the differences
Blue: (240-267)^2 / 267 = 2.73Red: (230-267)^2 / 267 = 5.12Green: (330-267)^2 / 267 = 14.86
2.73 + 5.12 + 14.86 = 22 = χ2
Wednesday, November 25, 2009
![Page 91: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/91.jpg)
Significant?
p < 0.001highly significant
Wednesday, November 25, 2009
![Page 92: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/92.jpg)
Wrap-up
- Can be a bit of work
- But can lead to amazing insights
- Plus, data analysis & visualization is really fun
Wednesday, November 25, 2009
![Page 93: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/93.jpg)
Google Analytics
Wednesday, November 25, 2009
![Page 94: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/94.jpg)
Why?
- It's free!
- For a small company or project, much better than rolling out some of these analysis tools for yourself
Wednesday, November 25, 2009
![Page 95: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/95.jpg)
Signing up
- http://analytics.google.com
Wednesday, November 25, 2009
![Page 96: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/96.jpg)
Integrating JS
Insert before </body>
Wednesday, November 25, 2009
![Page 97: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/97.jpg)
Tip: Tracking JS events
_trackevent function, takes(category, event, optional_key, optional_value)
Wednesday, November 25, 2009
![Page 98: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/98.jpg)
Using Web Optimizer
- Attached to AdSense
- Provides A/B testing tools with integrated confidence interval and significant different calculations
Wednesday, November 25, 2009
![Page 99: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/99.jpg)
Demo
Wednesday, November 25, 2009
![Page 100: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/100.jpg)
Final notes
Wednesday, November 25, 2009
![Page 101: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/101.jpg)
Where to go from here?
- Four fun things to explore
Wednesday, November 25, 2009
![Page 102: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/102.jpg)
Building full web apps in Django
- Templating
- Wrapping Request/Response
- Interfacing with the database
- Forms
Wednesday, November 25, 2009
![Page 103: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/103.jpg)
Some code from courseapp
Wednesday, November 25, 2009
![Page 104: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/104.jpg)
and the view...
Wednesday, November 25, 2009
![Page 105: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/105.jpg)
Trying out Google App Engine
- Great for weekend projects that could become something more
- Django templating built-in
- Can also use (most) of Django with app-engine-patch (http://code.google.com/p/app-engine-patch/)
- And, it'll scale if you need it to
Wednesday, November 25, 2009
![Page 106: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/106.jpg)
Adapting for native app
- PhoneGap lets you access native app features from JavaScript
- Can continue your class projects if you want to take them further
- Caveat: will have to get dev account
Wednesday, November 25, 2009
![Page 107: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/107.jpg)
Using Mechanical Turk for app feedback
- And quick testing of ideas
Wednesday, November 25, 2009
![Page 108: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/108.jpg)
Evaluating Scenarios
Wednesday, November 25, 2009
![Page 109: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/109.jpg)
Evaluating Scenarios
Wednesday, November 25, 2009
![Page 110: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/110.jpg)
Evaluating Scenarios
Wednesday, November 25, 2009
![Page 111: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/111.jpg)
Wednesday, November 25, 2009
![Page 112: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/112.jpg)
Wednesday, November 25, 2009
![Page 113: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/113.jpg)
Wednesday, November 25, 2009
![Page 114: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/114.jpg)
Reactions- “Personally I prefer the idea of storyboard one. This is
because the user freely walks around the museum as they would traditionally, yet automatically receive info about exhibits - a virtual guide without user input. Much more impressive.”
- “Personally I think the idea from storyboard one is more compelling. The reason for this is that I would be interested in finding out interesting information about a piece of artwork or a particular artist that I couldn't just get at the museum. The map of the museum is something that I can get at the museum on a piece of paper that doesn't require me to be pulling out my phone and wasting the battery to get to an exhibit.”
Wednesday, November 25, 2009
![Page 115: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/115.jpg)
Reactions
- “Students who are visiting for a school assignment and have limited time to partake in all the exhibits would definitely find that option helpful.”
- “I liked storyboard 2 because of [the use of] cell phones in a physical space.”
- Self-reported as non-designers
Wednesday, November 25, 2009
![Page 116: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/116.jpg)
Final plug
- We're hiring at Meebo!
- User Experience, Usability, UI...
- meebo.com/jobs or email me directly at [email protected]
Wednesday, November 25, 2009
![Page 117: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/117.jpg)
Thanks!
Wednesday, November 25, 2009
![Page 118: Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)](https://reader033.vdocuments.mx/reader033/viewer/2022050305/5f6d90ea88f5172fbb5fd2fc/html5/thumbnails/118.jpg)
Q’s?
Wednesday, November 25, 2009