user interfaces: evolution and revolution part one
DESCRIPTION
User Interfaces: Evolution and Revolution Part One. Andy van Dam Brown University April 25, 2011. My background. 1964 epiphany seeing Sketchpad movie 1966 “Computer-Driven Displays and their Use in Man-Machine Interaction” 1971- 86 - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/1.jpg)
Microsoft Research Asia - April 2011 1
User Interfaces:Evolution and Revolution
Part One
Andy van DamBrown University
April 25, 2011
4/25/2011
![Page 2: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/2.jpg)
Microsoft Research Asia - April 2011 2
My background 1964
– epiphany seeing Sketchpad movie 1966
– “Computer-Driven Displays and their Use in Man-Machine Interaction” 1971- 86
– work on graphics standards (SIGGRAPH Core, ISO PHIGS++); my group invented in logical (abstract) interaction devices: tasks such as inputting text strings, scalars, positions, picking/selection with pick path
Early 90s on: – desktop 3D widgets, post-WIMP UI’s, VR (BOOM, CAVE™) – interaction with
wands, gloves, trackers on body parts (feet), speech, pens on tablets, etc.. Also haptics for 3D GUI’s
1996 – 3D sketching w/ inferencing, mouse, then bi-mannual pen + “puck”– tabletPC apps for 2D visual languages: music, math, chem, 2D sketching
Current - Surface apps as driving apps, primarily in educational space: students and scholars. Useful apps, interesting new interaction techniques
4/25/2011
![Page 3: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/3.jpg)
Microsoft Research Asia - April 2011 3
Roadmap
1 - Current context – developments underway2 - Importance and brief history of UI/UX3 – WIMP post-WIMP/NUI4 – Invariants5 – Aspects of NUIs6 – Research Issues (a preview)7 – Bottom line
4/25/2011
![Page 4: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/4.jpg)
Microsoft Research Asia - April 2011 4
1 - Current context – developments underway
Evolution of desktop to mobile to ubiquitous computing
“Unreasonable effectiveness of data” Search Augmented reality/mixed reality Social computing
4/25/2011
![Page 5: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/5.jpg)
Microsoft Research Asia - April 2011 5
Continuation of "Moore's law" even by other means– multicore implies more CPUs, integrated GPUs and controllers
Evolution of desktop: to mobile and ubiquitous computing
4/25/2011
![Page 6: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/6.jpg)
Microsoft Research Asia - April 2011 6
Continuation of "Moore's law" even by other means– multicore implies more CPUs, integrated GPUs and controllers– GPUs are game-changing
Evolution of desktop: to mobile and ubiquitous computing
4/25/2011
![Page 7: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/7.jpg)
Microsoft Research Asia - April 2011 7
Continuation of "Moore's law" even by other means– multicore implies more CPUs, integrated GPUs and controllers– GPUs are game-changing– proliferation of platforms
Ubicomp – ecology of devices (society of appliances) and people
Evolution of desktop: to mobile and ubiquitous computing
4/25/2011
![Page 8: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/8.jpg)
Microsoft Research Asia - April 2011 8
Continuation of "Moore's law" even by other means– multicore implies more CPUs, integrated GPUs and controllers– GPUs are game-changing– proliferation of platforms
Ubicomp – ecology of devices (society of appliances) and people– iPhones/smartphones and iPads– game controllers, e.g., Wii and Kinect
Evolution of desktop: to mobile and ubiquitous computing
4/25/2011
![Page 9: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/9.jpg)
Microsoft Research Asia - April 2011 9
Continuation of "Moore's law" even by other means– multicore implies more CPUs, integrated GPUs and controllers– GPUs are game-changing– proliferation of platforms
Ubicomp – ecology of devices (society of appliances) and people– iPhones/smartphones and iPads– game controllers, e.g., Wii and Kinect– embedded computing in devices, appliances, vehicles, and buildings, including
sensors (RFID chips), QR (quick response) codes, ...
Evolution of desktop: to mobile and ubiquitous computing
4/25/2011
![Page 10: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/10.jpg)
Microsoft Research Asia - April 2011 10
Continuation of "Moore's law" even by other means– multicore implies more CPUs, integrated GPUs and controllers– GPUs are game-changing– proliferation of platforms
Ubicomp – ecology of devices (society of appliances) and people– iPhones/smartphones and iPads– game controllers, e.g., Wii and Kinect– embedded computing in devices, appliances, vehicles, and buildings,
including sensors (RFID chips), QR (quick response) codes, ... Browser-based and cloud computing
– monolithic shrink-wrapped apps cloud, with lightweight apps– AJAX and Web 2.0 mashups
Evolution of desktop: to mobile and ubiquitous computing
4/25/2011
![Page 11: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/11.jpg)
Microsoft Research Asia - April 2011 11
Unreasonable effectiveness of data
Core technologies (many worked on for decades)– voice recognition– hand writing recognition– machine learning– agents, e.g., e-commerce
Applications– data mining, visual analytics– machine translation– pattern detection and recognition– question answering
Jeopardy/Watson
4/25/2011
![Page 12: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/12.jpg)
Microsoft Research Asia - April 2011 12
Unreasonable effectiveness of data
Core technologies (many worked on for decades)– voice recognition– hand writing recognition– machine learning– agents, e.g., e-commerce
Applications– data mining, visual analytics– machine translation– pattern detection and recognition– question answering
Jeopardy/Watson Wolfram Alpha
4/25/2011
![Page 13: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/13.jpg)
Microsoft Research Asia - April 2011 13
Unreasonable effectiveness of data
Core technologies (many worked on for decades)– voice recognition– hand writing recognition– machine learning– agents, e.g., e-commerce
Applications– data mining, visual analytics– machine translation– pattern detection and recognition– question answering
Jeopardy/Watson Wolfram Alpha
Is this AI?– when it is in the research lab …4/25/2011
![Page 14: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/14.jpg)
Microsoft Research Asia - April 2011 14
Better output – Bing vs. Google
Search
4/25/2011
![Page 15: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/15.jpg)
Microsoft Research Asia - April 2011 15
Better output – Bing vs. Google
Contextualized search – history– geospatial location– social network– …
Image and video search– James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. "CueFlik: interactive
concept learning in image search" in ACM CHI '08, pp. 29-38. 2008. Voice search
Search
4/25/2011
![Page 16: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/16.jpg)
Microsoft Research Asia - April 2011 16
Mechanisms– superposition of computer-generated information and real
world Examples– Smartphone cameras (Layar’s Amsterdam)
Augmented reality/mixed reality
4/25/2011
![Page 17: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/17.jpg)
Microsoft Research Asia - April 2011 17
Mechanisms– superposition of computer-generated information and real
world Examples– Smartphone cameras (Layar’s Amsterdam)– SixthSense (MIT)
http://www.cs.brown.edu/~avd/UI/SixthSense-PranavMistry-part1Clip.mov
Augmented reality/mixed reality
4/25/2011
![Page 18: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/18.jpg)
Microsoft Research Asia - April 2011 18
Mechanisms– superposition of computer-generated information and real
world Examples– Smartphone cameras (Layar’s Amsterdam)– Sixthsense (MIT)– LeafView (Columbia & Smithsonian)
Augmented reality/mixed reality
4/25/2011
![Page 19: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/19.jpg)
Microsoft Research Asia - April 2011 19
Mechanisms– superposition of computer-generated information and real
world Examples– Smartphone cameras (Layar’s Amsterdam)– Sixthsense (MIT)– LeafView (Columbia & Smithsonian)– UNC visions –
Augmented reality/mixed reality
4/25/2011
![Page 20: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/20.jpg)
Microsoft Research Asia - April 2011 20
Mechanisms– superposition of computer-generated information and real
world Examples– Smartphone cameras (Layar’s Amsterdam)– Sixthsense (MIT)– LeafView (Columbia & Smithsonian)– UNC visions –
airplane repair
4/25/2011
![Page 21: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/21.jpg)
Microsoft Research Asia - April 2011 21
Mechanisms– superposition of computer-generated information and real
world Examples– Smartphone cameras (Layar’s Amsterdam)– Sixthsense (MIT)– LeafView (Columbia & Smithsonian)– UNC visions –
airplane repair Office of the Future – tele-presence, tele-immersion
Augmented reality/mixed reality
4/25/2011
![Page 22: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/22.jpg)
Microsoft Research Asia - April 2011 22
Social computing Social networking Teams– spectrum from single user to small team to large team to
crowdsourcing Social agents/avatars (Clippy!?!) – observing and mediating your interaction
Multiple demographics– digital natives and their expectations vs. aging boomer
population– one size does not fit all
4/25/2011
![Page 23: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/23.jpg)
Microsoft Research Asia - April 2011 23
Roadmap
1 - Current context – developments underway2 - Importance and brief history of UI/UX3 – WIMP post-WIMP/NUI4 – Invariants5 – Aspects of NUIs6 – Issues7 – Bottom line
4/25/2011
![Page 24: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/24.jpg)
Microsoft Research Asia - April 2011 24
2 - Importance and brief history of UI/UX
Distinctions between UI and UX
Is there an ideal user interface?
Historical trends
Projects and their legacies
4/25/2011
![Page 25: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/25.jpg)
Microsoft Research Asia - April 2011 25
Importance of UI/UX (1/2)
Distinctions between UI and UX– UX is more inclusive while UI is often restricted to
“look and feel” Key to productivity and differentiator– often the bulk of an app's code
4/25/2011
![Page 26: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/26.jpg)
26
Computing Capacity“Moore’s Law”
t
Human Capacity
t
Compute Graphics
CPU power vs. brain power
Microsoft Research Asia - April 2011
Use compute power in UI to increase b/w to the brain
4/25/2011
![Page 27: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/27.jpg)
Microsoft Research Asia - April 2011 27
Importance of UI/UX (2/2)
UIs much too hard to create and to use Style changes– controlled environment of 1980s MacApp - > wild wild
west, where every app has a different UI
4/25/2011
![Page 28: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/28.jpg)
Microsoft Research Asia - April 2011 28
Is there an ideal user interface?
None! Uis are a necessary evil Counterpoint: aesthetics of a good UI Want to communicate and control as
we do in and with the real world– objects– tasks– other participants (real and software agents)
Model: perfect Butler (Jeeves), but not HAL-9000 Future– ultimately: brain-machine interfaces,“cogito ergo fac”
4/25/2011
![Page 29: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/29.jpg)
Microsoft Research Asia - April 2011 29
Brown University’s BrainGate neural implant interface
Prof. John Donoghue et al– http://www.braingate2.org/
Deep brain implant– tapping about 100 neurons– cursor control for people who have NO
physical control
Others working on non-invasive technology (brain wave monitors)4/25/2011
![Page 30: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/30.jpg)
Microsoft Research Asia - April 2011 30
Is there an ideal user interface?
None! Uis are a necessary evil Counterpoint: aesthetics of a good UI Want to communicate and control as
we do in and with the real world– objects– tasks– other participants (real and software agents)
Models: Jeeves, HAL-9000 Future
– ultimately: brain-machine interfaces,“cogito ergo fac” Today: transparent, fluid UIs -> automaticity
4/25/2011
![Page 31: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/31.jpg)
31
History of UI: Punctuated Equilibrium
1960s… Batch
Microsoft Research Asia - April 20114/25/2011
![Page 32: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/32.jpg)
32
History of UI: Punctuated Equilibrium
1960s… Batch
1970s… Command Line on “Glass TTY”
Microsoft Research Asia - April 20114/25/2011
![Page 33: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/33.jpg)
33
History of UI: Punctuated Equilibrium
1960s… Batch
1970s… Command Line on “Glass TTY”
1980s… WIMP GUI
Microsoft Research Asia - April 20114/25/2011
![Page 34: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/34.jpg)
34
History of UI: Punctuated Equilibrium
1960s… Batch
1970s… Command Line on “Glass TTY”
1980s… WIMP GUI
Microsoft Research Asia - April 2011
1990s… Post-WIMP NUI
4/25/2011
![Page 35: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/35.jpg)
35
History of UI: Punctuated Equilibrium
Each paradigm shift led to a large increase in user base
Each paradigm persists
1960s… Batch
1970s… Command Line on “Glass TTY”
1980s… WIMP GUI
Microsoft Research Asia - April 2011
1990s… Post-WIMP NUI
4/25/2011
![Page 36: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/36.jpg)
Microsoft Research Asia - April 2011 36
Vannevar Bush’s Memex (1945) annotation, hypertext
Projects and their legacies (1/3)
4/25/2011
![Page 37: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/37.jpg)
Microsoft Research Asia - April 2011 37
Vannevar Bush’s Memex (1945) annotation, hypertext Ivan Sutherland’s SketchPad (1963) interactive graphics
– http://www.cs.brown.edu/~avd/UI/SketchpadEdited-avd-4.20.11.avi
Projects and their legacies (1/3)
4/25/2011
![Page 38: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/38.jpg)
Microsoft Research Asia - April 2011 38
Vannevar Bush’s Memex (1945) annotation, hypertext Ivan Sutherland’s SketchPad (1963) interactive graphics Donald Bitzer’s PLATO (1967) touch screens
Projects and their legacies (1/3)
4/25/2011
![Page 39: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/39.jpg)
Microsoft Research Asia - April 2011 39
Vannevar Bush’s Memex (1945) annotation, hypertext Ivan Sutherland’s SketchPad (1963) interactive graphics Donald Bitzer’s PLATO (1967) touch screens Doug Engelbart’s NLS (1968) bi-manual interaction with
mouse, keyboard, and chord; integrated audio/video telecollaboration
Projects and their legacies (1/3)
4/25/2011
![Page 40: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/40.jpg)
Microsoft Research Asia - April 2011 40
Vannevar Bush’s Memex (1945) annotation, hypertext Ivan Sutherland’s SketchPad (1963) interactive graphics Donald Bitzer’s PLATO (1967) touch screens Doug Engelbart’s NLS (1968) bi-manual interaction with
mouse, keyboard, and chord; integrated audio/video telecollaboration
Robert Anderson’s Math Recognizer(1968) pen-based, gesture recognition personal laptop/tablet– http://www.cs.brown.edu/~avd/UI/Anderson.mov
Projects and their legacies (1/3)
4/25/2011
![Page 41: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/41.jpg)
Microsoft Research Asia - April 2011 41
Alan Kay’s DynaBook (1972) personal laptop/tablet
Projects and their legacies (2/3)
4/25/2011
![Page 42: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/42.jpg)
Microsoft Research Asia - April 2011 42
Alan Kay’s DynaBook (1972) personal laptop/tablet Bolt’s Put That There (1979) multi-modal UI: gesture, voice
commands, graphical and audio feedback– http://www.cs.brown.edu/~avd/UI/Put-That-There-TwoPeopleClip.mov
Projects and their legacies (2/3)
4/25/2011
![Page 43: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/43.jpg)
Microsoft Research Asia - April 2011 43
Alan Kay’s DynaBook (1972) personal laptop/tablet Bolt’s Put That There (1979) multi-modal UI: gesture, voice
commands, graphical and audio feedback– http://www.cs.brown.edu/~avd/UI/Put-That-There-TwoPeopleClip.mov
Myron Kruger’s Video Place (1986) body-centric interaction using video and sensors– http://www.cs.brown.edu/~avd/UI/MyronKruegerCritterClip.mov
Projects and their legacies (2/3)
4/25/2011
![Page 44: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/44.jpg)
Microsoft Research Asia - April 2011 44
Roadmap
1 - Current context – developments underway2 - Importance and brief history of UI/UX3 – WIMP post-WIMP/NUI4 – Invariants5 – Aspects of NUIs6 – Issues7 – Bottom line
4/25/2011
![Page 45: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/45.jpg)
Microsoft Research Asia - April 2011 45
Advantages of WIMP GUIs
Xerox PARC’s legacy provides a “standard” – ease of X for users (learning, remembering, using…)
Layers of support software– ease of implementation, maintainability
toolkitsGUI buildersUIMS
– much more support for “look” (visual design) than for “feel” (interaction design)
“feel” involves domain-specific application semantics
4/25/2011
![Page 46: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/46.jpg)
Microsoft Research Asia - April 2011 46
Limitations of WIMP GUIs (1/2)
Limited vision (flat, 2D) No speech No gestures Limited audio and tactile
feedback One-handed interaction Limited movement System unaware of user System has no model of user
4/25/2011
![Page 47: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/47.jpg)
Microsoft Research Asia - April 2011 47
Limitations of WIMP GUIs (2/2)
Imposes sequential “ping-pong” dialog model: mouse and keyboard input, 2D graphics (sound?) output– deterministic and discrete– difficult to handle simultaneous input, even two mice– pure WIMP doesn’t use other senses: hearing, touch, ...– >50% of our neurons in visual cortex, but as humans it is very
difficult for us to communicate without speech, sound... Not usable in many environments – while walking or driving, in the shower,…– in immersive VR (e.g., head-mounted display or CAVE™) where
you are “in” the 3D scene: no keyboard, mouse…
4/25/2011
![Page 48: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/48.jpg)
Microsoft Research Asia - April 2011 48
Characteristics of post-WIMP user interfaces
Usually multiple simultaneous devices, sensory channels, users– multi-modal UIs
High bandwidth, continuous input– body part tracking (head, gaze, hand...) – gesture and speech recognition, often interlaced (“Put That There”)
Typically non-deterministic, probabilistic decoding and disambiguation (many issues, e.g., spatio-temporal resolution) – back-tracking to refine maximum-likelihood decoding– similar to differential diagnosis– in multi-modal UIs best done via mutual reinforcement
(Phil Cohen et al’s “unification”)
4/25/2011
![Page 49: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/49.jpg)
Microsoft Research Asia - April 2011 49
WIMP GUIs Will Be Augmented, Not Replaced UI Spectrum– direct control
direct manipulationby-example...
– indirect control agentssocial interfaces (avatars)
WIMP enhanced by– speech & gesture recognition
pens and multi-touchbody tracking and scene extraction via computer vision
technology, e.g., Kinect4/25/2011
![Page 50: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/50.jpg)
Microsoft Research Asia - April 2011 50
NUI (Natural User Interface)
Context-sensitive spectrum – “natural” refers to making the user “a natural” and the interface “invisible”
Goals– leverage full human sensorium, perceptual and cognitive
ability, even emotional state (affect)– minimize cognitive load by shifting from cognitive to perceptual
and motor skills Strategies
– multi-modal human interaction combined with intelligent computer collaboration
– shared human and machine initiative4/25/2011
![Page 51: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/51.jpg)
Microsoft Research Asia - April 2011 51
NUI (Natural User Interface) Examples
A movie NUI: Minority Report John Underkoffler of Oblong at TED 2010– http://www.cs.brown.edu/~avd/UI/UnderkofflerClip.mov
Apple’s vision “Knowledge Navigator”: 1989– http://www.cs.brown.edu/~avd/UI/KnowledgeNavigator.mp4
MSR’s Bill Buxton on NUIs– http://www.cs.brown.edu/~avd/UI/Buxton2.mp4
4/25/2011
![Page 52: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/52.jpg)
Microsoft Research Asia - April 2011 52
Agents: inferring user intent
Hierarchy of capabilities– helper /clerk
explicit instruction, e.g., commands, hi-level parameterized spec– concierge
higher level specification learn by example
Examples– Siri – iPhone Virtual Personal Assistant app
http://www.cs.brown.edu/~avd/UI/Siri.mp4
– MSR’s Bohus & Horvitz’ Virtual Receptionist 2009 situated interaction
http://www.cs.brown.edu/~avd/UI/Bohus-Horvitz-VirtualReceptionistClip.mov
4/25/2011
![Page 53: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/53.jpg)
Microsoft Research Asia - April 2011 53
Siri – iPhone Virtual Personal Assistant app
4/25/2011
![Page 54: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/54.jpg)
Microsoft Research Asia - April 2011 54
MSR’s Bohus & Horvitz’ Virtual Receptionist 2009situated interaction
4/25/2011
![Page 55: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/55.jpg)
Microsoft Research Asia - April 2011 55
Agents: inferring user intent Hierarchy of capabilities– helper /clerk
explicit instruction, e.g., commands, hi-level parameterized spec– concierge
higher level specification learn by example
– personal assistant reason based on observation of past activity unlike concierge, the assistant has a model of your context
therefore much higher level of reasoning and inferencing of intent
– perfect partner anticipation of intent
NUIs some combination of – direct manipulation plus indirect control via agents– some under explicit control, e.g., avatars, some autonomous4/25/2011
![Page 56: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/56.jpg)
Microsoft Research Asia - April 2011 56
Roadmap
1 - Current context – developments underway2 - Importance and brief history of UI/UX3 – WIMP post-WIMP/NUI4 – Invariants5 – Aspects of NUIs6 – Research issues7 – Bottom line
4/25/2011
![Page 57: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/57.jpg)
Microsoft Research Asia - April 2011 57
4 – Reflect: NUI’s evolve, what are some invariants? Humans in the loop NUIs will augment GUIs– iPhone and its SmartPhone equivalents, Pads are hybrids
Multiple media and formats– must co-exist and play well together
Rapid context-switching between tasks – system should help user manage and manipulate
complexity, e.g., contexts/”working sets”
4/25/2011
![Page 58: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/58.jpg)
Microsoft Research Asia - April 2011 58
Roadmap
1 - Current context – developments underway2 - Importance and brief history of UI/UX3 – WIMP post-WIMP/NUI4 – Invariants5 – Aspects of NUIs6 – Research Issues7 – Bottom line
4/25/2011
![Page 59: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/59.jpg)
Microsoft Research Asia - April 2011 59
5 – Aspects of NUIs
WIMP was the last one-size-fits-all UI UIs need to be much more adaptive and take into
account– context– tasks– time– user roles– input and output
4/25/2011
![Page 60: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/60.jpg)
Microsoft Research Asia - April 2011 60
Context as aspect of UI/UX (1/2)
Importance of context for– user– task– platform– environment– history– social network
Group structures– solo vs. small team vs. crowd– co-located vs. telecollaboration, both synchronous and
asynchronous
4/25/2011
![Page 61: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/61.jpg)
Microsoft Research Asia - April 2011 61
Context as aspect of UI/UX (2/2)
Locales– Home
shower studykitchen living room and denbedroom
– Mobilewalkingdriving
– Office
4/25/2011
![Page 62: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/62.jpg)
Microsoft Research Asia - April 2011 62
Tasks (1/2)
Type of task and data that is involved Structure– well-defined/structured/formal vs.
exploratory/unstructured Dedicated or shifting – get/supply information– browsing, searching– analysis vs. synthesis– level of abstraction and complexity
need to move from data to interpretation to knowledge
4/25/2011
![Page 63: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/63.jpg)
Microsoft Research Asia - April 2011 63
Tasks (2/2)
Size – measure of complexity
Landay’s Activity-Centered Design: Yang Li and James A. Landay. "Activity-based prototyping of ubicomp applications for long-lived, everyday human activities" in proceedings of ACM CHI '08
Time duration (e.g., “delete file” vs. “shuttle to building 8”vs. “getting fit”)– degree of having to save and restore state ...
Examples– data entry and form-filling– document composition– data mining– brainstorming– pattern searching– problem solving
4/25/2011
![Page 64: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/64.jpg)
Microsoft Research Asia - April 2011 64
Time as a first-class notion
Frequency of interaction Synchronous vs. asynchronous Defining, capturing, restoring, forking state/context
4/25/2011
![Page 65: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/65.jpg)
Microsoft Research Asia - April 2011 65
Time as a first-class notion
Frequency of interaction Synchronous vs. asynchronous Defining, capturing, restoring, forking state/context
4/25/2011
![Page 66: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/66.jpg)
Microsoft Research Asia - April 2011 66
User Roles
Occasional/casual vs. expert Research vs. ad-hoc Dynamically changing within session– learner– expert– explorer – communicator– manager/leader– critic– …
4/25/2011
![Page 67: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/67.jpg)
Microsoft Research Asia - April 2011 67
Input and Output
What do you need and why– data formats– lossage transfer issues
datastructuressemantics
4/25/2011
![Page 68: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/68.jpg)
Microsoft Research Asia - April 2011 68
Roadmap
1 - Current context – developments underway2 - Importance and brief history of UI/UX3 – WIMP post-WIMP/NUI4 – Invariants5 – Aspects of NUIs6 – Research Issues7 – Bottom line
4/25/2011
![Page 69: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/69.jpg)
Microsoft Research Asia - April 2011 69
6 – Research issues – a preview Component technology
– making speech and gesture recognition more robust; addressing discoverability, learnability…
Ecosystem of devices and users– moving from single user/device to multi-user/device/agent
Complexity management Situated interaction
– situated interaction task focus with situational awareness Acquiring knowledge of user intent, user modeling “Transparent movement” of context/state between environments
– environment-independent notion of context/state (>= “desktop”) Modeling NUI interaction
– building a NUI design discipline and NUI IDEs4/25/2011
![Page 70: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/70.jpg)
Microsoft Research Asia - April 2011 70
Roadmap
1 - Current context – developments underway2 - Importance and brief history of UI/UX3 – WIMP post-WIMP/NUI4 – Invariants5 – Aspects of UIs6 – Issues7 – Bottom line
4/25/2011
![Page 71: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/71.jpg)
Microsoft Research Asia - April 2011 71
7 – Bottom line
NUIs – will augment and expand, not replace existing Uis– much more multi-modal (especially bi-manual) interaction
Computer-human interaction – increasingly seamless and invisible: automaticity
Focus shift – from limited single user/computer UI to multi-everything UX
“smart offices, homes, …” with large interactive display surfaces, seamlessly interoperating with a host of portable devices
4/25/2011
![Page 72: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/72.jpg)
Microsoft Research Asia - April 2011 72
References
Upcoming NUI book:– Daniel Wigdor and Dennis Wixon. “Brave NUI World: Designing Natural
User Interfaces for Touch and Gesture”. Morgan Kaufmann, to be published in April 2011.
Conferences:– TEI (Tangible, Embedded, and Embodied Interaction)
http://www.tei-conf.org/11/hm/ NSF, ACM, SIGCHI, and Microsoft among the sponsors
– ICMI (International Conference on Multimodal Interaction) http://www.acm.org/icmi/2011/ ACM, SIGCHI, and Microsoft among the sponsors
Website repository: User Interfaces: Evolution and Revolution– http://www.cs.brown.edu/~avd/UI/
4/25/2011
![Page 73: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/73.jpg)
Microsoft Research Asia - April 2011 73
Garibaldi Panorama – 1860’s
4/25/2011
![Page 74: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/74.jpg)
Microsoft Research Asia - April 2011 74
Garibaldi Panorama – 1860’s
4/25/2011
Size– scroll approximately 90 meters long, and 1.5 meters tall,
painted on both sides Medium– watercolor on thin paper similar to wallpaper
Kept in a climate-controlled vault– panorama essentially inaccessible. – scanned at high resolution, best displayed on the
Surface for interactive viewing (like the famous Along the River During the Qingming Festival scroll)
![Page 75: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/75.jpg)
Microsoft Research Asia - April 2011 75
LADS (Large Art Displayed on Surface)
Generalization of "Garibaldi on the Surface” application
Any artwork, but especially meant for large artworks – currently 2D only
Modes– catalog with filters– artwork viewing (with metadata and other documents)
and manipulation Will run on any Windows 7 Touch platform
4/25/2011
![Page 76: User Interfaces: Evolution and Revolution Part One](https://reader035.vdocuments.mx/reader035/viewer/2022062316/56816930550346895de07c28/html5/thumbnails/76.jpg)
Microsoft Research Asia - April 2011 76
“To Infinity and Beyond…”
4/25/2011