gestural technologies and gesture interactionskane/classes/is760/fall2011/... · 2011-11-08 ·...
TRANSCRIPT
Gestural Technologies and Gesture Interaction
11/07/2011UMBC IS 760 Lisa Anthony, PhD
Plan for todayy
1. What are gestures?1. What are gestures?2. Hardware
– Sensing gestures made by users– Sensing gestures made by users
3. SoftwareUnderstanding gestures made by users– Understanding gestures made by users
4. UsabilityInteraction design concerns for gesture input– Interaction design concerns for gesture input
5. Interaction Case StudiesWhat orks and hat doesn’t ork for gest re interaction– What works and what doesn’t work for gesture interaction
6. Remaining ChallengesN f ti i t l t h l i & t i t ti– New frontiers in gestural technologies & gesture interaction
2Gestural Technologies and Gesture Interaction: Introduction
1. WHAT DO WE MEAN BY “GESTURES”?
Gestural Technologies and Gesture Interaction 3
What do we mean by “gestures”?y g
definitions? examples?…definitions? examples?
Gestural Technologies and Gesture Interaction: What do we mean by gestures? 4
SurfaceSurfaceGesturesGestures
Direct GesturesDirect GesturesDevice GesturesDevice Gestures
Gestural Technologies and Gesture Interaction: What do we mean by gestures? 5
Air GesturesAir Gestures
SurfaceSurfaceGesturesGestures
StylusStylus
Direct GesturesDirect GesturesDevice GesturesDevice Gestures
yywritingwriting
Gestural Technologies and Gesture Interaction: What do we mean by gestures? 6
Air GesturesAir Gestures
SurfaceSurfaceGesturesGestures
StylusStylus FingerFinger
Direct GesturesDirect GesturesDevice GesturesDevice Gestures
yywritingwriting multitouchmultitouch
Gestural Technologies and Gesture Interaction: What do we mean by gestures? 7
Air GesturesAir Gestures
SurfaceSurfaceGesturesGestures
StylusStylus FingerFinger
Direct GesturesDirect GesturesDevice GesturesDevice Gestures
yywritingwriting multitouchmultitouch
NintendoNintendo
Gestural Technologies and Gesture Interaction: What do we mean by gestures? 8
Air GesturesAir GesturesWiimoteWiimote
SurfaceSurfaceGesturesGestures
StylusStylus FingerFinger
Direct GesturesDirect GesturesDevice GesturesDevice Gestures
yywritingwriting multitouchmultitouch
MicrosoftMicrosoftKinectKinect
NintendoNintendo
Gestural Technologies and Gesture Interaction: What do we mean by gestures? 9
Air GesturesAir GesturesWiimoteWiimote
SurfaceSurfaceGesturesGestures
StylusStylus FingerFinger
Direct GesturesDirect GesturesDevice GesturesDevice Gestures
yywritingwriting multitouchmultitouch
MicrosoftMicrosoftKinectKinect
NintendoNintendo
Gestural Technologies and Gesture Interaction: What do we mean by gestures? 10
Air GesturesAir GesturesWiimoteWiimote
Sensing gestures made by users
2. HARDWARE
Gestural Technologies and Gesture Interaction 11
Types of sensing surfacesyp g
Technology How it works Accepted Real‐worldTechnology How it works Accepted input devices
Real world examples
Capacitivetouchscreen
Conductive material; when touched by human finger (a conductor) electrostatic
Finger, active stylus
iPhone, iPod, iPadtouchscreen human finger (a conductor), electrostatic
field is distorted measurably.stylus iPad
Resistiveh
Two layers of material; when touched, h h d l f h
Finger, passive l h l
LG Optimus,ktouchscreen they press together and location of touch
is registered by gridlines on material.stylus, whole hand
Nokia N97
Surface Sound waves are passed along surface of Finger, active / ATMs, public acoustic wave material; when touched, sound waves
are distorted measurably.passive stylus kiosks
Infrared / Grid of LED beams are projected over Finger, active / Microsoft /optical imaging
p jsurface; breaks in the beams correspond to touch locations.
g , /passive stylus Surface
( f l h h l i b h h )
Gestural Technologies and Gesture Interaction: Hardware 12
(surface gestures only; other approaches also exist but these are the most common)
Understanding gestures made by users
3. SOFTWARE
Gestural Technologies and Gesture Interaction 13
Capturing surface gesture inputp g g p
Input events are series of (X Y) coordinatesInput events are series of (X, Y) coordinates– Sometimes time or pressure can also be sensed for each point
Stroke = {Pen Down, Point1, … Pointn, Pen Up}Language Listeners Events
Java MouseListener: MousePressed, MouseDragged, MouseReleased
MouseEvent{MOUSE_PRESSED,MOUSE_DRAGGED, MOUSE RELEASED}MOUSE_RELEASED}
C# System.Windows.Forms.MouseEventHandler: MouseDown, MouseMove,MouseUp
MouseEventArgs
MouseUp
Android View.OnTouchListener.OnTouch MotionEvent{ACTION_DOWN,ACTION MOVE
Gestural Technologies and Gesture Interaction: Software 14
ACTION_MOVE, ACTION_UP}
Approaches to gesture recognitionpp g gSituational approaches Algorithmic approachespp
Online vs. offline
g pp
Machine learning
Unistroke vs. multistroke
Segmented vs. free‐form
– Finite state machines– Hidden Markov Models
(HMMs)g
Trainable model vs. fixed model
Mi d d l i l d l
( )– Neural networks– Statistical classifiers
Mixed model vs. single model
Size of vocabularyDynamic programming
Template matchingTemplate matching
Heuristic recognizers
Gestural Technologies and Gesture Interaction: Software 15
$1 and $N recognizers$ $ g
(Anthony & Wobbrock 2010)(Anthony & Wobbrock, 2010)
Template matchers
$1: unistroke, $N: multistroke
Open‐source (C#, Java, more)Open source (C#, Java, more)
(how $N represents multistrokes)
Gestural Technologies and Gesture Interaction: Software 16
(how $N represents multistrokes)
Improving recognition accuracyp g g y
Limited vocabulary setLimited vocabulary set
Boxed character entry
User‐specific training examples
Use of context (domain specific)Use of context (domain‐specific)
…others?
Gestural Technologies and Gesture Interaction: Software 17
Recognition challengesg g
Segmentation: Overlapping Orientation of symbolsSegmentation: Overlapping symbols
Orientation of symbols
Segmentation: Multistroke C f bl b lSegmentation: Multistrokesymbols
Confusable symbols
Gestural Technologies and Gesture Interaction: Software 18
! !
Operating system integrationp g y g
Google Android open‐source toolkit (Li 2009)Google Android open‐source toolkit (Li, 2009)
Gestural Technologies and Gesture Interaction: Software 19
Interaction design concerns for gesture input
4. USABILITY
Gestural Technologies and Gesture Interaction 20
Usability and interaction concernsy
ideas?…ideas?
21
Deciding what gestures to supportg g pp
(Morris et al 2010)(Morris et al, 2010)
Application specific vs. platform / interaction standards
Gestural Technologies and Gesture Interaction: Usability 22
Ergonomics and comfortg
issues with long‐term use of surface air gestures?…issues with long‐term use of surface, air gestures?
…how is gesture interaction different than using a keyboard and mouse?
Gestural Technologies and Gesture Interaction: Usability 23
Beware the dreaded gorilla arm!
Precision and touch point sizep
Stylus vs finger use affects desired size of interactorsStylus vs. finger use affects desired size of interactors
Recommendations on touch point size differ by platform:– Apple: 44 pixels x 44 pixels (164ppi)– Windows Phone: 9 mm x 7 mm– Nokia: 1 cm x 1cm
Microsoft Office Ribbon vs Classic Toolbar
Gestural Technologies and Gesture Interaction: Usability 24
Recognition tolerance and errorsg
Gesture input tolerance:Gesture input tolerance:– Size, orientation, speed, tremor– Multiple styles for symbolsMultiple styles for symbols
Gesture delimitation:How do you know the user is making a gesture vs just moving their– How do you know the user is making a gesture vs. just moving their arms?
– Mode switching, hardware buttons, special gestures, returning toMode switching, hardware buttons, special gestures, returning to “home” position…others?
Error recovery:Error recovery:– How much should we ask the user to help correct system input?– Depends on context, e.g., students learning math vs. mathematiciansp , g , g
Gestural Technologies and Gesture Interaction: Usability 25
Gesture recognition feedbackg
Method When would you use it? When would you not use it?
Immediately after receiving single strokereceiving single stroke (eager)
Recognition in background, waiting for enough information (lazy)
Upon user request (e.g., tap, special gesture, menu item)
Never (e.g., store raw strokes only)
Gestural Technologies and Gesture Interaction: Usability 26
To train or not to train?
User‐dependent vs user‐independentUser‐dependent vs. user‐independent
When to collect data to train the gesture recognizer?
Gestural Technologies and Gesture Interaction: Usability 27
$N recognition rates with more training examples (Anthony & Wobbrock, 2010)
What works and what doesn’t work for gesture interaction
5. INTERACTION CASE STUDIES
Gestural Technologies and Gesture Interaction 28
Minority Report http://www.youtube.com/wath?demonstration ch?v=NwVBzx0LMNQ
1. Type of gesture input:S f / Ai• Surface / Air
• Device / Direct
2. Pros:
3. Cons:
4. Other comments:
Gestural Technologies and Gesture Interaction: Interaction Case Studies 29
Air Guitar Hero http://www.youtube.com/wath? kdemonstration ch?v=pktVSTwC8qo
1. Type of gesture input:S f / Ai• Surface / Air
• Device / Direct
2. Pros:
3. Cons:
4. Other comments:
Gestural Technologies and Gesture Interaction: Interaction Case Studies 30
Hands‐on Math http://www.acm.org/uist/arch/ d / /demonstration ive/videos/2010/p17‐
zeleznik.wmv1. Type of gesture input:S f / Ai• Surface / Air
• Device / Direct
2. Pros:
3. Cons:
4. Other comments:
Gestural Technologies and Gesture Interaction: Interaction Case Studies 31
Double Flip http://www.youtube.com/wath? ddemonstration ch?v=NuN_d‐DiAKs
1. Type of gesture input:S f / Ai• Surface / Air
• Device / Direct
2. Pros:
3. Cons:
4. Other comments:
Gestural Technologies and Gesture Interaction: Interaction Case Studies 32
Iron Man http://www.youtube.com/wath? l flhdemonstration ch?v=PYjlMflhysc
1. Type of gesture input:S f / Ai• Surface / Air
• Device / Direct
2. Pros:
3. Cons:
4. Other comments:
Gestural Technologies and Gesture Interaction: Interaction Case Studies 33
Mouse Gestures http://www.youtube.com/wath?demonstration ch?v=WUYyCituz_U
1. Type of gesture input:S f / Ai• Surface / Air
• Device / Direct
2. Pros:
3. Cons:
4. Other comments:
Gestural Technologies and Gesture Interaction: Interaction Case Studies 34
Kinect Sign Language http://www.youtube.com/wath?demonstration ch?v=qFH5rSzmgFE
1. Type of gesture input:S f / Ai• Surface / Air
• Device / Direct
2. Pros:
3. Cons:
4. Other comments:
Gestural Technologies and Gesture Interaction: Interaction Case Studies 35
New frontiers in gestural technologies and gesture recognition
6. REMAINING CHALLENGES
Gestural Technologies and Gesture Interaction 36
Remaining Challengesg g
ideas about the future of gesture interaction?…ideas about the future of gesture interaction?
37Gestural Technologies and Gesture Interaction: Remaining Challenges
Combining gestural technologiesg g g(Hinckley et al, 2010)( y , )Design considerations:
– Pen vs TouchDiff ti t d I t h bl t k– Differentiated vs Interchangeable taskassignment
– Nonpreferred vs Preferred hand assignmentassignment
– Unimodal vs Multimodal input– Mobile vs Stationary use– Unimanual vs Bimanual interactionUnimanual vs Bimanual interaction– Interleaved vs Simultaneous input– Ink vs Command mode– Elementary input vs PhrasesElementary input vs Phrases
“Pen writes, touch manipulates”New tools and workflow enhancements are possibleNew tools and workflow enhancements are possible
38
Questions?QContact:
– Lisa Anthony, PhD– [email protected]
Resources:– Buxton, Bill. (2011). “Some Milestones in Computer Input Devices:
An Informal Timeline.” http://billbuxton.com/inputTimeline.html– Buxton, Bill. (2007‐2011). “Multi‐Touch Systems that I Have Known and Loved.”
http://www.billbuxton.com/multitouchOverview.htmlUIST Vid A hi (1995 2010) htt // / i t/ hi /ht l/ id ht l– UIST Video Archive. (1995‐2010). http://www.acm.org/uist/archive/html/videos.html
References:– Anthony, Lisa & Wobbrock, Jacob O. (2010). "A Lightweight Multistroke Recognizer for User Interface
Prototypes " Proceedings of Graphics Interface (GI '10) Ottawa Ontario (May 31‐June 2 2010)Prototypes. Proceedings of Graphics Interface (GI 10). Ottawa, Ontario (May 31 June 2, 2010). Toronto, Ontario: Canadian Information Processing Society, pp. 245‐252.
– Yang Li. (2009). "Beyond Pinch and Flick: Enriching Mobile Gesture Interaction." IEEE Computer 42, 12 (December 2009), pp. 87‐89.
– Morris, M.R., Wobbrock, J.O. and Wilson, A.D. (2010). "Understanding Users' Preferences for Surface , , , , ( ) gGestures." Proceedings of Graphics Interface (GI '10). Ottawa, Ontario (May 31‐June 2, 2010). Toronto, Ontario: Canadian Information Processing Society, pp. 261‐268.
– Hinckley, K., Yatani, K., Pahud, M., Coddington, N., Rodenhouse, J., Wilson, A., Benko, H. & Buxton, B. (2010). Pen + Touch = New Tools. Proceedings of the 23nd annual ACM symposium on User interface ft d t h l (UIST'10) 27 36software and technology (UIST'10), 27‐36.
Gestural Technologies and Gesture Interaction 39