complex gestures for mobile interaction with dynamic … · complex gestures for mobile interaction...
TRANSCRIPT
LFE Medieninformatik •Diploma Thesis – final report
Wolfgang Reithmeier
Complex Gestures for Mobile Interaction pwith Dynamic NFC-Displays
Betreuer: Dipl. Medieninf. Gregor Broll (LMU/DOCOMO)( )Dr. Matthias Wagner (DOCOMO)
Hochschullehrer: Prof. Hußmann
20. October 2009
Motivation
• Dynamic NFC displays– Grid of ordered NFC tags– Projected dynamic interface
• Comparison with public displays and touchscreens
Direct interaction– Direct interaction– Personalized interaction
• Work on dynamic NFC-display in the context of the MULTITAG-project (DOCOMO Lancaster University)
Wolfgang Reithmeier20. October 2009 2
(DOCOMO, Lancaster University)(Hardy et al., 2008)
Task Descriptionp
• Extension of the project thesis (NFC Display Framework)• Investigation of interaction techniques (e.g. Touch-Select,
Click-Select) and gestures on dynamic NFC-displays• Looking for suitable scenarios e g NFC-BillboardLooking for suitable scenarios, e.g. NFC Billboard• Comparison of interaction techniques and
gestures for different application features– Which interaction techniques and
gestures are technically feasible?– Which interaction techniques and gestures
are useful/intuitive/accepted/… ?– What is the best mapping between
interaction techniques, gestures
Wolfgang Reithmeier20. October 2009 3
q , gand application features?
OverviewOverview• Related Work
• Analysis & Requirements
• Use Case Scenario
• Implementation of Interaction Techniques
• User Study• User Study
• Outlook
Wolfgang Reithmeier20. October 2009 4
Selected Related WorkSelected Related Work
Public Display Interaction
(Wilson, 2004)
Public Display Interaction• TouchLight [Wilson, 2004]
Gesture Techniques• Presense [Rekimoto. 2003]• HoverWidgets [Grossmann, Baudisch. 2006]g [ , ]
Physical Mobile Interaction• Hovering [Välkkynen 2006]
(Rekimoto. 2003)
• Hovering [Välkkynen. 2006]• Marked-up Maps [Reilly. 2004 / 05]
Wolfgang Reithmeier20. October 2009 5
(Välkkynen. 2006)
Analysis & RequirementsAnalysis & RequirementsDesktop Interaction Modalities (comp. WIMP)• (Double ) Click Right Click / Context Menu• (Double-) Click, Right-Click / Context Menu,
Multi Select / Clear, Resize, Mouse Gestures…
Touch & Pen based Interaction&• Some special enhancements / adaptions
• Drag & Drop alternatives (e.g. Pick-and-Drop [Rekimoto, 97]),T & H ld P G t ( h t t k )Tap & Hold, Pen Gesture (characters, strokes, …)
• Mode switching techniques [Li et al., 2005]
NFC t h b d i t ti
(http://optimoz.mozdev.org/gestures/, 2009)
NFC touch based interaction[Reilly 2005, Vetter 2006, Hardy, Rukzio 2008/09]
• Touch Select / Hovering, Click-Select ,Path-Select, …
Wolfgang Reithmeier20. October 2009 6
• Multi-Select / Multi-Selection, Polygon-Select, ..
Use Case Scenario: NFC-BillboardUse Case Scenario: NFC Billboard
• Rich application for different interaction• Rich application for different interaction techniques and gestures
• Adopts concept of paper-based billboarddopts co cept o pape based b boa d• Creation of messages on mobile device
• Posting and retrieval of messages, pictures, etc. b t hi th NFC di lby touching the NFC-display
• Easy browsing of many messages
• Additional functionalities e.g. via toolbarAdditional functionalities e.g. via toolbar
• Use case already applied in practical –positive feedback
Wolfgang Reithmeier20. October 2009 7
Implementation of Interaction Techniques 1/7Implementation of Interaction Techniques 1/7• Techniques used by the Prototype
(Double )Touch Select– (Double-)Touch-Select– Press&Hold, Click-Select– Multiple Selection (indirect)
e g via Bounding-Boxe.g. via Bounding-Box– Single Selection (direct)
Mode Switch + Touch-Select– Drag & DropDrag & Drop
via Pick&Drop, Press&Hold,…– Mode Switch
via Phone-Menu and Toolbar– Gestures
OneStroke, Offset-(Shape)-Gesture– Cancel
Wolfgang Reithmeier20. October 2009 8
via key, gesture or toolbar
Implementation of Interaction Techniques 2/7Implementation of Interaction Techniques 2/7Pinboard Application-specific Features
It• Items• Different types (Messages,
Pictures, Coupons, Folders)
• Items have two states: collapsed / expanded
• Toolbarfilters
Toolbar• Provides features for all items
(views, filters and modes)viewsviews
modes
Wolfgang Reithmeier20. October 2009 9
modes
Implementation of Interaction Techniques 3/7Implementation of Interaction Techniques 3/7Pinboard Application-specific Features
• Views• Views• Three different view:
Overview, Standard-View and Exposé
• Three different options:Toolbar, phone-menu and gestures
Standard view (items collapsed
Overview(all items collapsed)
Exposé( i f d d
Wolfgang Reithmeier20. October 2009 10
(items collapsedor expanded)
(all items collapsed) (overview of expanded, overlapping items)
Implementation of Interaction Techniques 4/7Implementation of Interaction Techniques 4/7Pinboard Application-specific Features
T lti• Tooltip• Shows meta information
on the mobile device
• Collapse / expand items• Show / hide content of the item
Wolfgang Reithmeier20. October 2009 11
Implementation of Interaction Techniques 5/7Implementation of Interaction Techniques 5/7Pinboard Application-specific Features
C t / U l d it• Create / Upload items• Transfer items to the pinboard
• Place (and size) itemsPlace (and size) items
• Download items• Transfer items to the phone
Wolfgang Reithmeier20. October 2009 12
Implementation of Interaction Techniques 6/7Implementation of Interaction Techniques 6/7Pinboard Application-independent Features
Si l S l ti• Single Selection• (De-)Select one item per action
• Multiple Selection• Select multiple items per actionSelect multiple items per action
• Drag & DropDrag & Drop• Moving items around the board
Wolfgang Reithmeier20. October 2009 13
Implementation of Interaction Techniques 7/7Implementation of Interaction Techniques 7/7Pinboard Application-independent Features
C t t M / Ri ht Cli k• Context-Menu / Right-Click• Two different type of menus
(item and empty space)
• Cancel current gesture• Abort all interaction modalitiesAbort all interaction modalities
• Mode-SwitchM d S t i ti f t• Mode: Scope restriction of a gesture,allows reuse of e.g. Touch-Select
Wolfgang Reithmeier20. October 2009 14
User Study 1/6User Study 1/6Setup & Preparation
• Qualitative evaluation• Qualitative evaluation
• 11 subjects (8 students)
• 10 male, 1 female, average age 2710 male, 1 female, average age 27
• Comparison of interaction techniques and gestures for different application features
• Each feature triggered by exchangeable interactions (flexible rules / grammar)
• Fixed order of tasks, randomized order of interactions for each task
• Questionnaires to evaluate specific features
• Analysis of task completion time (logs), errors and attention shifts (video analysis)
Wolfgang Reithmeier20. October 2009 15
analysis)
User Study 2/6 Expand/Collapse & Download (Time)User Study 2/6Results
T h S l t 15
20
25
30
Touch-Select
Double-Touch-SelectClick-Select
P &H ld3
4
5
6
cs)
Touch-Select
Double-Touch-Select
• Touch-Select
• Subjects: Intuitive and fast0
5
10
15 Press&Hold
Toolbar
Phone-Menu
Kontext-menü0
1
2
3
Tim
e (s
ec Click-Select
Press&Hold
Gesture
• Adequate for simple actions
• Double-Touch-Select
-5
menüGeste
-1
2,5
Tooltip & Context (Errors)1Double Touch Select
• Subjects: Uncomfortable1
1,5
2
0,4
0,6
0,8
1
• Users seem to stick onbehavior of Double-Click
error-prone -0,5
0
0,5
-0,4
-0,2
0
0,2
Wolfgang Reithmeier20. October 2009 16
error prone-1
-0,6
User Study 3/6 Expand/Collapse & Context (Attention Shifts)User Study 3/6Results
Cli k S l t 4
5
6
7
2 5
3
3,5
4
4,5
Expand/Collapse & Context (Attention Shifts)
• Click-Select• Subjects: Different opinions
+ User feels in-control 1
2
3
4
0
0,5
1
1,5
2
2,5
- Additional attention shifts
• Press&Hold-1
0
-1
-0,5
Multiple Selection & Context (Time)
• Subjects: Problems with delay
• Not suitable for repetitive interactionslik M lti l S l ti
12
14
16
18
20
cs)
Touch-Select
Double-Touch-Select15
20
25
s)
ToolbarTouch&SelectToolbarPress&HoldHandy-Menülike Multiple Selection
• Prior Knowledge: Context-Menu(2nd place)
2
4
6
8
10
Tim
e (s
ec Click-Select
Press&Hold
Gesture
5
10Tim
e (s
ecs Handy-Menü
Touch-SelectHandy-MenüPress&HoldIndirect Touch-SelectIndirectPress&Hold
Wolfgang Reithmeier20. October 2009 17
0
2
0
User Study 4/6User Study 4/6Results
G t 3
3,5
4
4,5
7
8
9
View (Errors + Time)
• Gestures• Subjects: Very interesting, good alternative
1
1,5
2
2,5
3
3
4
5
6
Tim
e (s
ecs)
Geste
Toolbar
Handy-menü
• Top-ranked for downloading / open context menu
• Short and intuitive gestures preferred-0,5
0
0,5
0
1
2
Drag&Drop & Download (Time)g p
• Context-Menu• Subjects: Additional action not 12
14
16
18
20
)
Pick&Drop 20
25
30
Touch-Select
Double-Touch-SelectClick-Select
g p ( )
• Subjects: Additional action notalways comfortable
• Most suitable for directi t ti lik d l di
4
6
8
10
12
Tim
e (s
ecs) Press&Hold
Kontext-Menü
Touch-Select
0
5
10
15 Press&Hold
Toolbar
Phone-Menu
Kontext-ü
Wolfgang Reithmeier20. October 2009 18
interactions like downloading0
2
-5
0 menüGeste
User Study 5/6 Multiple Selection (Attention Shifts + Time)User Study 5/6Results
• Two different impl of Mode Switch20
25
ToolbarTouch&SelectToolbarPress&Hold
3
4
5
Multiple Selection (Attention Shifts + Time)
• Two different impl. of Mode-Switch• Useful for repetitive tasks
• Allows reuse of e.g. Touch-Select 5
10
15
Tim
e (s
ecs)
Press&HoldHandy-MenüTouch-SelectHandy-MenüPress&HoldIndirect Touch-SelectIndirect
0
1
2
g
• Toolbar• Most adequate method for Mode-Switch
0
Press&Hold
-2
-1
View (Errors)in general
• No need to switch focus
Ph M 2 5
3
3,5
4
4,5View (Errors)
• Phone-Menu• Subjects: Uncomfortable, additional
attention-shifts and complexity 0,5
1
1,5
2
2,5
Wolfgang Reithmeier20. October 2009 19
• More adequate for direct actions like switching views -0,5
0
User Study 6/6User Study 6/6Summary
• Touch-Select preferred for most interactionsTouch Select preferred for most interactions
• Easy to learn and use, fast, familiar and intuitive
• Alternatives needed for different interactions
• Mode switches, especially toolbar, suitable to differ between interactions
• Click-Select, Double-Touch, Press&Hold not well received
M tl d d t d t T h S l t• Mostly redundant compared to Touch-Select
• Often too much for simple interactions
• Only suitable for specific interactions, e.g. explicit triggering of actionsOnly suitable for specific interactions, e.g. explicit triggering of actions
• Mixed results for gestures
• Simple 2-tag-gestures well accepted
Wolfgang Reithmeier20. October 2009 20
• Effort for gestures should not be too great (see view)
OutlookOutlook
T t li fi di f th diff t i t ti• Try to generalize findings of the different interactions(e.g. Fitts’s Law tests)
• Exhaust abilities of the grammar, build new / different type of interactionsExhaust abilities of the grammar, build new / different type of interactions
• Go towards real multi-user environments
S l bl i h d i ( ll l j ti )• Solve problems concerning shadowing (e.g. small-angle projection)
Wolfgang Reithmeier20. October 2009 21
Details on ImplementationDetails on Implementation
• Formalize interactions (create Rules)– Constants: T=Touch, R=Release, Kpress / Krelease = down / up – Min / Max occurrence: 0n
– Sequences: [Const1 | Seq1, …., Constn | Seqn ]– Restrictions [restriction]:
[d < 1000] (delay), [p1=p2 (0,1,2...)] (point equality), [== (0,1,2,...)] (equality), [==] (self-equality), [x, y, w, h]
(area-restriction), [k=code] (key-code equality), [NN (0,1,2,...)] (Neighbouring) or [{1, -1} (0,1,2...)]
(offset)(offset) – (optional) CheckPoints
• Sample: – [T, R][p1=p2][d > 1000] (Press&Hold)– [[[T, R] [p1=p2]] [p1=p2, 0], [[T, R] [p1=p2]] 0*, [[T, R] [p1=p2]] [p1=p2, 0]] (Polygon-Select)
Wolfgang Reithmeier20. October 2009 23
Advantages of dynamic NFC-DisplaysAdvantages of dynamic NFC Displays• Direct interaction instead of remote driven interactions of the most public
displaysdisplays
• More personal than standard touch screens, uses the users’ own device instead of touching with the finger
• Additional advantages coming along with the NFC-technology used (Identification of users, ease of connection-establishment, environmental robustness )robustness, …)
• Having privacy-relevant information on the users’ display
• Combination of dynamic and static displays
• Already existing and supported technology with ongoing distribution
Wolfgang Reithmeier20. October 2009 24