1 jennifer mankoff coc & gvu center georgia tech programming support for natural interaction

1

Jennifer MankoffCoC & GVU Center

Georgia Tech

Programming Support for

Natural Interaction

2

Acknowledgements

Gregory Abowd & Scott Hudson FCE Group & GVU NSF & IBM

3

Natural Interaction

Mouse + Keyboard work great for killer apps of the GUI world

Problems Arise… Off the desktop (Ubiquitous computing) People with disabilities

My interest: Providing general support for interaction design in this space of problems

4

Outline of Talk

Output Augmenting physical objects and

spaces Input

Recognition in the face of errors Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas

Low bandwidth, ambiguous input

5

Output: Augmenting physical objects and spaces PALplates

User study Exploration

Domisilica Dual augmentation General infrastructure

6

PALplates (CHI’97)

Based on study of paper prototype Point of need Dynamic UI

Identity Location

(FX-Pal)

7

Domisilica

Dual Augmentation Augment virtual world with

information about real world state Augment real world with virtual

services, info (recipes, notes) General Infrastructure Example app: Cyberfridge (CVE’98)

8

Contents: an Instant Jello Pudding:French Vanilla an Instant Jello Pudding:French Vanilla a 16oz Ferrara Capellini-Angel Hair a Hamburger Helper: Beef Noodle...

9

Outline of Talk

Output Augmenting physical objects and

spaces General support for exploring new

domains Input

10

Input

Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas

Low Bandwidth, Ambiguous Input

11

Recognition

Recognition is becoming ubiquitous

Recognition is difficult to use A range of interface problems

result OOPS toolkit helps solve them

12

Research Method

Evaluators Study individual problems/solutions Design space of possible solutions

Builders Facilitate solutions to sets of problems

(Architectural / toolkit solutions) Design space exploration by a larger

community

13

Input

Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas

Low Bandwidth, Ambiguous Input

14

Definitions Recognizer

interprets user input creates ambiguity

Error mistake from user’s perspective represented with ambiguity

Mediation dialogue between user and computer used for resolving ambiguity

15

SILK (Landay, 1996)

16

Burlap

17

Design Space of Mediation Techniques

Multimodal Suhm (98) McGee et al

(98) Handwriting

Huerst, Yang & Waibel (98)

Speech Ainsworth &

Pratt (92) Baber & Hone

(93) IUIs

Horvitz (99) Many Others…

18

Design Space of Mediation Techniques

Two major classes Repetition Choice

19

Design Space of Mediation Techniques

Two major classes Repetition Choice

20

Input

Motivation: Alternatives to Keyboard/Mouse

Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas

Low Bandwidth, Ambiguous Input

21

OOPS Toolkit (CHI’00)

Toolkit-level support for handling ambiguity in recognition Library of mediators Architectural support

Based on subArctic

22

Architectural Support

INDEPENDENT of any specific toolkit

Separation of mediators, recognizers, and application

Communication by a common internal model (ambiguous hierarchical events)

Maintains ambiguity indefinitely

23

Related Work

GUI Garnet/interactors (Myers ‘90) X toolkits (Rosenberg et al. ’90)

GUI+Recognition SATIN (Hong et al ’00) Extended Amulet

(Landay & Myers ’93) PPSM (Hudson&Newell ’92) Artkit (Henry et al. ’90)

Context Context Toolkit

(Dey et al ’00) PARCPad, etc

(Schilit et al. ’94) Multimodal

OAA (Cohen et al ’94/Oviatt ’99)

PAC-Amodeus (Nigay& Coutaz ’95)

24

Three key pieces

Ambiguous hierarchical events Mediation subsystem Changes to event dispatch

25

down drag up• • • • • •

s

Ambiguous Hierarchical Events

strokestroke

down drag up• • • • • •

s c

Myers & Kosbie ‘96

26

Mediation Subsystem

Ambiguity is identified automatically Presence of multiple interactor nodes

Hierarchy is passed to mediators Recognizers, recipients informed of

accept/reject decisions Accept/reject modifies hierarchy

Application selects mediators from library

27

med1med2med3med4medn

rec1rec2rec3rec4recn

Event Dispatch

1. A sensed event arrives

eventInput

handler

2. It is dispatched to all recognizers3. It is mediated

•Problem: all recipients must support ambiguity

28

Modified Event Dispatch

1. A sensed event arrives 2a. It is dispatched to ambiguity

unaware components2b. It is dispatched to all recognizers3. It is mediated4. Any accepted leaf nodes are

dispatched to ambiguity unaware components

29

Comparison

stroke stroke

interactors

application

recognizers

application

Input handler

inter

recogs mediators

Input handler

meds

s c

s c

OOPS 1990’s GUI

30

Input

Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas

Low Bandwidth, Ambiguous Input

31

Validation

Doing what’s been done Burlap (complex app) ~15 Mediators (based on design space)

Architecture: 2 Toolkits “direct” hookup in GUI world Context Toolkit extensions (Dey)

Making new things possible: 4 problem areas

32

Problem Areas (UIST ’00)

Errors & Ambiguity rejection errors target ambiguity

Mediation adding alternatives occlusion

33

Target Ambiguity

Problem: There may be multiple targets of a user action

Example: clicking Kabbash (95) Worden (97)

34

Target Ambiguity

Problem: There may be multiple targets of a user action

Example: Clicking Solution: Give the

user a choice of all of the targets

35

Target Ambiguity

Problem: There may be multiple targets of a user action

Example: Clicking Solution: Give the

user a choice of all of the targets

Analysis: Any interface

involving mouse press/release

Requires separation of concerns

Works with all interactors

36

Adding alternatives

Problem: The correct choice isn’t always present

37

Adding alternatives

Problem: The correct choice isn’t always present

Example: word-prediction

38

Adding alternatives

Problem: The correct choice isn’t always present

Example: word-prediction

Solution: Allow the user to add choices

39

Adding alternatives

Problem: The correct choice isn’t always present

Example: word-prediction

Solution: Allow the user to add choices

Analysis: Closely related

choices (e.g. URL prediction)

Requires extensible mediators

Benefits from recognizer API

40

Contributions (Recognition) Resolution of ambiguity in

recognition through mediation Architectural support

(2 toolkits) Exploration of domain of mediators

41

Future Work (Recognition)

Testing Implicit Input (Current) Intelligent Mediation

“Lazy” alternative generation Meta-intelligence

General support for varied inputs

42

Input

Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas

Low Bandwidth, Ambiguous Input

43

Brain-computer interface (Current) Locked-in Syndrome Neural Mouse Control

44

Low-Bandwidth Input

No alternative modalities available Need for efficient error handling

Challenge: interpret brain signal in as rich a form as possible

45

Automatic Conversion for Low Bandwidth Input

Web Browser Example: Automatic URL

extraction Preview Area

Generalization –Next step for natural interaction

46

Conclusions Interaction design for ubiquity, disability

Dual augmentation Ambiguity in recognition Low bandwidth or error-prone input

Exploration of new interaction techniques Four mediation problems Low Bandwidth input

General, re-usable solutions Domisilica OOPS toolkit Extended Context Toolkit

47

Further Information

http://www.cc.gatech.edu/fce/errata/

jmankoff@cc.gatech.edu

48

What about Alternative Explosion?

Fairly independent, discrete interactions

Some stuff stays in recognizers Recognition API helps

Only cache ambiguous events Work to be done…

49

How general is “general”?

Two Toolkits Two complex applications Library of existing mediators Explorations of 4 problem areas

(UIST 00)

50

Is subArctic doing the work here?

No, our minimal requirements are common in today’s toolkits: An event-based toolkit An input-handling module that delivers

events to the appropriate places A library of interactors/widgets Access to source code (OOPS is not just

a library!)

51

Comparing SILK and Burlap

SI LK BURLAP

Modes Four One Combined

Only Draw Mode ThroughoutMediationSeparate Window I ntegrated

Editing Yes No

Storyboarding Yes No

52

Alternative Forms of Input

Cirrin (UIST 98)

53

Alternative Forms of Input

Cirrin (UIST 98) Cerebral Palsy

(Cursor Activity Recognition; Current)

54

Alternative Forms of Input

Cirrin (UIST 98) Cerebral Palsy

(Cursor Activity Recognition; Current)

Recognition

55

Interaction design for ambiguity: Supporting Creativity

Converting a the brain signal to music

Example

Two possible experiences Trying to play a piano with ones feet Having an artistic voice no one else can

reproduce

56

Library of mediators

Design space based on survey Generic and re-usable Three major classes

Repetition Choice Automatic

if (result is “dos”)reject it

elsedo nothing

57

58

Comparison

stroke stroke

interactors

application

recognizers

application

Input handler

inter

recogs mediators

Input handler

meds

s c

s c

OOPS 1990’s GUI