Nov 06 2000 1

Jason Hong and James LandayUniversity of California Berkeley

Group for User Interface Research

Nov 06 2000 2

Motivation• Sketching is useful in many settings

Nov 06 2000 3

• Creative / Design• SILK• Music Notepad• Knight• Elec. Cocktail

Napkin• Teddy• PatchWork

Survey of Informal Ink Apps• Capture

• ZenPad• NotePals• Dynomite• FiloChat

• Others• Pegasus• XLibris• PerSketch• Kramer's

Patches

• Whiteboard• Tivoli• Flatland

Nov 06 2000 4

Example - Elec. Cocktail Napkin

Gross and Do (UIST96)

Nov 06 2000 5

Example - Flatland

Igarashi, Edwards, LaMarca, and Mynatt (AVI2000)

Nov 06 2000 6

Common Features• Pen input as ink or gesture

• Stroke is pen input from dragging• Ink is a stroke that stays on-screen• Gesture is a stroke that activates a command

• Recognize ink as objects• Transformation / clean-up of ink

• Immediate / deferred processing of ink

erase me

Nov 06 2000 7

Common Features • Contain objects in addition to ink• Selecting and moving objects• Grouping of objects• Layering of objects

Nov 06 2000 8

Related Work• Commercial Software Support

– Newton OS– PenPoint– Microsoft Windows for Pen Computing– Windows CE– Palm OS

• Problems– Form-based and handwriting interfaces– Not easily extensible for building

informal ink apps

Nov 06 2000 9

Related Work• Research Software Support

– ArtKit (Henry et. al. 1990)– Garnet Pen Int. (Landay et. al. 1993)– Patches (Kramer, 1994)– OOPS (Mankoff et. al. 2000)

• Problems– Need more reusable libraries for

handling and processing strokes– Need more extensibility so lots of kinds

of informal ink apps can be built

Nov 06 2000 10

Problems• Software infrastructure support for

informal ink apps rudimentary• Pen-based interaction techniques

difficult to implement• Similar to GUI development in 1980s

– Lots of applications + interaction techniques

– Need cohesive frameworks + toolkits

Nov 06 2000 11

Goals of SATIN• Toolkit support for informal ink apps

– Provide reusable mechanisms for handling + processing strokes

– Separate mechanism from policy– Be extensible for new kinds of apps

• Reusable widgets for pens• Distribute this toolkit for use

Nov 06 2000 12

Overview Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions

Nov 06 2000 13

DENIM

Nov 06 2000 14

SketchySPICE

Nov 06 2000 15

SketchySPICE

Nov 06 2000 16

SketchySPICE

Nov 06 2000 17

Some Metrics

SATIN DENIM Sketchy SPICE

#source files 180 76 7#methods 2192 642 63#classes 220 131 32#lines of code

20000 9000 1000

time 1 person18 months

3 people 3 months

1 person 3 days

Nov 06 2000 18

Overview Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions

Nov 06 2000 19

Architecture Overview

Java Core Classes

Java Virtual Machine

Java2D

SATIN

Swing

ApplicationSATIN relies on Java2D and Swing

Applications use SATIN, can use Java2D and Swing

Nov 06 2000 20

Architecture OverviewSATIN can be broken into twelve

conceptsRecognizers

Interpreters Scenegraph

StrokeLibraries

Widgets

Views

Rendering

Transitions

Clipboard

Notification

Command

Events

Nov 06 2000 21

Recognizers• Problem - many recognition techniques• Recognizers let new recognition

technologies be plugged into SATIN• Given a stroke, return n-best list• No actions taken

• SATIN comes with Rubine's Recognizer (implemented by Chris Long)

Recognizer0.860.650.63

CutCopyUndo

Nov 06 2000 22

Interpreters• Problem – many ways of handling

strokes• Interpreters let new ways of handling

strokes be plugged into SATIN• e.g. straighten a line

• e.g. issue a command

Ink Interpreter

Gesture Interpreter erase me

Nov 06 2000 23

Interpreters and Recognizers• Decoupled recognition (Recognizers)

from actions (Interpreters)• Interpreters can use recognizers

• Both are modules that can be plugged into apps built with SATIN

Interpreter Recognizer

Cut

0.860.650.63

CutCopyUndo

Nov 06 2000 24

Multi-Interpreters• Problem – Hard to combine and extend

existing interpreters• Multi-Interpreters are a collection of

interpreters + dispatch policy– e.g. dispatch to chain of interpreters

– e.g. disable some interpreters on context

DefaultMulti Interpreter Intrp A Intrp Z…

Semantic ZoomMulti Interpreter

Intrp B

Intrp A Intrp Z…Intrp B

Nov 06 2000 25

Phrase(Patch)

Scenegraph

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

…

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Phrase(Patch)

Timed Stroke

Nov 06 2000 26

Stroke Dispatching• Needed an approach that enabled:

– Individual objects handle strokes on own– Priority to gestures over ink

• Default policy is top-down dispatch1. Process stroke with its gesture

interpreter2. Re-dispatch stroke to one of its children3. Process stroke with its ink interpreter4. Handle stroke in object itself

Nov 06 2000 27

Stroke Dispatching

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Nov 06 2000 28

Stroke Dispatching

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

1. Process stroke with gesture interpreterSheet

DefaultMulti Interpreter

Hold Select Intrp

Standard Gesture Intrp…

Nov 06 2000 29

Stroke Dispatching

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Nov 06 2000 30

Stroke Dispatching

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Sheet

DefaultMulti Interpreter

Hold Select Intrp

Standard Gesture Intrp…

1. Process stroke with gesture interpreter

Nov 06 2000 31

Stroke Dispatching2. Re-dispatch stroke to children

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Denim Sketch (Patch)

Nov 06 2000 32

Stroke Dispatching1. Process stroke with gesture interpreter

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Denim Sketch (Patch)

Nov 06 2000 33

Stroke Dispatching2. Re-dispatch stroke to children

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Denim Sketch (Patch)

Nov 06 2000 34

Stroke Dispatching3. Process stroke with ink interpreter

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Denim Sketch (Patch)

Phrase Interpreter

Nov 06 2000 35

Stroke Dispatching4. Handle stroke in object

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Denim Sketch (Patch)

Nov 06 2000 36

Stroke Dispatching

Phrase(Patch)

Sheet

Denim Label (Patch)Denim Sketch (Patch)

Timed Stroke

Nov 06 2000 37

Example - Cut Shallow

Nov 06 2000 38

Example - Cut Deep

Nov 06 2000 39

Reusable Stroke LibrariesStraighten

Merge

Nov 06 2000 40

Reusable Stroke LibrariesSplit

Simplify

Nov 06 2000 41

Widgets • Pie Menus• Pen PLAF

• Swing "Pluggable Look And Feel"• Larger and clickable sliders

• Single-click file opener

Nov 06 2000 42

Conclusions• SATIN - Java toolkit for informal ink

apps– Extensible thru Recognizers, Interpreters,

and Multi-Interpreters– Separates mechanism from policy for

strokes– Offers reusable libraries for manipulating

ink• Reusable widgets for pens

Nov 06 2000 43

Download SATIN

http://guir.berkeley.edu/projects/satin

http://guir.berkeley.edu

Download SATIN at

Nov 06 2000 44

Extra Slides

Nov 06 2000 45

DENIM

Nov 06 2000 46

DENIM

Nov 06 2000 47

Survey of Informal Ink Apps• Elec. Cocktail

Napkin• Flatland• Pegasus• FiloChat• Zenpad• PerSketch• Teddy• PatchWork

• XLibris• NotePals• Dynomite• SILK• Tivoli• Kramer's

Patches• Music Notepad• Knight

Nov 06 2000 48

Architecture Overview

• Recognizers• Interpreters• Scenegraph• Stroke Libraries• Widgets• Views

• Rendering• Transitions• Clipboard• Notifications• Commands• Events

SATIN can be broken into twelve concepts