Flexible Visual Authoring Using Operation HistorySara SuMassachusetts Institute of Technology

lApril 8, 2009

Committee in Charge:Prof. Frédo Durand (MIT, Supervisor)Prof. Maneesh Agrawala (UC Berkeley)Prof. Robert C. Miller (MIT)Dr. Sylvain Paris (Adobe)

Digital authoringPrecise and complex editing

Collaboration, dissemination of content

ExperimentationUndo lowers the cost of mistakes

2

Revisiting historyStoring and retrieving state

Hierarchical authoringGrouping, structure, selections

Operations and selections todayUses of history

System activity logs, instrumentation (not our focus)Operation history, undoVersion controlTutorials

Uses of selections and grouping

3

Uses of selections and groupingEfficient editing of sets of items (multiple selections)Hierarchical modeling, CAD

MotivationAddress limitations of standard techniques

Undo - sequentialSelections - not persistentGrouping – rigid structure expensive to modify

Thesis:

4

Thesis:Reusing operations, selections, and groups from a document’s

history can improve interaction for the end user.

Enhancing authoring and review

Visualizing history for non-linear interactionStoryboards: Interactive Visual Histories

Reusing complex selections for efficiencyHistory-Based Selection Expansion

Enabling bookmarking for flexible groupingSoft Groups: Multiple Selection Authoring and Reuse

5user operations

use

r se

lect

ions

Storyboards

SelectionExpansion

SoftGroups

Thesis contextDemonstrate techniques in context of visual authoring

Features in Inkscape vector graphics editor

Human componentEvaluations with beginner- and intermediate-level usersIterative design

6

Talk outlineInteractive Storyboards

Visualizing history for non-linear interaction

Selection ExpansionReusing complex selections for efficiency

Soft Groups

7

Soft GroupsFlexible authoring of multiple selections

Interactive StoryboardsVisualizing history for non-linear interaction

Selection Expansion

S ft G

8

Soft Groups

Motivation: Visual historiesEnable flexible browsing of history

Design a more intuitive interface to document’s editing historyShow history in spatial context

Enable flexible manipulation of historyInterface to selective undo

9

Related work: Undo

UndoRevisit historyUndo arbitrarily far backSequential

Selective undoText

[Kawasaki and Igarashi 2004]

10

TextSpreadsheetsGraphics

Amulet [Myers et al. 1997]

Related work: Graphical histories

Snapshots Editable graphical histories

11

[Meng et al. 1998][Kurlander and Feiner 1990]

[Agrawala et al. 2003][Goldman et al. 2006]

Film and schematic storyboardsAssembly diagrams

Our storyboard visualizationGraphically represents user editing actions

Assembly instructions for a document

Shows actions in context: action depictionsMust be descriptive, intuitive, and easy to select

12

13

Our storyboard visualizationGraphically represent user editing actions

Show actions in context: action depictions

Design considerationsDiscrete eventsB f d ft

14

Before and afterIn-place visualizationSummarization Figure-ground separation

ApplicationsSelective undo

User selects an action to undoConsider all later actions on the same objectCancel only those that are dependent

Spatial transforms: {translate, rotate}Appearance changes: {fill change, stroke change}Shape modifications: {scale, control point edit}

15

Alice

Bob

Depiction Key

Collaborative editing“Track changes”Asynchronous editing by multiple users

EvaluationGoals

Record users’ impressions after using storyboards for one hourEvaluate selective undo interface

Design12 beginner-level users of 2D drawing programsBackground interview, interactive tutorialR t “t i l” d i

16

Recreate a “typical” drawing

User feedbackStrengths

Free experimentationSpatial memory cuesPersistent history

LimitationsClutter, scalability

Addressing clutterPer-object history

“Magic lens” limits storyboard view

Multi-frame storyboardM lti l f i t b d

17

Multiple frames in a storyboard Multiple actions per frame

Summary: Interactive StoryboardsInteractive storyboards for visualizing history

Browsing history in spatial context

Composite, per-object, and multi-frame storyboardsSelective undo application

Collaborative editing

18

Selection ExpansionReusing complex selections for efficiency

Interactive Storyboards

S ft G

19

Soft Groups

Motivation: Selection reuseMultiple selections are fundamental in editing

Edit the same set of objects togetherReselecting the set can be repetitive, laborious

Esp. with overlapping, occluding objects

GroupsIntuitive, easy to build hierarchy

20

Intuitive, easy to build hierarchyAn item cannot belong to more than one group at a timeUngrouping/regrouping expensive

Related work: Selecting content

Transparency filters Multiblending [Baudisch and Gutwin 2004]Context-aware free-space transparency [Ishak and Feiner 2004]

Physical interaction metaphors“Paper peeling” windows [Beaudoin-Lafon 2001]Exposé [Apple 2003]

21

Exposé [Apple 2003]

Splatter [Ramos et al. 2006]Magic Lens [Bier et al. 1993]

Related work: Complex selections

Generalizing selectionsSelection guessing [Miller and Myers 2002]Selection classifier [Ritter and Basu 2009]Interactive query relaxation [Heer et al. 2008]

22

Related work: Adapting user interfaces

Resize/rearrange menus to reduce target acquisition timeFisheye menus [Bederson 2000]Flexcel [Thomas and Krogsæter 1993]

Dynamically organizing menu items – frequency, recency[Greenberg and Witten 1985][Mitchell and Shneiderman 1989]

23

[Mitchell and Shneiderman 1989]Split menus [Sears and Shneiderman 1994]

Selection expansionHypothesis: Items that have been edited together are likely to be edited

together again.

From an initial selection, expand to a larger set

B th i f f

24

Base the expansion on frequency of use

Greedy expansion strategyUser makes a selection (query)

Look in operation history for single best item to addCandidates = items that have been edited with the query setPick the item appearing most frequently

Expand the selection by one

25

Expand the selection by one

A simple example

Excerpt: Operations affecting {e}:

User’s initial selection is {e}

Compressing the matrix:

26

Query = {e}

Candidate object d:

Candidate object f:

Frequency = 5

Frequency = 2

27

Candidate object g:

q y

Frequency = 4

Query = {e,d}

Candidate object f:

Candidate object g:

Frequency = 2

Frequency = 2

28

q y

Query = {e,d,f}

Candidate object g:

Frequency = 2

Three expansions:

{e} {e,d} {e,d,f} {e,d,f,g}

29

Larger expansion steps

For efficiency, merge steps when we canLook for plateaus in maximum selection frequency

{e} {e,d} {e,d,f} {e,d,f,g}

30

Two expansions:

{e} {e,d} {e,d,f} {e,d,f,g}

selection size = 26 objectsselection size = 25 objectsselection size = 1 object

Implementation: QuickSelect

31

Evaluation of QuickSelectEleven subjects

Recruited from general populationAll familiar with at least one 2D drawing program (not Inkscape)

ApparatusControlled lab settingModified version of Inkscape

32

Two-part study1. Selection reuse with existing histories

Evaluate how QuickSelect affects selection speed and accuracy

2. Selection reuse in free drawingRecord users’ subjective preferences in unconstrained drawing

Study 1: Existing historiesTwo conditions: standard selection, QuickSelect20 tasks: edit existing drawings

Procedure:

33

Hypothesis: QuickSelect will reduce time to complete the trials and reduce number of editing errors.

Results of Study 1

34

Increasing complexity of task

Study 2: Free drawingTry selections in a more realistic setting

ProcedureRecreate “typical” drawing described during interviewUnstructured drawing with prompts to try different selectionsNo measure of success

35

FeedbackEasy to learn and usePerceived improvement in speedPerceived improvement in accuracyStudy 2 more convincing about applicability

ObservationsStrengths of QuickSelect

Performance savings larger for more complex designsRe-selecting occluded contentRe-selecting objects of differing size

36

LimitationsPredictability and error handlingCombining selection toolsAdditional expansion heuristics

Summary: Selection ExpansionReuse of multiple selectionsSimple yet effective history-based strategy

Easy to learn and applySelection reuse can increase efficiency

Greater savings for more complex designs

37

Expansion behavior can be difficult to predict soft groups

Soft Groups

Selection Expansion

Interactive Storyboards

38

Soft GroupsFlexible authoring of multiple selections

Motivation: Flexible groupingGroups

Easy to use, membership in only one group at a time

SelectionsMembership created as needed, ephemeral

Selection expansionReuse selections from history, lacks predictability

39

Related workSelecting, grouping, taggingFlexible grouping - ScanScribe [Saund et al. 2003]Relation building from history [Pedersen and McDonald 2008]

Soft groupsUsers bookmark multiple selections they wish to reuseLike groups, soft groups are persistent and reusable

An item can belong to more than one soft group

Like selections, soft groups appear on demand

40

Retrieval interaction similar to selection expansionExpansion steps determined by user

Create Soft Group

Group creation

41

Create Soft Group

2

Group creation

42

Retrieve Soft Group

Group retrieval

43

Exploratory evaluationGoals

Get user feedback on ease of learning and useCompare soft groups to standard selection and groupingCompare soft groups to selection expansion

Nine beginner- to intermediate-level users of 2D software

44

Procedure:Recreate “typical” drawing described during interviewUnstructured drawing with no measure of successFirst, asked to try soft groupsSecond, introduced to selection expansion

Observations from user studyStrengths of soft groups

Straightforward use, easy to learnSpatial memory cues: “visual reminder”Improves efficiency of authoring

Fixed cost to creating soft groups but faster retrieval

Limitations

45

Error handlingVisibility, responsiveness

Comparison to selection expansionQuickSelect “seems faster” than soft groupsIntermediate users concerned about cost of correcting QSSG offer more control

Summary: Soft GroupsBookmarking selections for reuseComplementary alternative to standard selection and groupingPersistent like standard groupsAppear on demand like standard selections

Easy to learn and use

46

Users preference divided by experienceBeginners: efficiency of selection expansionIntermediate-level users: control of soft groups

Summary of thesis contributionsPresented three uses of history for the end user

Interactive StoryboardsSelection Expansion (QuickSelect)Soft Groups

Demonstrated in the context of vector graphics editing

47

User evaluations suggest increased efficiency and flexibility in editing

Applications and open challengesPrototyping

Selection reuse for faster prototyping and testing of variationsStoryboards lower the cost of experimentation

CollaborationRecorded history for collaborators

EducationStoryboards as tutorials

48

Storyboards as tutorials

Future workOther domainsExpert users

Longer-term observationKeystroke-level modeling

ConclusionsBigger picture: Mining operation history to enhance HCI

Demonstrated history-based techniques for improving authoring and review processes

49

AcknowledgmentsProf. Frédo DurandProf. Maneesh AgrawalaProf. Rob MillerDr. Sylvain Paris

Adobe Systems: Fred Aliaga, Steve Johnson, Craig Scull

50

Inkscape, Open Clip Art LibraryMIT Computer Graphics GroupSu Family

NSF Graduate Research Fellowship ProgramMIT Presidential Fellowship Program