Direct ManipulationDirect Manipulation

User Interfaces and UsabilityUser Interfaces and Usability

Fall 2009Fall 2009

IntroductionIntroduction

► Interactive systems can Interactive systems can produce reactions that non-produce reactions that non-interactive systems are less interactive systems are less likely to producelikely to produce

► Truly pleased user! They Truly pleased user! They report…report… Master of the interfaceMaster of the interface Competency of task Competency of task

performanceperformance Ease of learning new and Ease of learning new and

advanced featuresadvanced features Confidence of retentionConfidence of retention EnjoymentEnjoyment Eagerness to show to Eagerness to show to

novicesnovices Desire to exploreDesire to explore

Direct Manipulation Direct Manipulation InterfacesInterfaces

► Interfaces that provide:Interfaces that provide: Visibility of objects of Visibility of objects of

interactioninteraction Rapid, reversible actionsRapid, reversible actions Instead of typed commands, Instead of typed commands,

graphic actions, such as graphic actions, such as pointing to the item of interestpointing to the item of interest

► Ex. Drag a file to a trash canEx. Drag a file to a trash can► What reasons is this better What reasons is this better

than ‘rm’?than ‘rm’?► Other areas of direct Other areas of direct

manipulation?manipulation? GamesGames Scientific VizScientific Viz VR/AR (gestures, gloves, VR/AR (gestures, gloves,

tracked devices)tracked devices) 2D/3D what’s the difference?2D/3D what’s the difference?

Direct Manipulation Direct Manipulation ExamplesExamples

► Drive a carDrive a car► If you want to turn If you want to turn

left, what do you do?left, what do you do?► What type of What type of

feedback do you feedback do you get?get?

► How does this help?How does this help?► Think about turning Think about turning

left using a left using a menu/text interfacesmenu/text interfaces

Command-line vs. Display Command-line vs. Display Editors vs. Word ProcessorsEditors vs. Word Processors

► Case Study: Word Case Study: Word Processors:Processors: Early 80s, only saw 1 line at Early 80s, only saw 1 line at

a timea time Editing was difficultEditing was difficult No global perspectiveNo global perspective

► Full-page Display EditorsFull-page Display Editors 2D cursor control2D cursor control Ex. WORDSTAR, emacsEx. WORDSTAR, emacs

► Researchers found:Researchers found: Increased performanceIncreased performance Decreased frustrationDecreased frustration Improved trainingImproved training

► What would be easier with What would be easier with command-line?command-line?

Command-line vs. Display Command-line vs. Display Editors vs. Word ProcessorsEditors vs. Word Processors

► Early 90s: What You See Is Early 90s: What You See Is What You Get (What You Get (WYSIWYGWYSIWYG)) Word, Corel’s WordPerfect, Word, Corel’s WordPerfect,

Lotus Word ProLotus Word Pro See a full page of textSee a full page of text Seen as it will appearSeen as it will appear Cursor action is visible Cursor action is visible

(attention focus)(attention focus) Cursor motion is natural Cursor motion is natural

(arrow/mouse vs. ‘Up 6’ – (arrow/mouse vs. ‘Up 6’ – requires converting)requires converting)

Labeled icons make frequent Labeled icons make frequent actions rapid (remind users of actions rapid (remind users of possible actions)possible actions)

Immediate display of the Immediate display of the results of an actionresults of an action

Rapid Response and Display Rapid Response and Display (sense of power)(sense of power)

Reversible Actions (lowers Reversible Actions (lowers anxiety)anxiety)

Technical Results from Empirical Technical Results from Empirical Studies of Word ProcessorsStudies of Word Processors

► Integration of multimodal Integration of multimodal information – graphics, information – graphics, sound, animation, data, sound, animation, data, photosphotos

► Desktop-publishing Desktop-publishing softwaresoftware

► Presentation softwarePresentation software► Hypermedia environments Hypermedia environments

and the WWWand the WWW► Improved macro/templates Improved macro/templates

facilitiesfacilities► Spell/grammar checkers & Spell/grammar checkers &

thesaurithesauri► Document AssemblersDocument Assemblers

VisiCalc SpreadsheetVisiCalc Spreadsheet

► 1979 – Dan 1979 – Dan Brickland (254 rows, Brickland (254 rows, 63 columns)63 columns)

►Direct ManipulationsDirect Manipulations► Users likeUsers like

Auto propagation of Auto propagation of their actionstheir actions

Alternate plansAlternate plans MacrosMacros

►Others:Others: Lotus 1-2-3, ExcelLotus 1-2-3, Excel

Spatial Data ManagementSpatial Data Management

► Geographical data Geographical data visualization and visualization and interactioninteraction

► Direct Manipulations Direct Manipulations Notion of using a joystick Notion of using a joystick

to navigate a map:to navigate a map: Idea: Nicholas Idea: Nicholas

Negroponte (MIT)Negroponte (MIT) App: Spatial Data App: Spatial Data

Management System Management System (’80)(’80)

Zoom-in on ocean map Zoom-in on ocean map and ship convoy markersand ship convoy markers

Spatial Data ManagementSpatial Data Management

► Others: Others: Xerox PARC Information Xerox PARC Information

VisualizerVisualizer► WalkthroughWalkthrough► File directories, org charts, File directories, org charts,

2d info2d info ArcGIS – Current map ArcGIS – Current map

viewer pg. 183viewer pg. 183 Google EarthGoogle Earth

► Success: Designer is very Success: Designer is very important! important! Icons, representations, Icons, representations,

understanding user needs. understanding user needs. Users typically enjoy the Users typically enjoy the

direct manipulationdirect manipulation

What is the most successful app What is the most successful app of Direct Manipulation?of Direct Manipulation?

► Video GamesVideo Games► PONGPONG

Low learning curveLow learning curve Mass appeal (which many Mass appeal (which many

current games don’t current games don’t have!)have!)

Let’s list a whole bunch of Let’s list a whole bunch of the most popular gamesthe most popular games

What are some What are some commonalities?commonalities?

► Direct ManipulationsDirect Manipulations Let’s list themLet’s list them

http://www.gamegum.com/game/663/pong/

http://www.engadget.com/2009/10/05/microsoft-research-

shows-off-multitouch-mouse-prototypes/

Video GamesVideo Games► Think about designing for Think about designing for

different platformsdifferent platforms AgeAge GenderGender PortabilityPortability Resolution/Computing PowerResolution/Computing Power GenreGenre MultiplayerMultiplayer CulturesCultures

► Different controllersDifferent controllers► The effect of having a score The effect of having a score

(public display, compare w/ (public display, compare w/ friends, competition, better friends, competition, better than encouragement)than encouragement)

► Direct manipulation for Direct manipulation for educationeducation SimCitySimCity The SimsThe Sims

Computer Computer Aided-DesignAided-Design

► Extensively uses Direct Extensively uses Direct ManipulationManipulation

► AutoCADAutoCAD► Structural engineer, Structural engineer,

landscaping, automobiles, landscaping, automobiles, etc.etc.

► Change design and evaluate Change design and evaluate designs quicklydesigns quickly

► Computer Aided Computer Aided Manufacturing (CAM)Manufacturing (CAM)

► Allows many of the Allows many of the specification tools to be used specification tools to be used for large designs (group for large designs (group review, etc.)review, etc.)

► Few complex commandsFew complex commands► Analogy/familiar designs Analogy/familiar designs

important (don’t change the important (don’t change the terminology, etc.)terminology, etc.)

Office AutomationOffice Automation► Xerox Star (1981)Xerox Star (1981)► Apple Lisa (1983) (precursor to Apple Lisa (1983) (precursor to

the Mac) the Mac) ► Direct manipulationDirect manipulation

Drag file to printerDrag file to printer Pull-down menusPull-down menus Window manipulationWindow manipulation

► Microsoft WindowsMicrosoft Windows► Command-line vs. GUICommand-line vs. GUI

Study result: Study result: task time (5.8 task time (5.8 vs. 4.8 minutes), errors (2.0 vs. vs. 4.8 minutes), errors (2.0 vs. 0.8) (’87)0.8) (’87)

Subjectively preferredSubjectively preferred novice/ computer naïve people novice/ computer naïve people

really benefitreally benefit Improved productivity, reduced Improved productivity, reduced

fatiguefatigue

Evolution of Evolution of Direct Direct

ManipulationManipulation► To create a good Direct To create a good Direct

Manipulation interfaceManipulation interface Model reality wellModel reality well Visual interface if Visual interface if

possiblepossible Know your usersKnow your users

► Personal Finance Personal Finance (Quicken)(Quicken)

► Aesthetic ComputingAesthetic Computing► Home automationHome automation► Robot programming Robot programming

(guide robots hand)(guide robots hand)

Evolution of Direct Evolution of Direct ManipulationManipulation

► Future:Future: VR/ARVR/AR Ubiquitous computingUbiquitous computing Wearable computingWearable computing Tangible interfacesTangible interfaces

► Goals: Goals: ComprehensiveComprehensive Rapid learningRapid learning Predictable actionsPredictable actions Appropriate feedbackAppropriate feedback

► Results:Results: RetentionRetention LearningLearning Lowered anxietyLowered anxiety Users feel empowered and Users feel empowered and

satisfiedsatisfied

Thoughts on Direct Thoughts on Direct ManipulationManipulation

► Principle of virtuality Principle of virtuality – users enjoy being able to – users enjoy being able to manipulate some version of reality (Nelson ’80)manipulate some version of reality (Nelson ’80)

► Principle of transparencyPrinciple of transparency – UI disappears and – UI disappears and allows user to apply intellect to task (Rutokwsiki allows user to apply intellect to task (Rutokwsiki ’82)’82)

► Logical thinking (which engineers are good at) Logical thinking (which engineers are good at) doesn’t always lead to good design (Heckel ’91) doesn’t always lead to good design (Heckel ’91)

►Gulf of execution Gulf of execution and and gulf of evaluationgulf of evaluation (Hutchins, Holland, and Don Norman ’86)(Hutchins, Holland, and Don Norman ’86)

► Related to psychology literature on problem-Related to psychology literature on problem-solving and learning researchsolving and learning research Ex. Use beads to teach math (better than abstract Ex. Use beads to teach math (better than abstract

terms)terms) Why people like the abacus over calc, esp. for teachingWhy people like the abacus over calc, esp. for teaching

Direct Manipulation problemsDirect Manipulation problems► Blind / Vision-Impaired Blind / Vision-Impaired - If - If

you develop you develop forfor a visual a visual interface, this group might interface, this group might be left out. Newer be left out. Newer technologies help.technologies help.

► ScreenspaceScreenspace Takes up plentyTakes up plenty Possible ‘abuse’Possible ‘abuse’ Multiple pages can slow Multiple pages can slow

user downuser down Bad design is Bad design is amplifiedamplified

► Detail Detail can be lost (graphs can be lost (graphs vs. tables)vs. tables)

► Learning curve Learning curve – users – users must learn meaning of must learn meaning of icons, etc. Different for icons, etc. Different for novice vs. experienced usersnovice vs. experienced users

Direct Manipulation problemsDirect Manipulation problems► Wrong conclusions Wrong conclusions – –

graphsgraphs► Slow Slow for fast typists for fast typists

(moving hand to mouse is (moving hand to mouse is relatively slow)relatively slow)

► Poor for Poor for some notations some notations (e.g. math)(e.g. math)

► Choosing Choosing the right the right icons/metaphors is difficulticons/metaphors is difficult

► Requires:Requires: Fast turnaround time Fast turnaround time

(100ms or less)(100ms or less) Reversibility (undo)Reversibility (undo) Both can be hard to codeBoth can be hard to code Difficult to do w/ HTML Difficult to do w/ HTML

(better w/ Java or Flash)(better w/ Java or Flash)

Direct Direct ManipulationManipulation

► Advantages Advantages Continuous visual representation Continuous visual representation

of objects and actions of interestof objects and actions of interest Physical actions instead of syntaxPhysical actions instead of syntax Rapid, incremental, and Rapid, incremental, and

reversible actions whose results reversible actions whose results are visible immediatelyare visible immediately

► Systems with Direct Manipulation Systems with Direct Manipulation usuallyusually have the following: have the following: Novices can learn basic Novices can learn basic

functionality quicklyfunctionality quickly Experts can work quickly to carry Experts can work quickly to carry

out a wide range of tasksout a wide range of tasks Intermittent users can retain Intermittent users can retain

conceptsconcepts Error messages are rarely neededError messages are rarely needed Immediate feedback if actions Immediate feedback if actions

furthered or hampered goalsfurthered or hampered goals Less anxiety due to Less anxiety due to

comprehension and reversibilitycomprehension and reversibility Gain confidence because users Gain confidence because users

initiate initiate action, feel in control, action, feel in control, and can predict outcomesand can predict outcomes

Object Action Interface Object Action Interface approach to Direct Manipulationapproach to Direct Manipulation► Ex: Ex: organizing digital organizing digital

photos, stock portfoliosphotos, stock portfolios► What are the objects? What are the objects? ► What are the actions? What are the actions? ► What is the interface?What is the interface?► Objects and actions are Objects and actions are

displayed close togetherdisplayed close together► Little need to break down Little need to break down

into complex syntaxinto complex syntax► Result: Result: Closeness of task Closeness of task

domain to the interface domain to the interface domain reduces cognitive domain reduces cognitive load and stress (stimulus-load and stress (stimulus-response compatibility in response compatibility in Human Factors research)Human Factors research)

OAI and DMOAI and DM► Actions are icons are more Actions are icons are more

‘natural’ (developed earlier) ‘natural’ (developed earlier) than languagethan language

► 7 to 11 yr old, can handle 7 to 11 yr old, can handle the DM approach (physical the DM approach (physical actions on an object)actions on an object) Concepts of conservation Concepts of conservation

and invarianceand invariance► 11+ is for formal operations 11+ is for formal operations

(symbol manipulation)(symbol manipulation) Math, programming, Math, programming,

languageslanguages Children often link early Children often link early

math, etc. to objectsmath, etc. to objects► Easier not only for kids but Easier not only for kids but

for everyone (Yet another for everyone (Yet another example!)example!)

Visual Thinking and IconsVisual Thinking and Icons► Visual Languages and Visual Visual Languages and Visual

Thinking (Arnheim ’72)Thinking (Arnheim ’72) Data viz and symbol people Data viz and symbol people

Reaches out to the right-Reaches out to the right-brained (look at all the users)brained (look at all the users)

Shunned by many a left-brainedShunned by many a left-brained► Read a paper by an Read a paper by an

algorithm/theory person lately?algorithm/theory person lately?► WIMP (WIMP (WWindows, indows, IIcons, cons, MMouse, ouse,

and and PPulldown-menu) interfaces ulldown-menu) interfaces have that nickname for a reasonhave that nickname for a reason

► No one styleNo one style People think differentlyPeople think differently Should provide several if Should provide several if

possiblepossible► Depend on expected user baseDepend on expected user base

Paint program (icons) vs. word Paint program (icons) vs. word processors (text menus)processors (text menus)

When should you use it? When should you use it? Ex. Road signs (left curve vs. Ex. Road signs (left curve vs.

bridge out vs stop). What bridge out vs stop). What factors play a part?factors play a part?

Icon Design ConsiderationsIcon Design Considerations► Stand out from Stand out from

background and each background and each otherother

► Limit the numberLimit the number► 3D not necessarily good3D not necessarily good► Familiarity (ex.)Familiarity (ex.)► Selected icons should be Selected icons should be

easily foundeasily found► Animations, shadows, etc. Animations, shadows, etc.

helphelp► Dynamic icons (size Dynamic icons (size

changes, thumbnails, changes, thumbnails, etc.)etc.)

► Interaction between iconsInteraction between icons

Icon Design ConsiderationsIcon Design Considerations► Components of icons:Components of icons:

Lexical – brightness, color, Lexical – brightness, color, blinking etc.blinking etc.

Syntatics – appearance and Syntatics – appearance and movements (lines, shape)movements (lines, shape)

Semantics – object Semantics – object representedrepresented

Pragmatics – legibility, Pragmatics – legibility, utilityutility

Dynamics – receptivity to Dynamics – receptivity to actionsactions

► Adding multimodal or subtle Adding multimodal or subtle affects helps users detect affects helps users detect anomaliesanomalies Phone dialingPhone dialing Hypothesis: Directories Hypothesis: Directories

played a song when openedplayed a song when opened

Direct Manipulation Direct Manipulation ProgrammingProgramming

► Instead of just affecting Instead of just affecting a simulation/system a simulation/system with DM, how about with DM, how about programming with it?programming with it?

► Alice, AVS, Car making Alice, AVS, Car making robotsrobots

► Other examples of Other examples of programming with DM?programming with DM? Car radio presetsCar radio presets Movie camera tracksMovie camera tracks MacrosMacros

► Systems observe the Systems observe the user and can replicate user and can replicate actions (chess)actions (chess)

Direct Manipulation Direct Manipulation ProgrammingProgramming

► PITUI – programming in the user PITUI – programming in the user interfaceinterface Sufficient generalitySufficient generality Access to data structures and Access to data structures and

operatorsoperators Ease in programming and editingEase in programming and editing Simplicity in execution and Simplicity in execution and

supplying argumentssupplying arguments Low-risk (low errors, reversibility, Low-risk (low errors, reversibility,

etc.)etc.)► Cognitive-dimensions framework Cognitive-dimensions framework

(Green and Petre ’96)(Green and Petre ’96) Analyzes design issuesAnalyzes design issues Viscosity – difficulty in making Viscosity – difficulty in making

changeschanges Progress evaluation – execute Progress evaluation – execute

partial programspartial programs Consistency, hidden dependences, Consistency, hidden dependences,

visbility, etc.visbility, etc.► Doesn’t try to guess user’s Doesn’t try to guess user’s

intentions, like Agentsintentions, like Agents

3D Interfaces3D Interfaces► We live in a 3D worldWe live in a 3D world► Natural interfaces are betterNatural interfaces are better► Therefore 3D interfaces will be Therefore 3D interfaces will be

the ultimatethe ultimate► What’s wrong with the above?What’s wrong with the above?

Natural interfaces aren’t always Natural interfaces aren’t always better!better!

Making the interface simple Making the interface simple (thus unnatural) often aids (thus unnatural) often aids performanceperformance► Constrains movementConstrains movement► Limiting possible actionsLimiting possible actions

Depends on application and Depends on application and goal of the user interfacegoal of the user interface► Surgery simulationSurgery simulation► Military simulation (general vs. Military simulation (general vs.

soldier training)soldier training)► Architecture, education, Architecture, education,

product designproduct design► Video gamesVideo games

3D Interfaces3D Interfaces► What we really want are What we really want are

enhancedenhanced interfaces interfaces► Give us powers we don’t Give us powers we don’t

normally havenormally have Flying, x-ray vision, Flying, x-ray vision,

teleportation, undo, etc.teleportation, undo, etc.► Be careful we don’t Be careful we don’t

become overzealousbecome overzealous Air traffic control 3D displayAir traffic control 3D display Library interfaces using Library interfaces using

books on shelves (what is it books on shelves (what is it good for? What is it poor good for? What is it poor for?)for?)

► Hurts performanceHurts performance Study results: Study results: 3D Bar 3D Bar

charts don’t help charts don’t help ► So what is helped by 3D?So what is helped by 3D?

Good 3DGood 3D► Social interfaces + 3D Social interfaces + 3D

can be very powerfulcan be very powerful MMORPG (EverQuest)MMORPG (EverQuest) ActivedWorldsActivedWorlds The Sims OnlineThe Sims Online

► ExperiencesExperiences Art gallaryArt gallary 3D Desktops (Mac’s latest)3D Desktops (Mac’s latest) Office metaphors did not Office metaphors did not

take off (BOB, Task take off (BOB, Task Gallary)Gallary)

3D Webbrowsing. Sure 3D Webbrowsing. Sure you can arrange 16 web you can arrange 16 web pages spatially, but why?pages spatially, but why?

► Compromises to provide Compromises to provide 3D interfaces might be 3D interfaces might be undermine usabilityundermine usability Think RTS gamesThink RTS games

► Discussion: Is the Discussion: Is the interface holding back interface holding back 3D?3D?

3D Interfaces3D Interfaces► Use occlusion, shadows, perspective carefullyUse occlusion, shadows, perspective carefully

Improves use of spatial memory (Ark ’98)Improves use of spatial memory (Ark ’98) Distracting and confusingDistracting and confusing

► Minimize navigation stepsMinimize navigation steps► Keep text readable (good contrast, 30 degree Keep text readable (good contrast, 30 degree

tilt max)tilt max)► Simple user movement (why lock to a floor?) Simple user movement (why lock to a floor?)

Descent vs QuakeDescent vs Quake► Prevent Errors (put in guides to help)Prevent Errors (put in guides to help)► Simplify object movement (connecting two Simplify object movement (connecting two

parts, for example, can be abstracted… most parts, for example, can be abstracted… most of the time)of the time)

► Organize groups of items into alignments Organize groups of items into alignments that facilitate visual search and recall (allow that facilitate visual search and recall (allow user choice)user choice)

3D Interface Development3D Interface Development► Developments that show promise:Developments that show promise:

3D sound3D sound Stereo display (Ware and Frank ’96)Stereo display (Ware and Frank ’96) Haptic feedback (mouse)Haptic feedback (mouse)

► 3D can help by:3D can help by: Provide overviews to see big pictureProvide overviews to see big picture Rapid teleportation (context shifts)Rapid teleportation (context shifts) Zooming (aid disabled)Zooming (aid disabled) Multiple coordinated views (3dsmax)Multiple coordinated views (3dsmax) 3D icons can represent abstract or recognizable concepts3D icons can represent abstract or recognizable concepts

► Homework: Homework: Find a UI to accomplish a 3D task. Describe the Find a UI to accomplish a 3D task. Describe the system, explain how DM is applied. system, explain how DM is applied. ( email me - 2 paragraphs max before Class on Tuesday)( email me - 2 paragraphs max before Class on Tuesday) Include a list of objects you can interact withInclude a list of objects you can interact with How it provides a global perspectiveHow it provides a global perspective Feedback mechanismFeedback mechanism Interaction mechanism (what does the user do to interact)Interaction mechanism (what does the user do to interact) How well it does/does not accomplish taskHow well it does/does not accomplish task

TeleoperationTeleoperation► Combines:Combines:

Direct ManipulationDirect Manipulation Process ControlProcess Control

► Human operators control physical Human operators control physical processes in complex environmentsprocesses in complex environments

► Example applications: Example applications: Mars rover Mars rover control, flying airplanes (Predator), control, flying airplanes (Predator), manufacturing, medicine (surgery)manufacturing, medicine (surgery)

► Supervisory control Supervisory control (Sheridan ’92)(Sheridan ’92) Different levels of human control Different levels of human control

(automation)(automation)► Direct Manipulation IssuesDirect Manipulation Issues

Adequate feedback (data quality, Adequate feedback (data quality, latency (transmission and operation latency (transmission and operation delays), incomplete, interference)delays), incomplete, interference)

PresencePresence Point and click or more natural Point and click or more natural

interaction vs. typinginteraction vs. typing► Example project: Nanomanipulator Example project: Nanomanipulator

(show video)(show video)

VR InteractionVR Interaction

► Trying to simulate reality or an experienceTrying to simulate reality or an experience Training, Learning, ExploringTraining, Learning, Exploring ExpensiveExpensive DangerousDangerous Logistically DifficultLogistically Difficult

► Best interaction?Best interaction? Flight simulators (they can cost $100 mil, but Flight simulators (they can cost $100 mil, but

that’s still a good deal!)that’s still a good deal!) Why?Why?

►Why do video game flight sims not cut it? (only $40!)Why do video game flight sims not cut it? (only $40!)►Okay, we have monitors that show 3D Okay, we have monitors that show 3D

worlds, what else do we possibly need?worlds, what else do we possibly need?