Dialog design

How do we communicate with computers?

Agenda

Exam review Project Part 3 Presentation: Moo Nam Dialogue styles

Command line WIMP Direct Manipulation Speech/Natural Language Pen/Mobile Device

Exam review

Top score: 97 Average: 79 Median: 86

Project Part 3

Due in 3 weeks Choose one design

Go more in depth on details Create testable prototype(s)

Demo for me Create evaluation plan

Discuss in class on 3/30

Dialog Design

How does a user interact with the interface?

Dialog Styles

1. Command languages 2. WIMP - Window, Icon, Menu,

Pointer 3. Direct manipulation

4. Speech/Natural language 5. Pen & PDA

General Issues in Choosing Dialogue Style Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required

Command Languages

Earliest UI interaction paradigm Examples: MS-DOS shell, UNIX, Linux Little or nothing is visible so…

Work primarily by recall, not recognition Heavy memory load

Poor choice for novices but...

CL Advantages

Advantages for experts Speed, conciseness

• % ls (hard to beat)

Can express actions beyond a limited set• Flags, piping one command to another

Repetition, extensibility• Scripting, macros

Easier implementation, less overhead Power

• Abstraction, wild cards

CL Dangers

With added power, comes added responsibility and dangerUNIX

• % rm -r *• Deletes every file that you have, and you

can’t get them back

CL Design Goals

ConsistencyHave options and arguments

expressed the same way everywhere Good naming and abbreviations

UNIX fails here because commands were developed by lots of different people at different organizations

• No guidelines provided

General Issues - CL

Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required

Direct Manipulation

Definition: 1) Continuous visibility of the objects

and actions of interest 2) Rapid, reversible, incremental

actions whose effect is immediately noticeable

3) Replacement of command language syntax by direct manipulation of object of interest (physical actions, buttons, etc.)

Shneiderman ‘82

DM Essence

Representation of reality that can be manipulated

The user is able to apply intellect directly to the task

The tool itself seems to disappear

Related to Directness Engagement

Direct Manipulation

See pgs 171-174 in DFAB Examples

WYSIWYG editors and word processors

VISICALC - 1st electronic spreadsheetCADDesktop metaphorVideo games

Example: Homefinder

DM Issues

Advantages Flexible, easily

reversible actions helps reduce anxiety in users

WYSIWYG provides context & instant visual feedback

Exploits human use of visual spatial cues

Limits types of errors that can be made

Disadvantages visual representation

not self-explanatory Mouse ops may be

slower than typing Not good at:

Repetition History keeping Certain tasks

(Change all italics to bold)

Abstract elements (variables)

General Issues - DM

Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required

WIMP

Windows, Icons, Menus, PointersFocus: Menus, Buttons, Forms

Predominant interface paradigm now (with some direct manipulation added)

Advantages:?

Menus

Advantages: 1 keystroke or mouse operation vs. many No memorization of commands Limited input set

Disadvantages: Less direct user control - have to find correct

menu / menu item Not so readily extensible Slower than keyboarding for experienced

users, at least without accelerators

Menu Items

Organization strategiesCreate groups of logically similar

itemsCover all possibilitiesEnsure that items are non-overlappingKeep wording concise,

understandable

A Good Example

Logical grouping Visual separation of

groups Disabled items

“grayed out” Shortcuts shown … indicates leads to

dialogue

Presentation Sequence

Forms, dialogue boxes, menus Use natural if available

Time• e.g. Breakfast, Lunch, Dinner

Numeric ordering• e.g. Point sizes for font

Other possibilities: Alphabetical Group related items Frequently used first Most important first

Other WIMP issues

Windows managementHow to locate, move, findTransfer information between

IconsNeed graphic design attention

Toolbars

General Issues - WIMP

Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required

Natural input

Universal design Take advantage of familiarity,

existing knowledge Alternative input & output Multi-modal interfaces Getting “off the desktop”

When to Use Speech

Hands busy Mobility required Eyes occupied Conditions preclude use of keyboard Visual impairment Physical limitation

Speech Input

Speaker recognitionTell which person it is (voice print)Monitoring, recording

Speech recognitionIdentify wordsIBM ViaVoice, Dragon Dictate, ...

Natural language understanding

Recognition Dimensions

Speaker dependent/independent Parametric patterns are sensitive to

speaker With training (dependent) can get better

Vocabulary Some have 50,000+ words

Isolated word vs. continuous speech Continuous: where words stop & begin Typically a pattern match, no context

usedDid youvs.Didja

Recognition Example:

My flight experience uh, I started off as a private pilot and then I spent 20 years flying as a navigator on C130s in the air force and I currently fly as a first officer on the SAAB 340 turboprop.

my flight experience a and I started off as a private pilot and I spent 20 years flying is a navigator on C-130s in the Air Force and I currently fly as a first officer on the sound 340 turboprop.

And another example

Here here's one of the

thoughts I'd throw in on

that. I'd agree with

Howard that the

combination of checklists

and flows is the way way

to manage things best.

hair and here's one of the

outside troodon and

an eye to agree with

power that the

domination checklists

and flows is the way

that managed things best

Errors

Systems make four types of errors:Substitution - one for anotherRejection - detected, but not

recognizedInsertion - addedDeletion - not detected

Problems with recovery

Natural Language Understanding

Putting meaning to the words

Input might be spoken or typed

Holy grail of Artificial Intelligence problems

Eliza - a “Faked” Understanding Person : Men are all alike.

ELIZA : In what way?Person : They're always bugging us about something or other.ELIZA : Can you think of a specific example?Person : Well, my boyfriend made me come here.ELIZA : Your boyfriend made you come here?Person : He says I'm depressed much of the time.ELIZA : I am sorry to hear that you are depressed.Person : It's true. I am unhappy.ELIZA : Do you think coming here will help you not to be unhappy?Person : I need some help, that much seems certain.ELIZA : What would it mean to you if you got some help?Person : Perhaps I could learn to get along with my mother.ELIZA : Tell me more about your family

http://www-ai.ijs.si/eliza/eliza.html Weizenbaum, J., "ELIZA -- A computer program for the study of

natural language communication between man and machine", Communications of the ACM 9(1):36-45, 1966

NL Factors/Terms

SyntacticGrammar or structure

ProsodicInflection, stress, pitch, timing

PragmaticSituated context of utterance, location,

time Semantic

Meaning of words

SR/NLU Issues

Advantages Easy to learn and

remember Powerful Fast, efficient (not

always) Little screen real estate

Disadvantages Assumes domain

knowledge Doesn’t work well enough

yet Requires confirmation And recognition will

always be error-prone Expensive to implement Unrealistic expectations

can generate mistrust

Speech Output

Tradeoffs in speed, naturalness and understandability

Male or female voice? Technical issues (freq. response of phone) User preference (depends on the application)

Rate of speech Technically up to 550 wpm! Depends on listener

Synthesized or Pre-recorded? Synthesized: Better coverage, flexibility Recorded: Better quality, acceptance

Speech Output

Synthesis Quality depends on software ($$) Influence of vocabulary and phrase choices http://www.research.att.com/projects/tts/demo.php

Recorded segments Store tones, then put them together The transitions are difficult (e.g., numbers)

Designing Speech Interaction Constrain vocabulary

Limit valid commandsStructure questions wisely (Yes/No)Manage the interactionExamples?

Slow speech rate, but concise phrases Design for failsafe error recovery Visual record of input/output Design for the user – Wizard of Oz

Speech Tools/Toolkits

Java Speech SDK FreeTTS 1.1.1

http://freetts.sourceforge.net/docs/index.php

IBM JavaBeans for speech Microsoft speech SDK (Visual Basic, etc.) OS capabilities (speech recognition and

synthesis built in to OS) (TextEdit) VoiceXML

General Issues – Speech/NL

Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required

Non-speech audio

Good for indicating changes, since we ignore continuous sounds Traditionally used for warnings, alarms or

status information Provides secondary representation

Supports visual interface Provides information that helps reduce error

Tradeoff in using natural (real) sounds vs. synthesized noises.

Non-speech audio examples

Error ding Info beep Email arriving ding Recycle Battery critical Logoff LogonOthers?

Pen & Mobile dialog

Stylus or finger Tradeoffs of each?

Pen as a standard mouse (doubleclick?) Variety of platforms

Desktop touch screens or input pads (Wacom)

Tablet PCs Handheld and Mobile devices Electronic whiteboards

Platforms often involve variety of size and other constraints

Mobile devices

More common as more platforms available PDA Cell phone Ultra mobile tablets

Smaller display (160x160), (320x240) Few buttons, different interactions

Free-form ink Soft keyboard Numeric keyboard => text Stroke recognition Hand printing / writing recognition

http://www.oqo.com/

http://www.blackberry.com/

http://www.intel.com/design/mobile/platform/umpc.htm

Ultra-Mobile PC (Samsung)Palm Z22 handheldhttp://www.palm.com

Soft Keyboards

Common on PDAs and mobile devicesTap on buttons on screen

Soft Keyboard

Presents a small diagram of keyboard You click on buttons/keys with pen QWERTY vs. alphabetical

Tradeoffs?Alternatives?

Numeric Keypad -T9

Tegic Communications developed You press out letters of your word, it matches the

most likely word, then gives optional choices Faster than multiple presses per key Used in mobile phones http://www.t9.com/

Cirrin - Stroke Recognition

Developed by Jen Mankoff (GT -> Berkeley CS Faculty -> CMU CS Faculty)

Word-level unistroke technique UIST ‘98 paper Use stylus to go

from one letterto the next ->

Quikwriting - Stroke Recogntion

Developed by Ken Perlin

Quikwriting Example

p l

http://mrl.nyu.edu/~perlin/demos/Quikwrite2_0.html

e

Said to be as fast as graffiti, but have to learn more

Hand Printing / Writing Recognition Recognizing letters and numbers and

special symbols Lots of systems (commercial too) English, kanji, etc. Not perfect, but people aren’t either!

People - 96% handprinted single characters Computer - >97% is really good

OCR (Optical Character Recognition)

Recognition Issues

Boxed vs. Free-Form input Sometimes encounter boxes on forms

Printed vs. Cursive Cursive is much more difficult

Letters vs. Words Cursive is easier to do in words vs individual

letters, as words create more context Usually requires existence of a dictionary

Real-time vs. off-line

Special Alphabets

Graffiti - Unistroke alphabet on Palm PDAWhat are your

experienceswith Graffiti?

Other alphabets or purposesGestures for commands

Pen Gesture Commands

-Might mean delete

-Insert

-Paragraph

Define a series of (hopefully) simple drawing gesturesthat mean different commands in a system

Pen Use Modes

Often, want a mix of free-form drawing and special commands

How does user switch modes?Mode icon on screenButton on penButton on device

Error Correction

Having to correct errors can slow input tremendously

StrategiesErase and try again (repetition)When uncertain, system shows list of

best guesses (n-best list)Others??

Free-form Ink

Ink is the data, take as is

Human is responsible forunderstanding andinterpretation

Often time-stamped Applications

Signature verification Notetaking Electronic whiteboards Sketching

Electronic whiteboards

Smartboard and Mimio Can integrate with projection Large surface to interact with

Issues?

http://www.smarttech.com/http://www.mimio.com/

Real paper

Anoto digital paper and pen technology (http://www.anoto.com/)

Issues?

http://www.logitech.com/

Logitech io Digital Writing System

General Issues – Pen input

Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required

Other interesting interactions

Gesture inputSpecialized hardware, or tracking

3D interactionStereoscopic displaysVirtual reality

• Immersive displays such as glasses, caves

Augmented realityHead trackers and vision based

tracking

What’s coming up

Upcoming related topicsMultimodal UIs: Ted3D user interfaces: AmyConversational agents: Evan

Next week: evaluation