Interaction Devices

Daroko blog:

• Daroko Blog-trending business and technology

Interaction Performance• 60s vs. Today

– Performance• Hz -> GHz

– Memory• k -> GB

– Storage• k -> TB

– Input• punch cards -> • Keyboards, Pens, tablets, mobile

phones, mice, digital cameras, web cams– Output

• 10 character/sec• Megapixel displays, color laser, surround

sound, force feedback, VR

• Substantial bandwidth increase!

Interaction Performance• Future?

– Gestural input– Two-handed input– 3D I/O– Others: voice, wearable, whole

body, eye trackers, data gloves, haptics, force feedback

– Engineering research!– Entire companies created around

one single technology• Current trend:

– Multimodal (using car navigation via buttons or voice)

– Helps disabled (esp. those w/ different levels of disability)

Keyboard and Keypads

• QWERTY keyboards been around for a long time– (1870s – Christopher Sholes)– Cons: Not easy to learn– Pros: Familiarity– Stats:

• Beginners: 1 keystroke per sec• Average office worker: 5

keystrokes (50 wpm)• Experts: 15 keystrokes per sec

(150 wpm)

• Is it possible to do better? Suggestions?

Keyboard and Keypads• Look at the piano for possible

inspiration• Court reporter keyboards (one

keypress = multiple letters or a word) – 300 wpm, requires extensive

training and use• Keyboard properties that matter

– Size • large - imposing for novices,

appears more complex• mobile devices

– Adjustable• Reduces RSI, better performance

and comfort– Mobile phone keyboards,

blackberry devices, etc.

Keyboard Layouts• QWERTY

– Frequently used pairs far apart– Fewer typewriter jams– Electronic approaches don’t jam.. why use

it?• DVOARK (1920s)

– 150 wpm->200 wpm– Reducing errors– Takes about one week to switch– Stops most from trying

• ABCDE – style– Easier for non-typists– Studies show no improvement vs. QWERTY

• Number pads– What’s in the top row? – Look at phones (slight faster), then look at

calculators, keypads• Those for disabled

– Split keyboards– KeyBowl’s orbiTouch (screenshot)– Eyetrackers, mice– Dasher - 2d motion with word prediction

Keys• Current keyboards have been

extensively tested– Size– Shape– Required force– Spacing

• Speed vs. error rates for majority of users

• Distinctive click gives audio feedback– Why membrane keyboards are

slow (Atari 400?)• Environment hazards might

necessitate • Usually speed is not a factor

Keys Guidelines• Special keys should be denoted• State keys (such as caps, etc.)

should have easily noted states• Special curves or dots for home

keys for touch typists• Inverted T Cursor movement keys

are important (though cross is easier for novices)

• Auto-repeat feature– Improves performance, but only if

repeat is customizable (motor impaired, young, old)

• Two thinking points:– Why are home keys fastest to

type?– Why are certain keys larger?

(Enter, Shift, Space bar)• This is called Fitt’s Law

Keypads for small devices• PDAs, Cellphones, Game consoles• Fold out keyboards• Virtual keyboard• Cloth keyboards (ElekSen)• Haptic feedback?• Mobile phones

– Combine static keys with dynamic soft keys– Multi-tap a key to get to a character– Study: Predictive techniques greatly improve

performance– Ex. LetterWise = 20 wpm vs 15 wpm multitap

• Draw keyboard on screen and tap w/ pen– Speed: 20 to 30 wpm (Sears ’93)

• Handwriting recognition (still hard)– Subset: Graffiti2 (uses unistrokes)

Pointing Devices• Direct manipulation needs some pointing device• Factors:

– Size of device– Accuracy– Dimensionality

• Interaction Tasks:– Select – menu selection, from a list– Position – 1D, 2D, 3D (ex. paint)– Orientation – Control orientation or provide direct 3D

orientation input– Path – Multiple poses are recorded

• ex. to draw a line– Quantify – control widgets that affect variables– Text – move text

• Faster w/ less error than keyboard• Two types (Box 9.1)

– Direct control – device is on the screen surface (touchscreen, stylus)

– Indirect control – mouse, trackball, joystick, touchpad

Direct-control pointing• First device – lightpen

– Point to a place on screen and press a button

– Pros: • Easy to understand and use• Very fast for some operations (e.g. drawing)

– Cons: • Hand gets tired fast! • Hand and pen blocks view of screen• Fragile

• Evolved into the touchscreen – Pros: Very robust, no moving parts– Cons: Depending on app, accuracy could

be an issue • 1600x1600 res with acoustic wave

– Must be careful about software design for selection (land-on strategy).• If you don’t show a cursor of where you are

selecting, users get confused– User confidence is improved with a good

lift-off strategy

Direct-control pointing

• Primarily for novice users or large user base

• Case study: Disney World• Need to consider those

who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.

Indirect-Control Pointing• Pros:

– Reduces hand-fatigue – Reduces obscuration problems

• Cons: – Increases cognitive load – Spatial ability comes more into play

• Mouse– Pros:

• Familiarity• Wide availability• Low cost• Easy to use• Accurate

– Cons:• Time to grab mouse• Desk space• Encumbrance (wire), dirt• Long motions aren’t easy or obvious (pick up and replace)

– Consider, weight, size, style, # of buttons, force feedback

Indirect-Control Pointing• Trackball– Pros:

• Small physical footprint• Good for kiosks

• Joystick– Easy to use, lots of buttons– Good for tracking (guide or

follow an on screen object)– Does it map well to your app?

• Touchpoint– Pressure-sensitive ‘nubbin’ on

laptops– Keep fingers on the home

position

Indirect-Control Pointing

• Touchpad– Laptop mouse device– Lack of moving parts, and

low profile– Accuracy, esp. those w/

motor disabilities• Graphics Tablet– Screen shot– comfort– good for cad, artists– Limited data entry

Comparing pointing devices• Direct pointing

– Study: Faster but less accurate than indirect (Haller ’84)• Lots of studies confirm mouse is best for most tasks for speed

and accuracy• Trackpoint < Trackballs & Touchpads < Mouse• Short distances – cursor keys are better• Disabled prefer joysticks and trackballs

– If force application is a problem, then touch sensitive is preferred– Vision impaired have problems with most pointing devices

• Use multimodal approach or customizable cursors• Read Vanderheiden ’04 for a case study

• Designers should smooth out trajectories• Large targets reduce time and frustration

Example• Five fastest places to click on for a right-handed

user?

Example• What affects time?

Fitts’s Law• Paul Fitts (1954) developed a model of human hand

movement• Used to predict time to point at an object• What are the factors to determine the time to point to an

object?– D – distance to target– W – size of target

• Just from your own experience, is this function linear?– No, since if Target A is D distance and Target B is 2D distance,

it doesn’t take twice as long– What about target size? Not linear there either

• MT = a + b log2(D/W + 1)– a = time to start/stop in seconds (empirically measured

per device)– b = inherent speed of the device (empirically measured

per device)– Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm

• Ans: 300 + 200 log2(14/2 + 1) = 900 ms– Really a slope-intercept model

Fitts’s Law• MT = a + b log2(D/W + 1)– a = time to start/stop in seconds (empirically measured per

device)– b = inherent speed of the device (empirically measured per

device)– Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm

• Ans: 300 + 200 log2(14/2 + 1) = 900 ms– Question: If I wanted to half the pointing time (on average), how much

do I change the size?• Proven to provide good timings for most age groups• Newer versions taken into account

– Direction (we are faster horizontally than vertically)– Device weight– Target shape– Arm position (resting or midair)– 2D and 3D (Zhai ’96)

Very Successfully Studied• Applies to

– Feet, eye gaze, head mounted sights– Many types of input devices– Physical environments (underwater!)– User populations (even retarded and drugged)– Drag & Drop and Point & Click

• Limitations– Dimensionality– Software accelerated pointer motion– Training– Trajectory Tasks (Accot-Zhai Steering Law)– Decision Making (Hick’s Law)

• Results (what does it say about)– Buttons and widget size?– Edges?– Popup vs. pull-down menus– Pie vs. Linear menus– iPhone/web pages (real borders) vs. monitor+mouse (virtual borders)

• Interesting readings:– http://particletree.com/features/visualizing-fittss-law/– http://www.asktog.com/columns/022DesignedToGiveFitts.html– http://www.yorku.ca/mack/GI92.html

Precision Pointing Movement Time

• Study: Sears and Shneiderman ’91 – Broke down task into gross and fine components for small targets– PPMT = a + b log2(D/W+1) + c log2(d/W)

• c – speed for short distance movement• d – minor distance

– Notice how the overall time changes with a smaller target.• Other factors

– Age (Pg. 369)• Research: How can we design devices that produce smaller

constants for the predictive equation– Two handed– Zooming

Novel Devices• Themes:

– Make device more diverse• Users• Task

– Improve match between task and device

– Improve affordance– Refine input– Feedback strategies

• Foot controls– Already used in music where

hands might be busy– Cars– Foot mouse was twice as slow as

hand mouse– Could specify ‘modes’

Novel Devices• Eye-tracking

– Accuracy 1-2 degrees– selections are by constant stare

for 200-600 ms– How do you distinguish w/ a

selection and a gaze?– Combine w/ manual input

• Multiple degree of freedom devices– Logitech Spaceball and

SpaceMouse– Ascension Bird– Polhemus Liberty and IsoTrack

Novel Devices• Boom Chameleon

– Pros: Natural, good spatial understanding

– Cons: limited applications, hard to interact (very passive)

• DataGlove– Pinch glove– Gesture recognition– American Sign Language,

musical director– Pros: Natural– Cons: Size, hygiene, accuracy,

durability

Novel Devices• Haptic Feedback

– Why is resistance useful?– SensAble Technology’s Phantom– Cons: limited applications– Sound and vibration are easier and can

be a good approximation• Rumble pack

• Two-Handed input– Different hands have different precision– Non-dominant hand selects fill, the

other selects objects• Ubiquitous Computing and Tangible

User Interface– Active Badges allows you to move

about the house w/ your profile– Which sensors could you use?– Elderly, disabled– Research: Smart House– Myron Kruger – novel user

participation in art (Lots of exhibit art at siggraph)

Novel Devices

• Paper/Whiteboards– Video capture of annotations– Record notes (special tracked pens

Logitech digital pen)

• Handheld Devices– PDA– Universal remote– Help disabled

• Read LCD screens• Rooms in building• Maps

– Interesting body-context-sensitive. • Ex. hold PDA by ear = phone call

answer.

Novel Devices

• Miscellaneous– Shapetape – reports 3D

shape. • Tracks limbs

• Engineer for specific app (like a gun trigger connected to serial port)– Pros: good affordance– Cons: Limited general use,

time

Speech and Auditory Interfaces• There’s the dream• Then there’s reality• Practical apps don’t really require freeform

discussions with a computer– Goals:

• Low cognitive load• Low error rates

• Smaller goals:– Speech Store and Forward (voice mail)– Speech Generation– Currently not too bad, low cost, available

Speech and Auditory Interfaces• Bandwidth is much lower than visual displays• Ephemeral nature of speech (tone, etc.)• Difficulty in parsing/searching (Box 9.2)• Types

– Discrete-word recognition– Continuous speech– Voice information– Speech generation– Non-speech auditory

• If you want to do research here, lots of research in the audio, audio psychology, and DSP field you should understand

Discrete-Word Recognition• Individual words spoken by a specific person• Command and control• 90-98% for 100-10000 word vocabularies• Training

– Speaker speaks the vocabulary– Speaker-independent

• Still requires– Low noise operating environment– Microphones– Vocabulary choice– Clear voice (language disabled are hampered, stressed)– Reduce most questions to very distinct answers (yes/no)

Discrete-Word Recognition• Helps:

– Disabled– Elderly– Cognitive challenged– User is visually distracted– Mobility or space restrictions

• Apps:– Telephone-based info

• Study: much slower for cursor movement than mouse or keyboard (Christian ’00)

• Study: choosing actions (such as drawing actions) improved performance by 21% (Pausch ’91) and word processing (Karl ’93)– However acoustic memory requires high cognitive load (> than hand/eye)

• Toys are successful (dolls, robots). Accuracy isn’t as important• Feedback is difficult

Continuous Speech Recognition• Dictation• Error rates and error repair are still poor• Higher cognitive load, could lower overall quality• Why is it hard?

– Recognize boundaries (normal speech blurs them)– Context sensitivity– “How to wreck a nice beach”

• Much training• Specialized vocabularies (like medical or legal)• Apps:

– Dictate reports, notes, letters– Communication skills practice (virtual patient)– Automatic retrieval/transcription of audio content (like radio, CC)– Security/user ID

Voice Information Systems• Use human voice as a source of info• Apps:

– Tourist info– Museum audio tours– Voice menus (Interactive Voice Response IVR systems)

• Use speech recognition to also cut through menus– If menus are too long, users get frustrated– Cheaper than hiring 24 hr/day reps

• Voice mail systems– Interface isn’t the best

• Get email in your car– Also helps with non-tech savvy like the elderly

• Potentially aides with– Learning (engage more senses)– Cognitive load (hypothesize each sense has a limited ‘bandwidth’)

• Think ER, or fighter jets

Speech Generation

• Play back speech (games)• Combine text (navigation systems)• Careful evaluation!– Speech isn’t always great

• Door is ajar – now just a tone• Use flash• Supermarket scanners

– Often times a simple tone is better– Why? Cognitive load

• Thus cockpits and control rooms need speech• Competes w/ human-human communication

Speech Generation• Ex: Text-to-Speech (TTS)• Latest TTS uses multiple syllabi to make generated speech sound better

– Robotic speech could be desirable to get attention– All depends on app– Thus don’t assume one way is the best, you should user test

• Apps: TTS for blind, JAWS• Web-based voice apps: VoiceXML and SALT (tagged web pages).

– Good for disabled, and also for mobile devices• Use if

– Message is short– Requires dynamic responses– Events in time

• Good when visual displays aren’t that useful. When?– Bad lighting, vibrations (say liftoff)

Non-speech Auditory Interface• Audio tones that provide information• Major Research Area– Sonification – converting information into audio– Audiolization– Auditory Interfaces

• Browsers produced a click when you clicked on a link– Increases confidence– Can do tasks without visual cognitive load– Helps figure out when things are wrong– Greatly helps visually impaired

Non-speech Auditory Interface• Terms:

– Auditory icons – familiar sounds (record real world sound and play it in your app)

– Earcons – new learned sounds (door ajar)

• Role in video games is huge– Emotions, Tension, set mood

• To create 3D sound– Need to do more than stereo– Take into account Head-related

transfer function (HRTF)• Ear and head shape

• New musical instruments– Theremin

• New ways to arrange music

Displays• Primary Source of feedback• Properties:

– Physical Dimension– Resolution– Color Depth and correctness– Brightness, contrast, glare– Power– Refresh rate– Cost– Reliability– # of users

Display Technology

• Monochrome displays (single color)– Low cost– Greater intensity range (medical)

• Color– Raster Scan CRT– LCD – thin, bright– Plasma – very bright, thin– LED – large public displays– Electronic Ink – new product w/

tiny capsules of negative black particles and positive white

– Braille – refreshable cells with dots that rise up

Large Displays• Wall displays– Informational

• Control rooms, military, flight control rooms, emergency response

• Provides– System overview – Increases situational awareness– Effective team review

• Old: Array of CRTs– Interactive

• Require new interaction methods (freehand sketch, PDAs)

• Local and remote collaboration• Art, engineering

Large Displays

• Multiple Desktop Displays– Multiple CRTs or Flat panels for large

desktops– Cheap– Familiar– Spatial divide up tasks– Comparison tasks are easier– Too much info?

• HMD• Eventually -> Every surface a pixel

Mobile device displays• Applications

– Personal• Reprogrammable picture

frames– Digital family portrait

(GaTech)– Business

• PDAs, cellphones– Medical

• Monitor patients– Research: Modality Translation

Services (Trace Center – University of Wisconsin)• As you move about it auto

converts data, info, etc. for you

Mobile device displays

• Actions on mobile devices– Monitor information and alert

(calendar)– Gather then spread out

information (phone)– Participate in groups and relate

to individual (networked devices)

– Locate services and identify objects (GPS car system)

– Capture and then share info (phone)

Mobile device displays• Guidelines for design

– Bergman ’00, Weiss, ’02– Industry led research and design case studies

(Lindholm ’03)– Typically short in time usage (except handheld

games)– Optimize for repetitive tasks (rank functions by

frequency)– Research: new ways to organize large amounts of

info on a small screen– Study: Rapid Serial Visual Presentation (RSVP)

presents text at a constant speed (33% improvement Oquist ’03)

– Searching and web browsing still very poor performance

– Promising: Hierarchical representation (show full document and allow user to select where to zoom into)

Animation, Image, and Video• Content quality has also greatly

increased• 3D rendering is near life-like• Digital Photography is common• Scanned documents• Video compression • Multimedia considerations for the

disabled• Printers

– 3D Printers create custom objects from 3D models