Can you see what I hear? The Design and Evaluation of a Peripheral Sound Display for the Deaf

F. Wai-ling Ho-Ching, Jennifer Mankoff,

James A. Landay

Presented by: Eshita Sharmin

Introduction People use sound in many subtle ways to gain

awareness of the state of the world around them In 1997 there were 3.4 millions Americans with

difficulty hearing, of which 227,000 were deaf Explored the ways in which hearing people use

sound in their everyday lives Conceptual drawings of sound by hearing

participants and exploration with a deaf participant using paper prototypes formed the basis for the design

Categories Visualizing Audio

Employ waveforms and spectrographs as visualizations, and are targeted at expert users.

Not meant to display non-speech sounds

Monitoring and Notification Supports monitoring of peripheral information without grabbing the

users attention Displays lie on the boundary between background and foreground

awareness Must notify the user of interesting information without impeding the

performance of a primary task Requires careful design

Sound Awareness ToolsTechnique Application Characteristics Cons

Vibration sensing Awareness of sounds that create vibration, e.g. sensing footsteps, feeling that a computer is on

Does not require focus of attention Supports some ambient and some notification sounds

Depends on infrastructure(e.g. having hardwood floors).

Flashing lights Awareness of telephones,doorbells (Notification sounds)

Supports notification sounds Fixed visual attentionMust hook up each device

Hearing Dogs Awareness of all sounds Supports notification sounds only Requires ongoing maintenanceRequires a priori training per sound

Visual Inspection Multiple applications, e.g. Steam for a kettle, Looking out the windowfor the arrival of a guest

Sometimes is the only alternativeSome ambient, some notification

Different for each soundPolling rather than interrupt based

Hearing Aids,CochlearImplants

Enhancing existing hearing, but not at the fidelity of “normal” hearing

Enhances awareness of all sounds Requires training for interpretation of soundsResults vary by case

Use of Sound in the Home and at Work

Needs of the Deaf Awareness of the presence of others Interaction with sound based appliances Leaving the home environment

Intuitive Visualization of Sound

Two Prototypes

Experiment- Participants and Apparatus

Procedure and Design

Results (1/2) Correct detection

Results (2/2) Distraction

Measured by counting the rate at which a participant could select 0’s in the primary task.

The performance of primary task was compared across three conditions

Monitoring of the Spectrograph prototype Monitoring of the Ripple prototype Absence of a secondary task

The rings visualization was perceived as less distracting than the Spectrograph

Learning Ripples visualization was easier to learn than Spectrograph

Qualitative Observations Field Study

Deployed the Spectrograph visualization for a week with a deaf graduate student

Findings are Robustness is extremely important Participant was able to detect a number of sounds

Pilot Study “This is great! … I’m learning to hear again after

30 years”

F. Wai-ling Ho-Ching Working as a research intern in the

Adaptive Systems and Interaction group at Microsoft Research (Feb 9, 2003).

Graduated in December 2002 from UC Berkeley with a Masters in Computer Science. During study at Berkeley, she worked on a visual display of sound for the deaf with Jennifer Mankoff and James Landay.

Recieved a Bachelors of Computer Science degree with an English minor at Simon Fraser University.

born and raised in British Columbia, Canada

Publications F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay. (2003) Using

peripheral displays to provide the deaf with awareness of environmental audio. Workshop paper accepted to the Elegant Peripheral Awarness Workshop at CHI 2003. 4 pages.

F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay, (2003). From Data to Display: the Design and Evaluation of a Peripheral Sound Display for the Deaf. In Proceedings of CHI 2003. 8 pages. In press.

Edward Deguzman, F. Wai-ling Ho-Ching, Tara Matthews, Tye Rattenbury, Maribeth Back, Steve Harrison, (2003). EEWWW!: Tangible Instruments for Navigating into the Human Body, In Extended Abstracts of CHI 2003. 2 pages. In press.

Inkpen, K, Ho-Ching, W., Kuederle, O. Scott, S and Shoemaker, G. (1999), This is Fun! We're All Best Friends and We're all Playing": Supporting Children's Synchronous Collaboration. CSCL'99 Proceedings. 252-259.

Ho-Ching, W., Inkpen, K. (1999), Multiple Mice: Realizing Single Display Groupware, Extended Abstract and Interactive Poster Demonstration presented at ASI 1999 (Vancouver, BC), CSCL 2000 (Palo Alto CA).

Jennifer Mankoff EECS Department

University of California, Berkeley

Received Ph.D. as a member of the Future Computing Environments research group in the College of Computing at Georgia Tech

Publications Mankoff, J., Dey, A.K., Hsieh, G., Kientz, J., Ames, M., Lederer, S. Heuristic

evaluation of ambient displays. CHI 2003, ACM Conference on Human Factors in Computing Systems, CHI Letters 5(1): 169-176. 2003.

J. Heer, N. Good, A. Ramirez, M. Davis and J. Mankoff. Presiding Over Accidents: System Mediation of Human Action. In Proceedings of CHI 2004, ACM Conference on Human Factors in Computing Systems, CHI Letters 6(1):To Appear.

Melody Y. Ivory and Jennifer Mankoff and Audrey Le. "Using Automated Tools to Improve Web Site Usage by Users with Diverse Abilities". In Information Technology and Society. 3(1), pp. 195-236.

Jennifer Mankoff and Gary Hsieh and Ho Chak Hung and Sharon Lee and Elizabeth Nitao. "Using Low-Cost Sensing to Support Nutritional Awareness". In Proceedings of Ubicomp 2002. G. Borriello & L.E. Holmquist (Eds.), October, 2002. LNCS 2498. Springer-Verlag. pp. 371-378.

A. Dey, and J. Mankoff, "Designing Mediation for Context-Aware Applications," ACM Transactions on Computer-Human Interaction, special issue on Sensing-Based Interactions. To Appear.

James A. Landay Associate Professor,

Computer Science Division, University of California, Berkeley

Carnegie Mellon University, Pittsburgh, PA (8/90-12/96) Ph.D. in Computer Science, 1996 Thesis: Interactive Sketching for the Early Stages of User Interface Design M.S. in Computer Science, 1993

University of California, Berkeley, CA (8/85-5/90) B.S. in Electrical Engineering/Computer Science with High Honors, 1990

Publications James A. Landay and Brad A. Myers, "Sketching Interfaces: Toward More Human Interface Design." IEEE

Computer, vol. 34, no. 3, March 2001, pp. 56-64. Jason Hong and James A. Landay, "WebQuilt: A Framework for Capturing and Visualizing the Web Experience."

To appear in Proceedings of The Tenth International World Wide Web Conference, Hong Kong, May 2001. Jason Hong and James A. Landay. "A Context / Communication Information Agent." In Personal and Ubiquitous

Computing, Special Issue on Situated Interaction and Context-Aware Computing. 5(1): Springer-Verlag. 2001, pp. 78-81.

Oviatt, S.L., Cohen, P.R., Wu, L.,Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J. & Ferro, D. "Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions," Human Computer Interaction, 2000, vol. 15, no. 4, 263-322 (to be reprinted in Human-Computer Interaction in the New Millennium, ed. by J. Carroll), Reading, MA.: Addison-Wesley, in press).

Hesham M. Kamel and James A. Landay. "A Study of Blind Drawing Practice: Creating Graphical Information Without the Visual Channel." In Assets 2000: Proceedings of the Fourth ACM Conference on Assistive Technologies, Arlington, VA, Nov. 2000, pp. 34-41.

Scott R. Klemmer, Anoop K. Sinha, Jack Chen, James A. Landay, Nadeem Aboobaker, Annie Wang, "SUEDE: A Wizard of Oz Prototyping Tool for Speech User Interfaces." CHI Letters: ACM Symposium on User Interface Software and Technology: UIST '00, 2000. 2(2): p. 1-10.

Jason I. Hong and James A. Landay, "SATIN: A Toolkit for Informal Ink-based Applications." CHI Letters: ACM Symposium on User Interface Software and Technology: UIST '00, 2000. 2(2): p. 63-72.

James Lin, Mark W. Newman, Jason I. Hong, and James A. Landay. "DENIM: Finding a tighter fit between tools and practice for web site design." CHI Letters: Human Factors in Computing Systems, CHI 2000, 2000. 2(1): p. 510-517.

Allan C. Long, James A. Landay, and Lawrence A. Rowe. "Visual Similarity of Pen Gestures." CHI Letters: Human Factors in Computing Systems, CHI 2000, 2000. 2(1): p. 360-367.

"Using Note-Taking Appliances for Student to Student Collaboration." James A. Landay, in Frontiers in Education '99, San Juan, Puerto Rico, November 1999.

?