The Audio User Experience

for Widgets

Todd Kloots@todd

Goals

1. Understand what users of screen readers expect or need to hear when interacting with widgets

2. Understand the common keyboard and touch interaction patterns for widgets

Screen Readers

Read the code (HTML)

Read the focused element

What is read:

Element/Control typeLabelPropertiesState changes

For web-based interfaces most of the information that screen readers announce originates from the code—the HTML.

Navigation

Some fundamentals. First, how do users of screen readers navigate through a page or application?

Navigating web pages or applications using a screen reader is very much the opposite of the experience for sighted users.

As a sighted person, when you first see a web page or application you see the gestalt.

Then you can focus your eyes on the task you want to accomplish, often ignoring the other parts of the page or application.

For example, if you want to search, you focus your eyes on the search box and ignore the rest of the page.

Or if you want to read a section of a page, you move your focus to that section.

On the other hand, users of screen readers begin with a blank canvas and discover the page or application one element or section at a time. It is not until they traverse through a section that they can begin to get a sense of the whole.

Imagine navigating through yahoo.com using a screen reader. You’d first encounter the Yahoo! logo.

Then the first of the search tabs at the top of the page: “Web”

Then “Images”

Then “Video”

Then “Travel”

Then “News”

Then the search box.

And finally the “Search” button.

And after navigating through those elements you might begin to perceive them as belonging to a section.

Widgets

That's navigation. Now, how do users of screen readers interact with wigets?

Demo

http://youtu.be/ZnlvBIfTQfI

Demo of interacting with a checkbox using VoiceOver—the screen reader for the Mac. Notice how VoiceOver announces the control type (“checkbox”), the label “Send me offers”, and the current state (“checked” and “not checked”).

All this text is announced to ensure that users of screen readers can perceive and operate the widget.

See Hear

That's the experience for existing widgets. For custom controls, most often problems are caused when there is a gap between the visual and audio user experience such that users of screen readers do not have access to the same information as sighted users.

http://youtu.be/lpU8IHiU8rw

This short video illustrates the screen reader user experience of a custom toolbar that doesn't use ARIA and whose buttons aren't properly labeled. Notice how the user is unaware of the fact that they are interacting with a toolbar, and how each button is indistinguishable for one another.

This demonstrates a common problem accessibility problem for custom controls built using JavaScript: a perception gap between the visual and audio user experience.

Visual rendering of the screen reader user experience when contextual information and labels are not available.

http://youtu.be/3AdvVg2Cs9Q

This short video illustrates the user experience of a custom toolbar that uses ARIA to improve the user experience for users of screen readers. Notice how there is no gap in what sighted users perceive and what the screen reader announces; the screen reader announces to the user they are in a toolbar and the label of each button.

This is made possible by ARIA—it provides developers with the necessary tools to close gaps between the visual and audio user experience.

http://youtu.be/WK6qN4OCwVU

This short video illustrates the screen reader user experience of a custom modal dialog that doesn't use ARIA. Notice how when the dialog is made visible the only information that is announced is the label of the "OK" button. Other contextual information such as the dialog's title, and message are not announced.

This demonstrates a common accessibility problem for custom controls built using JavaScript: a perception gap between the visual and audio user experience.

http://youtu.be/XqEwuV2HySs

This short video illustrates the user experience of a custom modal dialog that uses ARIA to improve the user experience for users of screen readers. Notice how when the dialog is made visible the same information visible to sighted users is also announced by the screen reader. This is made possible by ARIA—it provides developers with the necessary tools to close gaps between the visual and audio user experience.

Screen readers are available on a variety of devices including: desktop, phones and tablets.

As for mobile: Screen readers are available for both iOS and Android.

Configuring the Mac

Every Mac has a built-in screen reader called VoiceOver.

Cmd + F5(or Fn + Cmd + F5)

Start/Stop VoiceOver

To change the speech rate (how fast VoiceOver speaks)• Press Ctrl + Option + Command + → until you see/hear "Rate"• Press Ctrl + Option + Command + ↓ to make VoiceOver speak slower• Press Ctrl + Option + Command + ↑ to make VoiceOver speak faster

Speech Rate

Press Ctrl + Option + Command + → until you see/hear "Rate"

Press Ctrl + Option + Command + ↓ to make VoiceOver speak slower

Press Ctrl + Option + Command + ↑ to make VoiceOver speak faster

If you find VoiceOver is speaking too fast, you can adjust the rate.

VoiceOver Shortcuts Task Shortcut

Toggle VoiceOver on/off Command + F5 (or Fn + Command + F5)

Stop speaking Ctrl

Read the focused item Ctrl + Option + A

Move to next/previous control Tab / Shift + Tab

Activate the focused item Spacebar OR Ctrl + Opt + Spacebar

Toggle keyboard help Ctrl + Option + K

Toggle the display off/on Ctrl + Option + Shift + F11

Some of the very basic shortcuts you’ll need to know to begin using VoiceOver and to test custom widgets.

nvda-project.orgThe NVDA screen reader is a free screen readers for Windows.

Install NVDA

1. Download and install NVDA

2. Press Ctrl + Alt + N to start NVDA

3. After NVDA starts uncheck the checkbox labeled "Show this dialog when NVDA starts"

Disable Mouse Tracking

1. Right click on the NVDA icon in the system tray

2. Go to Preferences > Mouse Settings

3. Uncheck the checkbox labeled "Mouse Tracking"

Sighted users might want to disable mouse tracking in the interest of reducing "noise"

Save Preferences

1. Right click on the NVDA icon in the system tray

2. Choose the "Save Configuration" menu item

Configuring Speech Rate

Ctrl + Insert + ↓ Speak slower

Ctrl + Insert + ↑ Speak faster

If you find the screen reader is speaking too fast, you can adjust the rate.

NVDA Speech ViewerClick on the NVDA icon in the system tryGo to Tools > Speech Viewer

If you don't want to listen to the screen reader, the spoken text is available via NVDA's speech viewer.

NVDA Keyboard Shortcuts

Task Shortcut Stop NVDA from speaking Ctrl

Toggle Browse Mode on/off Insert + Spacebar

Read the focused item Insert + Tab

Activate the focused item Enter or Spacebar

Move between objects in the tab order Tab or Shift + Tab

Toggle NVDA on Ctrl + Alt + N

Toggle NVDA off Insert + Q

Some of the very basic shortcuts you’ll need to know to begin using NVDA and to test custom widgets.

Configuring Your iOS Device

Enable VoiceOver on iOS

Settings > General > Accessibility > Triple-click Home

Choose VoiceOver

As a developer, I recommend setting up VoiceOver on iOS such that triple-clicking the Home button toggles VoiceOver on and off.

Widget Touch Patterns

Task Gesture

Speak item or navigate to item Tap

Select the next or previous item Flick right or left

Activate (click) the selected item Double-tap

Change selection or value Flick up or down

Dismiss an alert Two-finger “scrub”

Some of the very basic shortcuts you’ll need to know to begin using VoiceOver and to test custom widgets for mobile.

Exercises

Some exercises to get you familiar with using existing web form controls and classic desktop controls using a screen reader.

Exercise

bit.ly/z86b0JUse this simple page to begin to get comfortable using either VoiceOver and Safari on the Mac, or NVDA and Firefox on Windows.

Observable keyboard patterns for existing HTML form controls: Tab and Shift+Tab navigate between widgets/controls Arrow keys change selection in a group Spacebar executes a click action Enter executes the default action Esc key closes menus and dialogs

Observable widget information announced by the screen reader: Widget type Widget label Widget properties Widget state

Exercise

Use the menubar with VoiceOver enabled. (Press Control + F2 to focus the menubar.)

Notice some of the menubar keyboard patterns: Arrows navigate the menubar and submenus Esc key closes the menu and blurs the menubar. When the menubar blurs, focus is restored to the control that previously had focus.

Notice how VoiceOver announces things like: The label of the current menu or menuitem The size of the menu when you first open it Submenu items are announced as "submenu" The state of menuitems—for example "dimmed" to indicate disabled menuitems When a menu closes

Exercise

Use the tree control in the Finder with VoiceOver enabled.

Notice some of the tree control keyboard patterns: Up and down arrow keys navigate between items Left and right arrow key expand or collapse items

Notice how VoiceOver announces when a tree item expands or collapses.

Exercise

Use one of the tab controls on the Mac with VoiceOver enabled.

Notice how VoiceOver announces things like: The selected state of a tab The total number of tabs

Notice the tab control keyboard patterns: Left and right arrows move focus between tabs Spacebar changes the selected tab

Exercise

Use the menubar in IE on Windows with NVDA enabled. (Press Alt + F to move focus to the menubar.)

Notice some of the menubar keyboard patterns: Arrows navigate the menubar and submenus Esc key closes the menu and blurs the menubar. When the menubar blurs, focus is restored to the control that previously had focus.

Notice how NVDA announces things like: The label of the current menu or menuitem The size of the menu when you first open it Submenu items are announced as "submenu" The state of menuitems—for example "unavailable" to indicate disabled menuitems

Exercise

Tab controls on Windows work similar to tab controls on the Mac.

Move focus to the control by pressing Tab or Shift + Tab. Change tabs via left/right arrow keys.

Widget Keyboard Patterns

Task Key

Focus Next or Previous Widget Tab / Shift + Tab Change selection →, ←, ↑, ↓ Expand / Collapse →, ←, ↑, ↓ Close or Exit Esc Click Spacebar Default Action Enter

Although there are differences across browsers and platforms, there are many similarities with respect to keyboard patterns for widgets.

Mobile Exercise

bit.ly/z86b0JUse this simple test page to begin learning how to interact with HTML form controls when VoiceOver is enabled.

Some observable patterns: Swipe left/right to move between widgets Once a widget has focus, double tab anywhere on the screen to execute a “click” or interact with the control For controls that support selection, swiping up or down changes the value

Widget Touch Patterns

Task Gesture

Speak item or navigate to item Tap

Select the next or previous item Flick right or left

Activate (click) the selected item Double-tap

Change selection or value Flick up or down

Dismiss an alert Two-finger “scrub”

Review

Screen Readers

Read the code (HTML)

Read the focused element

What is read:

Element/Control typeLabelPropertiesState changes

Widget Keyboard Patterns

Task Key

Focus Next or Previous Widget Tab / Shift + Tab Change selection →, ←, ↑, ↓ Expand / Collapse →, ←, ↑, ↓ Close or Exit Esc Click Spacebar Default Action Enter

Although there are differences across browsers and platforms, there are many similarities and common patterns for widgets, and these patterns span the web and the desktop on both platforms—Mac and Windows.

Widget Touch Patterns

Task Gesture

Speak item or navigate to item Tap

Select the next or previous item Flick right or left

Activate (click) the selected item Double-tap

Change selection or value Flick up or down

Dismiss an alert Two-finger “scrub”