managing remediation of web accessibility defects in a large enterprise daniel m. frank vice...
TRANSCRIPT
Managing remediation of Web accessibility defects in a large enterpriseDaniel M. FrankVice PresidentWells Fargo Bank, NA
Challenges of Defect-Based Metrics
• Simple counts don’t tell you how accessible something is or isn’t
• You don’t know what to fix first• You don’t have good information
for triage• Comparability with non-
accessibility defects is difficult to measure
• It’s difficult to set an achievable policy goal “My diagnosis is that
you are sick. Next?”
Developing an Effective Defect Management Program
Have an accessibility testing program
•Determine function points to be tested
•Map function points to organizational roles
•Specify supported configurations
•Manage skills and expertise
•Deploy testing tools, including AT
•Document adequate levels of information
•Ensure requirements are complete
Develop and test parameter definitions and weights
• Identify application(s) for pilot
•Create parameter definition strawman
•Hold whiteboard scoring sessions
•Adjust definitions on the fly
•Evaluate whiteboarding results and adjust definitions and weights as necessary
Implement scoring process
•Clean up defects and defect documentation standards
•Modify defect SOR to record parameters and weights
•Create weighting pipeline or process
•Train responsible staff and publish documentation
Develop “goodness” metrics and defect repair policy
• Identify denominator for “norming” defect scores
•Develop comparability metric
•Set thresholds or other policies for fix vs. leave as-is
•Measure and report regularly
Defect Prioritization Formula
• Weighting formula developed by Karl Groves of The Paciello Group, and described in his presentation for GAAD 2014
• Karl developed a set of weightings based on a Delphi process with industry experts. They can be re-weighted during implementation as necessary.
• The formula and weights yield a bias toward:– Impact on multiple populations– Defects that don’t require re-opening design or back end architecture decisions– Ease of code repair– Defects that occur in components or patterns used in multiple locations in the
application.
(Severity + Business Importance + Front-end Impact + Back-end Impact + Ease of Repair)
* Volume
Clear Parameter Definitions Are Key to Defect Scoring
• Discrete values between one and four
• Unambiguous and easily understood
• Adjusted for clarity and consensus during whiteboarding process
• Text of definitions incorporated into SOR during implementation
Severity
Experience of users
with disabilities
Multipliers for number of affected
populations
Volume and Importance
Occurrence in a
component or pattern
Importance of business
flow
Ease of Repair
Impacts on design
team, back end systems
Ease of code repair
Severity Definition and Multipliers Used In Wells Fargo Pilot
1. Blocker (4): User is completely prevented from accomplishing a task. There is no work-around.
2. High (3): User can perform task only through alternate path or workaround; Accomplishing the task likely only possible for power users.
3. Medium (2): User can perform task, but doing so will be frustrating or time consuming. User may require assistance from co-worker or support staff.
4. Low (1): User experiences inconvenience or moderate frustration
Basic Definitions Multipliers A “High”(3) impact defect for
two or more user populations should be regarded as a blocker and given a severity score of ‘4’
Otherwise the severity is the max of:o The count of affected
populations, with an upward bound of 4 (blocker)
o All severities assigned for this defect.
Priority Parameters Used in Wells Fargo Pilot
Impact on presentation logic1. Requires substantial redesign of both visual and
functional presentation design2. Requires substantial redesign of either functional or
visual design, but not both3. Requires minor redesign of either visual elements or
functional design, or both4. Resolution does not require changes to visual or
functional design
Impact on business logic1. Resolution requires changes to back end data
architecture, such as database or information bus schemas, with downstream effects on back end code implementing business logic, etc.
2. Resolution requires changes to back end code architecture and services schemas, but not back end data architecture
3. Resolution requires minor changes to back end logic, but does not impact code architecture or services schemas
4. No impact on business logic
Location of defect1. Defect is located in an error flow of a seldom-used or
non-critical business flow2. Defect is located in the functional flow of a seldom-used
or non-critical business flow3. Defect is located in an error flow of a frequently-used or
critical business flow4. Defect is located in a frequently-used or critical business
flow
Volume of defect 1. Defect is a one-off, not part of a component or pattern2. Defect occurs in a component or pattern, but only on
this page3. Defect occurs in a component or pattern
on multiple pages4. Defect occurs in a component or pattern
on every page
Ease and speed of repair 1. Defect is intermittent and/or very difficult to diagnose,
or requires global changes to resolve2. Defect is consistent and straightforward to diagnose,
but requires local changes to the design of underlying code to resolve
3. Defect requires changes to the underlying code but not its design
4. Defect requires correction of typos or minor parameter changes.
Who Has to Participate in Scoring?
Key Skills Required Typical Participants• Assess usability impact on
task completion by different user populations
• Understand relation to business function of task flow where defect occurs
• Estimate impact of repair on different aspects of the SDLC or architecture
• Predict the type of repair required to application code
• QA testers• Product managers• Application architects• Developers
Good Defect Hygiene is Critical
• A rigorous prioritization system demands more of your defect documentation and your testing team
• Accessibility is about usability; that’s what the severity metric really assesses, and your testers must approach testing with that mindset
• One defect per record!
– The same defect on a different page is a separate defect
– The same defect in different locations on the same page must be recorded separately
• Record the browser and assistive technology versions with the defect
– Many defects only appear with particular products or versions (e.g., IE11 with JAWS 16)
• When testing with screen readers, provide screen shots of transcript windows, and consider making video recordings
• Clean up your defect data before you try to score it.
Insights Gained From Whiteboarding
• Progress is slow at first, but speeds up quickly• A common language and methodology breaks
logjams and increases insight• So far the scores “feel” right, and the few
surprises make sense• Faster is better, make decisions and move on
Things to Think About When Evaluating Weights
• The initial scores are relative. The absolute values don’t mean anything without norming.
• Does there seem to be a reasonable spread of the scores? A good weighting scheme provides significant discrimination between high and low priority defects.
• Do the scores make sense in relation to each other? In other words, do highly-weighted defects seem like ones you would have chosen to fix first?
• Are you seeing bands of values that could be useful in setting thresholds in a defect repair policy?
• Are there any odd results? If so, figure out why.
Open Questions
• Can scoring be pipelined, or does it require a multi-disciplinary scrum?
• What is the right denominator for norming application scores? Lines of code? Function points? Pages/screens?
• What policies should we set around defect repair?• How do we compare our level of attention to
accessibility defects, to the effort we put into bugs affecting non-disabled users?
