Accessibility of websites and

mobile apps

A study on the current practices regarding

accessibility statements, reporting mechanisms and

mobile monitoring methodologies

Draft Final Deliverable

A study prepared for the European Commission

DG Communications Networks, Content & Technology

by:

Digital

Single

Market

This study was carried out for the European Commission by

Capgemini Consulting and Dialogic

For more information about this paper, please contact: Directorate-General for Communications

Networks, Content and Technology G – Learning, Multilingualism & Accessibility Unit G.3

Åsa Barton – Policy Officer E-mail: asa.barton@ec.europa.eu

Project team

Capgemini Consulting

Niels van der Linden – Principal Consultant

Executive lead

E-mail: niels.vander.linden@capgemini.com Remco van der Spiegel – Managing consultant

Project manager

E-mail: remco.vander.spiegel@capgemini.com

Wander Engbers – Consultant

Analyst

Christian le Clercq – Senior Consultant

Analyst

Sem Enzerink – Consultant

Analyst

Dialogic

Robbin te Velde – Principal Researcher

Quality assurance

Internal identification

Contract number: 30-CE-0631083/00-65

SMART number 2016/0089

DISCLAIMER

By the European Commission, Directorate-General of Communications Networks, Content & Technology.

The information and views set out in this publication are those of the author(s) and do not necessarily reflect the

official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this

study. Neither the Commission nor any person acting on the Commission’s behalf may be held responsible for

the use which may be made of the information contained therein.

© European Union, 2017. All rights reserved. Certain parts are licensed under conditions to the EU.

Reproduction is authorised provided the source is acknowledged.

3

European Commission Accessibility of websites and mobile apps – A study on the current

practices regarding accessibility statements, reporting mechanisms and mobile monitoring methodologies Luxembourg, Publications Office of the European Union

4

Executive Summary Technology, in particular connected technology, is rapidly changing our world. It changes every

industry, organisation and process. It also changes how citizens interact with their (local) public

sector: how they acquire information or how they use public services. To ensure all citizens, including

those with a disability, profit from digitisation the European Union enshrined Web Accessibility in its

legislation. Web Accessibility means that people with disabilities, such as blindness or motoric

limitations, can perceive, understand, navigate, interact with and contribute to the Web. With the

Web Accessibility Directive, it becomes mandatory for all Public Sector Bodies to have websites and

mobile applications adhering this principle.

Methodology: Examining the current accessibility practices in three studies To ensure that the Member States move towards an accessible web in a similar manner, the

Directive aims to harmonise accessibility legislation. In anticipation of the implementing acts

detailing this legislation, this study presents an overview of the current practices in three respective

fields: accessibility statements (study 1); reporting mechanisms (study 2); and, monitoring

methodologies for mobile applications (study 3). Based on the analyses of these practices and

discussions with experts, the study presents recommendations for the development of implementing

acts. The study collected the information via the following four means:

Landscaping of the European countries by asking them to participate in a questionnaire and

complementary interview. In total 28 Member States responded to the questionnaire and 25

representatives were interviewed.

Desk research to obtain a sound definition of the state-of-play on web accessibility

developments and to identify the key stakeholders, experts and interest groups.

Interviews with 25 international experts from the public, private and NGO sector. In

particular to discuss the possibilities to monitor the accessibility of mobile applications.

Survey addressing a global audience to identify existing monitoring initiatives covering the

accessibility of mobile applications for further examination. The survey did not identify any

existing initiatives.

Policy: Moving towards transposition, the currently heterogeneous position of

web accessibility Transcending the three individual studies, the report first examined the current legal position of web

accessibility and the (expected) progress in the transposition of the Directive. The legal position of

web accessibility strongly varies in Europe, ranging from no legislation, to general non-discrimination

acts, to binding legislation specifically on web accessibility. This heterogeneity affects the maturity of

the practices and the transposition preparations. To transpose the Directive, 21 countries started

preparations, ranging from composing commissions to drafting legislation. The countries mention

time and budget constraints as possible risks for a successful transposition. These constraints are

either the result of national developments, such as upcoming elections, or because of the absence of

implementing acts. In 24 of the surveyed countries, interest groups are involved in the policy

processes regarding digital accessibility. This ranges from ad-hoc advisory roles to executing

accessibility evaluations.

5

Study 1: Large variety of practices regarding accessibility statements At the moment, 54% of the reviewed countries have specific policies or regulations on accessibility

statements (implemented or in preparation). Member States miss overview on the use of statements

in their country. Only seven Member states could estimate the percentage of statements on public

sector body websites. Their estimations ranged from less than 25% to over 95%. This underlines that

the full use of accessibility statements has not been commonly achieved across all Member States.

Our analysis of 66 accessibility statements showed that statements generally include: a description of

inaccessible parts, a feedback mechanism, assistive user instructions, a statement date and the

achieved level of compliance. None of the statements provided a link to enforcement procedures or

the update frequency. The statements contained information relevant for end-users (35%),

monitoring purposes 9%) or a combination (16%). In 39% of the cases, the statements did not

sufficiently provide information to be helpful for end-users nor monitoring purposes.

The study found six practices how Member States develop (and update) accessibility statements:

1. Free formats that allow Public Sector Bodies to structure their own accessibility statements.

2. Content descriptions that prescribe the required topics of the accessibility statement.

3. Standard templates that define the format and structure of the accessibility statement so a

public sector body only provides particular content regarding its website.

4. Common Web Platform incorporations that automatically create an accessibility statement

within the commonly used (technical) web platform of public sector bodies.

5. Accessibility generators that collect particular information of the public sector bodies to

automatically generate an accessibility statement.

6. Statements based on a label that declare accessibility compliance by showing a logo and

further information related to that logo.

We examined the accessibility statement practices on their benefits and concerns, related to:

scalability, assumed impact, awareness, administrative burden and sustainability. The outcome of

this examination is displayed in Figure 1.

Free format (non-

standardised)

Content description (non-

standardised)

Standard template (standardised)

CWP incorporation (standardised)

Statement generator

(standardised)

Statements based on a label (standardised)

Be

ne

fits

Scalability

Administrative

burden

Sustainability

Scalability

Awareness

Administrative

burden

Sustainability

Scalability

Assumed impact

Administrative

burden

Sustainability

Assumed impact Scalability

Assumed impact

Administrative

burden

Sustainability

Assumed impact

Awareness

Co

nce

rns

Assumed

impact

Awareness

Assumed

impact

Awareness Scalability

Awareness

Administrative burden

Sustainability

Awareness Scalability

Administrative

burden

Sustainability

Figure 1: Benefits and concerns of practices regarding accessibility statements

The Directive proposes a harmonised accessibility statement model. Therefore, the benefits and

concerns of the standard template are particularly relevant, while gaining insights from the other

practices. On the level of individual accessibility statements, no single statement included all relevant

elements: containing all defined content items, combining end-user and monitoring purposes,

defining information on the update frequency, being easy to find and declaring accountability.

Therefore, the chapter provided good examples of different accessibility statements that cover

6

different accessibility statement elements. This selection of examples can inform further

development of a harmonised accessibility statement model.

This study recommends a standard statement in an accessible format with a fixed order of items that

serves as input for both end-users and monitoring purposes. It should include the mandatory items

defined in the Directive. The statement should also provide the following additional items: level of

compliance, planning of improvements and the test report. In order to be relevant, the statement

should contain as-is accessibility information meaning that the information provided should be

updated after each (internal or external) update with impact on the accessibility of the website or

mobile application. The statement should mention a senior executive or team that is political

accountable for the accessibility of the website or mobile application to show responsibility and

ensure ownership within the Public Sector Body. It is recommended to ensure findability, for

example by defining a uniform URL or location of the link to the statement. This link should be

available on a permanent body of pages of the website.

A standardised template to communicate the actual accessibility status of the PSB’s website, which serves end-users and monitoring purposes

Co

nte

nt

ite

ms

for

stan

dar

d t

em

pla

te

Use a harmonised standard template that serves as input for both end-users and monitoring.

Include additional content items: description of the level of compliance, planning of

improvements and a link to the test report, besides the mandatory items (descriptions and

explanation of inaccessible parts, alternatives, link to a feedback channel and a link to an

enforcement procedure).

The template should have a fixed order of items, taking into account the end-users point of

view.

Acc

ou

nta

bili

ty Include details of the senior executive or team accountable for accessibility in order to show

political ownership and sense of urgency.

Stan

dar

d

loca

tio

n Define a uniform location to ensure findability, for example by means of a uniform URL as

location for accessibility statements, or a uniform location of the link to the accessibility page,

and ensure visibility for search engines.

Fre

qu

en

cy o

f

up

dat

ing

The statement should reflect the as-is status and hence be updated at each update with

relevance to accessibility (change in the functioning of the website or app, this includes

website updates, browser and O.S. updates).

Acc

ess

ible

form

at

The standard template should be in an accessible (WAD mandatory) and machine-readable

format that can easily serve as input for monitoring.

Figure 2: Recommendations for accessibility statements

7

Study 2: How to collect information? Different ways to organise the reporting

on web accessibility In the absence of a policy on web accessibility in many countries, 11 out of 28 reviewed countries

have a formal reporting practice at this moment. With the addition of informal, planned or past

initiatives, the study identified and examined 16 reporting examples. Based on this examination, the

study found three types of reporting practices:

1. Decentralised transfer of data by the PSB via a (self-) declaration to a central body: a

reporting mechanism wherein it is the responsibility of the PSB to determine its level of

accessibility, via self- or third party evaluation, and to provide this information to a central

body.

2. Centralised monitoring based on third party evaluation: a reporting mechanism wherein a

third party evaluates the level of accessibility of multiple PSBs, either in commission of a

central government body or as NGO initiative.

3. Centralised monitoring based on a government agency evaluation: a reporting mechanism

wherein a central government body both monitors and reports on the level of accessibility of

PSBs.

Most of the existing reporting practices are organised according to the third type in which one body

measures and reports on the web accessibility of (a number of) public sector bodies. Of the

interviewed countries, 28% indicated to prefer or prepare decentralised reporting practices. The

current practices have a two-folded purpose: providing feedback to the audited entities and

reporting on overall developments using aggregated data. The practices regarding reporting were

also examined on their benefits and concerns in terms of scalability, assumed impact, awareness,

administrative burden and sustainability. This examination, displayed in Figure 3, showed that each

type has its own benefits and concerns. Nevertheless, decentralised and centralised reporting

primarily differ in scalability and administrative burden.

1. Decentralised reporting

2. Centralised reporting by third party

3. Centralised reporting by a central (government)

body

Be

ne

fits

Awareness

Administrative burden

Target group

involvement

Administrative burden

Scalability

Assumed impact

Co

nce

rns Assumed impact

Scalability

Sustainability

Scalability

Administrative burden

Figure 3: Benefits and concerns of practices regarding reporting

The study found three recurring trade-offs when examining the practices which pose relevant

questions for the development of a harmonised mechanism. The first trade-off concerns the use of

automated tools. Automated tools create efficiency gains and are complementary to manual

evaluation by end-users. The second is the trade-off between a detailed and valid but labour

intensive methodology and a more high-level but efficient methodology to measure the level of

accessibility. The third trade-off is between publishing the (individual) outcome of the reporting or

providing a safe-learning environment for the public sector bodies. Although not transparent, this

does enable continuous (peer-) learning.

8

Based on the current insights and preferences of Member States, the reporting mechanism will need

to strike a balance between flexibility and rigidity. Flexibility in the ‘how’, allowing Member States to

decide to organise the reporting centralised or decentralised. Rigidity in the ‘what’, to make sure

Member States collect the same data in a similar way via a harmonised monitoring methodology so

the data can be aggregated at the European Union level. We identified good practices to balance

flexibility and rigidity. The reporting could also make use of the other mandatory elements of the

Directive, in particular the accessibility statements. When these statements are valid and sound –

based on an approved and harmonised monitoring or evaluation methodology – the reporting body

could automatically harvest these for information. The report therefore recommends a form of

decentralised reporting scheme, entailing a strong role for the public sector bodies. This enhances

their awareness with web accessibility and provides them with better insights on their performance,

which could both help them improve. Furthermore, it allows for a bigger scale as all individual public

sector bodies are conducting evaluations. The figure below presents the recommendations that,

either as a framework or by themselves, provide the means to achieve a reporting mechanism with

the highest teleological impact.

Decentralised reporting scheme with a controlling sample

Fro

m M

em

ber

Stat

e t

o

Co

mm

issi

on

Harmonise the data by providing a fixed template for the Member States to report to the Commission on:

Monitoring data and enforcement procedure data (WAD mandatory)

Aggregated and segmented data in order to analyse differences in segments and allow for

Best practices and challenges to help Member States continuously improve

Market characteristics of accessibility products and services to analyse the impact of the

Directive on the web accessibility market

Wit

hin

Me

mb

er

Stat

e Monitoring based on decentralised evaluation by PSBs and transfer of data from PSB to

Member State’s monitoring body (self- of 3rd party evaluation)

Harmonised data by providing a fixed reporting template for the Public Sector Bodies to

report on:

o Level of accessibility

o Wider (planned) accessibility measures to keep track of policy developments

o End-user feedback

o Use automated tools for an efficient collection of data

Co

ntr

ol Centralised controlling function by the Monitoring Body, based on a random and purposive

sample to include sectors, services and functionalities that are most relevant to the end-users.

Consult end-users to determine the most relevant websites.

Fre

qu

en

cy Align the frequency of the transfer of data of PSBs to the Monitoring Body with the budgetary

cycle to encourage direct budget allocation.

Triennial reporting from Member State to EC (WAD mandatory).

Figure 4: Recommendations for a reporting mechanism

9

Study 3: heading towards a monitoring methodology for mobile applications The third study focused on methodologies to monitor the accessibility of native mobile applications

with the primary conclusion that there are currently no formal or structural initiatives monitoring the

accessibility of mobile applications, within nor outside the Europe Union. As an explanation, the

surveyed Member States and the global experts (from academia, public sector, private sector and

non-governmental organisations) indicated that the accessibility of mobile applications is not yet as

developed as the accessibility of websites.

Important conditions to stimulate the development of monitoring mobile applications in the future

are: an international mobile accessibility standard, a balance between automated and manual

evaluations, a clearly defined evaluation scope and the availability of source codes. Mobile

accessibility in general would benefit from: greater awareness among mobile app developers,

standardisation within the diversity of user interfaces and strengthened end-user feedback. The

experts were in near consensus that a monitoring methodology can only be successful when it covers

the entire life cycle of a mobile application. In other words, not only focussing on the accessibility

when the app goes live, but also in an earlier stage, during the development of the application.

Based on the gathered insights, Figure 5 presents five scenarios for a monitoring mechanism, with

respective benefits and concerns. The study outlines requirements for each scenario to implement it

as a monitoring methodology.

Development phase Live phase

Monitoring scenario 1: Mandatory self-evaluation. Developer is required to evaluate and document on accessibility during development process. Benefits: Developer can directly mitigate issues, (partly) use of automated tools, efficient evaluation. Concerns: Administrative burden for developer, dependent on uniform evaluations. Requirements: Reporting on accessibility by

developer, evaluation methodology.

Monitoring scenario 3: Evaluation by central body. All applications are evaluated by one central reporting body. Benefits: Consistent evaluation. Concerns: Labour-intensive and thus requires many resources from a central evaluation body. Requirements: Auditing expertise, evaluation

method.

Monitoring scenario 2: Process certification. Application owner is required to provide information that developer is certified. Benefits: Low costs, low administrative burden, promotional value. Concerns: Dependent on trust, no actual evaluations, missing insights on actual accessibility level. Requirements: Standards, certified developers, certification authority, authority to penalise.

Monitoring scenario 4: Evaluation by developer. Developer evaluates accessibility on request. Benefits: Developer has access to source code, developer can directly mitigate issues or during update. Concerns: Dependency on availability developer. Specific requirements: Evaluation method.

Scenario 5: Flagging by user community. Reactive enforcement based on user notification. Benefits: Only high priorities focused, target group involvement. Concerns: Not comparable, dependent on interest groups, no equal interest community. Requirements: Evaluation capability, enforcement

procedure, establish central contact points where

the feedback of end-users is being handled by

professionals.

Figure 5: Possible mobile monitoring scenarios

10

Based on these scenarios, the report recommends harmonised monitoring methodology combining

different scenarios. Mobile accessibility experts believe that evaluating the actual level of

accessibility is important, regardless of the scenario. Although the current automated evaluation

tools can measure different factors of mobile accessibility, none of these tools can currently evaluate

all accessibility factors. In existing evaluations automated and manual evaluations are

complementary. Depending on the monitoring scenario different tools can be used in different

phases of the application lifecycle: by developers in the development and testing phase, or by end-

users after deployment of an application. This translates into the recommendation to involve all

stakeholders in the monitoring process, from self-evaluation by the developers to end-user flagging

for focused evaluation. The study recommends a decentralised monitoring practice based on

mandatory developer evaluations in combination with control by a central body which is guided by

end-user feedback. The figure below presents the practical recommendations accompanying this

envisioned organisation.

A monitoring practice based on developer evaluation combined with central body control

Har

mo

nis

ed

mo

nit

ori

ng

me

tho

do

logy

Monitoring practice based on collection of decentralised evaluations. PSBs transfer the data

of the mandatory evaluation by the app developer to the Member State’s monitoring body.

Organise end-user feedback (post-market surveillance) as input for central body control.

Promote awareness on accessibility through Integrated Development Environments (IDEs).

Har

mo

nis

ed

eva

luat

ion

me

tho

do

logy

Clearly define the scope of the evaluation and use a modular design.

A combined evaluation approach where automated and manual evaluations are

complementary.

Clearly report on the evaluation process.

Fre

qu

en

cy Align the frequency of monitoring with the budgetary cycle of the app owner to encourage

direct budget allocation.

Align accessibility evaluations with the launch and major updates of mobile applications and

with updates of the operating system.

Sam

ple

Cover the entire population of mobile applications or use purposive samples.

Evaluate the full content of the app; only use internal sampling for extensive mobile

applications.

Figure 6: Recommendations for the monitoring of mobile applications

CA

TA

LO

GU

E N

UM

BER

Digital

Single

Market