this resource is made publicly available to anyone seeking ... file1 this resource is made publicly...

1

This resource is made publicly available to anyone seeking to adopt or design safer

and more usable AR solutions. It is provided by the members of the AREA at no

cost to the ecosystem of Augmented Reality technology providers, customers of

AR-enabled solutions and others participating the advancement of AR adoption

1

© 2014 - 2018 All Rights Reserved

www.theAREA.org

AREA AR Safety Framework Case

Study

The AREA © 2018. All rights reserved.

This is a proprietary report prepared under contract with the AREA.

As part of the AREA’s support of the development of sound information and

best practices for the introduction and adoption of AR, access to and use of

this proprietary report is provided to all members of the enterprise AR

ecosystem. If you wish to make your partners, suppliers and customers

aware of this AREA research report, you may share this resource provided

that this information and the content of the report are not edited.

The entire content of this proprietary report is protected by copyright. No

part of this publication may be reproduced, stored in a retrieval system or

transmitted in any form or by any electronic, mechanical, photocopying and

recording means or otherwise, without the prior written permission of the

AREA.

2


www.theAREA.org


Study

TABLE OF CONTENTS

Purpose ...................................................................................................................... 3

Statement of the Challenge ........................................................................................ 4

Case Study of an Aerospace Assembly Use Case ................................................. 4

AR Project Cycle ........................................................................................................ 5

Step 1: Requirements Capture ................................................................................... 6

Step 2: Design & Build - Analyze the Risk & Define Acceptable Safety Factors ...... 10

Step 3: Testing – Define Usability Testing ................................................................ 12

Step 4: Delivery – Document Spec & Training ......................................................... 13

Way Forward ............................................................................................................ 14

3


www.theAREA.org


Study

PURPOSE

This document is prepared by The Manufacturing Technology Centre, on behalf of the

AREA, to illustrate the proposed steps of a solution provider when assessing their

solution for safety in a manufacturing assembly process.

This resource is made publicly available to anyone seeking to adopt or design safer

and more usable AR solutions. It is provided by the members of the AREA at no cost

to the ecosystem of Augmented Reality technology providers, customers of AR-

enabled solutions and others participating the advancement of AR adoption.

The steps in this case study make use of the AREA Safety and Human Factors

Assessment Framework Tool and the proposed risk assessment framework in the

AREA report. The AREA Safety and Human Factors Assessment Framework is a

working model made available exclusively to AREA members for assessing AR design

and hardware for safety and usability.

An accompanying AREA member exclusive report provides Best Practices for

Wearable AR Technology Safety and Human Factors Assessment in the Enterprise.

This project and resultant model benefits project managers, solution providers,

developers and safety managers by:

- Encouraging collaboration between risk management and the project team - Offers specific approaches to mitigate risks that, in part due to lack of strong

human factors backgrounds may be introduced when AR is used in the workplace

- Provision of methods and tools for the assessment of safety and human factors, applied at the correct time, and aligned with the typical project lifecycle

To learn more about joining the AREA, these reports and other member benefits,

please visit http://thearea.org and/or contact it’s Executive Director, Mark Sage.

http://thearea.org/

mailto:[email protected]

4


www.theAREA.org


Study

STATEMENT OF THE CHALLENGE

Manufacturing companies are increasingly seeking to automate complex manual

assembly tasks in order to improve quality, efficiency and reduce cost.

Due to the mix of machines and employees, manufacturing workplaces require

proactive enforcement of safety and risk management protocols. The introduction of

wearable AR solutions in a typical workshop environment requires another level of

careful assessment and identification of possible safety issues before adoption. Then

mitigation measures must be incorporated into the design stages of a safe solution.

Case Study of an Aerospace Assembly Use Case

Augmented Reality can be an effective tool to ensure that procedures are followed

consistently and accurately as part of quality management processes and in

compliance with industry regulations. In assembly use cases, AR delivers value by

presenting information digitally in the user’s context to guide them through complex

assemblies. The costs and time associated with re-work and lack of compliance are

reduced when assembly is done correctly every time.

For this case study, an AR-assisted assembly project is commissioned by a fictitious

aerospace manufacturing company. An AR solutions provider is hired to build a

solution designed to reduce the time taken for complex assembly by delivering AR-

assisted instructions.

The company manufactures and assembles aerospace systems with strict assembly

compliance regulations at each step. The organization would like to improve

operational efficiency by ‘augmenting the manual worker’ through the use of portable

AR-assisted devices.

The work order instructions are accessed at a fixed workstation, however, assembly

can take place at another area of the factory floor or, in the case of large assemblies,

can require working at height.

The solution must prioritize safety and usability in its design to ensure the successful

adoption by the client.

5


www.theAREA.org


Study

AR PROJECT CYCLE

A project cycle is used as a reference to describe recommended actions and

methods/tools to capture and mitigate for safety risks and usability. In this case study

and the AREA Safety and Human Factors Assessment Framework Tool, this cycle

has been referred to as the ‘AR project cycle’.

Shown in the figure below, the AR project cycle emphasizes recommended actions to

promote safety and usability within the proposed AR solution. This case study is

described in context of a fictitious aerospace assembly company developing an AR

use case. For this scenario, the AR solution is to be deployed on both a wearable AR

display and a companion app running on tablet.

Figure 1 AR project cycle (source: AREA Safety and Human Factors Assessment Framework Tool)

The AR Project Cycle - Solutions Provider

Recommended Actions

Supporting Tools

Requirements Capture (Systems

Definition)Design Build Testing Training Delivery

• Identify SafetyRequirements by Defining:

• Context of use (process, task and environment)

• Currently used hardware,

software and equipment• Existing health and safety

hazards• Safety compliance and

regulatory factors• Key actors in the

project/implementation

• Define end-user demographics and characteristics

• Assess Solution Design for Safety and Usability

• Undertake risk assessment of solution design

• Identify mitigations for solution

• Assess and select device to safety requirements

• Define Safety and Usability Testing

• Define acceptable safety factors

• Conduct safety and

Usability trials• Analyse results and

incorporate feedback into design

• Document Solution Specification for the End-User Including:

• Instructions on proper use of solution

• Known risks• Residual risks

• AR Device Assessment

• AR Design Assessment

• AR Generic Risks Table

(Ref)

• AR Design Assessment

• AR Generic Risks Table (Ref)

• Usability and Safety Tools (Ref)

Commented [CP1]: I’m not sure what this means

6


www.theAREA.org


Study

STEP 1: REQUIREMENTS CAPTURE

During this stage, the project team or manager will describe the technology and the

processes to be modified. At the requirements capture phase of the project, the

following should be identified:

● Definition of the task, the process and the environment, ● Hardware, software and equipment currently used, ● Existing health and safety hazards

o Through discussions with the client’s side safety/human factors manager and/or project manager

o Through current risk management documents on the assembly process such as risk assessments or process failure mode effects analysis (PFMEAs)

● Safety compliance and regulatory factors including : o Known country/state regulations o Organizational policies

● Key actors in the project/implementation, ● End user demographics.

A brief summarized example of what the output of the requirements capture for the

case study is shown below.

Table 1. Output of the requirements capture process

Project Requirements Summary

Task/process Assembly of engine components. Manufacturer has defined the six work orders

in the process for assembling components on an aircraft engine. Each work

order is then broken down into smaller tasks known as operations

Environment Typical workshop environment

AR location of use Working at height for some tasks requiring climbing of scaffolding. Mainly

working on various areas of shop floor.

Hardware, software and

equipment currently used

Work instructions are accessed through a fixed work station computer. Some

tools required specific assembly operation.

Existing health and safety

hazards

Risk assessments and PFMEA are available.

Safety/Compliance

• Personal Protective Equipment o OSHA 1910.132

• Ergonomics o OSHA CFR 1910 General Duty Clause, Section 5(a)(1)

• Fall Protection (ensuring protection from fall hazards) o OSHA 1910 Subpart D State and organizational policies

benchmarking

Key actors in

project/implementation

• Solutions provider: project manager, development team

• Client –side: project manager, safety manager, human factors engineer,

User demographics (i.e. to

create end user personas,

and understand known

behaviors of end users)

Range between young apprentices to long-service employees, some workers

wear prescription lenses

Commented [CP2]: Not parallel

Commented [CP3]: Needs a caption

7


www.theAREA.org


Study

AR Assessment Tools

The ‘AREA Safety and Human Factors Assessment Framework’ produces results of

two assessments. These tools, consisting of a set of comprehensive questions and

related information, support the key roles involved in a project to systematically identify

and derive safety and human factors related issues and provide high level

recommendations to include in the design of the solution. The tools assist in analyzing

all the potential safety and usability requirements that may need consideration

surrounding the user, the environment, the context (the task) and its interactions within

the system.

AR Device Assessment tool

This tool assists in defining device requirements for the solution. For this assembly

use case, the following are some constraints derived from the assessment:

Environment

● The workshop floor is brightly lit ● The workshop floor can be noisy ● Operators must be aware of hazards in their surroundings such as forklifts. ● Operators must be aware of guarding and zone exclusion zones whilst using

the device. ● Health and safety notes should be displayed to the device before operation as

current standard process.

Use case

● Assembly operators are required to work at height for some operations –for e.g. climbing scaffolding and ladders

● Hanging cables from the device may get caught/detached in the assembly process.

● Some assembly operations require the use of small power tools for e.g. automated nut runners and hand-drills

User

● Some operators wear prescription glasses ● Experienced users will need to use the device for complex assembly operations

(such as when assembling harnesses). Less experienced operators may require longer use of the device even for simpler operations as they’re training.

● Device needs to be comfortable to wear for duration required- each operation lasts approx. 20mins.

● Voice interaction with the device would be ideal leaving hands-free

System

● Device needs to receive safety prompts and alarms

A sample of the recommendations is show below Table 1:

8


www.theAREA.org


Study

Table 1 Table 1 Device Assessment Results Sample from ‘AREA Safety and Human Factors Assessment Framework Tool’

Notes

Device Safety

Considerations Main AR Risk/Hazard

Will the user be working

at height?

Working at height can be a

high-risk situation and may

require full attention, and

situational awareness. A

device obscuring user's

FOV should be avoided.

Consider how the device's

form factor and fit for user

may increase

discomfort/distraction/obst

ruction to user view. Can

the device be used to alert

the user when they are in a

dangerous

orientation/location?

Low Peripheral Vision

Does the device need to

specially

robust/ruggedized?

There are risks of

damaging the device whilst

not worn by the wearer.

Device can be dropped or

handled rough that could

cause damage to the

external materials or

possibly internal circuitry.

Consider how the device

will be used by the user

and what its resting place

will be normally. Will the

device be carried or moved

around? These may

indicate that a ruggedized

device would be required

to prevent damage over

time.

Asset damage

AR Design Assessment

Similarly, the design assessment component of the framework supports in the

designing a safe solution. For this assembly use case, the following are some design

constraints derived from the assessment:

This assessment advises the Solutions Provider to consider the following when

designing the AR solution:

Environment

● The workshop is expected to be noisy - the design may need to consider multi-modal ways of interacting with the device. The solution may also need to consider a way to filter out noise so that safety critical alerts such as alarms can be heard.

● There is risk of asset and equipment damage in the surrounding environment. There are other components surrounding the operator such as equipment and materials stacked on shelves.

● The operator will be required to use the device every day, and at 20 minutes at a time. Design must consider reducing risk of eye strain.

User

● The end users have some visual impairments for e.g. an operator has color-blindness. UI will need designing to ensure color-blind users can view the content correctly.

9


www.theAREA.org


Study

● Operator’s collaboration/communication with other personnel, equipment or machines. Some operations in assembly require working in pairs. Shared experiences may be required.

● The solution must be intuitive and easy to set-up/calibrate. ● The solution must not obstruct the user’s field of view or peripheral view ● The solution must not distract the user from hazards/risks posed to them in the

environment

Use case

● Some operations will require AR-assisted work instructions whilst operating power tools. A solution must have a hands-free way of interacting with the AR device

System

● The solution must allow safety critical information to be received by the user for e.g. alarms, alerts or prompts

● The solution should give the user visibility of the state of the device for e.g. battery status, offline/online status or malfunction.

A sample of the recommendations are found in Table 2 below:

Table 2 Design Assessment Results sample from ‘AREA Safety and Human Factors Assessment Framework Tool’

Notes Device Safety Considerations

Main AR

Risk/Hazard

2.3 Do the end users

have any experience

with using AR?

Introduction to the emerging

technology and training be

considered before or as part of the

implementation phase. This should

include correct fitting/mounting of

device and use of functionality.

Reluctance to adapting new

technology may be an issue.

Consider appropriate calibration set

up.

Guided experience may be

required for new users or users

with little experience with new

technology interfaces. Design flow

may need to be intuitive and user

experience comfortable. Consider

a change management plan.

Inadequate training

3.2 Is the device

required to receive

safety critical

messages?

Alarms, messages require to be sent

and viewed by the headset by third

party IoT, then this information/data

needs to be presented in the most

effective way to the user.

Consider how this

information/data will be displayed

effectively and safely. Urgent

safety information visual/tactile

may need to capture user's

attention without obstructing their

FOV while being in the midst of a

task.

Safety critical

communication

3.3 Does the user

have/require visibility

of the system? E.g. if

system is offline

If device loses signal or

malfunctioning, the user being

unaware of this could be a risk

Consider how the user will have

visibility of the current device

status - battery, malfunctioning,

offline status etc.

Safety critical

communication

10


www.theAREA.org


Study

STEP 2: DESIGN & BUILD - ANALYZE THE RISK &

DEFINE ACCEPTABLE SAFETY FACTORS

There are no specific design standards for creating AR solutions, however industries

adopt their own systems design methodology however, typically the design cycle is

iterative and consists of a concept stage and detailed design stage.

In the design stage of the project, the development team needs to ensure the design

meets the safety/usability requirements defined and prioritized in the first stage. To

ensure the design is safe and robust for the application, standard practice dictates that

risk assessments are undertaken before the build/implementation of the design.

In the AREA Safety and Human Factors Assessment Framework Tool, A list of

commonly used and standardized risk assessment methods are listed in the AREA

Safety and Human Factors Assessment Framework Tool. A sample of the tools

captured are shown below in Table 3:

Table 3 Sample of risk assessment tools taken from AREA Safety and Human Factors Assessment Framework Tool

Assessment

Domain Tool Metrics When to use them in Design Cycle?

Failure Modes

and Effects

Analysis

Breakdown of

failures, scored by

severity and

likelihood

Design - Concept Usually conducted once during the technology

cycle stages

HAZOP

System is divided

into

subparts/subsystem

s and analyzed one

at a time. Severity

and Likelihood

scored. Risk is then

ranked.

Implementation -

Early

Systematic search for hazards which are defined

as deviations within these parameters that have

dangerous consequences.

Risk

Assessment

Severity and

Likelihood of risk is

scored

Implementation-

Early

Document systematically outlining the known

hazards , risks and controls for AR

The following (Table 3) is an example output of an FMEA for the current assembly use

case.

11


www.theAREA.org


Study

Table 3 FMEA example MTC, 2018

Process

Step

Potential

Failure

Mode

Potential

Failure

Effect

SEV Potential

Causes

Current

Process

Controls

OCC DET *RPN Action

Recommended

What is this

step?

In what ways

can the step go

wrong?

What is the

impact on the

user?

How

severe

is the

effect?

What causes

the step to go

wrong?

What are the

existing

control that

prevent/detect

the failure

mode from

occurring?

How

frequently

is the

cause

likely to

occur?

How

proba-

ble is

detecti

on of

failure

mode?

SEV x

OCC x

DET

What are the actions

for reducing the

occurrence of the

cause or improving the

detection?

Operator

wears

headset

Headset

is not

calibrated

Experience

is incorrect

7 User is

not aware

of

calibration

process

n/a 6 5 210 Train user in

setting up

headset or

prompt user in

application to

calibrate

headset on

first use.

Operator

climbs

scaffoldin

g to install

compone

nt

AR

content

obstruct

user view

Operator

visibility is

reduced,

risk of

tripping

8 User does

not switch

off AR

whilst

climbing

scaffoldin

g. User

keeps

headset

on while

climbing

scaffoldin

g

n/a 6 7 336 AR content to

automatically

switch off in

task mode.

User prompted

to take off

headset before

climbing.

Operator

goes to

store to

get a tool

FOV is

obstructed

, SA is

reduced

Distraction

from

hazards in

environme

nt –

moving

vehicles,

trip

hazards

7 User

keeps

headset

and AR

experienc

e active

n/a 7 7 343 AR content to

automatically

disappear

when away

from work

zone. AR

headset

tracked and

alarm/prompts

when outside

work zone.

*RPN – Risk priority number: The overall risk score of an event. It is calculated by multiplying the scores for severity,

occurrence and detection. An event with a high RPN demands immediate attention while events with lower RPNs

are less risky. Traffic lights color coding used to highlight severity of risk.

A team approach is used during these design risk assessment activities to identify

failure modes of the solution. Acceptable levels of risk are determined by the project

team. Team members may include:

● A person highly familiar with the assembly steps (such as the production manager or end user)

● A person (or same person as above) well-versed in the health and safety risks of the process (from client’s side such as the safety manager and/or human factors engineer)

Commented [4]: Is this a table caption?

Commented [AN5R4]: yes

12


www.theAREA.org


Study

● The AR solution’s development team ● A professional familiar with FMEA expertise but no specific knowledge of the

project or processes This process is iterative, and it will be repeated until the design adequately mitigates

for risks.

STEP 3: TESTING – DEFINE USABILITY TESTING

Testing is an essential step in the AR project cycle as this evaluates whether the AR

solution has met its functional and non-functional needs including safety and usability

requirements.

There are several methods that can be used in evaluating an AR solution including

subjective and objective measurements through human perception, observation and

expert analysis. A comprehensive evaluation would include a combination of methods

and measures.

In this assembly use case, distraction and cognitive strain and potential restriction in

field of view are three overlapping usability concerns. Common usability tools that

could be adapted to evaluate AR solutions in the required domain include:

• Questionnaires such as the NASA-TLX to assess the perceived mental workload on the user

• Observational methods such as Verbal Protocol that involves asking the end user to verbally explain what they see and what they do to understand the reasons behind their decisions whilst navigating through the AR application/experience. This method also allows the observer to record the number of errors made by the user for e.g. if the user closes a critical health and safety pop up in the AR application without reading it

• Objective methods such as gaze tracking and physiological response measures could give an indication of whether the user’s focus is on the expected object/area/task.

The following Table 4 summarizes usability assessment methods that could be used

in combination to provide feedback on the potential of user distraction and mental

workload.

13


www.theAREA.org


Study

Table 4 Usability Tools extracted from 'Usability & Safety Tools (Ref)' from Safety Framework Tool

Assessment

Domain Tool Metrics

When to

use them

in Design

Cycle?

Notes

Situational

Awareness

Eye-tracker

(Eye point of

gaze)

Heat map, areas of

interest, TTFF,

Fixation Sequences

Design -

Testing

To determine where the operator is

looking

Situational

Awareness

Think Aloud

(Verbal

protocol)

Error rate, user

behaviour,

observation

Design -

Testing

A method used to gain insight how the

user behaves whilst navigating through an

UX/UI/system

Mental

Workload

Assessment

NASA -TLX Subjective, 7 point

scale

Design -

Testing

Questionnaire used to assess mental

workload on user

A full list of safety and usability assessment tools can be found in the ‘AREA Safety

Framework Tool’.

STEP 4: DELIVERY – DOCUMENT SPEC & TRAINING

To support safe use of the final solution, documentation and training must be

performed during the deployment phase. The documentation would include the

solution specification ‘as built’ along with the following details:

● Instructions on using the AR system ● Instructions on calibrating the system and recommended frequency of

calibration/updates ● Any health and safety guidelines and warning ● Risk Assessment before use (this may be done by safety manager rather than

solution provider) ● Maintenance and cleaning instructions

Prior to rolling out an AR solution, training for end users is highly important. Training

delivered by the solution provider or the client (depending on contractual agreement),

should include:

• Introduction to the AR device and its functional use (including advantages and limitations)

• Duration of optimum time of use

• Key safety risks such as distraction and eye strain

• Findings on potential increased mental strain and distraction (or any other safety and human factor based on assessments/testing)

• Emphasis on technical and hardware limitations that may pose safety risks such as FOV, obstruction to peripheral view etc.

For easing adoption of wearable AR displays, an approach incorporating a transition

period in which both current and AR-assisted methods would be used to allow

operators to become comfortable and confident in using the technology on the

workshop floor.

Commented [6]: Please highlight the tools that would measure distraction and cognitive strain, then state that there are other tools provided in the framework.

Commented [7]: What is a “handover” If it is a technical term (part of the project cycle?) then explain. Otherwise could say “project deployment” or implementation on the workshop floor.

14


www.theAREA.org


Study

WAY FORWARD

Augmented reality can be an effective tool in ensuring that procedures are followed

consistently and accurately as part of quality management processes and in

compliance with industry regulations. In assembly use cases, AR delivers value by

presenting information digitally in context to the user to guide them through complex

assemblies. This case study has shown a mock run through the AR project cycle for

an aerospace assembly use case to show how a solution could be designed with the

usability and safety at the center of the design. The approach emphasizes the topic of

safety and usability however it should be noted that there are many other functional

and non-functional needs (UI/UX design, security, hardware software agnosticism

etc.) requiring attention to create higher quality AR experiences across enterprises.

The AREA is driving the development of best practices and policies that organizations

could be used to govern AR safety and human factors assessments within the

workplace.

Some of the current AREA activity includes a safety committee in which members

collaborate to highlight safety unknowns, identify and prioritize safety risks and

generate resources to educate and guide the members and the wider AR ecosystem

on the topic. Similarly, a human factors interest group has been established in the

AREA Research Committee to explore and discuss new approaches to address the

usability issues in design of AR systems.

This project has identified and engaged with companies, research institutes, private

and public safety organizations inside and outside the AREA member network. As the

AREA expands activities focusing on the safety agenda in AR, it will seek deeper

discussions and collaborate with governmental organizations.

this resource is made publicly available to anyone seeking ... file1 this resource is made publicly...

Documents