usoap continuous monitoring approach - coscap-sea · cooperative development of operational safety...

Cooperative Development of Operational Safety and

Continuing Airworthiness Programme COSCAP South East Asia

USOAP Continuous Monitoring Approach

Briefing



Objective

Review ICAO USOAP Continuous Monitoring Approach Introduce revised / updated PQs (May 2013) and impact on

States’ LEI Annex 19 Introduce ICAO CMA Online Framework Filing of Differences (EFoD) USOAP Reports Actions Required of States CMA Training

COSCAP 2013 2



COSCAP 2013 3

8 Resolution

of Safety Concerns

1 Legislation

2 Operating

Regulations 3

Organization, Safety Oversight

Functions 5

Guidance, Procedures

& Info.

6 Licensing & Certification Obligations

7 Surveillance & Inspection Obligations

ESTABLISH

IMPLEMENT



USOAP Comprehensive Systems

Approach CSA

COSCAP 2013 4



Comprehensive Systems Approach

Traditional approach audit Fixed cycle Each ICAO Member in turn 6 year cycle Concentrated preparation Infrequent follow-up Results fixed at time of Report

COSCAP 2013 5



[State]

COSCAP 2013 6



USOAP CMA

COSCAP 2013 7



USOAP CMA

COSCAP 2013 8



USOAP CMA Components

COSCAP 2013 9

• CSA audits • Safety audits • ICAO

Coordinated Validation Missions (ICVMs)

• Online monitoring activities

• Mandatory Information Requests (MIRs)

• Protocol findings • Significant Safety

Concerns (SSCs) • Corrective Action

Plans (CAPs)

• Analysis of safety risk factors

• Evaluation of State’s safety management capabilities

• States • Internal

stakeholders • External

stakeholders Collection of safety information

Determination of State safety risk profile

Update of LEI and status of SSCs

Prioritization and conduct of USOAP CMA activities



Collection of Safety Information

COSCAP 2013 10



Collection & Processing of CMA Info

COSCAP 2013 11




States provide: The State Aviation Activity Questionnaire (SAAQ) Compliance Checklists (CCs) or Electronic Filing of

Differences (EFOD) The self-assessment checklist Updated CAPs

COSCAP 2013 12




Internal stakeholders include: ICAO Secretariat Bureaus/Sections Regional Offices (ROs)

COSCAP 2013 13




External stakeholders include: EASA; EC; EUROCONTROL; IATA; CANSO; Interstate Aviation Committee (IAC); Airports Council International (ACI); and other national, regional, supranational and international organizations

recognized by ICAO. Note: These organizations conduct activities that generate safety information.

COSCAP 2013 14




COSCAP 2013 15



State Safety Risk Profile

Safety risk factors

Safety risk indicators

Safety risk profile

Safety risk factors

• Previous USOAP activity results

• Level of traffic in the State

• Progress made by the State in resolving USOAP deficiencies

Safety risk indicators

• LEI vs. traffic (exposure) • Existence of SSC(s) • Level of aviation activities

for each audit area • Projected growth of

aviation activities • Level of acceptability of

State’s CAPs • Progress in

implementation of State’s CAPs

• Ongoing or planned assistance projects

• Major changes in the organizational structure

Safety risk profile

• Continuous Monitoring Office (CMO) determines the safety risk profile which is generated by determining safety risk factors and indicators.

COSCAP 2013 16




COSCAP 2013 17



Prioritization and conduct of USOAP CMA activities States are prioritized by CMO, based on their safety risk profile and information including: Input from the States/ROs on States’ progress in

implementing their CAPs Input from the States/ROs on States’ progress in resolving

identified SSCs Specific requests from States/ROs for conduct of a USOAP

CMA activity

COSCAP 2013 18



Prioritization and conduct of USOAP CMA activities Scope of an ICVM based on:

Level of aviation activity in the State State’s self-assessment Level of progress reported by State to implement CAPs Level of progress reported by States to address not satisfactory PQs Request by a State (cost-recovery ICVM)

Duration of ICVM determined by scope.

COSCAP 2013 19



Prioritization and conduct of USOAP CMA activities Scope of an Audit (safety/CSA) based on:

All relevant PQs Level of aviation activity in the State State’s self-assessment Request by the State (cost-recovery audit)

Duration of Audit is determined by scope

COSCAP 2013 20



USOAP CMA Components

COSCAP 2013 21

• CSA audits • Safety audits • ICAO

Coordinated Validation Missions (ICVMs)

• Online monitoring activities

• Mandatory Information Requests (MIRs)

• Protocol findings • Significant Safety

Concerns (SSCs) • Corrective Action

Plans (CAPs)

• Analysis of safety risk factors

• Evaluation of State’s safety management capabilities

• States • Internal

stakeholders • External

stakeholders Collection of safety information


Update of LEI and status of SSCs




Update of LEI and

Status of SSCs

COSCAP 2013 22



Update of Lack of Effective Implementation (LEI)

ICAO Validates collected safety information Validation enables ICAO to continuously update State LEI

status State LEI for the previous audit cycle is reported on iSTARS

COSCAP 2013 23




PQs have been revised and updated Applicable in May 2013. New / amended PQs will impact State’s LEI

deletion of some PQs new PQs merging of existing PQs

Detail later in Presentation

COSCAP 2013 24




Mandatory Information Request (MIR) can be issued by CMO when: SAAQ, CCs and/or PQs are not submitted, are out-dated or

are contradictory to other available information CAPs are not submitted or are not kept up-to-date by State Available information is insufficient Concerns are raised by internal / external stakeholders.

COSCAP 2013 25




Mandatory Information Request (MIR) States required to provide status of compliance with PQs “Manage State self-assessment” tool on the OLF CMO will communicate with States through MIRs to seek

additional information

COSCAP 2013 26




Status of PQs may be changed through the CMO validation process based on: CAPs or other information received from States, supported by

appropriate evidence Information received from ICAO ROs, recognized

organizations and other stakeholders information received from States in response to MIRs

COSCAP 2013 27



With the new online monitoring activities, CMO may review and validate some PQs related to CE-1 to CE-5 off-site and from ICAO HQ. However, validation of PQs related to CE-6 , CE-7, and CE-8 will typically require an on-site activity.


COSCAP 2013 28



Status of Significant Safety Concerns (SSCs)

“SSC occurs when the audited State allows the holder of an authorization or approval to exercise the privileges attached to it, although the minimum requirements established by the State and by the Standards set forth in the Annexes to the Chicago Convention are not met, resulting in an immediate safety risk to international civil aviation.” Reference: EB 2010/7 dated 19 February 2010

COSCAP 2013 29



Status of SSCs

SSCs unresolved in 12 States

SSCs identified by ICAO

SSCs resolved through corrective actions taken by the States

SSCs resolved by immediate actions taken by the States prior to being posted on the ICAO website

34

15

14

5

COSCAP 2013 30



Status of SSCs

Continuous monitoring process

Ongoing monitoring of evidence and information collected from the State and other sources

USOAP CMA on-site activity

Evidence collected points to a SSC • Team leader brings it to the attention

of the State as soon as it is discovered

• State may initiate corrective actions immediately

• Team leader provides all relevant information to C/CMO

SSC Committee convened to validate

Preliminary SSC

identified

COSCAP 2013 31



Audit Areas And

Critical Elements of the State Safety Oversight System

COSCAP 2013 32



Civil aviation organization (ORG)

SAAQ

Primary aviation legislation & civil aviation regulations (LEG)

Chicago Convention

Personnel licensing & training (PEL)

Annex 1

Aircraft operations (OPS) Annexes 6, 9, 18

PANS-OPS

Aircraft accident and incident investigation (AIG)

Annex 13

Airworthiness of aircraft (AIR)

Annexes 6,7,8 and 16

Air navigation services (ANS)

Ann 2,3,4,5,10,11,12,15 PANS-ATM

Aerodromes and ground aids (AGA) Annex 14

COSCAP 2013 33



Critical Elements of State Safety Oversight System

ICAO Audits to determine States safety oversight capabilities include: Assessing the effective implementation of the 8 CEs of a

safety oversight system Verifying State implementation of:

all safety-related ICAO SARPs associated procedures guidance materials best safety practices

COSCAP 2013 34



COSCAP 2013 35

8 Resolution

of Safety Concerns

1 Legislation

2 Operating

Regulations 3

Organization, Safety Oversight

Functions 5

Guidance, Procedures

& Info.

6 Licensing & Certification Obligations

7 Surveillance & Inspection Obligations

ESTABLISH

IMPLEMENT



CE-1: Primary aviation legislation The State shall promulgate a comprehensive and effective

aviation law, consistent with the size and complexity of the State’s aviation activity and with the requirements contained in the Convention on International Civil Aviation, that enables the State to regulate civil aviation and enforce regulations through the relevant authorities or agencies established for that purpose.

The aviation law shall provide personnel performing safety oversight functions access to the aircraft, operations, facilities, personnel and associated records, as applicable, of service providers.

COSCAP 2013 36



CE-2: Specific operating regulations The State shall promulgate regulations to address, at a

minimum, national requirements emanating from the primary aviation legislation, for standardized operational procedures, products, services, equipment and infrastructures in conformity with the Annexes to the Convention on International Civil Aviation.

Note The term “regulations” is used in a generic sense and includes but is not limited to instructions, rules, edicts, directives, sets of laws, requirements, policies, and orders.

COSCAP 2013 37



CE-3: State system and functions The State shall establish relevant authorities or agencies, as

appropriate, supported by sufficient and qualified personnel and provided with adequate financial resources. Each State authority or agency shall have stated safety functions and objectives to fulfill its safety management responsibilities.

The State shall ensure that inspectors are provided with guidance that addresses ethics, personal conduct and the avoidance of actual or perceived conflicts of interest in the performance of official duties.

Note In addition, Appendix 5 to Annex 6, Part I, and Appendix 1 to Annex 6, Part III, require the State of the Operator to use such a methodology to determine its inspector staffing requirements. Inspectors are a subset of personnel performing safety oversight functions.

COSCAP 2013 38



CE-4: Qualified technical personnel The State shall establish minimum qualification requirements

for the technical personnel performing safety oversight functions and provide for appropriate initial and recurrent training to maintain and enhance their competence at the desired level.

The State shall implement a system for the maintenance of training records.

COSCAP 2013 39



CE-5: Technical guidance, tools and provision of safety-critical information The State shall provide appropriate facilities, comprehensive

and up-to-date technical guidance material and procedures, safety critical information, tools and equipment, and transportation means, as applicable, to the technical personnel to enable them to perform their safety oversight functions effectively and in accordance with established procedures in a standardized manner.

The State shall provide technical guidance to the aviation industry on the implementation of relevant regulations.

COSCAP 2013 40



CE-6: Licensing, certification, authorization and/or approval obligations The State shall implement documented processes and

procedures to ensure that personnel and organizations performing an aviation activity meet the established requirements before they are allowed to exercise the privileges of a license, certificate, authorization and/or approval to conduct the relevant aviation activity.

COSCAP 2013 41



CE-7: Surveillance obligations The State shall implement documented surveillance

processes, by defining and planning inspections, audits, and monitoring activities on a continuous basis, to proactively assure that aviation license, certificate, authorization and/or approval holders continue to meet the established requirements. This includes the surveillance of personnel designated by the Authority to perform safety oversight functions on its behalf.

COSCAP 2013 42



CE-8: Resolution of safety issues The State shall use a documented process to take appropriate

corrective actions, up to and including enforcement measures, to resolve identified safety issues.

The State shall ensure that identified safety issues are resolved in a timely manner through a system which monitors and records progress, including actions taken by service providers in resolving such issues.

COSCAP 2013 43



Critical Elements of State Safety Oversight System

Definitions of the 8 CEs of a safety oversight system are now an Appendix of Annex 19

Guidance on the eight CEs is provided in ICAO Doc 9734, Part A

COSCAP 2013 44



Annex 19 – Safety Management

COSCAP 2013 45



The Air Navigation Commission, at the fourth and fifth meetings of its 190th Session on 8 May 2012, considered proposals developed by the Safety Management Panel (SMP) to transfer the provisions on safety management responsibilities and processes from existing Annexes for consolidation in new Annex 19 — Safety Management, and related consequential amendment proposals to existing Annexes developed by the Secretariat.

The proposed new Annex 19 and consequential amendments to Annexes 1, 6, 8, 11, 13 and 14, Volume I are envisaged for applicability on 14 November 2013.


COSCAP 2013 46




Given the increasing complexity of the global air transportation system and the interrelated nature of aviation activities required to assure the safe operation of aircraft, this Annex provides the means to support the continued evolution of a proactive strategy to improve safety performance.

The Standards and Recommended Practices (SARPs) in this Annex are intended to assist States in managing aviation safety risks, with the objective of continuously reducing the number of aviation accidents and incidents.

The foundation of a proactive safety strategy is based on the implementation of the State safety programme (SSP) of a State and will continue to rely on the successful implementation of existing international provisions and accident/incident investigation activities.

COSCAP 2013 47




The 4 components of the SSP framework are: State safety policy and objectives; State safety risk management; State safety assurance; and State safety promotion.

COSCAP 2013 48



SSP Roll-out

The roll-out will include details on how USOAP CMA will monitor the implementation of the SSP.

The ICAO Secretariat will be proposing to Council a detailed roll-out plan of the SSP to guide States in its implementation.

COSCAP 2013 49



USOAP CMA Audit Areas

COSCAP 2013 50



1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013+

PUBLIC

STATES

1997: Voluntary Assessment Programme, Fully Confidential (Annexes 1-6-8)

1999: USOAP Audit Summary Reports to all States (Annexes 1-6-8)

2005: USOAP CSA Audit results full transparency to all States

2006: SSC introduced, fast track notification to all States (restricted web site)

2001: Generic, non-State specific LEI results globally and by region

2005: Public access to LEI, Critical Element results by State. All States provided consent

2006: Mechanism to make full USOAP results available to the public with State consent. 1st cycle audits 45% of States

SSCs published on the CMA on line framework Proposed layout of the SSCs for the public to receive State feed back

2014 Unresolved SSCs to be made available to the public in the format and conditions approved by Council

ICAO has identified a significant safety concern with respect to the ability of [State] to properly oversee the [insert airlines (air operators); airports; aircraft; or air navigation services, as applicable] under its jurisdiction. This does not necessarily indicate a particular safety deficiency in the [insert airlines (air operators); airports; aircraft; or air navigation services, as applicable] but, rather, indicates that the State is not providing sufficient safety oversight to ensure the effective implementation of applicable ICAO Standards. Full technical details of the ICAO findings have been made available to [State] to guide rectification, as well as to all ICAO Member States to facilitate any actions that they may consider necessary to ensure safety. [State] has undertaken to regularly report progress on this matter to ICAO.

As of January 2013, safety oversight information is available on the ICAO public website.

COSCAP 2013 51



Revision of Protocol Questions (PQs)

COSCAP 2013 52



Revision of Protocol Questions (PQs)

PQ revision Change in PQ methodology Applicability (May 2013) PQ mapping and addressing revised PQs Impact LEI Score Actions required by States

COSCAP 2013 53



Overview of the PQ revision

In preparation for USOAP CMA, all PQs from the CSA cycle were revised and updated to reflect the latest changes in ICAO provisions.

The review also took into consideration comments from States, ICAO (ROs, ANB, etc.), USOAP audit team members and external stakeholders.

Under the USOAP CMA, PQs will be reviewed and updated on a periodic basis to reflect emerging issues and changes in Annex provisions and guidance materials.

COSCAP 2013 54



Overview of the PQ revision

The CSA cycle had a total of 975 PQs addressing the 8 audit areas of LEG, ORG, PEL, OPS, AIR, AIG, ANS, AGA.

The new USOAP CMA PQs have a total of 1016 PQs, including several revised PQs

Deleted PQs: 15 PQs merged with other PQs: 53 New PQs: 105

COSCAP 2013 55

5 PQs without a CE assignment have been deleted.



Finding & Recommendation based to

Protocol Based

Change in Methodology

COSCAP 2013 56



2.011 3.101 3.103 4.027 5.021 5.023 5.045 7.031 7.053 7.203 7.271 7.327

…

One CAP/CAP

update including:

Item 1 . . . . . . . . . . .

Item 20

Finding text

The current organizational structure of the

CAD is not determined in sufficient detail as to provide a

clear definition of the regulatory

and safety oversight

functions and responsibilities related to PEL, IOS, AIR< ANS

and AGA. In addition, the

available organizational

charts have not been approved

yet.

ORG/01

One CAP assessment

ORG/01 ORG

PEL

ANS

. . .

Finding text

One CAP/CAP

update

2.011 One CAP assessment

Finding text

One CAP/CAP

update


Finding text

One CAP/CAP

update


COSCAP 2013 57



PQ Mapping &

Addressing Revised PQs

COSCAP 2013 58



PQ Mapping & Addressing Revised PQs

Revisions to PQs have resulted in minor changes in LEIs of all States.

LEIs on the CMA OLF should be considered live, since their status can change on a continuous basis.

With the introduction of the revised PQs, States will regularly update their status of compliance and provide evidence on the CMA OLF (through the “Manage State self-assessment” and “Manage CAP” functionalities).

COSCAP 2013 59

States may also be required to respond to MIRs, when applicable.



PQ Mapping & Addressing Revised PQs

PQs that are currently “satisfactory” will remain satisfactory, unless new evidence results in a change of status.

Status of “not satisfactory” PQs may be changed by CMO on the basis of existing evidence provided for some PQs related to CE-1 to CE-5.

PQs related to CE-6, CE-7 and CE-8 will typically require an evaluation through an on-site USOAP CMA activity to verify effective implementation.

COSCAP 2013 60



Highlights of Changes Per

PQ Area

COSCAP 2013 61



General

Addition of new PQs on new ICAO Standards and Recommended Practices (SARPs) and guidance material

Moving of certain LEG and ORG PQs to other audit areas (i.e. OPS and ANS)

Harmonization of text for all PQs

COSCAP 2013 62



LEG

Area-specific PQs previously under LEG are now moved to the relevant audit areas, such as:

2 PQs on interception of civil aircraft → OPS and ANS; PQ on units of measurement → ANS; and PQ on journey log book and documents to be carried on board an aircraft → OPS.

COSCAP 2013 63



ORG

New PQs have been added on mechanisms to avoid perceived or potential conflict of interest of inspectorate staff.

5 PQs have been deleted that were not associated with any CEs (3 ORG, 1 PEL and 1 OPS; related information now obtained from State Aviation Activity Questionnaire or SAAQ).

COSCAP 2013 64



PEL

ICAO Doc 9379 — Manual of Procedures for Establishment and Management of a State’s Personnel Licensing System (Second Edition) has been published in 2012.

All references (in “ICAO ref.” column) have been revised to include ICAO Doc 9379.

COSCAP 2013 65



OPS

2 PQs on flight data recorders have been moved to AIR (AIR PQs 5.204 and 5.205)

These PQs have been moved from LEG to OPS: 1 PQ on interception of civil aircraft 1 PQ on journey log book 1 PQ on documents to be carried on board an aircraft

COSCAP 2013 66



AIR

New PQs have been added Examples:

Aircraft registration (identification plate and use of registration marks – AIR PQs 5.120 and 5.130); Safety Management System (SMS – AIR PQs 5.372 and 5.373); and AIR-related approval of CAT II and III instrument approaches (AIR PQs 5.330 and 5.331).

COSCAP 2013 67



AIG

New PQs have been added on procedures for inclusion of safety recommendations in draft final report forwarded to States and organizations concerned.

COSCAP 2013 68



ANS

Revisions include PQs on: New RVSM monitoring requirements; Performance-based navigation (PBN) implementation; Implementation of SMS/State Safety Programme (SSP); and Runway safety programme.

COSCAP 2013 69



AGA

PQs associated with multiple CEs (CE-2, CE-5, CE-6 and CE-7) are split into PQs associated with single CEs

New CE-2 PQs have been added to address all chapters of Annex 14, Volume I

New PQs have been added on minimum qualifications for aerodrome inspectors

COSCAP 2013 70



Actions Required by

States

COSCAP 2013 71



Actions required by States

Manage user accounts of the CMA online framework Keep up-to-date SAAQ and CC/EFOD Provide a CAP update and/or progress per corresponding

protocol finding Provide self-assessment result on each PQ with a priority on

the PQs that have a “Not Satisfactory” status Respond to MIR in a timely manner

COSCAP 2013 72



CMA On-line Framework

A suite of web-integrated applications and centralized database systems to support the implementation of CMA activities

Includes: Collection of safety-related information and documents from different sources Monitoring and reporting of safety oversight activities by Member States

COSCAP 2013 73



On-line Framework

COSCAP 2013 74



Access Control

NCMCs Have a full control of user accounts for the State

Issuing additional user accounts Deactivating invalid user accounts Setting individual user rights/permissions

Individual Users Manage user profile information Personalize password

Note: If you are not an NCMC and want to have access to the online framework, please contact the NCMC(s) of your State

75 COSCAP 2013



Access Control

User account composed of “ID or Email address” and password duplicate IDs or email addresses are not accepted

Access rights 3 types (no access, read-only or read/write) CC/EFOD : access rights can be granted per Annex PQ, CAP and F&R : access rights are permitted for all technical areas

76 COSCAP 2013



State Dashboard

States (NCMCs) to view at a glance: Status of CMA information completion Status of overall LEI/EI Status of CAP progress

77 COSCAP 2013



Update on Electronic Filing of Differences

(EFoD)

COSCAP 2013 78



States ICAO

Not at all or not properly file differences

Not publish significant

differences in AIPs

Misleading, inconsistent, insufficient guidance

Limited value of

Supplements

Frequent Annex Am’t

and unrealistic

applicability dates

Difficulty in comprehendin

g and interpreting

Annex provisions

Lack of knowledge

and expertise

Lack of effective national

process and procedures

Lack of States’

commitment

COSCAP 2013 79



Filing Differences – Transition

March 2011, Council agreed to use EFOD as an alternative means of filing of differences

Jan 2011, CC data from 153 States were migrated into the EFOD system

COSCAP 2013 80

Keep up-to-date compliance checklist pursuant to MOU for USOAP CMA

File differences pursuant to Article 38 of Chicago Convention



Filing Differences – Transition

In April 2011, States were requested to (SL AN 1/1-11/28): Recognize EFOD as an alternative means of filing of

differences Verify and confirm the data in EFOD which had previously

been submitted through CC under USOAP. Over 100 States have been updating data in EFOD 45 States sent replies (34 States completed verification, 11 States submitted plan for completion)

COSCAP 2013 81



Procedures on the Use of EFOD

Considered and approved by Council at its 195th Session

To be Published by Secretary General in 2nd Quarter 2013

Table of Contents 1. Introduction/Scope 2. Access to the System 3. Languages 4. Annex Information 5. Compliance/Differences Information 6. System performance 7. References

COSCAP 2013 82



Previous USOAP Report

Static Report Final reports based on onsite activities

conducted since the first cycle of USOAP

Published according to predetermined timelines after the closing meeting of an onsite activity

COSCAP 2013 83



CMA USOAP Report

Dynamic Report Based on the most recent online and/or onsite activity for each

audit area/CE

84 COSCAP 2013



USOAP CMA Computer Based Training (CBT)

Understanding of the USOAP CMA methodologies Essential knowledge required to participate in USOAP CMA

activities Enhance the competencies in the areas addressed by USOAP

CMA NCMC and others

COSCAP 2013 85



Questions & Comments

Kim Trethewey Chief Technical Advisor

COSCAP South East Asia

[email protected] +66 8 8686 0002

COSCAP 2013 86

mailto:[email protected]

usoap continuous monitoring approach - coscap-sea · cooperative development of operational safety...

Documents