tsa trainingprojects.erg.com/conferences/ambientair/...aug 16, 2018 · tsa overview (part 1)...

TSA Training NATIONAL AMBIENT AIR MONITORING CONFERENCE (NAAMC)

AUGUST 16, 2018

PORTLAND, OREGON

National Ambient Air Monitoring Conference August 2018


Images from Real Technical Systems Audits

All photographs and/or images are from real audits! They are for educational purposes only. No names or affiliations will be mentioned to protect the guilty parties, but hopefully they were all corrected and are in good shape now through a robust and effective corrective action program!

Not so fine print…

TSA Overview

• What is a TSA?• TSA Guidance Document• The TSA Process


What is TSA?


What is a Technical Systems Audit (TSA)?

A systematic and objective examination of an organization’s ambient air monitoring program to determine its compliance with both the ambient air monitoring regulations and the quality system requirements formalized in the organization’s QAPP


A primary objective of a TSA is to assess the suitability and effectiveness of the quality system being implemented by the

monitoring organization on behalf of EPA

In order to do that, the auditor must evaluate the monitoring organization against the specific policies and practices

established within its QAPPs and SOPs


• Looks at the monitoring program as a whole

• Assesses the auditee’s compliance with requirements governing the data operation

• Collection• Analysis• Validation• Reporting

What is a TSA?

Generated

Collected

Verified

Validated

Reported to AQS


What is a TSA?

•Highlights areas within a monitoring organization where it has shown innovation and improvement

• Identifies areas where monitoring programs can be strengthened

•Provides feedback to the organization


What Requires EPA to do a Technical Systems Audit?

40 CFR Part 58, Appendix A,§2.5

Technical systems audits of each PQAO shall be conducted at least every 3 years by the appropriate EPA Regional Office and reported to the AQS. If a PQAO is made up of more than one monitoring organization, all monitoring organizations in the PQAO should be audited within 6 years (two TSA cycles of the PQAO)

9National Ambient Air Monitoring Conference August 2018

Why Conduct TSAs?

•To ensure the monitoring organization is meeting regulatory requirements•To ensure the monitoring organization is meeting the requirements of its QAPP•To ensure the monitoring organization has a solid, functioning quality system •To ensure monitoring data is of adequate quality, defensible, and can stand up to public and legal scrutiny


TSA Benefits•Identifies problems and/or vulnerabilities•Increases confidence in the information / data obtained •Improves monitoring programs, networks, and quality systems•Provides a formal record of an independent assessment of the monitoring program•Provides decision-makers / data users context for their decisions•Promotes national consistency and data comparability•Serves as a mechanism to share good monitoring practices between organizations


TSA Overview (Part 1)


The TSA Quality Assurance Guidance Document (QAGD) provides guidance on how to conduct TSAs and includes tools for auditors to use to assess the various aspects of an air monitoring program

TSA QAGD


What is a TSA?

• Document finalized in November 2017• Product of a two-year effort by the EPA TSA Workgroup• TSA Workgroup members include:

• Representatives from all 10 EPA Regional Offices and OAQPS• Combination of QA and ambient air monitoring staff who

conduct TSAs• Created to foster national consistency and stringency among the

EPA Regions• Also prepared in response to feedback from S/L/Ts

TSA QAGD


What is a TSA?

•TSA Objectives•Roles and Responsibilities•Pre-Audit Activities•On-Site Audit Activities•Report Development and Corrective Action Process

•TSA Training•EPA Audit Communication

TSA QAGD Contents


What is a TSA?

•Revised TSA Questionnaire•Site Visit Logbook, including Appendix E siting criteria•Low-volume gravimetric laboratory audit checklist•Lead laboratory audit checklist•TSA Report Template•TSA Close-Out Letter Template

TSA QAGD Appendices


TSAs Have Two Phases

Phase 1 - Conducting the TSA

Phase 2 - Corrective Actions


Summary of EPA Regional Office

Pre-Audit Activities

~90 days total


TSA Scheduling•EPA Regional Office (RO) typically sets TSA schedule at beginning of fiscal year

• Varies per EPA Region, based on number of PQAOs • Each PQAO is audited once every 3 years per CFR

•RO notifies PQAOs of the TSA schedule after it is formulated•RO management assigns lead auditor for each TSA and assigns audit team •Team members selected based on scope of individual TSA and expertise of staff

• For example, if agency has a gravimetric laboratory, an auditor is assigned to the team with particulate laboratory expertise

• Minimum of 2 individuals per audit, best practice•Lead auditor is responsible for planning, coordinating, and managing the overall audit (i.e., project), as well as leading the audit team


TSA Planning • Typically begins 3 months in advance of the scheduled on-site investigation

• Lead auditor determines the scope of the TSA by answering 3 basic questions:

• What aspects of the monitoring program will be covered during the on-site visit? • What time period is under review? • What issues were identified during the last TSA that warrant follow-up?

• Previous TSA report and other resources, such as the monitoring organization’s Annual Monitoring Network Plan, are reviewed in order to answer these questions and direct focus of the audit

• Number of days allotted by management and size of the audit team will also impact the scope of the TSA


TSA Planning • Lead auditor establishes communications with auditee (~3 months in advance)

• Confirms specific dates for on-site investigation

• Requests documents (QMP, QAPPs, SOPs), maps, organizational charts, etc to help with planning

• Provides TSA questionnaire to auditee and establishes due date (~1 month prior)

• Lead auditor (solo, or collectively with audit team) devises a specific audit plan based upon scope of TSA

• Anticipated dates/times for entrance and exit briefings

• Specific facilities, laboratories, and air monitoring stations to inspect on site

• Refined audit plan is shared with monitoring organization typically ~1 month prior to audit


Technical Preparations•AQS Data Review

• Various assessments pulled• List of specific data points to target

prepared for the audit of data quality• Will include data that appears to be

anomalous or suspect in AQS• Will include data randomly selected for the

purposes of data tracking• May also include data that appear to exceed

pollutant standards


Technical Preparations•TSA Questionnaire responses reviewed upon receipt

•Responses highlighted that necessitate further review or discussion during the on-site TSA

•Examples earmarked may include:• Ambiguous responses• Unanswered questions• Answered “no” when “yes” was expected (or vice

versa)


Technical Preparations•Quality System Document Review

• Includes QMPs, QAPPs, and SOPs• Number and size of documents will vary by PQAO,

depending on the complexity of the monitoring network• Audit team members should begin studying documents

immediately upon receipt• ~3 months in advance, recommended• Ideally, should read these documents before completing

data review activities – b/c documents should shed light on the coding that will be present in the data!


During the TSA, the auditor is charged with assessing whether or not the monitoring organization is implementing the QAPP and its referenced SOPs

To do so, the auditor should become fluent in the documents prior to the on-site TSA, in order to effectively assess how procedures are implemented on the ground


Audit Team Meetings •Held throughout the 3 months of preparation, two minimum

• Kick-off meeting to discuss logistics and delegate team responsibilities

• Final meeting ~1 week prior to on-site visit• Review and verify audit logistics• Ensure team members have necessary tools, logbooks, etc• Discuss concerns and issues identified during technical review

activities – including documents, data, and questionnaire responses• Prepare preliminary findings summary to help guide the on-site

investigation


Break time!

On-Site Investigation


Flow Diagram of Typical On-Site

TSA


On-Site Investigation •The on-site TSA will cover a lot of territory in a short amount of time• Number of days on site varies per

PQAO/scope• Typically 1 business week

•Thorough pre-audit preparation and planning promotes efficiency and effectiveness during the on-site TSA


Core On-Site Activities•Entrance Briefing

•Tour of Facilities

•Questionnaire Discussion

•Staff Interviews

•Records Review

•Data Review

•Ambient Monitoring Station Inspections

•Exit Briefing


What is a TSA?

•Proper siting of monitoring stations, samplers, and probes (Appendix E requirements)

•Use of approved analyzers and samplers for monitoring objective (FRM,FEM)

•Operation of analyzers and samplers according to FRM/FEM requirements

•Adherence to SOPs

•Site housekeeping

•Age of equipment

Field Inspections•Maintenance capabilities

•Shelter condition & safety concerns

•Operator cross-training

•Documentation


What is a TSA?

•Use of appropriate analytical method

•Use of appropriate analytical equipment

•Adherence to analytical SOPs

•Analyst cross-training

•Maintenance capabilities

•Laboratory housekeeping

•Age of equipment

•Sample storage

•Documentation

Laboratory Inspections


What is a TSA?

Certification RecordsNIST-TRACEABILITY

A review of documentation of individual standards utilized within the network. Includes: • Photometers• MFCs• Calibration gases• Flow orifices• Mass references standards• Others

•Demonstrate a continuous chain of certification records for the time period under review (e.g., 3 years)

•Auditor will review the certification / expiration dates on the certificates of traceability (calibration), looking for gaps in time sequence

•If gaps are identified, further investigation takes place


What is a TSA?

Data Review and AQS Reporting

•Methodically trace the complete history of specific data points to confirm that all steps in the collection process pass QC checks, calibration requirements, traceability requirements, and collection requirements

•Ensure records and documentation justify and support the validity, flagging, or invalidation of the specific data points reviewed

Review records / documentation to see if it supports the AQS coding for the sample in question. Includes:

•Logbook documentation

•Data forms

•Chain-of-custody logs

•Analyzer chart traces (minute data)

•Other agency information / records

DATA TRAIL TRACKING GENERAL DATA INQUIRIES


Method WitnessWhen possible during a TSA, the auditor should observe procedures performed by the site operator in the field or the analyst in the laboratory, in order to witness techniques and confirm adherence to SOPs.


What is a TSA?

Records ReviewTYPES OF RECORDS QUERIED

•Paper and/or electronic

•Certification (traceability) records

•QA/QC records (calibrations, QC checks, maintenance)

•Logbooks

•Chain-of-Custody

•Audit Reports / Assessments

REVIEW INCLUDES:

•Are records current and available for all years under review?

•Are records organized and can be retrieved quickly?

•Are records completely documented, and contain enough detail to support and justify decisions?

•Are records secured?

•Archival system in place?


What is a TSA?

Note about Certification RecordsNIST-TRACEABILITY

A review of documentation of individual standards utilized within the network. Includes: • Photometers• MFCs• Calibration gases• Flow orifices• Mass references standards• Others

•Demonstrate a continuous chain of certification records for the time period under review (e.g., 3 years)

•Auditor will review the certification / expiration dates on the certificates of traceability (calibration), looking for gaps in time sequence

•If gaps are identified, were the standards used for QA / QC checks?


What is a TSA?

Exit Briefing•Face-to-face meeting with monitoring organization personnel (usually upper management) and the EPA audit team

•Additional EPA personnel, such as RO management, may attend

•Agenda should include:

• Introductions of staff in attendance

• Presentation of audit results

• Discussion of the TSA report process

• Discussion of the required Corrective Action Plan and follow-up process

• Closing dialogue to address questions and comments from the monitoring organization


Break Time!

What is a TSA?

Report Development •Goal is to provide written feedback to the monitoring organization within 30 calendar days of the on-site investigation

•Feedback can be in the form of a briefing sheet, Excel spreadsheet, or Word document watermarked as “draft”

•Auditee is provided the written feedback for ~15 calendar days for fact-checking

•Auditee should submit any necessary corrections to the EPA RO in writing

•Report finalized within ~30 calendar days of receipt of any comments from auditee


Summary of Post-Site Visit

Activities


Phase 2: Corrective Actions


Corrective Actions•EPA Regional Office Responsibilities

• Oversight of Corrective Action Process• Continued Communications and

Follow-Up• Tracking Completion of Action Items• Assess evidence submitted• Documentation of each item using

RO’s established mechanism• TSA Close-Out Letter and AQS Update

•Monitoring Organization Responsibilities

• Performing Corrective Actions• Corrective Action Plan (CAP)

and implementation• Progress Reports / Status

Updates


•Corrective action phase critical to the success of the monitoring organization

•Organization develops a CAP to address TSA findings

•EPA auditor provides continued oversight and support• Reviews CAP upon receipt & assesses adequacy• Tracks progress of action items• Assesses adequacy of implemented corrections• Periodic conference calls to check in

Corrective Action Plan (CAP)


Multiple ways to track completion of action items. Examples include:• Audit Finding Response Forms• Spreadsheets• Databases

Serves as a mechanism to ensure all identified findings are resolved

Action Item Tracking


Once all identified TSA findings have been satisfactorily addressed, the EPA Regional Office should issue an official “close out” letter to the audited organization

The letter states that the organization has provided the necessary documentation to illustrate that required corrective action measures were successfully implemented; with that, the TSA is considered officially “closed”

TSA Close-Out


What is a TSA?

TSA Close-Out•The written TSA report serves as official documentation to demonstrate that the requirements of 40 CFR Part 58, Appendix A, §2.5 have been fulfilled – for both EPA and the audited organization

•CFR-required TSAs must be reported to AQS

•TSA close-out information should be entered into AQS upon issuance of the close-out letter


What is a TSA?

TSA Close-Out in AQSFive TSA-related parameters are reported:

•Auditee

•Auditor

•Audit begin date

•Audit end date

•Close-out date

Allows AQS users to quickly see when TSAs have been conducted, and when corrective actions have or have not been completed


Conducting Internal Systems Audits

(ISAs)


TSAs aren’t just for EPA…

•One of the best practices an air monitoring organization can implement is to conduct internal systems audits (ISAs) on a routine basis!•Include ISAs in the QAPP


Mirror the EPA!

•Use same approach as the federal TSA!

•Develop a TSA audit form based on your organization’s quality system requirements (QAPPs and SOPs)

•Develop an audit schedule • Recommend annual audits, at minimum

•Document findings in internal reports routed through organization’s chain-of-command


Internal Systems Audit Benefits

•Illustrates areas where supplemental training may be beneficial

•Prevents data loss

•Improves overall data quality•Enhances quality system

•Small issues won’t become big issues!

•Significantly reduces EPA findings during the regulatory TSA!

National Ambient Air Monitoring Conference August 2018National Ambient Air Monitoring Conference August 2018National Ambient Air Monitoring Conference August 2018

Audit Scenarios•Multi-organizational PQAO

• ISA of individual monitoring agencies• ISA of specific pollutant operations across PQAO

•Large PQAO consisting of district/regional offices• ISA of individual district offices & the sites within their jurisdiction

•Local agency PQAO• ISA of operations at individual sites• ISA of individual operators

•Or, all sites / operators once per year


Audit Scenarios

•In-house laboratory• Operator or monitoring staff weighing PM10 or PM2.5 samples

•Inter-departmental laboratory• Toxics analyses• Criteria Pb analyses• Gravimetric method

•Contract laboratories


Gather Your ToolsQAPP / SOPsCFRValidation TemplatesFRM/FEM DesignationsAudit FormsAQS Data Cameras, range finders, measuring tapes, compasses, etcTime

Conducting TSAs: Pre-Audit


Review previous EPA TSA Report(s) – or other external audit reports, if available (e.g., NPAP, PEP, APEs) – and familiarize yourself with the findings

Look for those same issues during your audit to ensure they have been completely resolved – or

that they have not recurred!


TSA QAGD Appendix A: TSA Questionnaire

Use the questionnaire to audit yourself!


Or, develop your own unique questionnaire using your QAPP as a guide!

It can be completed by the auditee or used as a checklist for the auditor. Or both!


Developing ChecklistsDecide what you plan to audit!•The agency as a whole?•Individual sites?•Individual operators?

Potential Audit Categories:•Housekeeping

•Grounds & Safety

•Instrument-specific operations

•Calibrations

•Routine QC & Maintenance

•Documentation


Use your agency’s SOP requirements to develop questions for audit checklist(s) that are specific to your program!


Specific QuestionsSOP Requirements

Audit Checklists


Individual Method Checklists

Utilize those available in the TSA QAGD or make your own!



Know the methods you are auditing!Auditor selected should be proficient in the method under review Will help ensure technical components of the program are being implemented correctly!


Certification Records Review

• Traceability records!• Look for a continuous, unbroken chain of certifications! • Select a percentage of instruments to review – including site

(QC) and audit (QA) devices• Certifications for an individual instrument – are there any

gaps in dates? If so, was the instrument used during the time it was expired?

• Select an individual site – and trace the calibrator(s) at the site to ensure all procedures were performed with a certified device



Standards Needing Certification

•Photometers•Calibration gases (cylinders)•Calibrator Mass Flow Controllers (MFCs)

•Other flow rate devices•Thermometers •Barometers•Mass reference standards (i.e., check weights)

•Others


Documentation should be available to track the “life” of

all valid sample concentrations, as well as

justify concentrations which were flagged or invalidated

An audit of data quality (ADQ) tracks this chain of records!

NIST-Traceable Calibration

NIST-Traceable QA/QC Checks

Supporting Documentation

Valid Pollutant Concentration


What is a TSA?

Data Review and AQS Reporting

•Methodically trace the complete history of specific data points to confirm that all steps in the collection process pass QC checks, calibration requirements, traceability requirements, and collection requirements

•Ensure records and documentation justify and support the validity, flagging, or invalidation of the specific data points reviewed

Review records / documentation to see if it supports the AQS coding for the sample in question. Includes:

•Logbook documentation

•Data forms

•Chain-of-custody logs

•Analyzer chart traces (minute data)

•Other agency information / records

DATA TRAIL TRACKING GENERAL DATA INQUIRIES


Develop Your Target List!• Pull your organization’s data from your agency’s data management software• And/or, pull data out of AQS!

• Mark data that catch your eye! • Review your ear-marked selection for trends or issues• Common / repeating issues? (Systemic problem)• Anomalies?

• Develop a list of data points to scrutinize based on issues seen in the data (i.e., general inquiries)

• Some of these data points can also used for the AQD (i.e. data trail tracking)



Helpful AQS Reports• AMP 350 (Raw Data Report)• AMP 251 (QA Raw Data Assessment Report)• AMP 430 (Data Completeness Report)• AMP 503 (Extract Sample Blank Data)• AMP 504 (Extract QA Data)• AMP 256 (QA Data Quality Indicator Report)



Sort the Data in a Spreadsheet

Action Type Site Param POC Date Days Method UnitMonitor

ConcActual Conc

% Diff

1-Point QC 3002 44201 1 17-Jul-13 15 047 008 73.4 75 -2.1%1-Point QC 3002 44201 1 22-Jul-13 5 047 008 74.4 75 -0.8%1-Point QC 3002 44201 1 22-Jul-13 0 047 008 74.8 75 -0.3%1-Point QC 3002 44201 1 31-Jul-13 9 047 008 73 75 -2.7%1-Point QC 3002 44201 1 8-Aug-13 8 047 008 82.4 75 9.9%1-Point QC 3002 44201 1 8-Aug-13 0 047 008 75 75 0.0%1-Point QC 3002 44201 1 21-Aug-13 13 047 008 74.4 75 -0.8%1-Point QC 3002 44201 1 9-Oct-13 49 047 008 105 75.6 38.9%1-Point QC 3002 44201 1 9-Oct-13 0 047 008 74 75.6 -2.1%1-Point QC 3002 44201 1 14-Oct-13 5 047 008 41.8 76.2 -45.1%1-Point QC 3002 44201 1 14-Oct-13 0 047 008 75.8 75.8 0.0%1-Point QC 3002 44201 1 30-Oct-13 16 047 008 75 75.8 -1.1%1-Point QC 3002 44201 1 19-Nov-13 20 047 008 74 74.8 -1.1%



Or Use Other Tools Available on AMTIC!https://www3.epa.gov/ttn/amtic/qareport.html


AMP 504 Extract Tool


Things to Look for in Your Data…

•75% data completeness •1-point QC checks % difference acceptance criterion•1-point QC checks frequency and concentration range•Performance Evaluation (audit )frequency •Audits concentrations (e.g., 3x MDL, 99th percentile)•Flow rate verifications % difference acceptance criteria•Flow rate verifications frequency•Flow rate audit frequency and acceptance criteria



More Things to Look for in Your Data…

•Determine the highest concentration site for each pollutant

•Identify any sites that have incomplete design values (DVs)

•Identify site with DVs that are uncommonly low•Review 1-hour SO2 data against its associated 5-minute data

•Examine PM2.5 blank data for exceedances

•Examine PM2.5 lab blank data (request from lab!)•Examine ambient concentration data for adherence to SOPs!



The AMP 350 Tells a Story

•Missing data (i.e., gaps in reported concentrations)?

•Repeating concentration values?

•Unusual maximums concentration values?•Consistency in application of null value codes and qualifier flags?

•Do the null value codes, through the “story” they convey when read, indicate that SOPs have been followed?



Example AQS AMP 350Code change?

What malfunctioned?Where is maintenance & recalibration?

Data should be coded in a manner that most accurately represents what happened


Visit the Sites – and take measurements!Appendix E siting criteria! Can take measurements during: • A systems audit at a site• A performance audit of

an instrument• A Method witness of an

operator • Or, all sites once per

year as a stand alone activity!


Unannounced AuditsConducting TSAs: Pre-Audit

• One of the best ways to truly assess the day-to-day monitoring operations is to conduct audits without prior scheduling

•Allows the auditor to see the true condition of the shelters / facilities / programs before they’re been “cleaned up” in preparation for a scheduled audit


Network!•Talk to personnel in other monitoring organizations to learn how they complete internal systems audits • Share ideas, techniques, and tools

•Read the TSA QAGD

•Talk to your EPA Regional Office Auditor!• EPA audit staff are willing to assist you in enhancing your QA

programs• Your auditor can offer suggestions on various audit techniques


Questions?Greg Noah,OAQPS

Stephanie McCarthy,EPA Region 4


tsa trainingprojects.erg.com/conferences/ambientair/...aug 16, 2018 · tsa overview (part 1)...

Documents