2018 focus on compliance - college of american...

2018 Focus on Compliance

Karl Voelkerding, MD, FCAP Birgit Funke, PhD, FACMG Avni Santani, PhD, FACMG

September 12, 2018

Next Generation Sequencing: What Does Compliance Look Like?

Disclosure

© 2018 College of American Pathologists. All rights reserved.

The following authors/planners/reviewers have financial interests/ relationships to disclose:

Andrew Goodwin, MD, FCAP

Haematologic Technologies

Medical Director Monetary stipend

Lee Hilborne, MD, MPH, DLM(ASCP) Avni Santani, PhD, ABMGG Karl Voelkerding, MD, FCAP

Quest Diagnostics CHOP Veritas Agilent PierianDx

Employee Consultant Consultant License agreement Scientific advisor

Compensation, stock Consulting fee Consulting fee Fees Consulting fee

2

Disclosure, cont’d.


Birgit Funke, PhD, ABMGG, ACMG

Michael Talbert, MD

Gregory Gagnon, MD, FCAP

Carlos Machicao, MD, FCAP

S. Robert Freedman, MD, FCAP

Jacob Mayer, Jr PhD, HCLD

Kenneth Klein, MD, FCAP

Justin Rueckert, DO, MLS(ASCP)cm

Ritu Nayar, MD, MBBS, FCAP Marian Briggs, MS, MT(ASCP)

The following authors/planners/reviewers have no financial interests/relationships to disclose:

3

Accreditation The College of American Pathologists (CAP) is accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide continuing medical education for physicians.

CE (CONTINUING EDUCATION FOR NON-PHYSICIANS) The CAP designates this educational activity for a maximum of 1 credit/hour of continuing education. Each participant should only claim those credits/hours he/she actually spent in the activity.

ASCP STATEMENT The American Society for Clinical Pathology (ASCP) Board of Certification (BOC) Certification Maintenance Program (CMP) accepts this activity to meet the continuing education requirements.

CALIFORNIA AND FLORIDA STATEMENT This activity is approved for continuing education credit in the states of California and Florida.

© 2018 College of American Pathologists. All rights reserved. 4

Today’s Presenter Karl Voelkerding, MD, FCAP

Dr. Voelkerding is a professor of pathology at the University of Utah School of Medicine and past president of the Association for Molecular Pathology. He is the former chair of the College of American Pathologists’ Next-Generation Sequencing Project Team, which developed laboratory accreditation requirements and proficiency testing programs for clinical next-generation sequencing. Dr. Voelkerding is currently chair of the College of American Pathologists’ Genomic Medicine Resource Committee. Dr. Voelkerding received his MD from the University of Cincinnati College of Medicine and is board certified in clinical pathology and molecular genetic pathology. His research focuses on the translation of genomics technologies into clinical diagnostics.


Today’s Presenter


Birgit Funke, PhD, FACMG • Associate Professor of Pathology

(part-time), Harvard Medical School • Vice President of Clinical Affairs,

Veritas Genetics

6

Today’s Presenter Avni Santani, PhD, FACMG

• Director - Clinical Laboratories, Strategic Partnerships and Innovation

• Director - Center for Applied Genomics, The Children’s Hospital Of Philadelphia

• Associate Professor of Clinical Pathology & Lab Medicine

• Education - Perelman School of Medicine, University of Pennsylvania


Objectives

• Explain the accreditation requirements for Next Generation Sequencing (NGS)

• Describe practical steps to ensure appropriate implementation of NGS testing

• Demonstrate compliance with the accreditation requirements


• First Published in the 2012 edition of the Molecular Pathology Checklist

• Revised Yearly as NGS Clinical Testing Has Advanced

CAP NGS Laboratory Accreditation Requirements


• Guiding Principle – Finding the Right Balance

• Foster Innovation and Technology Adoption – Assure Patient Safety



Wet Bench Process • Sample Handling

• Library Preparation • Sequence Generation

Bioinformatics “Dry Bench” Process • Sequence Alignment to Reference

• Variant Identification • Variant Annotation (eg CFTR

p.Arg205Lys)

Data Interpretation and Reporting

Medical Practice

FASTQ File

Annotated Variants

Analytical

Laboratory Process

CAP Definition of a NGS Test


Majority of Laboratories Perform NGS Testing

In One Primary Physical Laboratory

Wet Bench Process Sequencing

Bioinformatics Process

Interpretation and Reporting

Single CLIA License

Data Integrity Security Privacy


Some Laboratories Outsource Individual Process Steps



Interpretation Reporting

Primary Laboratory

Reference Laboratory

Primary Lab Receives Patient Sample

Outsource




Primary Laboratory



Outsource




• Written to Accommodate Diversity of Testing Configurations

• Single Laboratory to Primary and Reference Laboratory



Diversity

15





Primary Laboratory



Outsource

“Distributive” Testing Model Concept

CAP Requires Integrated Methods Based Validation © 2018 College of American Pathologists. All rights reserved. 16



Primary Laboratory



Outsource



“Distributive” Testing Model Concept

CAP Requires Integrated Methods Based Validation © 2018 College of American Pathologists. All rights reserved. 17

Primary/Referring Laboratory

NGS Reference Lab

Policy

NGS Confirmatory Testing Policy

NGS Sample Provenance

Tracking

General

Exception Log/Record

Data Storage Local/Cloud

Data Transfer Confidentiality

Wet Bench

NGS Wet Bench Process

Documentation


Validation

Quality Management

Plan

Laboratory Records

Monitoring of Upgrades

Bioinformatics

NGS Bioinformatics

Pipeline Documentation

NGS Bioinformatics

Pipeline Validation

Quality Management

Plan

Pipeline Version Traceability


Interpretation

Sequence Variants


Reporting of Incidental Genetic Findings

Current CAP NGS Laboratory Accreditation Requirements - 2018



NGS Reference Lab

Policy



Tracking

General




Wet Bench


Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan



Interpretation

Sequence Variants









p.Arg205Lys)


FASTQ File

Annotated Variants

CAP Requires an Integrated Methods Based Validation

Integrated

Methods Based Validation






p.Arg205Lys)


Integrated

Methods Based Validation

FASTQ File

Annotated Variants

Validation Specimens Contain a

Representative Spectrum of Variant

Types Test is Designed to Detect

Establish Assay Performance



NGS Test Validation – Choice of Specimens

Validation Specimens

Contain a Representative Spectrum of Variant Types Test is Designed to Detect

For Gene Panels Include Specimens that

Contain Common Mutations

(e.g. Cancer Hot Spots, CFTR p.Phe508del)

Methods Based

Gene or Analyte Based

Hybrid


Add Extra Specimens that Contain Variants That are Technically

Difficult to Detect


• We are sequencing every base instead of assaying known disease variants -- novel variant detection – Requires new validation approaches

– Methods–based validation

• The amount of data exceeds ability to process it manually – Increased reliance on computational resources and bioinformatics

• Changing role of laboratory director – Moving from predominantly technical focus to increasingly interpretive role

• Ability to sequence large numbers of genes – Increased burden on test content design

NGS has changed many things….

We need to develop new approaches and standards


Adapted from: Rehm et al. 2013

LIFECYCLE OF AN NGS BASED CLINICAL TEST


TARGET

CATPURE

TEST VALIDATION

LIBRARY PREP CAPTURE SEQ ANALYSIS

QUALITY MANAGEMENT

PATIENT TESTING

QC PT

GEN

E P

NL

- A

GEN

E P

NL

- B

GEN

E P

NL

- C

E X

O M

E

G E

N O

M E

P L A T F O R M (cumulative data)

Optimize components and entire test • General assay conditions • Coverage • Sample pooling • Analysis setting/thresholds Establish protocol for entire workflow

Validate entire test using set conditions • Sensitivity • Specificity • Robustness • Reproducibility Include samples representing • Different variant types (in/dels, substitutions, CNVs) • Disease specific aspects (common pathogenic variants)

TEST OPTIMIZATION

Quality Control (QC) • Every run

Proficiency Testing (PT) • Periodically

TEST DESIGN

Technical design

Content design

Added dimension Greatly increased complexity

24

A GROWING LIST OF PROFESSIONAL NGS GUIDELINES

Organization/Entity PMID Year Title

CDC 23138292 2012 Assuring the quality of next-generation sequencing in clinical laboratory practice.

CDC 26154004 2015 Good laboratory practice for clinical next-generation sequencing informatics pipelines.

ACMG 23887774 2013 ACMG clinical laboratory standards for next-generation sequencing.

ACMG/AMP 25741868 2015

Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology

AMP/CAP 28341590 2017

Guidelines for Validation of Next-Generation Sequencing-Based Oncology Panels: A Joint Consensus Recommendation of the Association for Molecular Pathology and College of American Pathologists.

AMP/CAP 29154853 2018

Standards and Guidelines for Validating Next-Generation Sequencing Bioinformatics Pipelines: A Joint Recommendation of the Association for Molecular Pathology and the College of American Pathologists.

CAP 25152313 2015 College of American Pathologists' laboratory standards for next-generation sequencing clinical tests.

CAP 28322587 2017 Development and Validation of Targeted Next-Generation Sequencing Panels for Detection of Germline Variants in Inherited Diseases.

CAP 28362156 2017 Development and Validation of Clinical Whole-Exome and Whole-Genome Sequencing for Detection of Germline Variants in Inherited Disease.

CLSI N/A 2014 MM09: Nucleic Acid Sequencing Methods in Diagnostic Laboratory Medicine, 2nd Edition

New York State N/A 2015 Guidelines for Validation Submissions of Next Generation Sequencing (NGS) assays under the NYS Testing Category of Genetic Testing– Molecular

FDA N/A 2016 Use of Standards in FDA Regulatory Oversight of Next Generation Sequencing (NGS)-Based In Vitro diagnostics (IVDs) Used for Diagnosing Germline Diseases

Santani et al. 2018 (manuscript under review, JMD) © 2018 College of American Pathologists. All rights reserved. 25

THERE IS A NEED FOR MORE….

….. concrete guidance • Labs want to do the right thing but it’s not easy to

translate recommendations into an SOP for how to set up high quality NGS testing

• Need step by step instructions that “get into the weeds”



THE CAP CHECKLIST HAS TRIED TO ACCOMMODATE THIS NEED

From the 2017 CAP checklist:

This is a SINGLE requirement

27

EVOLUTION OF THE CAP NGS CHEKLIST (MOL.36010 – Analytical Wet Bench Process Documentation)


2013 2015

The complexity of NGS is not compatible with what the checklist was designed for

2017

28


**REVISED** 08/21/2017 MOL.36015 NGS Analytical Wet Bench Process Validation Phase II The laboratory validates the analytical wet bench process and revalidates the entire process and/or confirms that the performance of the components of the process is acceptable when modifications are made. NOTE: The output of the NGS analytical wet bench process is a collection of sequence data that requires additional bioinformatics processing and analysis to determine whether the sequence is of sufficient quality and quantity for the intended test. To determine this, and to ensure acceptable beginning-to-end test performance, validation of the NGS analytical wet bench process must be integrated with the bioinformatics process validation for the intended test (see MOL.36115). The analytical wet bench process and the bioinformatics process for a test may occur within a single laboratory, or in a combination of primary and referral laboratories (see MOL.35840). Whether performed in a single laboratory, or in a distributive model involving primary and referral laboratories, the validations of the wet bench and bioinformatics processes for an intended test must be integrated to ensure acceptable beginning-to-end test performance. It is the responsibility of the laboratory director or designee meeting CAP director qualifications to review and approve all validations relevant to the intended test for processes performed within their laboratory, and to review all validations relevant to the intended test for processes performed in referral laboratories, if applicable. Laboratories validating NGS tests must comply with MOL.31015 and/or MIC.64770, as applicable, and meet the requirement that validations are performed with samples for each type of specimen expected for the assay (e.g. blood, fresh/frozen tissue, paraffin embedded tissue, prenatal specimens, saliva, culture isolates). Validations can be augmented by, but not supplanted with, additional reference standards (e.g. cell lines such as NIST NA12878). • Analytical validations must consist of a baseline methods-based validation that establishes the test's general

performance for the detection of the sequence variant type(s) that the test is designed to identify. For tests that are offered for specific clinical indications (e.g. diagnostic gene panels) the laboratory should investigate whether any analyte, disease or gene specific needs exist that necessitate inclusion of additional specimens or reference samples in the validation. For example, this may include samples containing prevalent pathogenic variants, especially if they are technically difficult to accurately detect (e.g. large indels). For detailed requirements on validation see MOL.36115 (NGS Analytical Bioinformatics Process Validation).

“…I understand what a methods-based validation approach is - but how exactly do I go about this?”

YET….WE ARE DEFINING MOSTLY WHAT NEEDS TO BE DONE, BUT NOT HOW…

29

A “COMPANION RESOURCE” FOR THE CAP CHECKLIST

Manuscript in revision at the Journal of Molecular Diagnostics


FILLABLE WORKBOOKS THAT GUIDE THE WAY THROUGH THE NGS TEST LIFECYCLE


TEST DESIGN


Background Detailed Instructions 32


This information is critical for appropriate test design, examples: - CNVs rare: supplemental assay less important - Gene not well associated with disease –careful when including

into gene panel !

Goal: ensure both clinical and technical validity

TEST DESIGN

33


Now you’ve chosen your genes – preparing for technical design

TEST DESIGN

34


TEST VALIDATION

35


Formulas to help translate

Helpful comments and concrete examples

Common sense translation

TEST VALIDATION

36


TEST VALIDATION

37


TEST VALIDATION

• Build your validation runs • Start with assembling sample list • Annotate with pertinent information

- Which validation “layer” is served by the sample - Specific variants of interest present? - Which metric is served (intra/inter-run variability)

38


• Place samples on a run grid • Example here:

• HiSeq 2500 • 10 samples/lane • Run 1: Same libraries run 3X • Run 2+3: New libraries of same samples

TEST VALIDATION

39


METHODS-BASED TEST VALIDATION -PERFORMANCE METRICS SUMMARY

40

NEXT STEPS? Short Term • Make worksheets publicly available • Invite comment from users (public access, CAP

website) • Adapt to incorporate other applications (somatic

NGS)

Medium Term: Make it a “living document” • Iterate on first version based on user feedback and

evolving evolution of NGS

Long Term • Convert to web based application


Demonstrate Compliance with the Accreditation Requirements

• Preparing for an inspection of an NGS laboratory

• Integrated inspection (MOL, COM, GEN, TLC)

- Refer to latest checklist edition • The highlights of NGS section of the MOL

Checklist will be covered • This webinar is not a substitute for the

comprehensive CAP Checklists


PREPARATION

Think like an Inspector!


Review the Checklists

• New or revised questions? • Requirement AND NOTES are applicable • Evidence of Compliance suggests ways to

ensure compliance • Review R.O.A.D instructions to inspector;

documents and records must be available • Is the available documentation acceptable to

you if you were the inspector?


• High volume tests • Low volume tests • High patient impact tests • Tests from variant PT report • Concerns from previous deficiencies • New analyte, new instrument

Most Likely Choices During an Inspection


Top Molecular Deficiencies

MOL.30785 For each test there is a validation summary addressing analytical and clinical performance parameters.

MOL.34495 Quality control data are reviewed and assessed at least monthly by the laboratory director or designee

MOL.49570 The final report includes an appropriate summary of the methods, the loci or variants tested and the analytic interpretation. When appropriate, the final report includes the clinical interpretation.

MOL.35350 Nucleic acid amplification procedures (e.g. PCR) are designed to minimize carryover (false positive results) using appropriate physical containment and procedural controls.

MOL.34229 For qualitative tests, positive, negative and sensitivity controls are included for each assay, when appropriate, in every run and as specified in the manufacturer's instructions (as applicable) and laboratory procedure.


MOL.36105 The laboratory has a written procedure that describes the steps included in recording the bioinformatics process used to analyze, interpret, and report NGS test results.

MOL.36115 The laboratory validates the analytical bioinformatics process (also termed pipeline) and revalidates the entire process and/or confirms the performance of the components of the process as acceptable when modifications are made.

MOL.31145 The results of each validation study include a sufficient number of samples with repeated analyses to provide a high degree of assurance of the test's precision or reproducibility

MOL.34352 The results of controls are reviewed for acceptability before reporting of results.

MOL.36010 The laboratory has a written procedure for performing the analytical wet bench process used to generate next generation sequencing data.

Top Molecular Deficiencies cont.




NGS Reference Lab

Policy



Tracking

General




Wet Bench

NGS Wet Bench Process Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan

Pipeline Version

Traceability


Interpretation

Sequence Variants




Demonstrating Compliance • Reference Laboratory: Records of evaluations of referral

laboratories for NGS referral testing and copies of valid CLIA and/or CAP certificates

• Specimen Provenance: Specimen handling, chain of custody and data transfer starting from initial NGS test order to the final report

• Confirmatory Studies: Determine during validation if and when confirmatory testing of NGS identified variants should be performed

• If certain variants are NOT confirmed, then provide rationale for, and data supporting the rationale

• Records of review of correlation of NGS test results and confirmatory test results over time





NGS Reference Lab

Policy



Tracking

General




Wet Bench

NGS Wet Bench Process Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan

Pipeline Version

Traceability


Interpretation

Sequence Variants



Demonstrating Compliance cont. • Review of exception log and corrective action • Data Security: records of policies and protocols to ensure

patient confidentiality, security and data integrity • Encryption, control of physical and virtual access to data,

system backups and redundancy • Records of audit trails and copies of HIPAA Associate

Agreements • Data Storage: Policy must be in accordance with local, state

and federal requirements for storage of data • Written policy that describes files, type of data to be

retained and length of retention





NGS Reference Lab

Policy



Tracking

General




Wet Bench


Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan



Interpretation

Sequence Variants



Demonstrating Compliance cont.

• Review all centrally located records • Review documentation of maintenance, reagent labeling,

testing performed, QC, and results reporting • Ensure that policies and procedures are available to testing

personnel


54

Demonstrating Compliance cont. • Monitoring of Proficiency Testing (PT)

– Required for all analytes regardless of CLIA complexity classification

– PT must be specific to the analyte and system matrix – If PT is not available, another mechanism must be

defined to assess accuracy and reliability of the system (alternative assessment)

– PT must be performed by, and rotated among, the same individuals who perform the routine testing

– Corrective action must be documented and reviewed for unacceptable results and for ungraded results


Demonstrating Compliance cont. • Quality Control

– QC must be verified prior to reporting patient results – Corrective action must be documented and reviewed – QC must be handled in the same manner as patient

samples and performed by those performing patient testing


Quality Control EXAMPLE CriteriaSpecimen quality Rejected and cancelled samples

Wrong sampleWrong type of tubeInsufficient quantityClottedMislabeled

Specimen cancellation Wrong testInsufficient specimenDegraded specimen

DNA quality OD 260/280 ratioOD 260/230 ratio

DNA quantification and integrity Gel electrophoresisSpectrophotometric Analysis

PRE-ANALYTIC QC FOR NGS - Examples


ANALYTIC QC FOR NGS - Examples Genomic DNA fragmentation Fragment size and distribution

Library preparationPre and post target enrichment size and concentration

Cluster Generation and Sequencing Cluster density and clusters passing filtersBase quality: Percentage of bases with Q>30 Error rate using Phi X-174 internal controlTotal reads passing filter

Sequence read alignment % Reads Aligned to Target% Reads Off target% of duplicated readsAverage Target CoverageDistribution of Coverage

Variant CallingAverage number of SNVs and indelsTransition to transversion ratio

Specimen verification

Specimen identity confirmation using independent genotyping approachSpecimen contamination identification

Variant confirmation by orthogona Rate of confirmation


POST-ANALYTIC QC FOR NGS - Examples

Case Statistics Average turnaround time

Test Specific ResultsNumber of positive and negative cases

Amended or corrected reports Number of clerical errorsChanges to variant classificationNumber of cases with changes to interpretation





NGS Reference Lab

Policy



Tracking

General




Wet Bench


Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan



Interpretation

Sequence Variants



Demonstrating Compliance cont. • Variant Interpretation: Provide validated algorithm for

classifying and interpreting variants • The ACMG guidelines for classification and

interpretation of germline variants in inherited disorders (Richards et al. 2015)

• Standard nomenclature (eg., HGVS and HUGO) – also consider: Infectious disease, somatic variants

• Incidental and Secondary findings: Describe approach for reporting findings unrelated to the clinical purpose of testing


College of American Pathologists (CAP) – led effort Creating practical, step-by-step guidance for

molecular laboratories Manuscript in revision at the Journal of Molecular Diagnostics


Remaining 2018 Focus on Compliance Webinars



64

2018 focus on compliance - college of american...

Documents