establishing validity, reproducibility, and utility of highly scalable genetic tests: challenges and...

© 2015 Invitae Corporation. All Rights Reserved. | 1

Improving the accuracy and reproducibility of clinical genetic tests

STEPHEN E. L INCOLN

HVP Meet ingMay 2016

2 © 2015 Invitae Corporation. All Rights Reserved.

1. Analytic Performance– Analytic Sensitivity: Can you find the DNA variants a

patient has?– Analytic Specificity: Of the variants you find, are they

really there?

2. Clinical Performance– Clinical Sensitivity: Can you find the complete

spectrum of clinically relevant variants for a patient’s condition?

– Classification: Can you interpret the medical impact of these variants appropriately?

Questions about any genetic test…


1. Analytical Performance


Lions and Tigers and Bears, Oh My!


Clinical utility of panel testing for HBOC

Desmond et al., JAMA Oncol. 2015Swisher, JAMA Oncol. 2015


Companion technical paper

A systematic comparison of traditional and multigene panel testing for hereditary breast and ovarian cancer in more than 1000 patientsStephen E. Lincoln1, Yuya Kobayashi1, Michael J. Anderson1, Shan Yang1,Andrea J. Desmond2, Meredith A. Mills3, Geoffrey B. Nilsen1, Kevin B. Jacobs1, Federico A. Monzon1, Allison W. Kurian3, James M. Ford3, Leif W. Ellisen2,4

1. Invitae, San Francisco, CA2. Massachusetts General Hospital Cancer Center,

Boston, MA3. Stanford University School of Medicine, Palo Alto,

CA4. Harvard Medical School, Boston, MA Lincoln et al., J Mol Diag 2015


A Significant Fraction of Pathogenic Variants in Representative Clinical Cases are Technically Challenging

Pathogenic variants among 1062 clinical cases, by type:

Lincoln et al., J Mol Diag 2015

Small Indel

i.e. CNVs or “Large Rearrangements” (Myriad term)

“Hard stuff” (Steve’s term)


Analysis of 30,000 sequential cases with clinical testing across over 1,000 known disease genes (in total)…

Percentage Pathogenic Variant Type2.9% Single-exon del/dups (CNVs)

1.8% Large indel or complex sequence change

5.8% In high-GC, low-complexity, or poorly mapabie regions of the genome

Of the last 5,000 patients with pathogenic findings…

Rebecca Truty, Invitae


BRCA2: c.9342_9343insAlu– Novel Alu insertion (at the time we first saw it)– Not the Portuguese founder mutation (that’s c.156_157insAlu)

BRCA2: c.9203_9328del126 – Very large indel (or a very small CNV)

MSH2: c.942+3A>T– Splice-affecting mutation next to 25 bp homopolymer-A

MSH2: inv exons 1-7– Breakpoint detection in intronic regions

CDKN2A: c.9_32dup24– Third copy of a wilt-type tandem duplication in 5’ CpG island

And many other examples...

The spectrum of pathogenic variants in patients includes cases that the typical NGS workflow does not handle


Analytic validity in N=1105 individuals

NGS vs. Traditional MethodsIn 1105 Individuals

Sensitivity 100.0%

Specificity 100.0%

750 Comparable Variants (Pathogenic or Otherwise)

Sequence Changes 721

Del/dups (CNVs) 29


Single Nucleotide 549

Small Indel 156

Large Indel* 13

Complex** 6

To achieve this, specialized NGS methods, biochemical and bioinformatics, are required.

The other challenging classes of variation tend to be not well represented in other validation studies.

* Large Indel is deletion≥10bp, insertion≥5bp

** Complex includes homo-polymer associated variants, indels in low-complexity sequence, short

range haplotypes, etc.


NGS vs. Traditional Methods in 250 Patients

Sensitivity 100.0%

Specificity 99.99%

3025 Variants Appropriate to Measure Sensitivity

Sequence Changes 3021

Del/dups (CNVs) 4 ?

Chong et al., PLOS One 2014

Single Nucleotide 3010

Small Indel 11 ?

Large Indel * 0

Complex ** 0* Large Indel is deletion≥10bp, insertion≥5bp** Complex includes homo-polymer associated

variants, indels in low-complexity sequence, short range haplotypes, etc.

Note: These numbers assume that Table 2 in the paper includes all of the indels and del/dups, and that the remaining variants in the are benign SNPs. This is not entirely clear to us in the paper text.

A different lab’s study: Analytic validity in 250 individuals


What is one to do?

Build Awareness

Develop/Implement Methods

Improve Validation Resources

Strengthen Validation Standards


2. Clinical Interpretation


ClinVar top submitters

www.ncbi.nlm.nih.gov/clinvar > Statistics > Submitter List (as of May 3, 2016)

http://www.ncbi.nlm.nih.gov/clinvar




ClinVar entry for BRCA1:c.4410A>T


NM_007294.3(BRCA1):c.4410A>T (p.Glu1470Asp)


This sequence change replaces glutamic acid with aspartic acid at codon 1470 of the BRCA1 protein (p.Glu1470Asp). The glutamic acid residue is moderately conserved and there is a small physicochemical difference between glutamic acid and aspartic acid. This variant has been published in the literature and is present in population databases (rs80357075, 0.05%). This variant has been reported in individuals affected with breast cancer. Due to limited data, whether this variant segregates with disease remains uncertain (PMID: 20104584, 22682623). This variant has also been reported in a healthy individual not affected with cancer (PMID: 24728327). ClinVar contains entries for this variant (RCV000131557, RCV000077572, RCV000074594, RCV000120257). Algorithms developed to predict the effect of missense changes on protein structure and function (SIFT, PolyPhen-2, Align-GVGD) all suggest that this variant is likely to be tolerated, but these predictions have not been confirmed by published functional studies. In summary, this is a rare missense change that is not predicted to affect protein function or cause disease. However the evidence is insufficient at this time to prove that conclusively. Therefore, it has been classified as a Variant of Uncertain Significance.

27 © 2015 Invitae Corporation. All Rights Reserved.Vail et al., J Community Genet 2015

Finding: 3-14% disagreement rate between databases

(BIC, HGMD, LOVD, UMD, and ClinVar)


The disagreement in public databases “precludes their wider use in clinical practice”

Finding: 3-14% disagreement rate between databases

(BIC, HGMD, LOVD, UMD, and ClinVar)


The disagreement in public databases “precludes their wider use in clinical practice”

“Public databases [are] fraught with errors”“Interpretation accuracy impossible with public databases”

Analyst Day Presentation, September 2015investor.myriad.com > Events & Presentations > Financial Events

Accessed 2/1/2016


This issue can be misrepresented to clinicians…

• Experienced and responsible lab directors never simply copy variant classifications from any public database

• Instead, they critically evaluate underlying evidence and report classifications following established guidelinesQuestion: What is the clinical impact of discordance in public databases?


BRCA1/2 variant classification concordance

Positive vs. not positive result for BRCA1/2

Agree 99.8%

Disagree 0.2%

% of patients with one or more VUS in BRCA1/2

Our test 4.1%

Traditional test 3.2%

975 patients with both Myriad Genetics and independent test results. Variants uncovered by the independent test were blindly classified:

• Using only publicly available resources (literature and databases)

• Following a system based on the ACMG/AMP 2015 guidelines

All amended reports provided to clinical sites were incorporated, so the Myriad variant classifications were up to date



Positive vs. not positive results

Pathogenic (P)

Likely Pathogenic(LP)

VUSLikely Benign (LB)

Benign (B)

P Concordant Discordant LP

VUS Discordant Concordant LB

B

Count differences that could substantially change patient

management decisions


Inclusion Criteria were ClinVar submitters that:– Are an established and licensed diagnostic laboratory

• or are submitting data from such a lab

– Submitted at least 200 classified variants in BRCA1/2– Most classifications were from the last 5 years

• Using “last evaluation date”, not “submission date” in ClinVar

Thus we excluded data from:– Research projects, software vendors– Smaller labs, consortia (e.g. ENIGMA)– Old sources (e.g. BIC)

• Mostly pre-2007 Myriad Genetics data

Examining the larger data set in ClinVar

Shan Yang, Invitae


ClinVar based data set

Shan Yang, Invitae

Name Classified Variants

ComparableVariants

Submitter Name in ClinVar

Ambry 2793 1502 AMBRY GENETICS

Myriad(via SCRP)

2067 1184 SHARING CLINICAL REPORTS PROJECT (SCRP)

Invitae 1479 1082 INVITAE

GeneDx 1214 937 GENEDX

Counsyl 272 256 COUNSYL

CHEO 257 216 MOLECULAR GENETICS DIAGNOSTIC LABORATORY, CHILDREN'S HOSPITAL OF EASTERN ONTARIO

Emory 203 183 EMORY GENETICS LABORATORY

Total 4725 1800 Non-redundant list from the above

Data from ≥2 labs


BRCA Exchange

Benedict Paten, Melissa Cline, Molly Zhang, David Haussler, et al., UCSC

www.brcaexchange.org

• Replicated our results • Open-source code to help mine ClinVar

• Integrating many other sources of BRCA1/2 data


Ambry Invitae GeneDx Counsyl CHEO Emory

Myriadvia SCRP

98.7%939/951

97.9% - 99.3%

99.2%619/624

98.3% - 99.7%

99.5%569/572

98.6% - 99.9%

99.4%171/172

97.3% - 100%

99.5%139/142

94.5% - 99.4%

97.2%103/106

92.6% - 99.2%

Ambry99.2%860/867

98.4% - 99.6%

99.6%780/783

99.0% - 99.9%

99.6%223/224

97.9% - 100%

98.3%176/179

95.6% - 99.5%

98.8%161/163

96.1% - 99.7%

Invitae99.8%593/594

99.2% - 100%

99.1%214/216

97.1% - 99.8%

98.2%161/164

95.2% - 99.5%

99.3%144/145

96.8% - 100%

GeneDx99.5%221/222

97.9% - 100%

97.9%138/141

94.4% - 99.4%

99.3%149/150

96.9% - 100%

CounsylConcordanceConcordant/All

Conf. Int.

100%82/82

97.0% - 100%

100%105/105

97.6% - 100%

CHEO 98.3%

57/5892.2% - 99.9%


Ambry Invitae GeneDx Counsyl CHEO Emory

Myriadvia SCRP

98.7%939/951

97.9% - 99.3%

99.2%619/624

98.3% - 99.7%

99.5%569/572

98.6% - 99.9%

99.4%171/172

97.3% - 100%

99.5%139/142

94.5% - 99.4%

97.2%103/106

92.6% - 99.2%

Ambry99.2%860/867

98.4% - 99.6%

99.6%780/783

99.0% - 99.9%

99.6%223/224

97.9% - 100%

98.3%176/179

95.6% - 99.5%

98.8%161/163

96.1% - 99.7%

Invitae99.8%593/594

99.2% - 100%

99.1%214/216

97.1% - 99.8%

98.2%161/164

95.2% - 99.5%

99.3%144/145

96.8% - 100%

GeneDx99.5%221/222

97.9% - 100%

97.9%138/141

94.4% - 99.4%

99.3%149/150

96.9% - 100%

Counsyl100%82/82

97.0% - 100%

100%105/105

97.6% - 100%

CHEO 98.3%

57/5892.2% - 99.9%

On a per-variant basis:Pairwise concordance: 97.2% -

100.0%Overall concordance: 98.5%

Only 27/1800 variants have a significant classification discordance

between any two reporting laboratories


How many individuals does this represent?

≈22,000 patients give ≈1800 non-redundant

variants

Melissa Cline, UCSC


Did SCRP data bias results from other labs?

Other lab classifications Concordance

Pre-dating SCRP release 99.0%

Post-dating SCRP release 98.9%

Apparently not

Compare ClinVar submission date of Myriad data via SCRP to evaluation date of same variant from other

labs


All of the discordant variants are rare

All of the 27 variants with classification discordances:– Have population allele frequencies ≤ 0.05% in all of:

• ExAC• ESP • 1000 Genomes• Invitae patient database

Only 16.6% of patients carry any such rare variant(s)

Most (98.4%) of these rare variants are completely concordant when seen by more than one laboratory


Per-variant and per-patient concordance are different

16.4%

0.2%

83.4%

Patients with 1 or more rare variants, all concordant (16.4%)

Patients with 1 or more rare variants having a discordance (0.2%)

Patients with no rare variants (83.4%)

Steve Lincoln, Invitae and Benedict Paten UCSC

Estimated per-patient concordance 2 orthogonal ways:– From population allele frequencies– From patient database (n≈20,000)

99.8%Concordant


Do not over-generalize these results

Discordance in other genes can be higher– e.g. some cardiovascular genes

Discordance from other sources can be higher

44 © 2015 Invitae Corporation. All Rights Reserved.Van Driest et al., JAMA 2016

“There was low concordance in designating SCN5A and KCNH2 variants as pathogenic.In an unselected population, the putatively

pathogenic genetic variants were not associated with an abnormal phenotype.”


Discussion

While substantial disagreements in BRCA1/2 are infrequent, they are important

We must resolve differences collaboratively, not competitively, to deliver the best patient care, as is done in other areas of medicine

Labs that maintain data as a proprietary asset can make arbitrary and unverifiable claims regarding interpretation accuracy

Data submission ClinVar facilitates peer review and interlaboratory quality control on a global scale, as exemplified by this study


Desmond et al., 2015 (open access) Swisher, 2015 (commentary)

Lincoln et al., 2015 (open access)

Jim Ford Allison Kurian Meredith Mills

Leif Ellisen Andrea Desmond Kristen Shannon

Nadine Tung

Steve Lincoln Shan Yang Yuya Kobayashi Scott Topper Bob Nussbaum

Melissa Cline Molly Zhang David Haussler Benedict Paten

For copy of slides:steve.lincoln

@[email protected]

or www.invitae.com

mailto:[email protected]




establishing validity, reproducibility, and utility of highly scalable genetic tests: challenges and...

Science