EPA’s WET Interlaboratory Variability Study Findings

Marion KellyUSEPA Office of Science & Technology

Office of Water

Office of Water

Interlaboratory Variability Study Design Overview

National Study of WET test methods which demonstrated that the methods were scientifically sound

Test methods -12 of the 17 approved WET test methods evaluated in the study

Laboratories - 56 different laboratories involved, each in an average of 3 methods

• Base study design = 9 EPA-sponsored labs, up to 11 non-EPA-sponsored labs

• Extended study design = additional non-EPA-sponsored labs

Office of Water

Study Design Overview (cont.)

Samples - over 700 samples analyzed

• Participant laboratories each tested 3 or 4 blind samples

• Sample types included: effluents, receiving waters, blanks, and reference toxicants

Objectives - study assessed the following parameters for each of the 12 WET test methods:

• Test completion rate

• False positive rate

• Interlaboratory precision

Office of Water

Peer Review Process

Peer review involvement• Study plan - independently peer reviewed 10/9/1998 - 12/9/1998• Study results and report – independently peer reviewed 1/01 – 4/01

Peer reviewers• Three experts in the field of aquatic toxicology and biometrics• Identity blinded to EPA• Experts not associated with the generation of the WET methods

final rule Peer review charge

• Ensure that study design and results are scientifically acceptable within the context of the intended use

Office of Water

WET Test Methods Evaluated

Acute Freshwater acute

• fathead minnow, Pimephales promelas

• cladoceran, Ceriodaphnia dubia

Marine acute• sheepshead minnow, Cyprinodon

variegatus• inland silverside minnow, Menidia

beryillina• mysid, Holmesimysis costata

Chronic Freshwater chronic

• fathead minnow, Pimephales promelas

• cladoceran, Ceriodaphnia dubia• green algae, Selenastrum

capricornutum

Marine chronic• sheepshead minnow, Cyprinodon

variegatus• inland silverside minnow, Menidia

beryillina• mysid, Mysidopsis bahia • red macroalga, Champia parvula

Office of Water

Participant Laboratory Procurement Solicitation

• 319 labs solicited for participation in the study Prequalification

• All participant laboratories were required to prequalify• Prequalification required documentation of capacity and

capabilities, experience and proficiency, and quality assurance / quality control

• Only 8 laboratories failed to prequalify Selection / level of participation

• Dependent upon bid cost, sponsorship, and random selection• Base study design = 9 EPA sponsored labs, up to 11 non-EPA

sponsored labs• Extended study design = additional non-EPA sponsored labs

Office of Water

Participant Laboratories (freshwater)

Test Method EPA-sponsored Non-EPA-sponsored Total

Base Base Extended

Fathead acute 9 11 9 29

Fathead chronic 9 11 7 27

Ceriodaphnia acute 9 11 8 28

Ceriodaphnia chronic 9 11 15 35

Selenastrum chronic 9 2 0 11

Office of Water

Participant Laboratories (marine)

Test Method EPA-sponsored Non-EPA-sponsored Total

Base Base Extended

Silverside acute 9 0 0 9

Silverside chronic 9 1 0 10

Sheepshead acute 7 0 0 7

Sheepshead chronic 7 0 0 7

Mysidopsis chronic 9 2 0 11

Homsesimysis acute 2 Interlaboratory testing canceled

Champia chronic 1 Interlaboratory testing canceled

Developed Study PlanFunded Study

Managed Study

STUDYPLAN

STUDYRESULTS

SOPsCollected, Prepared, & Shipped Samples

ConductedPreliminary Testing

Procured LabsManaged Day-to-Day ActivitiesTracked SamplesReviewed & Analyzed Study Data

TESTDATA(Over700

tests)

Conducted WET Tests on 3 or 4

Blind Samples

7 to 35Participant Labs

per Method

PARTICIPANTLABS SAMPLES

BlankReference Toxicant

EffluentReceiving Water

REFEREELAB

WETVariability

Study Design

Office of Water

Roles and Responsibilities

EPA• Assemble WET technical workgroup to develop study plan and to

provide technical oversight during the study• Provide overall study management

Sample Control Center (SCC) operated by DynCorp I&ET• Procure referee and participant laboratories• Coordinate and provide day-to-day management of referee and

participant laboratories• Track sample shipment and receipt• Review, validate, and analyze study data

Office of Water

Roles and Responsibilities (cont.)

Referee Laboratories• Conduct preliminary testing to determine the appropriateness of

samples for interlaboratory testing• Collect, prepare, package and ship test samples to participant

laboratories Participant Laboratories

• Conduct WET tests on blind samples during interlaboratory testing

Office of Water

Interlaboratory Variability Study Timeline

Task Date

Settlement agreement signed 7/24/98

Preliminary study plan published 8/24/98

Expert peer review conducted on preliminary study plan 10/9/98 – 12/9/98

Final study plan published 6/11/99

Interlaboratory testing conducted 9/28/99 – 4/4/00

Preliminary study results published 10/10/00

Expert peer review conducted on preliminary study results

1/01 – 4/01

Final study report and proposed rule published 9/28/01

Office of Water

Peer Review Process

Peer review involvement• Study plan - independently peer reviewed 10/9/1998 - 12/9/1998• Study results and report – independently peer reviewed 1/01 -

4/01 Peer reviewers

• Three experts in the field of aquatic toxicology and biometrics• Identity blinded to EPA• Experts not associated with the generation of the WET methods

final rule Peer review charge

• Ensure that study design and results are scientifically acceptable within the context of the intended use

Office of Water

Study QA

Laboratory prequalification - EPA required that laboratories prequalify for participation in the study

Test-specific QC criteria - standard quality control measures for WET testing were required in the study, including test acceptability criteria as stated in the methods manual, reference toxicant testing, and test condition monitoring

Data reporting standards - EPA required that laboratories submit all bench-level data electronically in pre-designed standard reporting templates

Independent result recalculation - EPA independently recalculated all test results from reviewed electronic data

Office of Water

Results - Successful Test Completion Rates

Test Method Test completion rateCeriodaphnia dubia acute 95.2%Ceriodaphnia dubia chronic 82.0%Fathead minnow acute 100%Fathead minnow chronic 98.0%Selenastrum capricornutum chronic (with EDTA) 63.6%Mysidopsis bahia chronic 97.7%Sheepshead minnow acute 100%Sheepshead minnow chronic 100%Inland Silverside minnow acute 94.4%Inland Silverside minnow chronic 100%

Office of Water

Results – False Positive Rates

Test Method Survival Endpoint

Sublethal Endpoint

Ceriodaphnia dubia acute 0.00% NACeriodaphnia dubia chronic 0.00% 3.70%Fathead minnow acute 0.00% NAFathead minnow chronic 0.00% 4.35%Selenastrum capricornutum chronic (with EDTA) NA 0.00%Mysidopsis bahia chronic 0.00% 0.00%Sheepshead minnow acute 0.00% NASheepshead minnow chronic 0.00% 0.00%Inland Silverside minnow acute 0.00% NAInland Silverside minnow chronic 0.00% 0.00%

Office of Water

Results – Interlaboratory Precision (CV%)

Test Method LC50 IC25Ceriodaphnia dubia acute 29.0% NACeriodaphnia dubia chronic 21.5% 35.1%Fathead minnow acute 20.0% NAFathead minnow chronic 13.4% 20.9%Selenastrum capricornutum chronic (with EDTA) NA 34.3%Mysidopsis bahia chronic 31.2% 41.3%Sheepshead minnow acute 26.0% NASheepshead minnow chronic 8.72% 10.6%Inland Silverside minnow acute 38.5% NAInland Silverside minnow chronic 40.6% 43.8%

Office of Water

Study Results Summary

Successful test completion rate - • 8 of 10 methods had successful test completion rates greater than

90%

False positive rate - • All 10 methods had false positive rates less than 5% (0.00% -

4.35%)

Interlaboratory variability -• Ranged from 8.72% to 40.6% for LC50s and from 10.6% to 43.8%

for IC25 values• Comparable to chemical methods promulgated at 40 CFR Part 136

• Below interlaboratory variability previously cited for the WET methods

Comparison of WET Method Variabilityat Method Promulgation and Reported

in the WET Variability Study

Reported in WET Variability StudyReported at time of promulgation

Inte

rlab

orat

ory

Vari

abili

ty (

%CV

)

Ceriodaphniadubiaacute

Ceriodaphniadubia

chronic

Fatheadminnowacute

Fatheadminnowchronic

Sheepsheadminnowacute

Sheepsheadminnowchronic

Inlandsilverside

acute

44.2

%

42%

35%

34%

42%

44.2

%

42.2

%

29.0

% 35.0

%

20.0

%

20.9

% 26.0

%

10.5

%

38.5

%

Office of Water

For More Information

Contact:Marion Kelly

U.S. EPA Office of Water Engineering and Analysis Division (4303T)

1200 Pennsylvania Ave., N.W. Ariel Rios Building

EPA WestWashington, DC 20460

(202) 566-1045kelly.marion@epa.gov