guide to selecting qualified wet laboratories robert n. brent, ph.d. dyncorp science and engineering...

Guide to Selecting Qualified WET Laboratories

Robert N. Brent, Ph.D. DynCorp Science and Engineering Group

Why is Lab Selection Important?

You, the permittee, are responsible

for:

• Meeting monitoring frequency requirements

• Certifying the quality of data

– “I certify under penalty of law …the information submitted is, to the best of my knowledge and belief, true, accurate, and complete…”

• Living with the test results

– Test failures can initiate additional testing, TIEs, TREs, enforcement actions, fines, public scrutiny


You are the one paying

• In a 1998 WERF Study

– C.dubia chronic test: $100 - $2,300

– Fathead minnow chronic test: $238 - $5,500

• With multiple outfalls, monthly testing requirements, and multi-species monitoring, test costs can be significant

– 3 outfalls X 12 samples/yr X 3 species X $1000/test = $108,000

• Know what you are paying for!


Quality does matter

• There is a wide range of laboratory performance

– Ex: EPA’s Variability Guidance Document

Percentiles of Intralaboratory CVs (coefficient of variation) for C. dubia Chronic Test

25th percentile CV17%

0

5

10

15

20

25

0 10 20 30 40

Test

Res

ult


0

5

10

15

20

25

0 10 20 30 40

Test

Res

ult


0

5

10

15

20

25

0 10 20 30 40

Test

Res

ult

A Tale of Two Labs

Ex: WET Interlaboratory Variability Study

0

10

20

30

40

50

60

70

80

1 2 3 4 1 2 3 4SampleLab

Va

ria

bili

ty (

Co

ntr

ol %

CV

)

0

5

10

15

20

25

30

Co

ntr

ol P

erf

orm

an

ce

(M

ea

n)

Variability

ControlPerformance

inva

lid

inva

lid

inva

lid

false +

-------------A------------- -------------B-------------

A Tale of Two Labs

Ex: WET Interlaboratory Variability Study

0

10

20

30

40

50

60

70

80

1 2 3 4 1 2 3 4SampleLab

Va

ria

bili

ty (

Co

ntr

ol %

CV

)

0

5

10

15

20

25

30

Co

ntr

ol P

erf

orm

an

ce

(M

ea

n)

Variability

ControlPerformance

inva

lid

inva

lid

inva

lid

false +

-------------A------------- -------------B-------------

$936 / test

$936 / test

$500 / test

$500 / test

Obstacles to Selecting Qualified Labs

Procurement Regulations

• Many permittees are required to select lowest responsive and responsible bidder

• This doesn’t always mean selecting lowest bidder

– Must be responsive to everything that you request in the solicitation

– Must be responsible for delivering exactly what you specify in the contract

• Solution: You determine the exact specifications for what you want

– Government Ex: $600 hammer

– WET Ex: lab must demonstrate intralab variability of <27%

• Talk to your procurement specialist about incorporating detailed specifications into the lab solicitation process

Obstacles to Selecting Qualified Labs

Insufficient knowledge and resources for identifying qualified laboratories

• Solution: This workshop will provide

– Practical tools for evaluating laboratory quality

– Things to look for in a qualified lab

– Questions to ask

– Sample test data sets to quiz your lab

General Evaluation Criteria

Capacity

• Will the lab be able to test your samples when you need them tested?

• What is the labs maximum and typical capacity (tests per week)?

– Look for maximum capacity to greatly exceed typical capacity

– Quality usually decreases when operating near capacity limits

• What limits capacity? (space, staff, organisms, equipment)

– Look for limits that can be easily remedied


Staff

• What is the level of experience and education of staff from top to bottom?

– Remember: it is the staff at the bottom (technicians) that will have the most contact and interaction with your samples

– Experienced and well-trained staff is critical to consistently generating high quality data

• What is the turnover rate?

– Look for low turnover at the top

– Turnover at the bottom may be high, but look for rigorous training program


Organizational Structure

• What is the chain of command?

• Who will you be interacting with?

– Look for a dedicated account manager

– Look for a laboratory contact as well

Certification

• Does the lab carry any State or national certifications?

– Not all States have lab certification programs, but beware of labs without certification in a State where there is a certification program


Historical Performance

• How many tests does the lab run annually?

– Look for 50 – several hundred (for common test methods)

• What percentage of tests were successfully completed (met TAC) without retesting?

– Look for 90 -100%

• How many other clients do they have with similar waste streams (municipal, industrial, etc.) to yours?

– Familiarity with similar waste streams may lead to better, quicker problem resolution


Reporting

• Does the lab’s reporting standards and format meet the requirements of your regulatory authority?

• Does the lab’s reporting format meet your needs

– Look for simple but detailed reports

Ex: You should be able to find the test result (NOEC, IC25, etc.) within 60 seconds

Ex: Report should be detailed enough for someone to completely recalculate the test results

Ex: You should be able to find the temperature of treatment A on Day 4 of the test


Awareness

• Is the lab aware of recent developments in WET testing? – Look for lab to be aware of:

Proposed WET method changes and upcoming new versions of WET manuals

WET Method Guidance Document (EPA 821/B-00/004)

WET Variability Guidance Document (EPA 833/R-00/003)

Quality Control Evaluation Criteria

Organism Source and Quality

• Where does the lab get their organisms?

– In-house cultures vs. commercial suppliers

– Look for consist source

• How many organisms are available on a daily basis?

– Look for capacity to exceed usage

• What QC measures are used to assess organism health?

– Look for reference toxicant testing and other measures (reproduction and survival in cultures)

• How often are culture crashes experienced?


Dilution Water Source and Quality

• What dilution water types does the lab use?

• Where does the dilution water come from or how is it prepared?

– Look for consistent high quality source

– Look for experience with different dilution water types

• How does the lab assess dilution water quality?

– Look for chemical testing and toxicity testing of new dilution water batches


Reference Toxicant Control Charts

• Reference toxicant control charts are one of the best ways to assess laboratory performance (if you know what to look for)

• Labs are required to conduct monthly reference toxicant tests

• Test results are plotted on a chart to show variability over time

0

0.5

1

1.5

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Ref

. To

x. C

on

c. IC25

Mean

-2STD

+2STD

Evaluating Reference Toxicant Control Charts

Look for control charts of LC50 (for acute methods) or IC25 (for chronic methods), not NOECs

Look for 20 points

Look for curved limits

0

0.5

1

1.5

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Ref

.To

x. C

on

c. IC25

Mean

-2STD

+2STD


Look at scale

0

0.5

1

1.5

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Re

f. T

ox

. Co

nc

. IC25

Mean

-2STD

+2STD

-0.50

0.51

1.52

2.53

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Re

f. T

ox

. Co

nc

. IC25

Mean

-2STD

+2STD

0

0.5

1

1.5

2

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Re

f. T

ox

. Co

nc

. IC25

Mean

-2STD

+2STD

0

0.5

1

1.5

2

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

DateR

ef.

To

x. C

on

c. IC25

Mean

-2STD

+2STD

DifferentScale

SameScale


Look at CV

• CV = Standard deviation / Mean * 100%

• Compare CV to national percentiles (Variability Guidance Doc Table B-1)

0

0.5

1

1.5

2

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Re

f. T

ox

. Co

nc

. IC25

Mean

-2STD

+2STD

0

0.5

1

1.5

2

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Re

f. T

ox

. Co

nc

. IC25

Mean

-2STD

+2STD

CV = 17% CV = 40%

0

0.5

1

1.5

2

7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01

Date

Re

f. T

ox

. Co

nc

. IC25

Mean

-2STD

+2STD


Look for results beyond control limits

• 1 in 20 will exceed limits by chance alone

• Beware of >2 exceedances in 20


Control Charts for Additional QC Measures

• Look for control charts of:

– Control performance over time (i.e., control reproduction)

– Control CV over time

– PMSDs (percent minimum significant difference) over time

• Compare with national percentiles from Variability Guidance Document (Tables B-7 to B-8)

Evaluation of PMSD Control Charts

Look for PMSD mean versus national percentiles

Look for PMSDs that exceed 90th percentile

0

10

20

30

40

50

7/24/98 12/6/99 4/19/01 9/1/02

Date

PM

SD

(%

)

PMSD

Mean

50th %tile

90th %tile


Statistical Analysis and Endpoint Calculation

• Can the laboratory correctly perform the recommended statistics and properly calculate test endpoints?

– EPA found that a large percentage of laboratories in the WET Interlaboratory Variability Study made one or more errors in the calculation of test results

• Test laboratories with sample data sets

– 3 sample data sets provided

– Ask laboratories to calculate LC50, survival NOEC, sublethal NOEC, and IC25 using the recommended statistical flowcharts and concentration-response evaluation guidance (where appropriate)

Sample Test Data Set #1

Sample Test Data Set #1 - Ceriodaphnia dubia Survival and Reproduction Test

Survival NOEC 25LC50 40.613

Reproduction NOEC 12.5Reproduction IC25 22.969

1-tail, 0.05 levelof significance

0

5

10

15

20

25

30

D-C

ontr

ol

6.25

12.5

*25 50 100

Da

y 6

Re

pro

du

cti

on


Requires use of concentration-response guidance

Sample Test Data Set #2 - Ceriodaphnia dubia Survival and Reproduction Test

Survival NOEC 100LC50 >100

Reproduction NOEC 100Reproduction IC25 >100*

* IC25 calculated as 86.82, but based on concentration-response evaluation, this result was determined to be anomalous and IC25 reported as >100

0

5

10

15

20

25

30

35

40

D-C

ontr

ol

6.2

5

12.5 25

50

100

Da

y 6

Re

pro

du

cti

on


Requires use of concentration-response guidance

Sample Test Data Set #3 - Fathead Minnow Larval Survival and Growth Test

Survival NOEC 50LC50 51.404

Growth NOEC 50*Growth IC25 51.446

* 6.25% treatment was significantly different from control, but based on concentration-response evaluation, this treatment was determined to be anomalous and NOEC reported as highest concentration not significantly different from the control

0

0.1

0.2

0.3

0.4

0.5

0.6

D-C

ontr

ol

*6.2

5

12.5 25 50 100

7 D

ay B

iom

ass

Conclusions

It’s your money and your reputation at stake:

• Choose a quality laboratory

When you find a quality laboratory, stick with them

When you don’t know, ask

guide to selecting qualified wet laboratories robert n. brent, ph.d. dyncorp science and engineering...

Documents

test costs

lab selection important

lab certification programs

test resultstest failures

fathead minnow chronic

qualified laboratoriessolution

qualified labquestions

maximum capacity