1 of 40

How Many Samples do I Need?Part 2

Presenter: Sebastian Tindall

60 minutes(15 minute 2nd Afternoon Break)

DQO Training CourseDay 1

Module 5

2 of 40

Summary Use Classical Statistical sampling approach:

• Very likely to fail to get representative data in most cases

Use Other Statistical sampling approaches:• Bayesian• Geo-statistics• Kriging

Use M-Cubed Approach: Based on Massive FAM

Use Multi-Increment sampling approach:• Can use classical statistics• Cheaper• Faster• Defensible: restricted to surfaces (soils, sediments, etc.)

MASSIVE DATA Required

3 of 40

[µ – AL] ≥ 3σ

then almost never fail

(Run Simulations)

Use Classical Statistical sampling approach:

Very likely to fail to get representative data in most cases, except if…

4 of 40

Analytical +

Sampling & Sub-sampling +

Natural heterogeneity of the site=

Total Uncertainty

Uncertainty is Additive!Remember the uncertainty is additive for

all steps in sampling and analysis

5 of 40

What is the one phenomenon that causes

ALL sampling error?

HETEROGENEITY

6 of 40

The SYSTEM functions as if it believes that…

{Data

Uncertainty Automatically

Managed

Data Quality

PrescriptiveAnalytical Methods ={

AnalyticalUncertainty

Automatically Managed

Decision Quality= {

DecisionUncertainty

Automatically Managed

7 of 40

Take-Home Message

Non-Non-Representative Representative

SampleSample

Perfect Perfect Analytical Analytical ChemistryChemistry

++

““BAD” DATABAD” DATA

8 of 40

Representativeness

Diamond Ring Costume Jewelry

Can an analyst tell the difference? Yes.

9 of 40

Representative Soil Sample

Representativeness

Can an analyst tell the difference? No.

Non-Representative Soil Sample

SoilSample

SoilSample

10 of 40

Sample vs. Analytical Certainty

16

5 4

3

27

Analytical = 5%

TOTAL ERRORSampling = 95%

1,280 Onsite1,220 Lab

331 Onsite286 Lab

500 Onsite416 Lab

164 Onsite136 Lab

24,000 Onsite27,700 Lab

27,800 Onsite42,800 Lab

39,800 Onsite41,400 Lab

Note: Above sample locations are 12” apart

11 of 40

Dilemma!

• None of the equations for the number of samples, or the average, or the standard deviation include a term for size: Area or Volume

• Some guidance suggests 1 sample/20 cu yd but this is indefensible

• Must decide on the scale of the decision or exposure unit to represent the population of interest

• Must sample within the scale of the decision unit

12 of 40

Typical Sampling Design

EPA “Methods for Evaluating the Attainment of Soil Cleanup Standards - Vol 1”, 1989

Equation 6.6

Wrong Often

Estimate of σ usually way off

orunknown

Assumed Normal Distribution

2

121

211

2

5.0

ZAL

ZZn

13 of 40

n 5 1. Will usually fail to truly capture heterogeneity…. of population(s)

2. Results in large uncertainty which is seldom:- Identified- Quantified- or even Acknowledged

Typical Sampling Design (cont.)

14 of 40

Uncertainty

Mo = Md = Mn

Normal

Mo Md Mn

Lognormal

M0 = modeMd = medianMn = mean

% of time when x < is high, (when n is small)

15 of 40

Classical Statistics Burdens

Reasonably accurate estimate of PDF (Histogram) Reasonably accurate estimate of the SD Correct selection of appropriate statistical

sampling method (equation) Correct selection of appropriate statistical method

(equation) for calculating a UCL

Required for each COPC within each Decision Unit:

All this is almost never possible and almost never done.

16 of 40

Classical Statistics Burdens

Student’s t UCL Approximate Gamma UCL Adjusted Gamma UCL H-UCL (Lands Method) Chebyshev (MVUE) UCL CLT UCL Adj-CLT UCL (Adjusted for skewness) Mod-t UCL (Adjusted for skewness) Jackknife UCL Standard Bootstrap UCL Bootstrap-t UCL Hall's Bootstrap UCL Percentile Bootstrap UCL BCA Bootstrap UCL

Which UCL to use?

List of above UCLs taken from ProUCL

17 of 40

Classical Statistics BurdensProblem: Which UCL to use?

Say you want to calculate an UCL on the average rainfall in your area for the purpose of building a dike to protect your town. So you get data for 5 out of a 100 years. You enter those 5 data points into ProUCL and use the 95% UCL it calculates. You build the dike. The next year the river overflows the dike and kills all the townsfolk.

What happened?

Answer: GIGO; your 5 data points did not include data for heavy rainfall years.

(ProUCL uses bootstrap techniques on small data sets. But remember, Statistics cannot create information where there is none.)

18 of 40

Why Decisions are suspect Failure to define population accurately Failure to collect representative samples from the population

of interest Failure to obtain representative data from the population of

interest Failure to accurately determine the frequency distribution of

the COPCs Failure to accurately determine the standard deviation of the

COPCs Failure to select the appropriate statistical method for

generating adequate samples Failure to use the appropriate UCL in making the decision

19 of 40

Definitions of Representativeness

A sample collected in such a manner that the sampling error is less than a specified amount.

A sample of a universe or whole that can be expected to exhibit the average properties of the universe or whole (40 CFR 260.10).

A sample that answers a question about a population with a specified confidence

Sampling for Environmental Activities, Envirostat, 2003

20 of 40

Definitions of Representativeness Representativeness expresses the degree to which sample

data accurately and precisely represents a characteristic of a population, parameter variations at a sampling point, or an environmental condition. Representativeness is a qualitative parameter which is most concerned with the proper design of the sampling program. The representativeness criterion is best satisfied by making certain that sampling locations are selected properly and a sufficient number of samples are collected. Representativeness is addressed by describing sampling techniques and the rational used to select sampling locations.

DQOs for Remedial Response Activities: Development Process, US EPA 1987

21 of 40

Definitions of Representativeness

A sample is representative when it is taken by a selection method that is both accurate and reproducible. Thus representativeness is characterized by the absence of bias and an acceptable variance. As far as the author is aware, this is the only possible objective and scientific definition of representativeness.

Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 30

22 of 40

Definitions of Representativeness

A correct sampling method is always structurally accurate. In addition, its variance is minimal so that its representativeness is maximal.

Non-correct sampling is always structurally biased. It may be accurate over short periods, but these cannot be forecast and so are unusable. This makes the tests of accuracy recommended by certain standards (the so-called bias tests) not only useless but also dangerous as they offer a false sense of security.

As well as having a negligible bias, representativeness requires reproducibility, i.e. a minimum variance, which itself depends on the quantitative properties of the sample (e.g. the mass and the number of increments).

Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 31

23 of 40

Definitions of Representativeness A sample is representative when the mean square,

r2 SE , of Sampling Error (SE) is not larger than a

certain standard of representativeness regarded as acceptable.

Representativeness is the sum of the square of the mean of SE (mSE), and the variance of the SE (s2

SE).

Preparation of Soil Sampling Protocols: Sampling Techniques and Strategies, EPA/600/R-92/128, July 1992

r2 (SE) m2 (SE) + s2 (SE) ≤ r2o (SE)

24 of 40

Typical Values of Bias Primary sample (non-probabilistic): up to

1000% Secondary sample (probabilistic but

incorrect): up 50% (and probably much more) Analysis: 0.1-1.0%

Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 32

Thus it is pointless and illusory to return an analytical result to three or four supposedly significant decimal places if the sample analyzed is insufficiently representative and even more pointless if it is biased.

25 of 40

Concepts

Homogeneous: when all its units are strictly identical to each other.

Homogeneity is an abstract mathematical concept that does not exist in the real, material world.

Heterogeneous: when all the units are not identical to each other. Heterogeneity is the only state in which a set of material units or

groups of units can be observed in practice. Heterogeneity is seen as the sole source of all sampling errors Homogeneity is the inaccessible condition of zero Heterogeneity

Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 24-25

26 of 40

Quantity of Data Matters. Why?

WARNING: The Statistician General has determined that drawing conclusions from insufficient data may

be hazardous to your decisions.

27 of 40

Sample Size Rules of Thumb

“Samples of less than 10 are usually too small to rely on sample estimates even in ‘nice’ parametric cases.”

“In many practical contexts, the number 30 is used as a ‘minimum’ sample size.”

M.R. Chernick in Bootstrap Methods: A Practitioner's Guide, 1999, pp. 150, 151.

28 of 40

Sample Size Rules of Thumb•In order to choose a specific classical statistical method (equation) information regarding the distribution of the contaminant within the decision unit is usually required.

•Such information allows one to select a method and calculate the number of sample needed to meet the specified error tolerances, providing a reasonably accurate estimate of the variance in known.

•However, certain assumptions must be presented and TESTED in order to show the selected method was appropriate. These tests are performed using data generated from the sampling event, i.e, AFTER sampling has occurred. Herein lies the requirement for ~30-50 or more samples. It is usually not possible to make definite statements (e.g. frequency distribution) with small sample sizes.

•If the tests fail, then the sampling results are in jeopardy and the data maybe invalidated, which could lead to another round of sampling.

29 of 40

Sample Size Rules of Thumb(continued)

•The sampling data is presented graphically (usually in the form of a histogram) in order to assess the distribution of the contaminant.

•Based on the distribution of the contaminant, a method to calculate an UCL follows.

•It is inappropriate to calculate a 95% UCL using the method based on a normal distribution if Data Quality Assessment cannot show that the contaminant is distributed normally.

30 of 40

Sample Size Rules of Thumb

“Although it is always dangerous to set ‘rules of thumb’ for sample sizes, I would suggest that in most cases it would be wise to take n ≥ 50.”

M.R. Chernick in Bootstrap Methods: A Practitioner's Guide, 1999, p. 151.

31 of 40

Sample Size Rules of Thumb

“For practical purposes it will be assumed here that a ‘too small number’ is less than 30, and a ‘large number’ is at least 50.”

Pierre Gy in Sampling for Analytical Purposes, 1998, p. 70.

32 of 40

Sample Size Rules of Thumb

“In practice, there appears to be no simple rule for determining how large n should be….If the distribution is highly skewed, an n of 50 or more may be required.”

Richard Gilbert in Statistical Methods for Environmental Pollution Monitoring, 1987, p. 140.

33 of 40

Quantity of Data Matters. Why?

“If the sample size is ‘large’ then most traditional estimators will yield the same conclusions and simple estimators suffice.”

H. Lacayo, Jr. in Environmental Statistics: Handbook of Statistics Volume 12, 1994, p. 891.

34 of 40

“Lacking distribution information, it is impossible to devise an optimal sampling strategy.”

- Jenkins, et. al. 1996. “Assessment of Sampling Error Associated with Collection and Analysis of Soil Samples at Explosives-Contaminated Sites” U.S. Army Corps of Engineers, Cold Regions Research & Engineering Laboratory, p. 1.

http://www.crrel.usace.army.mil/techpub/CRREL_Reports/reports/SR96_15.pdf

35 of 40

How Many Samples do I Need?

Begin with the End in Mind

Optimal Sampling Design

Alternative Sample Designs

, , , Correct Equation for n (Statistical Method)

Population Frequency Distribution

Contaminant Concentrations in the Spatial Distribution of the Population

The end

DATA

36 of 40

Q: Where do you obtain the contaminant distribution information in order to select the correct sampling design to ensure representativeness, etc?

A: From sampling data.

Q: How much sampling data do you need?

A: Depends upon the consequences of making the wrong decision.

37 of 40

Sample Representativeness

Are we honestly addressing Heterogeneity(sampling uncertainty)?

Now we are finally able to address this issue,

defensibly and affordably! Use cheaper analytical technologies that allow you to

increase sample density Use real-time measurements at the site of the

sample to support real-time decision-making IF we are willing to honestly balance analytical

uncertainty against overall data uncertainty

38 of 40

Managing UncertaintyManaging Uncertainty

Systematic Planning

Dynamic Work Plan

Real-Time Measurement Technologies

39 of 40

Managing Uncertainty

Systematic planning– Identify decision goals w/ tolerable overall

uncertainty– Identify major uncertainties (cause decision error)– Identify strategy to manage each major uncertainty

Use the Field Analytical Method (FAM) and a Dynamic Work Plan (DWP) to effectively manage sampling uncertainty (ensure sample representativeness)

40 of 40

End of Module 5

Thank you

Questions?

We will now take a

15 minute break.

Please be back in 15 minutes