a rationale for the assessment of errors in the sampling ... · a rationale for the assessment of...
TRANSCRIPT
il
A RATIONALE FOR THE ASSESSMENT OF ERRORS IN THE SAMPLING OF SOILS
by
J. Jeffrey van Ee and Louis J. BlumeExposure Assessment Division
Environmental Monitoring Systems LaboratoryLas Vegas, Nevada 89193
and
Thomas H. StarksEnvironmental Research Center
University of Nevada - Las VegasLas Vegas, Nevada 89154
Environmental Monitoring Systems LaboratoryOffice of Research and Development
Las Vegas, Nevada 89193-3478
NOTICE
The information in this document has been funded wholly or inpart by the United States Environmental Protection Agency undercooperative Agreement CR814701-01 . It has been subject to theAgency’S peer and administrative review, and it has been approvedfor publication as an EPA document.
TABLE OF CONTENTS
LIST OF FIGURES
LIST OF TABLES
ACKNOWLEDGMENT
INTRODUCTIONPurpose
BACKGROUNDSources of ErrorRepresentative SamplingData Quality Levels for Analytical Measurements inSuperfund
NUMBER OF QUALITY ASSESSMENT SAMPLESBackgroundConfidence Levels for the Assessment of MeasurementVariability
Quantitative Assessments of BiasRecommendations for the Assessment and Control of Biasand Measurement Variability
DEFINITIONS AND TERMSQuality Assessment SamplesSplit SamplesSpiked SamplesBlanksBatch
AVAILABILITY OF QUALITY ASSESSMENT SAMPLES
A RATIONALE FOR ASSESSING ERRORSQuality Assessment Sample Design
Field DuplicatesPreparation SplitsField Evaluation SamplesExternal Laboratory Evaluation SamplesPreparation Rinsate BlanksField Rinsate Blanks
Equations for Estimating Bias and Precision
AN ALTERNATIVE QA DESIGN THAT DOES NOT EMPLOY FES AND ELES
EXAMPLE OF QUALITY ASSESSMENTQA/QC from a Pilot StudyPost Pilot Study Data AnalysisDevelopment of the QA Plan for the Primary Study
Pageiv
v
vi.
11
335
7
99
1011
12
151519191919
21
2222252627282930
31
34
38383944
ii
RECOMMENDATIONS FOR FUTURE WORK
REFERENCES
APPENDIX AKEY COMPONENTS
TrainingPilot StudyAuditsDocumentation
APPENDIX BDEFINITIONS
Quality AssuranceQuality ControlQuality AssessmentSoilStandard Additions
APPENDIX CSoil Sampling Methods Table
APPENDIX DThe Lognormal Distribution and Logarithmic
Transformations
APPENDIX ENon-blind Quality Assessment Samples in the Contract
Laboratory Program
APPENDIX FUpper Confidence Limits for the Variance, s2, as a
Function of the Number of Degrees of Freedom forthe Variance Estimate,ss.
iii
45
46
484848484950
51515151515152
5353
54
54
55
55
57
LIST OF FIGURES
Figure
1. Quality Assessment Samples
2. Alternative Quality Assessment Sample Design
3. Scatter Plot of QA Data from ASSESS
4. Error Plot from ASSESS
24
37
43
44
iv
LIST OF TABLES
1. Sources of Bias in Soil Sampling Studies 4
2. Sources of Variability in Soil Sampling Studies 5
3. Some 95 Percent Confidence Intervals for Variance 10
4. Types of Quality Assessment Samples or Procedures 17-18
5. Equations for Determining Precision and Bias 32-33
6. Equations for Determining Precision without FES and ELES 35
7. Quality Assessment Data 41
8. Transformed Quality Assessment Data 42
v
ACKNOWLEDGEMENT
During the initial development of this document, Kenneth W. Brownand Daniel T. Heggem of the Exposure Assessment Divisioncontributed their knowledge, words, and constructive suggestionsto the development of the first drafts. Later, Ann M. Pitchfordof the Exposure Assessment Division made further constructivechanges to the developing document. The document was thenreviewed by by variety of people with a variety of backgroundssince the guidance provided in this document is meant to apply toan entire soils study. With involves a diverse group of people.The resulting document is meant to serve as an initial "roadmap"for people to use in assessing the errors and uncertainties in asoils study. With time the importance of the guidance providedby this document is expected to become greater as more experienceis gained in applying the rationale to greater numbers ofhazardous waste site investigations. Further revisions areanticipated, but those revisions should not detract from thesignificant contributions that were made in the development ofthis document to the assessment of total quality of data from asoils investigation.
vi
Considerablequality assurance
INTRODUCTION
guidance has been provided on the importance of(QA), quality control (QC), and quality
assessment procedures for assessing and minimizing errors inenvironmental studies. QA/QC terms, such as quality assuranceproject plans and program plans, are becoming a part of thevocabulary for remedial project managers (RPM's). Establishmentof data quality objectives (DQOs) early in the process of a siteinvestigation has been stressed in EPA QA/QC guidance documents.Quality aSSeSSment practices, such as the use of duplicate,split, spiked, and reference samples, are becoming widelyaccepted as important means for assessing errors in measurementprocesses. Despite the existence of numerous and various foresof guidance for hazardous waste site investigations, there havebeen no clear, concise, well-defined strategies for precisely howrecommended QA/QC practices can be utilized.
Purpose
The purpose of this document is to provide a foundation foranswering two basic questions:
How many, and what type, of samples are required toassess the quality of data in a field sampling effort?(quality assessment samples)
How can the information from the quality assessmentsamples be used to identify and control sources oferror and uncertainties in the measurement process?
This document expands upon the guidance for quality controlsamples for field sampling as contained in Appendix C of EPA'sData Quality Objectives for Remedial Response Activities-Development Process (9). This report outlines, in greaterdetail, strategies for how errors may be assessed and minimizedin the sampling of soils with emphasis on inorganic contaminants.
Basic guidance for soil sampling, which includes adiscussion of basic principles, may be found in EPA’s Revised
User's Guide (15). The UsersGuide is intended to be revised on a periodic basis. It isanticipated that some of the guidance provided in this documentwill eventually be incorporated into the Users Guide.
The primary audience for this document is assumed to beRPM’s who have a concern about the quality of the data beingcollected at Superfund sites but have little time to understandthe complexities of the processes used to assess the quality ofdata from the total measurement process. The approach outlinedin this document for assessing errors in the field sampling of
1
inorganic in soils may be transferable, with modification, toother Contaminants in other media.
The example offered at the end of this document illustratesthe planning process for determining a reasonable number ofquality assessment samples.The examples also demonstrates howthe information from the process may be used to document thequality of the measurement data,and how this data may be used tomake adjustments to the monitoring program.
2
BACKGROUND
Superfund and RCRA site investigations are complicated by:the variety of media being investigated, an assortment ofmethods, the diversity of people, the variety of contaminants,and the numerous risks and effects to human health and theenvironment. Many phases exist in Superfund site investigations.An initial phase, generally described as a “preliminaryinvestigation, “ consists of collecting and reviewing existingdata and data from limited measurements using practically anyavailable method. The next phase, generally described as "sitecharacterization, ” uses selected methods and prescribedprocedures to characterize the magnitude and areal extent of thecontamination. Final phases include an examination of remedialactions, which involve an examination of treatment technologies,and monitoring to assess the degree of cleanup at a site. Afinal phase may require long-term monitoring to substantiate thatno new or additional threats occur to human health and theenvironment. Throughout Superfund site investigations QA/QCprocedures change as data quality objectives vary and differentphases occur.
Sources of Error
In many of the phases of Superfund and RCRA siteinvestigations, errors and uncertainties occur. During themeasurement process, random errors will be induced from:sampling; handling, transportation and preparation of the samplesfor shipment to the laboratory; taking a subsample from the fieldsample and preparing the subsample for analysis at thelaboratory, and analysis of the sample at the laboratory(including data handling errors). The magnitude of these errorscan be expected to vary during the measurement process and makeit more difficult to determine the natural variability ofcontaminants in the environment. Errors introduced in theinterpretation and analysis of data are not considered in thisdocument.
Typically, errors in the taking of field samples are muchgreater than preparation, handling, analytical, and data analysiserrors; yet, most of the resources in sampling studies have beendevoted to assessing and mitigating laboratory errors. It may bethat those errors have traditionally been the easiest toidentify, assess and control. This document adopts theapproaches used in the laboratory, e.g. the use of duplicate,split, spiked, evaluation and calibration samples, to identify,assess and control the errors in the sampling of soils.
Systematic errors, termed bias (B), can accumulate during ameasurement process. Bias may result from: faults in samplingdesign, sampling procedure, analytical procedure, contamination,losses, interactions with containers, deteriorations,
3
displacement of phase or chemical equilibria, and inaccurateinstrument Calibrations. (Table 1) Bias causes the mean value ofthe sample data to be either consistently higher or consistentlylower than the “true” mean value. Laboratories usually introducevarious quality control samples into their sample load to detectpossible bias. Bias in soil sampling is difficult to detect.components of bias can be discovered by the technique describedas standard additions or by using evaluation samples. On theother hand, it is difficult to demonstrate that bias is notpresent because an apparent lack of bias may be the result of aninability to measure it rather than its actual absence.
Table 1. Sources of Bias in Soil Sampling Studies
Bs = Measurement bias introduced in sample collection not causedby contamination
Bsc= Measurement bias introduced in sample collection caused by
contaminationBh = Measurement bias introduced in handling and preparation not
caused by contaminationBhc
= Measurement bias introduced in handling and preparationcaused by contamination
Bss = Measurement bias introduced in subsampling not caused bycontamination
Bssc = Measurement bias introduced in subsampling caused bycontamination
Ba = Measurement bias introduced in the laboratory analyticalprocess not caused by contamination
Bsc= Measurement bias introduced in the laboratory analyticalprocess caused by contamination
Bn = Total measurement bias
NOTE : It is necessary to realize that biases, other thancontamination biases in the measurement of a sample, will oftenbe dependent on the original concentration of the contaminantbeing measured and on the sample matrix. Biases caused bycontamination are listed separately because some QA samples, suchas rinsate samples, detect only contamination bias.
Also, variability occurs in the measurement process from theheterogeneity of the soil and random errors throughout themeasurement process. The variability caused by any type ofrandom error is frequently described quantitatively by thevariance, σ2, of the random error, or by the positive squareroot, the standard deviation, σ, of the random error. Variancesof independent random errors are additive in that the variance ofthe sum of errors is the sum of the variances of the individualerrors (Table 2). Other quantifications of variability do nothave this useful, additive property.
4
Table 2. Sources of Variability in Soil Sampling Studies
NOTE : It is assumed that the data are normallydistributed or that a normalizing data transformation hasbeen performed.
Biases and variability can accumulate during a measurementprocess to the point where the data are unsuitable for meetingthe objectives of the study. Often at the end of, but preferablyduring the planning of a study, a question arises as to whetherthe data are acceptable in terms of accuracy, precision,representativeness, and completeness, i.e. DQOs. Qualityassessments, i.e., systematic investigations of the measurementprocess, can be performed to try to assess-and identify theextent of biases and variability in the measurement process andto determine whether the DQOs are being met.
Representative Sampling
Soils are extremely complex and variable which necessitatesa multitude of sampling methods. The sample collector mustselect methods that best accommodate specific sampling needs, andthat satisfy the stated sampling objectives. In addition, thesample collector is responsible for providing the appropriatesample for laboratory analysis. A soil sample must satisfythe following:
1. Provide an adequate amount of soil to meet analyticalrequirements and be of sufficiently large volume as tokeep short range variability reasonably small,
2. Provide material < 2 mm in size,
3. Be a member of the population to be evaluated and, when
5
taken in association with the other samples, berepresentative of that population.
Deposition of airborne contaminants, especially thoserecently deposited, is often evident in the surface layer of thesoil. Contaminants that have been deposited by liquid spills orby long-term disposition of water soluble materials may be foundat depths up to several meters. Plumes emanating from hazardouswaste dumps or leaking storage tanks may be found at considerabledepths. The methods of sampling each of these may be different;but all make use of one of the following three basic samplingtools : (1) scoops, (2) coring, or (3) augering devices.
Two major considerations must be addressed when selecting aspecific sampling tool. These two considerations include soilconditions and the contaminant(s) that are to be analyzed fromthe collected material. Soil condition can be extremely variablefrom location to location. For example, soils can be wet or dry,stony, cohesive (e.g., clay) or cohesionless (e.g., sand).Similarly, contaminants are extremely diverse, varying betweenmetals which in most cases are relatively immobile, to highlymobile water soluble substances, to contaminants that arevolatile.
Improper use and selection of sampling tools may result indata that are not representative of the soil environment beingsampled (See Appendix C). Measurement errors can result from atool being either inappropriate for the particular task, orimproperly used. Results based on previous experience, or froman equivalency test, may be used to evaluate and select theproper tool for a specific sampling objective. Operationalmeasurement errors are identified and assessed by implementingand utilizing a number of field QA samples. The optimal numberand timing of QA samples depend in part on the proper soilsampling method being utilized.
A variety of sampling methods may be used to obtain ameasurement of inorganics in soil. EPA's Soil Sampling Quality
User's Guide notes that the concentrations measured inan heterogeneous medium such as soil are related to the volume ofsoil sampled and the orientation of the sample within the volumeof earth that is being studied. (The term ‘support” is used todescribe this concept.) A RPM not only wants to know theconcentration of contaminants, but their location. Frequently,an average concentration of contaminants in the soil is soughtand compared against some standard. Depending on the samplingmethod used, the location of the samples collected, the number ofsamples taken, and the time the samples were collected, thereported concentrations can vary considerably even whenrelatively stable contaminants such as inorganic in soil aremeasured.
6
The processes involved in collecting representative samplesof inorganic in soil can be complicated. (The Soil Sampling
Assurance Users Guide should be consulted for furtherinformation on these processes. ) The problem of measuring thenatural variability of contaminants, such as inorganic, in soiland adequately representing the site to be studied is also aproblem for traditional QA/QC programs which have emphasized theassessment and minimization of errors and variabilities in theanalytical process.
A major problem in obtaining a “representative" sample isthe spatial scale chosen for the study. Geostatisticaltechniques, such as kriging, may also be used to estimate thenatural variability of contaminants in soil. A measure of thespread of the distribution of contaminant concentrations about
Data Quality Levels for Analytical Measurments in Superfund
As many as five different levels of quality have beenassigned by EPA in the Superfund program to analytical data.These levels have been generally associated with different phasesof a site investigation (9); however, it may prove to benecessary to have all five levels of data quality in any onephase of a site investigation. Levels III and IV involve off-site analytical laboratory measurements with Level IV in theContract Laboratory Program (CLP) having the most rigorous QA/QCprotocols and documentation. Levels III and IV are assumed inthis document in the development of a strategy for assessingfield sampling errors.
Since the assumption is made in this document that CLPlaboratories are involved in the analysis of samples from a soilsampling study, errors and biases from those laboratorymeasurements are presumed to be small and known. Theseassumptions allow greater emphasis to be given to theidentification of errors and biases in the field sampling ratherthan in the laboratory analysis.
In a pilot study, within a particular phase of a Superfundsite investigation, it may be necessary to utilize Level III andIV analytical levels to identify, assess and reduce errors infield sampling even though these analytical levels might not beneeded for every sample and measurement. For example, a fieldportable x-ray fluorescence instrument, which measures inorganicin soil, is frequently used to identify sampling locations forsamples to be sent to the CLP. The data quality from theportable x-ray fluorescence instrument is not classified as beingLevel 111 or IV; however, data from the instrument is used toscreen samples for subsequent analysis by Level III and IVmethods. It may be advantageous to compare the performance of thefield-screening, portable x-ray fluorescence instrument against
7
more rigorous, well characterized laboratory analytical methodseven though the level of data quality desired from each method isdifferent.
The assessment of errors from non-conventional "sampling"and analytical methods, such as the portable x-ray fluorescenceinstrument, are not specifically addressed in this document.
Two important factors must be considered in the collectionof environmental data. These are the probability that thecollected data will yield the correct assessment or solution foran environmental problem and the costs. The strategy developedin this document recognizes that these important factors must beconsidered in the implementation of QA/QC measures in thesampling of soils for inorganics.
8
NUMBER OF QUALITY ASSESSMENT SAMPLES
Background
A key question for a RPM is: how many samples must becollected to adequately characterize the site? A question for aQA/QC officer is: how many quality assessment samples must betaken to adequately characterize the errors and uncertainties inthe measurement process? The timing and type of those qualityassessment samples in the measurement process determines the typeof information that is obtained. The number of qualityassessment samples is determined by the available resources andthe degree to which investigators need to be sure that they haveadequately characterized the measurement process. The simplestcase is when one method, one sampling crew, and one laboratoryare used to analyze the soil samples. A more difficult, andprobably more typical, case is when more than one batch ofsamples are either collected or analyzed at various times or byvarious laboratories.
The percentage of the total monitoring effort allocated toQA/QC activities will depend on many factors including the sizeof the project, the available knowledge concerning sampling andanalytical procedures, the relationship of risk to human healthand the environment at various pollutant concentrations, thenearness of action levels to method detection limits, and thenatural variability and distribution of the contaminants.Typically the smaller the project, the larger will be theproportion of cost allocated to QA/QC. New, untried procedureswill typically require pilot-study runs and additional trainingfor personnel. If the action level is near the method detectionlimit, there will be little room for error in the measurements,and the QA/QC effort may have to be large to assure thatmeasurement errors are kept small. If the natural variability ofthe contaminants is relatively large, it may be necessary tocollect more samples rather than collect more quality assessmentsamples. One should not specify a certain percentage of aproject’s costs be allocated to QA/QC without considering theabove factors.
Previous EPA guidance for the number of quality assessmentsamples has been one for every 20 field samples (9). However,such rules of thumb are oversimplifications and should be treatedwith great caution. A better approach is to determine how eachtype of QA sample is to be employed and then determine the numberfor that type based on the use. For example, field duplicates,i.e., duplicate samples at the same location, are used toestimate the combined variance contribution of several sources ofvariation. Hence, the number of field duplicates to be obtainedin the study should be dictated by how precise one wants thatestimate of the total measurement variance to be.
9
Confidence Levels for the Assessment of Measurement Variability
The precision of an estimate, s’, of the "true" variance,σ2, depends on the number of degrees of freedom for the estimatewhich is directly related to the number of quality assessmentsamples. Table 3 gives the 95% confidence intervals for variousnumbers of degrees of freedom, based on an assumption that thedata are, or have been transformed to, normally distributed data.Methods for obtaining such confidence intervals for any number ofdegrees of freedom are given in most statistics texts.
If it is decided that 20 degrees of freedom givessatisfactory precision for the estimate of the total measurementvariance, one might equally space 20 field-duplicate samplesamong the routinely collected field samples so as to have 20duplicates by the end of the study. (Note: Each pair, fieldduplicate sample and associated routine sample, provides one
10
degree of freedom in the estimation of the between-collocated-samples variance. ) Alternatively, one might take duplicatesamples at a fairly high frequency at the start of the studyuntil 10 duplicate pairs are obtained and then obtain theremaining ten duplicate pairs at a reduced rate over theremainder-of the study. This second procedure would allow anearly estimate of the variance based on 10 degrees of freedom todetermine whether the QA plan is resulting in error variances inthe range expected, and the remaining ten pairs would allow theafter-study variance estimate to take the entire study intoaccount.
The confidence intervals in Table 3 are called two-sidedconfidence intervals because they put both upper and lower limitson the value of σ2. However, in practice, one is particularlyinterested in the upper limit; that is, one is interested in howmuch larger than the estimated variance might the true variancebe. If the true measurement error variance is seriouslyunderestimated in a pilot study, it may cause one to proceed toan expensive final study with an inadequate protocol. If themeasurement error variance is underestimated in a final study itmay cause the RPM to put more reliance on the study results thanare warranted and may also cause an inadequate protocol to becopied in a following study. For these reasons, some may be moreinterested in one-sided confidence intervals that provide only anupper limit on the true variance (i.e. , since a variance cannotbe negative, a one-sided confidence interval is between zero andan upper limit). Such upper limits for one-sided confidencelimits are provided in Appendix F for confidence levels of 90, 95and 99 percent. (Intervals between zero and the upper limits inTable 3 would be 97.5 percent one-sided confidence intervals;that is, for 2 degrees of freedom, one would be 97.5 percentconfident that σ2 is between zero
Quantitative Assessments of Bias
Quantitative assessments ofdifferent types of bias that may
and 39.21s2.)
bias are complicated by thebe present in a study and the
different times when those biases may occur. There may beconsistent additive-constant biasing error (e.g., the datahandling algorithm might add a constant to all measurementsentered into the database). There may be consistentmultiplicative biasing error (e.g., a recovery error in theanalytical system). There are also random biasing errors of theadditive and multiplicative types (e.g., sample taking, recovery,contamination, and calibration errors). In sample taking, thefield crew may occasionally have the bottom portion of a soilcore drop out of the sampler prior to bagging the core; ifconcentration decreases with depth, this would be a randombiasing error that would increase the expected concentrationabove “true” value. The rate of recovery of a chemical fromsamples may depend on the individual matrix properties of the
11
samples. The randomand to quantify. If
biasing errors are most difficult to detectcontamination occurs in one of 20 samples on
average, the use of 20 contamination blanks would have a 36(=100[1-0.05] 20) percent chance of not encountering acontamination incident. Random biasing errors contribute tomeasurement variance as well as the bias of the measurementsystem. The number of samples required to detect random biaswill depend on the distribution of the biasing errors, and thisdistribution will generally be unknown. A major problem in dataanalysis is the separation of random biasing errors from randomnonbiasing errors. Estimation of the magnitude of bias and itseffect on the estimates and decisions, would require many morequality assessment samples than are required for the detection ofbias. Bias estimates that are reported in the literature areoften only estimates of the analytical bias obtained either asthe difference in recovery rate from 100% obtained by the methodof standard additions, or the average difference between reportedand reference values of performance evaluation samples.
With these considerations in mind, it would seem that thebest one can do is to include some bias-detecting qualityassessment samples in each batch of routine samples and hope thatthey will detect bias if it is present. If bias is detected, aneffort should be made to eliminate the source of bias, ratherthan attempt to correct routine-sample measurements for bias.
Non-blind samples, such as the calibration check standardsat the analytical laboratory (Appendix E), are used to assessbias in the laboratory and provide a quality control function.That is, if measurements of these check standards differ by toomuch from their reference values, the instrument is declared "outof control" and is re-calibrated. Then, samples between the lastin-control reading and the out-of-control reading may bereanalyzed. The frequency of use of samples of this qualitycontrol type should be based on the consequences associated without-of-control data and the costs of the analyses of thesesamples versus the costs of re-analyzing routine field samples inout-of-control situations. This frequency of use is usuallyrelated to the probability of obtaining an out-of-controlsituation in the laboratory with the objective being to minimizeexpenditures of both time and money while obtaining data ofquality, suitable for the intended end-use of the data.
Recommendations for the Assessment and Control of Bias andMeasurement Variability
A two-phased approach is suggested to answer the questionsposed in the introduction:
How many, and what type, of samples are required toassess the quality of data in a field sampling effort?(quality assessment samples)
12
How can the information from the quality assessmentsamples be used to identify and control sources oferror and uncertainties in the measurement process?
The first phase involves the acquisition of data to estimatetotal measurement variability and bias. The second phaseinvolves identification of the sources of the bias andvariability.
The required number of samples will vary depending on thedata quality objectives and available resources. The morequality assessment samples that are used, the better theassessment of measurement errors. Five field-duplicate samples,as demonstrated in Table 3, will yield an estimate of themeasurement variability that may be high by a factor of 6 or lowby a factor of 2.5 with a confidence level of 95%. As notedearlier, accurate assessments of measurement bias are morecomplicated. A recommendation of one quality assessment samplefor bias per batch would allow for the plotting of bias on acontrol chart to determine if the measured bias is withinacceptable limits. (Bias may be random, constant, or varying. )In either the measurement of bias, or measurement variability,the accumulation of historical data is extremely important injudging the appropriate number of quality assessment samples andthe relative importance of that data to the overall objectives ofthe study.
It must be emphasized that estimates of measurement-errorvariance components are of little value if they are based on sofew degrees of freedom that they may differ from the truevariances by large factors (Table 3). Hence, even in pilotstudies with few routine samples, it is important to obtainmeasurement-error component variance estimates that are based ona sufficient number of quality assessment samples (i.e., based ona sufficient number of degrees of freedom) that the user can havesome confidence that the large estimates actually represent largevariances and small estimates represent small variances.Otherwise, corrective actions taken to improve precision may be awaste of money, and failure to take corrective action may resultin the failure of the subsequent study.
Experience indicates that an effective quality assuranceprogram is negated if certain key elements of a sampling effortare not adequately addressed. Those elements are: training,pilot studies, audits and documentation. More detaileddiscussion of those key components is provided in Appendix A:however, the importance of pilot studies to the overallmonitoring effort cannot be stressed enough. The experience anddata gained during the pilot study can be extremely important tothe success of the larger monitoring effort.
13
Frequently, time and financial constraints lead to a minimumnumber of samples being collected in the initial measurementphase of a hazardous waste study. If sufficient historicalquality assessment data has not been collected for the selectedsampling crews, sampling methods, and analytical laboratoriesinvolved in the initial measurement phase, an accurate assessmentof performance during the “pilot study" will require a relativelylarge number of quality assessment samples. However, even withan experienced sampling crew, tested sampling methods,and well-characterized analytical laboratories, unique characteristics ofa Particular site may require an increased number of qualityassessment samples to measure performance against stated dataquality objectives.
14
DEFINITIONS AND TERMS
Problems can often occur when important terms inenvironmental sampling are not defined, poorly defined, or notwell understood. Commonly accepted terms such as soil can havemany different definitions depending upon the interests andbackground of an investigator. Major errors can occur ifeveryone involved in an investigation has a differentunderstanding of a term.
While the definitions contained in this document may not beuniversal, it is important in an understanding of the overallprocess of assessing errors and identifying their sources thatthe terms be defined early. Important steps and samples that arerequired for the assessment of variability and bias are definedin the main document. Basic definitions that are less criticalto an understanding of the quality assurance process are inAppendix B.
Quality Assessment Samples
Many terms have been used to describe quality assessmentsamples. (Quality evaluation is also used.) Quality assessmentsamples are defined as those samples that allow statements to bemade concerning the quality of the measurement system. Thesesamples have two primary reasons for utilization. They allowassessment of the quality of the data, and most importantly, theyallow for control of data quality to assure that it meets theoriginal objectives.
The objective of this section is to identify the varioussample types, define what they are used for, and how they havebeen previously used in characterizing the measurement process.
Quality assessment samples include samples from threegroups, based upon whether they are double-blind, single-blind,or non-blind to the analytical laboratory. Double-blind samplesare samples that cannot be distinguished from routine samples bythe analytical laboratory. Single-blind samples are samples thatcan be distinguished from routine samples but are of unknownconcentration. Non-blind samples are samples that have aconcentration and origin that are known to the analyticallaboratory. Some references state that quality control samplesare only those that are blind to the analytical laboratory (9).Other documents refer to quality control samples as non-blind tothe analytical laboratory (3,13). The intent of categorizingquality assessment samples into these categories, i.e., double-blind, single-blind, and non-blind, is to avoid confusion due toterminology. The key point is that all of the samples discussedhere refer to those samples that make some statement about thequality of the measurement system. This discussion will notinclude samples such as background samples or critical samples
15
(9) because they are required for characterization of thecontamination at a site and not for characterization of theerrors in the measurement system.
Table 4 and Appendix E list typical quality assessmentsamples and describe how measurements of these samples are usedin the control of the measurement process and in the evaluationof the quality assurance procedures. To obtain an unbiasedmeasure of the internal consistency of the samples and theiranalyses, the individual quality assessment samples must bedouble-blind and should be labeled as routine samples so that theanalyst (and preferably also the laboratory) does not know therelationship between the samples. This reduces the chances ofconscious or unconscious efforts to improve the apparentconsistency of the analyses.
16
Table 4. Types of Quality Assessment Samples or Procedures====================================================
Double-blind Samples
1. Field Evaluation Samples (FES)
These Samples are of known concentration, subjected to the samemanipulations as routine samples and introduced in the field atthe earliest stage possible. They can be used to detectmeasurement bias and to estimate precision.
Field Evaluation Samples (LLFES)
These samples are essentially the same as field evaluationsamples, but they have very low or non-existent concentrations ofthe contaminant. They are used for determination ofcontamination in the sample collection, transport, and analysisprocesses. They can also be used for determination of the systemdetection limit (13).
This sample is similar to the field evaluation sample except itis sent directly to the analytical laboratory without undergoingany field manipulations. It can be used to determine laboratorybias and precision if used in duplicate. We recommend using thesame sample as the FES to allow isolation of the potentialsources of error. Spiked soil samples have been used as externallaboratory evaluation samples in past studies for dioxin,pesticides, and organics (1,6), and natural evaluation sampleshave been used for metals analysis in soil and liquid samples(3,13).
This sample is similar to the LLFES except it is sent directly tothe analytical laboratory without undergoing any fieldmanipulations. It is used to determine method detection limit,and the presence or absence of laboratory contamination. Werecommend using the same sample source as for the LLFES to allowisolation and identification of the source of contamination.
This is a routine sample spiked with the contaminant of interestin the field. Because of the inherent problems associated withthe spiking procedure and recovery it is not recommended for usein field studies (9).
An additional sample taken near the routine field sample todetermine total within-batch measurement variability. The
17
differences in the measurements of duplicate and associatedsamples are in part caused by the short-range spatial variability(heterogeneity) in the soil and are associated with themeasurement error in the field crew’s selection of the soilvolume to be the physical sample (i.e., two crews sent to thesame sampling site, or the same crew sent at different times,would be unlikely to choose exactly the same spot to sample).
7. Split (PS)
After a routine field sample is homogenized, a subsample is takenfor use as the routine laboratory sample. If an additionalsubsample is taken from the routine field sample in the same wayas the routine laboratory sample, this additional sample iscalled a preparation split. The preparation split allowsestimation of error variability arising from the subsamplingprocess and from all sources of error following subsampling.This sample might also be sent to a reference laboratory to checkfor laboratory bias or to estimate inter-laboratory variability.These have also been called replicates (5).
Single-Blind Samples
1. Field Rinsate Blank (PRB)
These samples, also called field blanks (9), decontaminationblanks (14,15), equipment blanks (5), and dynamic blanks (5), areobtained by running distilled, deionized (DDI) water through thesampling equipment after decontamination to test for any residualcontamination.
2. Rinsate Blank (PRB)
These samples, also called sample bank blanks (12,14,15), areobtained by passing DDI water through the sample preparationapparatus after cleaning in order to check for residualcontamination.
These samples are used when volatile organics are sampled, andconsist of actual sample containers filled with ASTM Type IIwater, and are kept with the routine samples throughout thesampling event. They are then packaged for shipment with theroutine samples and sent with each shipping container to thelaboratory (9). This sample is used to determine the presence orabsence of contamination during shipment.
Non-blind Samples
These samples (e.g. Laboratory Control Samples (LCS)) are used inthe Contract Laboratory Program to assess bias and precision.For convenience, these samples are described in Appendix E withthe definitions being adapted from the CLP Inorganic Statement ofWork #788 (10).
Split Samples
Samples can be split to provide:
samples for both parties in a litigation or potentiallitigation situation;a measure of the within-sample variability;materials for spiking in order to test recovery; anda measure of the analytical and extraction errors.
The location Of the sample splitting determines the components ofvariance that are measured by the split. A split made in thesample bank (i.e., at the sample preparation facility to whichsamples are sent from the field) measures error introduced fromthat level onward. A split made in the field includes errorsassociated with field handling. A split or series of subsamplesmade in the laboratory for extraction purposes measures theextraction error and subsequent analytical errors.
Spiked Samples
Spiked samples are prepared by adding a known amount ofreference chemical to one of a pair of split samples. Comparingthe results of the analysis of a spiked member to that of thenon-spiked member of the split measures spike recovery andprovides a measure of the analytical bias. Spiked samples aredifficult to prepare with soil material itself. Frequently thespike solution is added to the extract of the soil sample. Thisavoids the problem of mixing, but does not provide a measure ofthe interaction of the chemicals in the soil with the spike;neither does it provide an evaluation of the extractionefficiency. A predigest spike, as utilized in the CLP (9) wouldallow a check of the-extraction or digestion efficiency: -
addition, if the laboratory does the spiking, the spikingblind to the laboratory.
Inis non-
Blanks
Blanks provide a measure of various cross-contaminationsources, background levels in the reagents, decontaminationefficiency, and other potential error that can be introduced fromsources other than the sample. For example, a blank introducedat the earliest point in the field can measure input fromcontaminated dust or air into the sample. A rinsate blank, i.e.,decontamination sample, measures any chemical that may have beenon the sampling and sample preparation tools after thedecontamination process is completed.
Batch
A batch is defined as a group of samples which are sampled,shipped and analyzed under similar conditions. The total number
19
of routine and quality assessment samples in a batch is dependenton the desired frequency of quality assessment sampling that abudget will allow. The term batch is synonymous with the term“sample delivery group" as used in the CLP (9).
20
AVAILABILITY OF QUALITY ASSESSMENT SAMPLES
Presently, performance-evaluation-materials (PEMs) forinorganic in soils are not readily available. PEMs closelyresemble routine samples, are well characterized, and areprovided as unknown samples to a laboratory to demonstrate thatthe laboratory can produce analytical results within specifiedlimits of performance. ) Soil performance-evaluation-materialsare available for routine soil characterization for aciddeposition purposes (13), and performance materials have beendeveloped for dioxin analysis (1,6). These materials areavailable as quarterly blind samples; however, adequate PEMs donot exist for analysis of inorganic in hazardous waste soils.
To meet the growing needs for PEMs the EnvironmentalMonitoring Systems Laboratory - Las Vegas (ESML-LV) inconjunction with EPA’s Office of Emergency and Remedial Response(OERR) has begun a project1 to develop, test, and produce "case-specific” or "site-specific” PEMs. Samples taken at Superfundsites are organized into groups called “Superfund Cases". Eachcase of samples is sent to a specific CLP laboratory foranalysis. The objective is to provide a multi-matrix, multi-analyte, multi-level library or shopping list of PEMs which theRegional site-managers could order by telephone or from acatalog. Each PEM would be included as just another samplewithin the Case. Ultimately the PEMs would be double-blind tothe laboratories. The PEMs would be tailored-made for eachSuperfund Case of analytical samples to enable more reliable,accurate decision making about Superfund sites.
PEMs are an important component of the rationale that isused to assess variability and bias throughout the measurementprocess; however, variability and biases may also be assessedwithout the use of these materials since present availability islimited. An alternative QA design that does not rely on the useof FES and ELES is provided after the discussion of the rationalethat is based on the use of PEMs.
1 Butler, L., 1989. Personal communication.Environmental Monitoring Systems Laboratory. Las Vegas,NV.
21
A RATIONALE FOR ASSESSING ERRORS
Quality Assessment Sample Design
An effective quality assurance program should ensure thatthe uncertainty associated with the measurement system will beinsignificant when compared to the uncertainty allowed for thepopulation of interest. As stated by J.K. Taylor (7):
"When the uncertainty of a measured value is one-thirdor less of the permissible tolerance for its use, itcan be considered as essentially errorless for thatuse. "
Therefore it is critical that a quality assurance system providesfor quantification of total measurement error. Measurement errorconsists of three major components, i.e. sample collection,preparation, and analysis. Each of these phases can then bedivided into smaller components depending on the specific designof the operation.
It is important to realize that if the error associated withthe sample collection or preparation phase is large, then thebest laboratory quality assurance program is inadequate. Thus amanager should apply the greatest amount of emphasis to the phasethat contributes the largest component of error; this will not bepossible if the quality assurance design does not provide forerror evaluation of the major measurement phases.
The following sample design (Figure 1) is proposed as acomplete quality assurance design that will allow determinationand control of the various components of measurement bias andprecision. It is assumed that only one analytical laboratory isutilized; nevertheless, the design can be applied to multiplelaboratories. A multiple-laboratory approach is not discussedhere for simplicity. The samples, discussed in the design, weredefined previously in Table 4.
Figure 1 depicts how quality assessment samples of severaltypes are treated at the sample collection, preparation, andanalysis stages. Starting at the left of the diagram, thecollection of a field duplicate is shown. At the locationselected for the duplicate, two collocated samples are collected.One is designated as the routine sample (RS), the other as thefield duplicate (FD). During the preparation phase, after aroutine field sample has been homogenized, a subsample is takento be used as the routine laboratory sample. If an additionalsubsample is taken from the routine field sample, this additionalsample is called a preparation split. In a similar manner, a subsample is also obtained from a field duplicate to provide thelaboratory sample from which a concentration measurement for the
22
FD will be obtained. These samples are then forwarded to theanalytical laboratory for analysis.
Moving to the middle of the diagram, paired evaluationsamples are shown entering the process at two stages. Ingeneral, fleld evaluation samples (FES) are introduced as earlyin the collection and packaging process as possible. However, inthe case of soil sampling, it 1S normally not possible to passthem over the sampling tools, so they enter the processimmediately after the collection step. These samples are thensubjected to the same handling and analysis procedures as theother samples. The external laboratory evaluation samples (ELES)are introduced after the preparation stage in such a way thatthey cannot be identified as QA samples by the laboratory andthereby serve as double-blind samples. These samples are thensubjected to the same analytical procedures as routine samples.
On the right side of the diagram, two types of rinsateblanks are shown. The field rinsate blank (FRB) is used to checkfor sample-collection equipment contamination, and thepreparation rinsate blank (PRB) is used to check for preparationequipment contamination.
Some consideration must be given to how quality assessmentsamples should be assigned to batches of routine samples. Eachbatch should contain either one pair of field evaluation samplesor none. Typically, external evaluation samples will only beassigned to batches of samples containing field evaluationsamples, and, in such cases, only one pair will be assigned to abatch. Any particular batch may contain zero, one, or severalfield duplicates and their associated routine samples. However,some attempt should be made to distribute field duplicatesthroughout the batches from the beginning to the end of thestudy . Rules for the assignment of preparation splits, andassociated routine samples, to batches are the same as for fieldduplicates. As a general rule, each batch should contain onefield rinsate blank and one preparation blank.
23
Figure 1
QUALITY ASSESSMENT SAMPLES
24
Field Duplicates (FD)
Function of Field Duplicates:To provide data required toestimate total measurementerror variance minus between-
In other words, fieldduplicates can be used toestimate the sum of allmeasurement-error variancecomponents except thebetween-batch error variancecomponent. To assessbetween-batch errors, fieldevaluation samples orexternal lab evaluationsamples may be used.
Since fieid duplicates are employedmeasurement error variance and since an estimate of this variance
in the estimation of total
is required, at least 20 pairs (i.e., routine sample and fieldduplicate (co-located) sample) or 10 triples (i.e., routinesample and two field-duplicate samples) must be obtained to meetthe minimal 20 degrees of freedom objective.
25
Preparation Splits (PS)
Function of PreparationSplits:To provide data required toestimate the sum ofsubsampling and analytical
accomplish this, the splitmust be performed before thesample arrives at theanalytical laboratory.
NUMBER REQUIRED:If the estimation of the components of variance is an importantobjective of the project, then one should follow the 20 degreesof freedom rule and run at least 20 preparation pairs (i.e. ,routine subsample and preparation-split subsample). However,unlike estimation of the total measurement error variance,estimation of variance components may be unnecessary in thequality evaluation of some projects. If the estimation ofvariance components is unnecessary, then the only reason forpreparation splits might be for quality control purposes and thenumber would be determined in terms of the quality controlrequirements.
26
Field Evaluation Samples (FES)
Function of Field Evaluationsamples (in pairs )To provide data which when takenin conjunction with the dataobtained from field duplicatesamples and their associatedroutine samples allows one toobtain unbiased estimates of thetotal measurement error variance,
the between-batch error
The data from the FES also allowestimation of the handling error
and of the totalmeasurement bias minus the samplecollection biases (Bn-Bs-Bsc).
NUMBER REQUIRED:Since field evaluation samples are employed in the estimation ofthe total measurement error variance and since an estimate ofthis variance is required, at least 21 field-evaluation pairsmust be obtained to meet the minimal 20 degrees of freedom forall variance estimates. (The number of required pairs is 21rather than 20 because one of the variances being calculated fromthe data is the variance of the paired sample averages;therefore, 21 averages are required to obtain a variance estimatewith 20 degrees of freedom.)
27
External Laboratory Evaluation Samples (ELES)
Function of ELES:To provide data required toestimate the sum of the biasesdue to analysis and to datahandling (Ba+Bsc), and analytical
NUMBER REQUIRED:If the estimation of the components of variance is an importantobjective of the project, then one should follow the 20 degreesof freedom rule and run at least 20 laboratory evaluation pairs.However, unlike estimation of the total measurement errorvariance, estimation of variance components and bias componentsmay be unnecessary in the quality evaluation of some projects.If the estimation of variance and bias components is unnecessary,then the only reason for laboratory evaluation samples might befor quality control purposes and the number would be determinedin terms of the quality control requirements.
28
Preparation Rinsate Blanks (PRB)
Function of PRB:To provide data required toestimate the sum of biascaused by contamination,analysis and data handling( Ba+ Bh c+ Bssc+ Bs c) .
29
Field Rinsate Blanks (FRB)
Function of FRB:To provide data required toestimate the sum of the bias causedby contamination at the time ofsampling and at the laboratory andby analysis and data handling(Ba+Bac+Bsc)
30
Equations for Estimating Bias and Precision
Once the analytical results are received, computation ofbias and precision values is the next step. Bias may beexpressed as the difference between the measured concentration ofthe evaluation Samples and the reference or known value for theevaluation sample. A reference value and an expected range ofvalues are usually available for evaluation samples. If themeasured values are within this range, then one can say that biashas not been detected. If the measured values are outside thisrange, then bias may be present and its amount may be estimatedfrom the differences between the measured and the referencevalues.
Precision is usually described by variance, althoughstandard deviations are sometimes used. However, standarddeviations are not additive, while variances are. Table 5provides equations for the variance estimates. Most of theseequations are based on the statistical definitions of variancefor the difference between paired values. Subscripts "WFES" and"
SFES" refer to within and between-batch variances, respectively,which are computed from paired field-evaluation samples.Subscripts "1" and "2" refer to individual samples in a pair. Indeveloping these equations, it is assumed that splits and fieldduplicates were assigned to sample locations such that nolocation had both a field-duplicate and a preparation split. Ifduplicates and splits are assigned to the same locations, some ofthe above variance formulas must be modified. However, all theabove variance components can be estimated in either case. Thesymbol ‘n” always represents the number of pairs involved. Itis also assumed, as will typically be case, that the fieldevaluation samples are of the same size (weight or volume) as theroutine laboratory samples forwarded from the preparation phase:this means that there is no subsampling of the FES in thepreparation phase. Triples are not considered here.
Details for computing variance estimates for totalmeasurement error sample collection, sample handling,subsampling, and analytical error are provided in Table 5. Ifthe estimates of variance components, involving differences ofvariance estimates, S2, yield negative values, the reportedestimate is zero.
31
Table 5
EQUATIONS FOR DETERMINING PRECISION AND BIAS
===================================================PRECISION
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Table 5 (continued)
SUGGESTED BIAS FORMULAE=====================================================
1. Field Evaluation Sample (FES): Bias = 100(F-R)/R %
where R is the reference value for the FES, and F is thereported measurement of the FES.
2. Field Rinsate Blank (FRB): Bias = 100 X/CRDL %
where X is the measured value of the FRB, and CRDL is theContract Required Detection Limit.
3. Preparation Rinsate Blank (PRB): Bias = 100 P/CRDL %
where P is the measured value of the PRB, and CRDL is asdefined above.
4. Pre-Digest Spike: Bias = 100 ( SSR-SR-SA) /SA %
where SSR is the spiked sample result, SR is the sampleresult, and SA is the spike amount.
5. Post-digest Spike: (same formula as for the pre-digest spike)==============================================
33
AN ALTERNATIVE QA DESIGN THAT DOES NOT EMPLOY FES AND ELES
The quality assurance design given in the preceding sectionemployed field evaluation samples (FES) and external laboratoryevaluation Samples (ELES). It was assumed that these sampleswould be double blind samples of homogeneous soil and that thesoil would be very similar (i.e., similar in soil type, inconcentrations of the pollutants of concern, and inconcentrations of other possible chemical interferents) to thatto be sampled in the study. This usually implies that a fairlylarge quantity (or quantities) of soil should be collected fromthe study site, sent to a laboratory to be dried, mixed, sieved,and split into homogeneous subsamples to be used as FES and ELES.It also requires analysis of a sufficient number of samples bythe laboratory to establish the homogeneity of the samples, andthe sending of samples to a number of other laboratories toestablish a reference value for the FES and ELES. This is atime-consuming process, and the time required for the process maynot be available to the RPM prior to the start of the soilsampling study. This section addresses how one may plan thestudy without FES and ELES so as to still be able to search outbias sources, to estimate some error variance components, and toestimate total measurement error variance.
The basic use of the FES in the preceding section was toestimate between batch variance. As an alternative, it issuggested that additional field duplicates may be employed forthis purpose. One may go back to a particular sampling location(e.g., a point at which one sample of soil is taken), and take afresh (collocated) sample to include with each batch (or with atleast 21 randomly selected batches if there are a larger numberof batches). If it is difficult to take so many collocatedsamples from one sampling location, one might use two or threesuch locations and take collocated samples to include in thebatches, alternating between locations (e.g., for two samplinglocations A and B, batch 1 has a collocated sample from locationA, batch 2 from location B, batch 3 from location A,...). Bycomparing the variability between collocated samples that arecollected and analyzed in different batches, with the variabilitywithin the field-duplicate-and-associated-routine-sample pairs,one can estimate the variability contributed by changes in themeasurement process between batches. These collocated samplesare actually field duplicates, but because they are used in adifferent way than the field duplicates encountered in theprevious section, they will be identified as batch fieldduplicates (BFD). Equations for determining the varianceestimates using this procedure are given in Table 6. Theassumption stated for Table 5 that field duplicates andpreparation splits are not associated with the same samplinglocation is again applied in Table 6. It is shown in Table 6that the variance component associated handling cannot beseparated from that associated with sample collection, and that
34
Table 6
EQUATIONS FOR DETERMINING PRECISION WITHOUT FES AND ELES
===============================================
Field Duplicatcs
Prep. splits
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
a This equation is appropriate when the m batch field duplicate samples arc all taken fromare sampling location. BFD is the sample mean of the m samples.
b This equation is appropriate with mj(>1) BFDs coming from sampling location j. BFDj is the sample mean of the
mj samples taken for location j.
35
the variance component associated with subsampling cannot beseparated from that associated with analysis. This loss ofinformation is a consequence of not using FES and ELES in thestudy .
The bias detection allowed by use of FES and ELES may beagain obtained at least in part by the introduction of well-characterized single-blind samples, containing the contaminantsof interest, that are already available from previous studies orfrom EMSL-LV. These are single-blind samples, since thelaboratory analyst will probably be able to distinguish them fromthe routine samples. These samples will be denoted here by FES1and ELES1. It will not be necessary to run these samples inpairs as they will not be used in variance estimation. Figure 2is a diagram of this alternative study that plays the same roleas Figure 1 did for the procedure involving FES and ELES.
36
Figure 2
Alternative Quality Assessment Sample Design
37
EXAMPLE OF QUALITY ASSESSMENT
The purpose of this example is to show how the guidance inthis document can be implemented. Data used in this example wereobtained from an actual Superfund site which was contaminated bylead deposition from a smelter; however, arrangement of the datainto batches and data from field evaluation samples are fictionaland are included for illustrative purposes.
QA/QC from a Pilot Study
A pilot study was conducted over a representative area todetermine spatial variability and extent of the leadcontamination in order to develop an efficient sampling networkfor obtaining representative measurements of contamination over alarge area. Since measurement variability is known to contributeto the overall variability of data from a sampling effort, aquality assessment program was implemented to assess thevariability from the collection, handling, and analysis of thesamples. Data from the quality assessment program were intendedfor use in determining if the measurement variability was so highas to prevent accurate assessments of the spatial variabilityfrom being made and whether corrective actions would be requiredto reduce the measurement variability. One sampling crew and onelaboratory were selected for collection and analysis of thesamples.
The quality assessment samples are identified in Figure 1and defined in Table 4. A laboratory control sample wasrecommended at a rate of one per batch. This sample was obtainedfrom the EMSL-LV EPA laboratory (9), and the acceptableconcentration range was provided to the laboratory. This samplewas used by the analytical laboratory as a quality controlsample; thus, it could not be used to estimate analyticallaboratory bias because it was a non-blind sample.
Field evaluation samples were made by sampling 50 kg of asoil type, which was the same as that in the contaminated zone,but was located 5 miles away in an area known from past studiesto have background concentrations of lead. The bulk material wasthen processed as follows:
-the material was air-dried for a one week period-the material was then sieved, and all material that passed a 2-mm sieve was saved (40 kg of air dried material)-16 grams of lead were added to the 40 kg of soil (400 ppm)-the sample was homogenized by rolling the material in a Teflon-coated drum for 48 hours-100 subsamples were made by using a closed-bin riffle splitter-10 subsamples, chosen at random, were then shipped to areferee laboratory to check the lead concentrations and to
38
verify thatsubsample
The total numberas follows:
the lead was equally distributed in each
of samples, by type, utilized in this study were
routine samples: 180field evaluation (FES) samples: 6 (3 pairs)field duplicates (FD): 10field rinsate blanks (FRB): 10preparation-split (PS) samples: 10laboratory control samples (LCS): 10total samples analyzed: 226total quality assessment samples: 36total quality control samples: 10percentage of QA/QC: 22%
High concentrations of lead were encountered in the fieldrinsate blank (FRB) from the second batch of samples sent to theanalytical laboratory. This problem was not detected until afterthe 4th batch of samples was sent to the analytical laboratory,Fortunately, this problem was not observed with later batches.Nevertheless, the sampling crew was advised of this problem andtold to be more careful. In addition, all samples associatedwith that batch were resampled and reanalyzed. This problem wasnot evident with the field evaluation samples (FES) because theycould not be used with a split-spoon sampling device. The fieldevaluation samples were introduced after the sample was taken outof the ground. It was also evident that the contamination didnot come from the preparation phase because the preparationblank was acceptable.
Post Pilot Study Data AnalySiS
After all data were received from the analytical laboratorythe equations defined in Table 5 were utilized to calculateestimates of total measurement variance, the sum of sample-collection and sample-handing variances, and between-batchvariances.
A computer program, entitled “ASSESS”2, was developed fromthe equations in Tables 5 and 6, and data were entered into theprogram to estimate measurement-error variance components. Datalisted in Table 7 were entered into the program to facilitate thecalculation of the terms described in Tables 5 and 6. Themeasured lead concentrations in soil (in mg/kg) are given for 10
2 This is a public-domain program written in Fortran foruse on an IBM PC. It may be obtained by writing to theExposure Assessment Division, Environmental MonitoringSystems Laboratory, P.O. Box 93478, Las Vegas, NV 89193.
39
preparation-split pairs and for 10 field-duplicate pairs. Theamount of data used has been kept small to make it easier to readand to illustrate the use of the computer program to calculatevariances.
The first step is to determine whether a transformation(e.g., taking the natural log (ln) of the values) is needed tostabilize the variance. The estimation of variance componentsimplies that there are unique variances to be estimated thatdescribe measurement-error variance for all measurements. Thisis not the case if measurement error variances change with sampleconcentrations. The dependence of measurement error variances onsample concentrations is frequently encountered. Fortunately,this problem can be overcome through the appropriate selectionand use of such transformations as are discussed by Hoaglin,et al. (1983) and Box and Cox (1964). A typical rule of thumbused by statisticians is that if the ratio of the maximumobservation to the minimum observation is less than 20, novariance stabilizing transformation is needed; otherwise, theneed for a variance stabilizing transformation should beinvestigated. The information provided by the field-duplicatepairs and the preparation splits is useful in deciding whether atransformation is required to stabilize variance with respect tosample concentration.
For each field-duplicate pair and each preparation-splitpair in Table 7, the sum and absolute difference of the twomeasurements is calculated. One compares how the pair absolutedifferences change as the pair sums change, which is equivalentto comparing how the sample standard deviations change as theassociated sample means change. For the field-duplicate pairs,one observes that the differences associated with the larger sumstend to be larger than those associated with smaller sums. Forexample, the median difference associated with the five largestsums is 96, while the median difference associated with the fivesmallest sums is only 28. Similarly, for the preparation-splitpairs, one finds the median difference associated with the fivelargest sums is 28, while the median difference associated withthe five smallest sums is only 1.3. This is reasonably clearevidence measurement error variances are changing with sampleconcentrations and that a transformation is required. There isinsufficient information available in the table to choose anappropriate variance stabilizing transformation. However, leadconcentration data from other soil sampling studies indicate thatthe simple logarithmic transformation (Y=ln(lead concentration))satisfactorily stabilizes the variances. The log-transformationwas performed on the data in Table 7, and the results are givenin Table 8 along with the variance component calculations andestimates from the ASSESS program.
40
Figure 3 from the ASSESSprogram further illustratesthe need for a transform ofthe original data. Thestandard deviation of the datafrom routine samples and fieldduplicates increases with theaverage concentration. Alogarithmic transform of thedata will stabilize thestandard deviation of the dataover the measuredconcentration range. Afterthis has occurred, thevariances may be computed forthe purpose of assessingvariability throughout themeasurement process. TheASSESS program allows thesecalculations to be performedeasily.
The variances reported in Table 8 indicate that the sum ofthe variances arising from sample collection and from samplehandling amount to about 7/8ths of the total measurement-errorvariance. The estimate of the sum of the variances caused bysample collection and sample handling is given by (see Table 5)
sFD
2 - spe
2 = 0.0730 - 0.0045 = 0.0685,
while the estimate of total measurement-error variance is givenby
sFD
2 + (sBFES - sWFES)/2 = 0.0730 + (0.0100 - 0.0025)/2 = 0.0768
However, the estimate sBFES, based on only 2 degrees of freedom (3
of 30 or more (Table 3). The other estimates, s’, of varianceare based on 10 or fewer degrees of freedom and may underestimatethe true variances by factors of 3 or more. If all of theestimates of variance had been based on at least 20 degrees offreedom ach, one would have much more confidence in theestimates and would certainly be justified in instituting a morerigorous training program for the sample-taking crews and inconsidering an increase in the volume of soil in each sample. Inpoint of fact, the sum of the variances caused by samplecollection and sample handling was such a large portion(approximately 1/3 ) of the total (measurement plus spatial)variation that action was taken to reduce its contribution in theprimary study. While more data from the use of field evaluationsamples and external laboratory evaluation samples would have
43
been useful in implementing the rationale for assessing errorsand variability in all phases of the pilot study, sufficientinformation was provided from the existing quality assessmentsamples to begin making some changes. Tighter adherence to therationale during the main part of the study would ensure thatsufficient data were available to accurately assess thesignificance and sources of variability during the study of theentire site.
Figure 4 from the ASSESSprogram illustrates the rangein which the estimates of thevarious variance componentscan be expected to occurwithin a 95% confidenceinterval. It is clear fromthe length of the line for sBFES
2
that greater use of fieldevaluation samples would haveimproved the assessment ofbetween-batch variability aswell as of the totalmeasurement error andcollection variances.
Development of the QAthe Primary Study
The DQOs for the
sample
Plan for
primary I Istudy were developed with the Figure 4. Error Plot frombackground data from the pilot ASSESSstudy . Goals were establishedfor accuracy, precision, completeness, comparability andrepresentativeness of the data to be collected during theexpanded study. Historical data on variability in major portionsof the measurement process were input into the ASSESS program forreference. Based upon the limited number of FES, lack of ELES inthe pilot study and the importance of having reliable assessmentsof data quality throughout the measurement process, Table 3 wasused to determine the added number of QA samples that wererequired to better estimate variability during sample collection,handling, transportion, subsampling and analysis. Even thoughthe personnel, procedures, and analytical equipment would beidentical in the primary study to that in the pilot study, thedecision-makers felt that 20 degrees of freedom were needed forall quality assessment samples to permit assessments ofvariability to be within a factor of roughly two to the actualvalue, at least to the 95% confidence interval. Theseassessments would confirm that the changes made during the pilotstudy were effective in reducing sampling and measurementvariability to acceptable levels, i.e., to permit spatialvariability to be accurately assessed.
44
RECOMMENDATIONS FOR FUTURE WORK
1.
2.
3.
4.
5.
6.
7.
Greater attempts to define and standardize QA/QC terms needto be made in conjunction with the Quality AssuranceManagement Staff (QAMS) within the Office of Research andDevelopment (ORD) and the program offices.
Protocols and materials for the preparation of QA/QC samplesfor the field need to be reviewed further and described ingreater detail. In some cases, new materials and protocolswill need to be developed and standards established.
The rationale presented in this document needs to bedeveloped further to integrate it with the work of QAMS.Data quality objectives (DQOs) are important in determiningthe level of QA/QC for a study, and QAMS effort to develop astandardized approach to the development of DQOs through theuse of computer software could incorporate the rationalepresented in this document. It appears that this rationalecould be translated into a spreadsheet or expert systemscomputer program.
Greater characterization of commonly used sampling methodsneeds to be made. The choice of a sampling methoddetermines to some extent the amount of QA/QC involved in astudy . For methods such as the portable x-ray fluorescenceinstrument, the volume of earth sampled, minimum detectionlevel, interferences, and range of contaminants detected aresome of the characteristics that need to be defined, whenpracticable, on a scale common to the other samplingmethods.
The rationale presented in this document needs to beevaluated at several actual Superfund site investigations.If the rationale proves to be workable and worthwhile, therationale needs to be adopted for use at all Superfund siteinvestigations to try to achieve a uniform measure of dataquality from all of the investigations of inorganic insoil .
Training may be conducted prior to a study or during astudy. The optimum approach is to complete the training andevaluation of sampling crews prior to the initiation of amajor study. The feasibility of establishing a nationaltraining/certification program for sampling crews should beconsidered further.
Specific sub-sampling techniques need to be defined anddeveloped for utilization in the field and the laboratory.
45
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(l0)
(11)
(12)
REFERENCES
Bennet, Tom. 1989. Standard Operating Procedures. RegionIV Analytical Services Branch, U.S. Environmental ProtectionAgency.
Box , G.E.P., and D.R. Cox. 1964. The Analysis ofTransformations. Journal of the Royal Statistical Society,Series B. Vol. 26(2):211-243.
Drouse, S. K., D.C. Hillman, L.W. Creelman, and S. J. Simon,1986. National Surface Water Survey, Eastern Lake Survey(Phase 1 - Synoptic Chemistry) Quality Assurance Plan.EPA/600/4-86/008 . U.S. Environmental Protection Agency,Environmental Monitoring Systems Laboratory, Las Vegas,Nevada.
Hoaglin, D.C., F. Mosteller, and J.W. Tukey. 1983.Understanding Robust and Exploratory Data Analysis. JohnWiley & Sons. New York, NY
La Barge, R. 1989. “A Programmatic Approach to AchievingData Quality Objectives." presented at 2nd. Annual Symposiumfor Quality Assurance for Ecological Monitoring, Feb. 1989.U.S. Environmental Protection Agency, EnvironmentalMonitoring Systems Laboratory, Las Vegas, Nevada.
Morey, Debra.Environmental
Taylor, J.K.Measurements w
1989. “Standard Operating Procedures. U.S.Protection Agency, Region VIII.
1987. “Quality Assurance of Chemical
U.S. DOE. 1987. The Environmental Survey Manual. DOE/EH-0053.
U.S. EPA. 1987. Data Quality Objectives for RemedialResponse Activities - Development Process. EPA/540/G-87/003 .
U.S. EPA. 1988. USEPA Contract Laboratory Program -Statement of Work for Inorganic Analysis - Multi-MediaMulti Concentration. SOW No. 788.
U.S. EPA. 1988. Sampling for Hazardous Materials.Environmental Response Team, Office of Emergency andRemedial Response, Hazardous Response Support Division,Washington, D.C.
U.S. EPA. 1988.LC1-619-026, EPA
Quality Assurance Plan Love Canal Study.Contract 68-02-3168.
46
(13) U.S. EPA. 1989. Direct/Delayed Response Project: QualityAssurance Plan for the Mid-Appalachian Region of the UnitedStates (Draft). Environmental Monitoring SystemsLaboratory, Las Vegas, Nevada.
(14) U.S. EPA. 1984. Soil Sampling Quality Assurance Users Guide(lst. Edition). Environmental Monitoring Systems Laboratory,Las Vegas, Nevada. EPA/600/4-84-043
(15) U.S. EPA. 1989. Soil Sampling Quality Assurance Users Guide(2nd. Edition). Environmental Monitoring Systems Laboratory,Las Vegas, Nevada. EPA 600/8/89/046
47
APPENDIX A
KEY COMPONENTS
A rigorous program to assess the quality of data cannot bedeveloped and implemented if key components for a field study areneglected. Those components include training, pilot studies, avariety of audits to assess the effectiveness of the QA/QCprogram, and documentation.
Training
Training is an integral part of an effective qualityassurance program. It should furnish the essential knowledgeneeded by all study participants to assure that plans, methods,and procedures are all accomplished as designed. The trainingprogram should review key principles and point out changes inprotocols to experienced workers. At the same time it shouldorient the new employee to all the study methods. Trainingshould always occur at the earliest possible time before a studyin order to give personnel time to make adjustments. However,training typically occurs, in varying degrees, throughout multi-phased studies as personnel change and more is learned about thesite, methods and procedures. Training may be formal, orinformal, in the classroom, or on the job. Training shouldinclude lectures on sampling principles, a demonstration ofprocedures, question and answer sessions, and hands-on samplingpractice.
A practical way to implement a training program is to integrateit into a ‘preliminary” study. However, additional training mayalso occur during a “pilot” study as personnel are evaluated onthe methods and procedures that are expected to be used duringthe main study.
Pilot Study
Pilot studies may serve as the impetus for further trainingbefore the full-scale study. The purpose of a pilot study is toevaluate the logistics, equipment, sampling plans and analyticalprotocols prior to implementation of the full study. It shouldalso provide a test of the study design, quality assurancedesign, and data interpretation plan. Pilot studies arerecommended for all programs so that state-of-the-art measurementmethods and study designs can be fully tested before the fullstudy begins. In order to be useful, the pilot study shouldemploy all of the plans for the full-scale project, including thesame personnel, management structure, equipment, and procedures.After the pilot study is complete, the study methods can becarefully assessed to see whether or not changes need to be madebefore the full-scale study starts. The data must be interpretedso that the study methods may be evaluated in relation to how the
48
study objectives are being met. It is critical that managersprovide time and resources to evaluate the pilot study andincorporate changes before the start of the next phase of a siteinvestigation.
The guidance in this document, to assess errors in fieldsampling, may be used to assess the proficiency of a trainingeffort by allowing the measured errors to be compared againststated data quality objectives.
Audits
An adequate QA program ensures that the quality of the finalproduct meets the DQOs. Audits are an integral part of the QAprocess and are vital for assuring that program procedures arebeing implemented.
Audits are performed to document the implementation of thequality assurance program plan, quality assurance project planand/or associated operational protocols. Four specific kinds ofaudits can be used to determine the status of the measurementsystems, the adequacy of the data collection systems, thecompleteness of documentation of data collection activities andthe abilities of the program management to meet the mandated datacollection and data quality objectives. These four audit typesare respectively Performance Audits, Technical System Audits,Data Quality Audits and Management System Audits.
Performance Audits (PA) are generally based on QualityAssessment or Evaluation (QE) samples. Samples havingknown concentrations may be tested as unknowns in thelaboratory or a sample may be analyzed for the presenceof certain compounds. Performance audits are used todetermine objectively whether an analytical measurementsystem is operating within established control limits atthe time of the audit.
Technical System Audits (TSA) are qualitative on-siteaudits that evaluate the technical aspects of fieldoperations against the requirements of the approvedprotocols and QA plans. TSA reports will note anyproblems, allowing corrective action to be taken toprotect the validity of future data.
Data Quality Audits (DQA) are evacuations of thedocumentation associated with data quality indicators ofmeasurement data to verify that the generated data are ofknown and documented quality. This is an important partof the validation of data packages showing that themethods and SOPS designated in the QA plans werefollowed, and that the resulting data set is a functional
49
part of satisfying the established DQOs.The results arevital to decisions regarding the legal defensibility ofthe data should it be challenged in litigation.
A Management System Audit (MSA)is a formal review of anentire program, e.g. ,a review of a state’s QA program,or a review of a state-contracted Laboratory. In a MSA,key elements in the program, e.g., lab certificationprogram, QC in field operations, and QC in the certifiedlab, are evaluated to see if QA is being implemented. Ifdeficiencies are detected, corrective actions aresuggested and implementation monitored.
The guidance in this document is particularly useful for theperformance audit.
Documentation
Documented procedures should be developed prior to a studyand followed. Errors can increase and blunders can occur in anymeasurement program through inadequately prepared and revieweddocumentation. Data transcribed onto paper or recorded onmagnetic media must be checked for accuracy on a timely basis byqualified personnel. Measures to assess and minimize errors, asdescribed in this document, are not going to be effective ifadequate documentation is not developed and reviewed on a timelybasis.
It is important that field personnel follow specified,documented procedures. If changes in program execution anddesign (e.g., sample site selection, number of samples to becollected, sampling intervals, and tools) are required, thesechanges, with appropriate rationale, must also be documented.
50
APPENDIX B
DEFINITIONS
Quality Assurance
A system of activities whose purpose is to provide to theproducer or user of a product or service the assurance that itmeets defined standards of quality. It consists of two separate,but relate activities, quality control and quality assessment.
Quality Control
The overall system of activities whose purpose isquality of the measurement data so that they meetthe user.
Quality Assessment
to control thethe needs of
The overall system of activities that provide an objectivemeasure of the quality of data produced.
Soil
The soil referred to in this document encompasses the mass(surface and subsurface) of unconsolidated mantle of weatheredrock and loose material lying above solid rock. Further, adistinction must be made as to what fraction of theunconsolidated material is soil and what fraction is not. Thesoil component here is defined as all mineral and naturallyoccurring organic material that is 2 mm or less in size. This isthe size normally used to distinguish between soils (consistingof sands, silts, and clays) and gravels. In addition, the 2-mmsize is generally compatible with analytical laboratory methods,capabilities, and requirements.
The non-soil fraction (e.g., automobile fluff, wood chips,various absorbents and mineral/organic material greater than 2-mmin size) must also be addressed in sampling and monitoring. Thisfraction may contribute and/or contain a greater amount ofcontaminant(s) than the associated soil fraction. At sites inwhich this occurs, reporting contaminant levels only in the soilfraction will ultimately lead to inappropriate and incorrectdecision making. Decision makers must realize that a number ofproblems are normally encountered in obtaining and using datafrom the non-soil components. For example, questions ariseconcerning the validity of data obtained from the analysis ofmaterials that do not meet the size and volume requirements forwhich the analytical processes were validated. Also, standardreference and audit materials are not available to substantiateand validate the analytical results.
51
The current recommended procedures are to identify andrecord the type and volume of non-soil material for each samplecollected with a minimum of 10 percent (%) of these non-soilsamples submitted for analysis. Data from the non-soil materialare important to the assessment of the representativeness of thesoil sampling/monitoring program. The behavior of contaminantsin the soil environment is a functiOn of the contaminant’s andsoil’s physical and chemical properties. Soil sorption (theretention of substances by adsorption or absorption) is relatedto properties of the contaminant (e.g., solubilities, heats ofsolution, viscosity, and vapor pressure) and to properties ofsoils (e.g., clay content, organic content, texture,permeability, pH, particle size, specific surface area, ionexchange capacity, water content, and temperature). The soilcomponents that are most associated with sorption are claycontent and organic matter. The soil particle surfacecharacteristics thought to be most important in adsorption aresurface area and cation exchange capacity (CEC).
Standard Additions
A procedure called standard additions is commonly used to detect bias in chemical analysis. In this procedure, knownamounts of standard solutions are added to aliquots of soilsamples. It is recommended that this be done in the field or ina field laboratory. The main problem encountered is that mixingsoils to obtain homogeneity is difficult in a laboratory, andeven more so in the field. Several known quantities of thestandard are added to the aliquots of the soil samples. Theanalytical results should follow a straight line:
y = a + b x ,
where x is the increase in concentration caused by the additionand y is the value obtained by the laboratory. Bias is indicatedif the data do not follow a straight line, or if a < 0. If theunits of x and y are the same, the value of b should be near one,and a significant deviation from one would indicate apropotiional bias.
52
APPENDIX D
THE LOGNORMAL DISTRIBUTION AND LOGARITHMIC TRANSFORMATIONS
with variance
and with standard deviation
Note that the relative standard deviation (rsd) for X is
(The logarithmictransformation is used to stabilize measurement variance relativeto concentration when the data indicates that the standarddeviation is a linear function of the sample concentration.)Further, note that the series expansion about zero of the square
of the rsd is sometimes called the rel-variance.
9 Johnson, N.L., and S. KotZ. Continuous UnivariateDistributions -1. Houghton Mifflin Co., Boston, MA 1970 300 pp.
54
APPENDIX E
NON-BLINDQUALITY ASSESSMENT SAMPLES IN THE CONTRACT LABORATORY PROGRAM
1.
2.
3.
4.
5.
6.
7.
8.
- A sample of well-characterized soil, whose analyte concentrations are knownto the laboratory, is used for internal laboratory control(10,15). This sample is also called a quality control auditsample (13).
- A routine sample in which a knownquantity of analyte is added to an aliquot of the sample.It is used to determine bias from the digestion and analysisof components.
- A routine sample in which a knownquantity of analyte is added to an aliquot after thedigestion process is completed. It is used to determinebias from the analytical or detection phase. When used incombination with a pre-digest spike sample, the bias fromthe digestion phase may be determined by difference.
Analytical Laboratory Duplicate (ALD) - This sample is asubsample of a routine sample which is analyzed by the samemethod. It is used to determine method precision, butbecause it is a non-blind sample, or known to the analyst,it can only be used by the analyst as an internal controltool and not as an unbiased estimate of analyticalprecision.
- These areprepared solutions containing known concentrations ofanalytes that originated from a different source as thecalibration standards. They are used as an independentcheck of the instrument calibration accuracy. The CCVsamples are normally run in an ordered fashion after aspecified number of routine samples.
These are blank samples run at thesame frequency as the ICV and CCV, and they are used tocheck for instrument baseline drift.
This is a solution standardat a concentration of two times the CRDL, or two times theIDL, whichever is greater. It is used during each run inplace of a formal instrumental detection limit determinationto assure the instrument is running properly.
- This standard is
a solution of known analyte at concentrations within theupper limit of the linear range. Above this range, thesamples must be diluted.
9. ICP Interference Check Sample - This sample contains twoparts. Part A contains potential interfering analytes, andPart B contains both the analytes of interest and the targetanalytes. Part A and B are analyzed separately to determinethe potential for interferences.
56
