11

Continuous Pollutant and Continuous Pollutant and Meteorological DataMeteorological Data

22Routine Data Operations—Continuous Routine Data Operations—Continuous (gas, PM, and met)(gas, PM, and met)

33

QC Check Data EntryQC Check Data Entry

• We recommend you We recommend you startstart with QC check data entry with QC check data entry (except for meteorological data)(except for meteorological data)

• Then you can review monitoring data with its relevant Then you can review monitoring data with its relevant QC data as part of the data import process QC data as part of the data import process

• This allows you to manage potential maintenance and This allows you to manage potential maintenance and other issues with samplerother issues with sampler

• A set of data (two weeks worth, for example) has at A set of data (two weeks worth, for example) has at least one QC check before and one after, if you least one QC check before and one after, if you routinely download data from the sampler each time routinely download data from the sampler each time you perform a QC checkyou perform a QC check

44

Routine is to 1Routine is to 1stst enter QC check, enter QC check, then import data:then import data:

55

QC Data Entry—One Form QC Data Entry—One Form

QC checks: QC checks: 66

AQS-related terms:AQS-related terms:77

Different types of QC checks:Different types of QC checks:88

Continuous PM Checks:Continuous PM Checks:99

1010

Click on Linear RegressionClick on Linear Regression

1111

QC check results that are out of QC check results that are out of EPA limits cause flags, which are EPA limits cause flags, which are

internal to your datainternal to your data

Flag for level one

span check

1212

After QC entry, the actual After QC entry, the actual Data:Data:

• Data files to import are comma delimited text files Data files to import are comma delimited text files with field (column) names with field (column) names

• If data files not in this format, easy to convert (see If data files not in this format, easy to convert (see SOP)SOP)

• The text file name must be changed to NewImport.txt The text file name must be changed to NewImport.txt and saved on C drive (this step is the same for every and saved on C drive (this step is the same for every imported file)imported file)

• Keep a copy of file elsewhere with its original nameKeep a copy of file elsewhere with its original name• Take notes of what you are importing, where the Take notes of what you are importing, where the

original file is, date, etc. in your Database Logbookoriginal file is, date, etc. in your Database Logbook

1313

Import Raw DataImport Raw Data

Data are moved thru “pipeline” of tables

Data Import Data Verification Archive

1414

The Data Import Table The Data Import Table (pipeline intake-the 1(pipeline intake-the 1stst table) table)

• Data go Data go temporarilytemporarily in the Data Import in the Data Import

tabletable

• Data retain field (column) names from text Data retain field (column) names from text

filefile

• This Data Import table takes whatever is in This Data Import table takes whatever is in

the NewImport.txt filethe NewImport.txt file

1515

First Step is to import the NewImport.txt First Step is to import the NewImport.txt file into the Data Import Tablefile into the Data Import Table

Then, transfer to the middle Then, transfer to the middle table:table:

• Verification table is an empty shell with Verification table is an empty shell with many columnsmany columns

• You tell the Toolbox which columns You tell the Toolbox which columns from the Import table go into which from the Import table go into which columns in the Verification table, by columns in the Verification table, by field field mappingmapping

1616

1717Transfer to Data Verification Table Transfer to Data Verification Table (middle step of the pipeline)(middle step of the pipeline)

1818

Example of Field MappingExample of Field Mapping

Columns in the

verific’n table

1919

The Data Verification Table The Data Verification Table (middle step in pipeline)(middle step in pipeline)

• You will do most of work on data while it is in You will do most of work on data while it is in Data Verification table Data Verification table

• You will add information such as Site IDs, You will add information such as Site IDs, Sampler IDs, Units, etc.Sampler IDs, Units, etc.

• You may have to clarify sampling dates and You may have to clarify sampling dates and times to be consistent with AQS specstimes to be consistent with AQS specs

• You will do initial data flagging in this tableYou will do initial data flagging in this table

2020Check ranges, error codes, units, Check ranges, error codes, units, dates in this middle Verification Tabledates in this middle Verification Table

2121

The Final Table: Data Archive The Final Table: Data Archive TableTable

• Data are permanently stored in this tableData are permanently stored in this table

• Table includes all data you have previously Table includes all data you have previously archived (unlike Data Verification and Data archived (unlike Data Verification and Data Import that you clear once you have archived Import that you clear once you have archived the data in your import)the data in your import)

• Over years, table will get large but database Over years, table will get large but database can handle 2 gigabytes (one Toolbox we can handle 2 gigabytes (one Toolbox we know has 3 years of data from four sites and know has 3 years of data from four sites and the entire database is still only 35 megabytes)the entire database is still only 35 megabytes)