steps 3 & 4: evaluating types of evidence for the truckee river case study

Steps 3 & 4: Evaluating types of evidence for the Truckee River case study

2

Define the Case

List Candidate Causes

Evaluate Data from Elsewhere

Identify Probable Cause

Detect or Suspect Biological Impairment

As Necessary: Acquire Data

and Iterate Process

Identify and Apportion Sources

Management Action: Eliminate or Control Sources, Monitor Results

Biological Condition Restored or Protected

Decision-maker and

Stakeholder Involvement

Stressor Identification

Step 3: Evaluate Data from the Case

Step 4: Evaluate Data from Elsewhere

3

Types of evidence using data from the case

• Spatial/temporal co-occurrence

• Evidence of exposure or biological mechanism

• Causal pathway

• Stressor-response relationships from the field

• Manipulation of exposure

• Laboratory tests of site media

• Temporal sequence

• Verified predictions

• Symptoms

Types of evidence using data from elsewhere

• Stressor-response relationships from other field studies

• Stressor-response relationships from laboratory studies

• Stressor-response relationships from ecological simulation models

• Mechanistically plausible cause

• Manipulation of exposure at other sites

• Analogous stressors

Use all available types of evidence to make an inferential assessment

italics indicates commonly available types of evidence

4

Basic analysis strategy

• Develop as many types of evidence, for as many candidate causes, as you can

– you won’t have all types of evidence, for all candidate causes– most effective when you can compare results across candidate

causes

• Work through one type of evidence, then set it aside – avoid cognitive overload

• Show your work– make your process transparent & reproducible – make use of appendices

5

Let’s begin by figuring out what types of evidence we have for the Truckee…

6

General vs. specific causation

• General – Does C cause E?– Does smoking cause lung cancer?

– Does increased water temperature reduce bull trout abundance in rivers?

• Specific – Did C cause E?– Did smoking cause lung cancer in

Ronald Fisher?

– Did increased water temperature reduce bull trout abundance in my stream?

7

Specific causation: using data from the case

8

Spatial/temporal co-occurrence

SUPPORTS

WEAKENS

Want paired measurements of proximate stressors & biological impairments, at locations where impairments are & are not observed.

9

Causal pathway

Want paired measurements of other steps in causal pathway & biological impairments, at locations where impairments are & are not observed.

10

Stressor-response relationships from the field

Want paired measurements of proximate stressors (or other steps in causal pathway) & biological impairments, at varying levels of exposure.

11

Other types of evidence using data from case

TYPE OF EVIDENCE SUPPORTING EVIDENCE

Manipulation of exposure Impairment improves after stressor is removed

Laboratory tests of site media

Exposure to site media in lab tests results in effects similar to impairment

Evidence of exposure or biological mechanism

Measurements of biota (e.g., biomarkers, tissue residues) show proposed mechanism of exposure has occurred

Verified predictions Predictions based on stressor’s mode of action are made & confirmed at site

Temporal sequence Exposure to stressor precedes impairment

Symptoms Only one stressor supports observed symptom

12

What types of evidence do we have, using data from the case?

13

General causation: using data from elsewhere?

Stressor-response relationships from other field studies

15

Stressor-response relationships from the lab

16

Other types of evidence using data from elsewhere

TYPE OF EVIDENCE SUPPORTING EVIDENCE

Stressor-response relationships from ecological simulation models

Stressor is at levels associated with impairment in mathematical models simulating ecological processes

Manipulation of exposure Impairment improves after stressor is removed at another site

Mechanistically plausible cause

Relationship between stressor & impairment is consistent with current scientific knowledge

Analogous stressors Stressor is structurally similar to other stressors known to cause impairment

Verified predictions Predictions based on stressor’s mode of action are made & confirmed at other sites

17

What types of evidence do we have, using data from elsewhere?

18

Now that we know what data we have, how do we analyze it?

19

Spatial co-occurrence

Do your impairment and your stressor co-occur in space?

To Do:1. Load relevant data file2. Merge files3. Make boxplots for each

candidate causeSelect ‘reference’ and impaired sites

4. Fill in worksheet

20

Causal Pathway

Does your data support the steps in the causal path between the stressor and the impairment?

To Do:1. Return to the conceptual

diagram2. Identify the steps in the causal

pathway3. Construct table to show

whether data supports the steps between the stressor and the impairment

4. Fill in worksheet

21

Verified Prediction - Traits

Do data support predictions based on stressor’s mode of action?

To Do:1. Load relevant data file2. Merge files3. Make boxplot

Select ‘reference’ and impaired sites

4. Fill in worksheet

22

Verified Prediction - PECBO

Do data support predictions based on stressor’s mode of action?

To Do:1. Load relevant data file2. Merge files3. Run PECBO4. Load PECBO results file into

CADStat5. Merge files6. Make boxplot

1. Sed2. STRMTEMP

7. Fill in worksheet

23

Stressor-response from elsewhere

Does impairment decrease as exposure to the stressor decreases (or increases as exposure increases)?

To Do:1. Listen and ask lots of

questions2. Fill in the worksheet

24

Randomized, controlled experiments

Key elements:• Replication: use of multiple

test units (e.g. tanks, sites)

• Controls: differ only by absence of the treatment

• Randomization: random assignment of test units to “control” or “treated” status

• Statistical analysis: estimate treatment effect (causal)

The scientific standard for establishing cause and effect

0

1

2

3

4

5

6

7

1 2 3 4

Toxin

Su

rviv

ors

ip

25

Observational studies

Key elements:• Replication: collect data

from multiple test units

• Controls: ?• Randomization: ?• Statistical analysis: identify

associations among variables of interest (non-causal)

Often the only option for large-scale field studies

None

None

26

Trade offs: control vs. realism, scale

Lab Experiment

Field Experiment

Observational Study

control

realism,scale

27

Biomonitoring = Observational

Issues for causal analysis:

• Estimates of stressor effects are confounded by covarying factors

• Analyst can’t randomly assign treatments (stressors) to sites

* Reference sites are not experimental controls

28

Analogous to clinical trials

Does smoking cause lung cancer?

• Estimates of stressor effects are confounded by covarying factors

• Analyst can’t randomly assign treatments (stressors) to subjects

* Non-smokers without lung cancer are not experimental controls

29

Example using western EMAP*

Using propensity scores to infer cause-effect relationships in observational data

– Analysis and slides by Lester Yuan (USEPA), Amina Pollard (USEPA), and Daren Carlisle (USGS)

– Original presentation given at North American Benthological Society conference, May 2008

*EPA Environmental Monitoring and Assessment Program (EMAP)

30

EMAP-West Study Area

Measurements Collected:

•Macroinvertebrates

•Substrate composition

(SED)

•Stream temperature

(STRMTEMP)

•N = 838

Data collected by the EPA Environmental Monitoring and Assessment Program (EMAP)

31

Total N vs. total taxon richness

Data from EMAP Western Pilot

SLOPE = -16.5

32

Total N covaries with many other factors

33

Multiple linear regression

Include covariates in the regression model to control for their effect.

SLOPE = -9.4

Regression model includes: %agriculture, %urban, grazing intensity, %sands/fines, stream temperature, and log conductivity.

SLOPE = -16.5

Correlation of Total Richness and Total N (ug/L)

34

Potential issues with multiple regression

• Must assume that linear relationships are appropriate for all covariates.

• Regression model may extrapolate.

• Inclusion of certain variables may “mask” true effect:– e.g., part of the effect of agriculture may be

attributed to total N

35

Alternate approach: Stratify dataset

r = -0.01 r = 0.15 r = 0.27

r = 0.64

36

Model richness vs. total N within strata

How do we simultaneously stratify on many different covariates?

SLOPE = -10.7 SLOPE = -12.3 SLOPE = -9.7

37

Propensity Score Matching

• Method developed in epidemiology to retroactively control for confounding effects in observational studies

• Sometimes called a quasi-experiment• Intuitively:

1. Model the magnitude of treatment (e.g. nutrient concentration) as a function of the covariates. The predicted magnitude of treatment at each site is its propensity score.

2. Stratify the total set of observations by the propensity scores (i.e., group sites with similar scores). Six strata are typically used.

3. Within each stratum, sites having different treatment levels (e.g. high vs. low nutrients) may be considered to have been “randomly assigned” to those treatment levels, because covariates have effectively been controlled by propensity score matching of “treated” and “control” sites.

38

Propensity Score Model

Total N = f(percent agriculture, percent urban, grazing intensity, percent sand/fines, stream temperature, log conductivity, elevation, log catchment area, canopy cover, sampling day)

39

Define 6 strata based on propensity scores

40

Covariate values within strata

Grazing intensity

Percent agriculture

41

Stratification by propensity score controls covariance of all modeled variables

Original r Min r Max r

Percent agriculture 0.60 0.06 0.24

Percent urban 0.29 -0.12 0.40

Grazing intensity 0.64 -0.21 0.18

Percent sand/fines 0.64 -0.16 0.13

Stream temperature 0.48 -0.13 0.22

log conductivity 0.68 -0.12 0.16

Elevation -0.26 -0.38 0.20

log catchment area 0.48 -0.26 0.21

Canopy cover 0.45 -0.13 0.09

Sampling day -0.17 -0.21 0.10

After stratification

42

Total N vs. total taxon richness

Data from EMAP Western Pilot

SLOPE = -16.5

43

Total N vs. total richness: Stratified

SLOPE = -3.3 (n.s.)

SLOPE = -10.0***

SLOPE = -7.1*SLOPE = -7.1*

SLOPE = -10.5*** SLOPE = -8.1***