Clinical Trials Versus Health Clinical Trials Versus Health Outcomes Research: Outcomes Research:

SAS/STAT Versus SASSAS/STAT Versus SASEnterprise MinerEnterprise Miner

Patricia B. CerritoPatricia B. Cerrito

pcerrito@louisville.edu

University of LouisvilleUniversity of Louisville

ObjectivesObjectives

To examine some issues with traditional To examine some issues with traditional statistical models and their basic statistical models and their basic assumptionsassumptions

To examine the Central Limit Theorem To examine the Central Limit Theorem and its necessity in statistical modelsand its necessity in statistical models

To look at the differences and similarities To look at the differences and similarities between clinical trials and health between clinical trials and health outcomes researchoutcomes research

Surrogate Versus Real Surrogate Versus Real EndpointsEndpoints

Because clinical trials tend to be short Because clinical trials tend to be short term, they use high risk patients and term, they use high risk patients and surrogate endpointssurrogate endpoints

Use of statins reduce cholesterol levels Use of statins reduce cholesterol levels but do they increase longevity and disease but do they increase longevity and disease free survival?free survival?

Health outcomes data can examine real Health outcomes data can examine real endpoints from the general populationendpoints from the general population

One Versus Many EndpointsOne Versus Many Endpoints

Clinical trials generally have one survival Clinical trials generally have one survival endpoint-time to recurrence, time to death, endpoint-time to recurrence, time to death, time to disease progressiontime to disease progression

Health outcomes can examine multiple Health outcomes can examine multiple endpoints simultaneously using survival endpoints simultaneously using survival data miningdata mining

Homogeneous Versus Homogeneous Versus Heterogeneous DataHeterogeneous Data

Clinical trials generally use Clinical trials generally use inclusion/exclusion criteria to define a inclusion/exclusion criteria to define a homogeneous samplehomogeneous sample

Health outcomes have to rely upon Health outcomes have to rely upon heterogeneous dataheterogeneous data Populations are more gamma distributions Populations are more gamma distributions

than normal and this must be taken into than normal and this must be taken into considerationconsideration

Large Versus Small SamplesLarge Versus Small Samples

Clinical trials tend to use the smallest sample Clinical trials tend to use the smallest sample possible to achieve the desired powerpossible to achieve the desired power Database designed for analysis and data are Database designed for analysis and data are

very cleanvery clean Health outcomes have an abundance of data Health outcomes have an abundance of data

and variablesand variables Power not an issuePower not an issue Data are very messy and require considerable Data are very messy and require considerable

preprocessingpreprocessing

Rare OccurrencesRare Occurrences

Clinical trials not large enough to find all Clinical trials not large enough to find all potential rare occurrencespotential rare occurrences

Health outcomes have enough data to find Health outcomes have enough data to find rare occurrences and to predict the rare occurrences and to predict the probability of occurrenceprobability of occurrence Requires modifications to standard linear Requires modifications to standard linear

modelsmodels Predictive modeling much better at actual Predictive modeling much better at actual

predictionprediction

Example 1Example 1

Ottenbacher, Kenneth J. Ottenbacher, Ottenbacher, Kenneth J. Ottenbacher, Heather R. Tooth, Leigh. Ostir, Glenn V.Heather R. Tooth, Leigh. Ostir, Glenn V.

A review of two journals found that articles A review of two journals found that articles using multivariable logistic regression using multivariable logistic regression frequently did not report commonly frequently did not report commonly recommended assumptions. recommended assumptions. Journal of Clinical Epidemiology. Journal of Clinical Epidemiology. 57(11):1147-52, 2004 Nov.57(11):1147-52, 2004 Nov.

continued...

Example 1Example 1

Statistical significance testing or Statistical significance testing or confidence intervals were reported in all confidence intervals were reported in all articles. Methods for selecting articles. Methods for selecting independent variables were described in independent variables were described in 82%, and specific procedures used to 82%, and specific procedures used to generate the models were discussed in generate the models were discussed in 65%. 65%.

continued...

Example 1Example 1

Fewer than 50% of the articles indicated if Fewer than 50% of the articles indicated if interactions were tested or met the interactions were tested or met the recommended events per independent recommended events per independent variable ratio of 10:1. variable ratio of 10:1.

Fewer than 20% of the articles described Fewer than 20% of the articles described conformity to a linear gradient, examined conformity to a linear gradient, examined collinearity, reported information on validation collinearity, reported information on validation procedures, goodness-of-fit, discrimination procedures, goodness-of-fit, discrimination statistics, or provided complete information on statistics, or provided complete information on variable coding.variable coding.

Example 2Example 2 Brown, James M. O'Brien, Sean M. Wu, Brown, James M. O'Brien, Sean M. Wu,

Changfu. Sikora, Jo Ann H. Griffith, Bartley P. Changfu. Sikora, Jo Ann H. Griffith, Bartley P. Gammie, James S.Gammie, James S.Title: Isolated aortic valve replacement in North Title: Isolated aortic valve replacement in North America comprising 108,687 patients in 10 America comprising 108,687 patients in 10 years: changes in risks, valve types, and years: changes in risks, valve types, and outcomes in the Society of Thoracic Surgeons outcomes in the Society of Thoracic Surgeons National Database.National Database.Source: Journal of Thoracic & Cardiovascular Source: Journal of Thoracic & Cardiovascular Surgery. 137(1):82-90, 2009 Jan.Surgery. 137(1):82-90, 2009 Jan.

continued...

Example 2Example 2

108,687 isolated aortic valve replacements 108,687 isolated aortic valve replacements were analyzed. Time-related trends were were analyzed. Time-related trends were assessed by comparing distributions of risk assessed by comparing distributions of risk factors, valve types, and outcomes in 1997 factors, valve types, and outcomes in 1997 versus 2006.versus 2006.

Differences in case mix were summarized by Differences in case mix were summarized by comparing average predicted mortality risks comparing average predicted mortality risks with a logistic regression model.with a logistic regression model.

Differences across subgroups and time were Differences across subgroups and time were assessed.assessed.

continued...

Example 2Example 2

RESULTS:RESULTS: There was a dramatic shift There was a dramatic shift toward use of bioprosthetic valves. toward use of bioprosthetic valves.

Aortic valve replacement recipients in Aortic valve replacement recipients in 2006 were older (mean age 65.9 vs 67.9 2006 were older (mean age 65.9 vs 67.9 years, P < .001) with higher predicted years, P < .001) with higher predicted operative mortality risk (2.75 vs 3.25, P operative mortality risk (2.75 vs 3.25, P < .001)< .001)

Observed mortality and permanent stroke Observed mortality and permanent stroke rate fell (by 24% and 27%, respectively). rate fell (by 24% and 27%, respectively).

continued...

Example 2Example 2

Female sex, age older than 70 years, and Female sex, age older than 70 years, and ejection fraction less than 30% were all ejection fraction less than 30% were all related to higher mortality, higher stroke related to higher mortality, higher stroke rate and longer postoperative stay. rate and longer postoperative stay.

There was a 39% reduction in mortality There was a 39% reduction in mortality with preoperative renal failure.with preoperative renal failure.

Central Limit TheoremCentral Limit Theorem

As the sample size increases to infinity, the As the sample size increases to infinity, the distribution of the sample average approaches distribution of the sample average approaches a normal distribution with mean a normal distribution with mean μμ and variance and variance σσ22/n. /n.

As n approaches infinity, the variance As n approaches infinity, the variance approaches zero. approaches zero.

Therefore, the distribution of the sample Therefore, the distribution of the sample average starts to look like a straight line at the average starts to look like a straight line at the point point μμ if n is too large. if n is too large.

continued...

Central Limit TheoremCentral Limit Theorem

In addition, the sample mean is very In addition, the sample mean is very susceptible to the influence of outliers. susceptible to the influence of outliers.

Moreover, the confidence limits are Moreover, the confidence limits are defined based upon the assumption of defined based upon the assumption of normality and symmetry. Therefore, the normality and symmetry. Therefore, the existence of many outliers will skew the existence of many outliers will skew the confidence interval.confidence interval.

Nonparametric StatisticsNonparametric Statistics

Nonparametric models still require Nonparametric models still require symmetry.symmetry.

Many populations are highly skewed so Many populations are highly skewed so that these models also have problemsthat these models also have problems

DatasetDataset

We use data from the National Inpatient We use data from the National Inpatient Sample from 2005Sample from 2005

A stratified sample from 1000 hospitals A stratified sample from 1000 hospitals from 37 statesfrom 37 states

Approximately 8 million inpatient staysApproximately 8 million inpatient stays

Distribution of Patient StaysDistribution of Patient Stays

Normal EstimateNormal Estimate

0.15 4.65 9.15 13.65 18.15 22.65 27.15 31.65 36.15 40.65 45.15 49.65

0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Percent

Length of stay (cleaned)

Kernel Density EstimationKernel Density Estimation

Instead of assuming that the population Instead of assuming that the population follows a known distribution, we can follows a known distribution, we can estimate it.estimate it.

Kernel density estimation is an excellent Kernel density estimation is an excellent method to use to do thismethod to use to do this

continued...

Kernel Density EstimationKernel Density Estimation

n

j

n

j

na

xXK

naxf

1

1)(

Proc KDEProc KDE

procproc kdekde data=nis.diabetesless50los; data=nis.diabetesless50los;

univar los/gridl=univar los/gridl=00 gridu= gridu=5050 method=srot method=srot out=nis.kde50 bwm=out=nis.kde50 bwm=33;;

runrun;;

Kernel Estimate of Length of Kernel Estimate of Length of StayStay

Sampling from NISSampling from NIS

Given that the National Inpatient Sample Given that the National Inpatient Sample has 8 million records, we can consider it to has 8 million records, we can consider it to be an infinite population. Therefore, we be an infinite population. Therefore, we can sample from this population to see if it can sample from this population to see if it can be estimated by the Central Limit can be estimated by the Central Limit TheoremTheorem

We start with extracting 100 different We start with extracting 100 different samples of size N=5samples of size N=5

Examine Central Limit TheoremExamine Central Limit Theorem

PROC SURVEYSELECT DATA=nis.nis_205 PROC SURVEYSELECT DATA=nis.nis_205 OUT=work.samples METHOD=SRS N=5 OUT=work.samples METHOD=SRS N=5 rep=100 noprint;rep=100 noprint;

RUN;RUN;

proc means data=work.samples noprint;proc means data=work.samples noprint;

by replicate;by replicate;

var los;var los;

output out=out mean=mean;output out=out mean=mean;

run;run;

Sample Size=5Sample Size=5

Sample Size=30Sample Size=30

Sample Size=100Sample Size=100

Sample Size=1000Sample Size=1000

Confidence LimitConfidence Limit

The confidence limit excludes much of the actual population distribution

Confidence Limit With Larger nConfidence Limit With Larger n

DiscussionDiscussion

An over-reliance on the Central Limit An over-reliance on the Central Limit Theorem can give a very misleading Theorem can give a very misleading picture of the population distribution. picture of the population distribution.

Kernel density estimation (PROC KDE) Kernel density estimation (PROC KDE) allows an examination of the entire allows an examination of the entire population distribution instead of just using population distribution instead of just using the mean to represent the population.the mean to represent the population.

Without the assumption of normality, we Without the assumption of normality, we need to use predictive modeling.need to use predictive modeling.

DiscussionDiscussion

This is true for both logistic and linear This is true for both logistic and linear regression where the assumption of regression where the assumption of normality is required.normality is required.

The two regression techniques do not The two regression techniques do not work well with skewed populations. work well with skewed populations.

We first look at logistic regression for rare We first look at logistic regression for rare occurrencesoccurrences

Problems With RegressionProblems With Regression

Logistic regression is not designed to Logistic regression is not designed to predict rare occurrencespredict rare occurrences

With a rare occurrence, logistic regression With a rare occurrence, logistic regression will predict virtually all observations as will predict virtually all observations as non-occurrencesnon-occurrences

The accuracy will be high but the The accuracy will be high but the predictive ability of the model will be predictive ability of the model will be virtually nil.virtually nil.

Regression EquationRegression Equation

25252221

... XXX

Threshold ValueThreshold Value

For Logistic regression, a threshold value For Logistic regression, a threshold value is defined, and regression values above is defined, and regression values above the threshold are predicted as 1the threshold are predicted as 1

Regression values below the threshold are Regression values below the threshold are predicted as 0predicted as 0

Choice of threshold value optimizes error Choice of threshold value optimizes error raterate

Simple RegressionSimple RegressionTable of pneumonia by DIED

pneumonia DIED

Frequency Row Pct Col Pct

0 1

Total

0 7431129 98.21 94.97

135419 1.79

81.02

7566548

1 393728 92.54

5.03

31731 7.46

18.98

425459

Total 7824857 167150 7992007

Frequency Missing = 3041

Classification TableClassification Table

Classification Table

Correct Incorrect Percentages Prob Level

Event Non- Event

Event Non- Event

Correct Sensi- tivity

Speci- ficity

False POS

False NEG

0.920 782E4 0 167E3 0 97.9 100.0 0.0 2.1 .

0.940 743E4 31731 135E3 394E3 93.4 95.0 19.0 1.8 92.5

0.960 743E4 31731 135E3 394E3 93.4 95.0 19.0 1.8 92.5

0.980 743E4 31731 135E3 394E3 93.4 95.0 19.0 1.8 92.5

1.000 0 167E3 0 782E4 2.1 0.0 100.0 . 97.9

Classification With 3 VariablesClassification With 3 VariablesClassification Table

Correct Incorrect Percentages Prob Level

Event Non- Event

Event Non- Event

Correct Sensi- tivity

Speci- ficity

False POS

False NEG

0.480 782E4 0 167E3 0 97.9 100.0 0.0 2.1 .

0.500 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.520 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.540 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.560 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.580 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.600 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.620 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.640 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.660 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.680 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

0.700 781E4 4907 162E3 11633 97.8 99.9 2.9 2.0 70.3

continued...

Classification With 3 VariablesClassification With 3 Variables Event Non-

Event Event Non-

Event Correct Sensi-

tivity Speci-

ficity False POS

False NEG

0.760 775E4 26363 141E3 78618 97.3 99.0 15.8 1.8 74.9

0.780 775E4 26363 141E3 78618 97.3 99.0 15.8 1.8 74.9

0.800 775E4 26363 141E3 78618 97.3 99.0 15.8 1.8 74.9

0.820 775E4 26363 141E3 78618 97.3 99.0 15.8 1.8 74.9

0.840 775E4 26363 141E3 78618 97.3 99.0 15.8 1.8 74.9

0.860 775E4 26363 141E3 78618 97.3 99.0 15.8 1.8 74.9

0.880 775E4 26363 141E3 78618 97.3 99.0 15.8 1.8 74.9

0.900 768E4 41608 126E3 149E3 96.6 98.1 24.9 1.6 78.1

0.920 757E4 51297 116E3 258E3 95.3 96.7 30.7 1.5 83.4

0.940 757E4 51297 116E3 258E3 95.3 96.7 30.7 1.5 83.4

0.960 757E4 51297 116E3 258E3 95.3 96.7 30.7 1.5 83.4

0.980 634E4 103E3 64219 149E4 80.6 81.0 61.6 1.0 93.5

1.000 0 167E3 0 782E4 2.1 0.0 100.0 . 97.9

ModelsModels

Linear regression:Linear regression: Y = βY = β0 0 + β+ β1 1 XX1 1 + β+ β22XX2 2 …….+…….+ββk k XXkk

Logistic regression:Logistic regression: loglogee(p/1− p) = β(p/1− p) = β0 0 + β+ β11ΧΧ1 1 + β+ β22ΧΧ2 2 …….β…….βnnΧΧnn

Poisson regressionPoisson regression logloge e (Y) = β(Y) = β0 0 + β+ β11ΧΧ1 1 + β+ β22ΧΧ2 2 …….β…….βnnΧΧnn

Poisson DistributionPoisson Distribution

The parameter of the Poisson Distribution, The parameter of the Poisson Distribution, λλ, will represent the average mortality rate, , will represent the average mortality rate, say 2%.say 2%.

Then the sample size times 2% will give the Then the sample size times 2% will give the estimate for the number of deaths, say estimate for the number of deaths, say 1,000,000*0.02=20,0001,000,000*0.02=20,000

However, the problem still persists.However, the problem still persists. For example, septicemia has a 26% For example, septicemia has a 26%

mortality rate, pneumonia has a 7.5% ratemortality rate, pneumonia has a 7.5% rate

ParametersParameters

The three conditions include approximately The three conditions include approximately 25% of total hospitalizations, leaving 75% 25% of total hospitalizations, leaving 75% not accounted for. not accounted for.

The Poisson distribution can be accurate on The Poisson distribution can be accurate on those patients but cannot determine those patients but cannot determine anything about the remaining 75%anything about the remaining 75%

If more patient conditions are added, the If more patient conditions are added, the 25% will increase but not to the point that 25% will increase but not to the point that the model will have good predictabilitythe model will have good predictability

Predictive ModelingPredictive Modeling

Takes a different approachTakes a different approach Uses equal group sizesUses equal group sizes

100% of the rarest level100% of the rarest level Equal sample size of other levelEqual sample size of other level Randomizes the selection of the samplingRandomizes the selection of the sampling

Uses prior probabilities to choose the Uses prior probabilities to choose the optimal modeloptimal model

50/50 Split in the Data50/50 Split in the Data

Target Outcome Target Percentage Outcome Percentage Count Total Percentage 0 0 67.8 80.1 54008 40.4 1 0 32.2 38.3 25622 19.2 0 1 23.8 19.2 12852 9.6 1 1 76.3 61.7 41237 30.8

Filter data to mortality outcome

Filter data to non-mortality outcome

Use PROC SURVEYSELECT to extract a subsample of non-mortality outcome

Append the mortality outcome data to subsample

75/25 Split in the Data75/25 Split in the Data

Target Outcome Target Percentage

Outcome Percentage

Count Total Percentage

0 0 80.4 96.6 10070 72.5 1 0 19.6 70.9 2462 17.7 0 1 25.6 3.3 348 2.5 1 1 74.4 29.1 1010 7.3

90/10 Split in the Data90/10 Split in the Data

Target Outcome Target Percentage Outcome Percentage Count Total Percentage 0 0 91.5 99.3 31030 89.4 1 0 8.5 83.5 2899 8.3 0 1 27.3 0.7 216 0.6 1 1 72.6 16.5 574 1.6

ValidationValidation

The reduced sample is partitioned into The reduced sample is partitioned into training/validation/testing setstraining/validation/testing sets

Only need training/testing sets for Only need training/testing sets for regression modelsregression models

Model is validated on the testing setModel is validated on the testing set

Sampling NodeSampling Node

Misclassification in RegressionMisclassification in Regression

Target Outcome Target Percentage

Outcome Percentage

Count Total Percentage

Training Data 0 0 67.8 80.1 54008 40.4 1 0 32.2 38.3 25622 19.2 0 1 23.8 19.2 12852 9.6 1 1 76.3 61.7 41237 30.8 Validation Data

0 0 67.7 80.8 40498 40.4 1 0 32.3 38.5 19315 19.2 0 1 23.8 19.2 9646 9.6 1 1 76.2 61.5 30830 30.7

ROC CurvesROC Curves

Rule Induction ResultsRule Induction Results

Target Outcome Target Percentage Outcome Percentage Count Total Percentage Training Data 0 0 67.8 80.8 54008 40.4 1 0 32.2 38.3 25622 19.2 0 1 23.8 19.2 12852 9.6 1 1 76.3 61.7 41237 30.8 Validation Data 0 0 67.7 80.8 40498 40.4 1 0 32.3 38.5 19315 19.2 0 1 23.8 19.2 9646 9.6 1 1 76.2 61.5 30830 30.7

Variable SelectionVariable Selection

ROC CurvesROC Curves

DecileDecile

Data are sorted and divided into decilesData are sorted and divided into deciles True positive patients with highest True positive patients with highest

confidence come firstconfidence come first Next, positive patients with lower Next, positive patients with lower

confidence. confidence. True negative cases with lowest confidence True negative cases with lowest confidence

come nextcome next Next, negative cases with highest Next, negative cases with highest

confidence. confidence.

LiftLift

Target densityTarget density =number of actually =number of actually positive instances in that decile\ the total positive instances in that decile\ the total number of instances in the decile.number of instances in the decile.

TheThe lift lift =the ratio of the target density for =the ratio of the target density for the decile to the target density over all the the decile to the target density over all the test data. test data.

Way to find patients most at risk for Way to find patients most at risk for mortality (or infection)mortality (or infection)

DiscussionDiscussion

Predictive modeling in Enterprise Miner Predictive modeling in Enterprise Miner has some capabilities that are possible, has some capabilities that are possible, but extremely difficult in SAS/Statbut extremely difficult in SAS/Stat Sampling a rare occurrence to a 50/50 splitSampling a rare occurrence to a 50/50 split Partitioning to validate the resultsPartitioning to validate the results Comparing multiple models to find the one Comparing multiple models to find the one

that is optimalthat is optimal Variable selectionVariable selection

SummarySummary

Clinical trials do differ from health outcomes Clinical trials do differ from health outcomes research and the statistical techniques research and the statistical techniques required must be adapted to outcomes required must be adapted to outcomes researchresearch

Model assumptions are important, but too Model assumptions are important, but too often ignoredoften ignored

We need to look at results in detailWe need to look at results in detail Superficial consideration of results can lead Superficial consideration of results can lead

to very erroneous conclusionsto very erroneous conclusions