understanding inferential statistics

Understanding Understanding Inferential Statistics—Inferential Statistics—An Overview of An Overview of Important ConceptsImportant Concepts

Important DefinitionsImportant Definitions

PopulationPopulation– The complete set of individuals or objects that the The complete set of individuals or objects that the

investigator is interested in studyinginvestigator is interested in studying SampleSample

– A subset of the population that is actually being A subset of the population that is actually being studiedstudied

VariableVariable– A characteristic of an individual or object that can A characteristic of an individual or object that can

have different values (as opposed to a constant)have different values (as opposed to a constant) Independent variableIndependent variable

– The variable that is systematically manipulated or The variable that is systematically manipulated or measured by the investigator to determine its measured by the investigator to determine its impact on the outcome.impact on the outcome.

Important DefinitionsImportant Definitions

Dependent variableDependent variable– The outcome variable of interestThe outcome variable of interest

DataData– The measurements that are collected by The measurements that are collected by

the investigatorthe investigator StatisticStatistic

– Summary measure of a sampleSummary measure of a sample ParameterParameter

– Summary measure of a populationSummary measure of a population

Two branches of the Two branches of the science of statisticsscience of statistics Descriptive StatisticsDescriptive Statistics Inferential StatisticsInferential Statistics

Descriptive StatisticsDescriptive Statistics

Concerned with describing or Concerned with describing or characterizing the obtained characterizing the obtained sample data sample data

Use of summary measures—Use of summary measures—typically measures of central typically measures of central tendency and spreadtendency and spread

Descriptive StatisticsDescriptive Statistics

Measures of central tendency include Measures of central tendency include the mean, median, and modethe mean, median, and mode

Measures of spread include the Measures of spread include the range, variance, and standard range, variance, and standard deviation.deviation.

These summary measures of These summary measures of obtained sample data are called obtained sample data are called statisticsstatistics

Inferential StatisticsInferential Statistics

Involves using obtained sample Involves using obtained sample statistics to estimate the statistics to estimate the corresponding population corresponding population parametersparameters

Most common inference is using a Most common inference is using a sample mean to estimate a sample mean to estimate a population mean (surveys, population mean (surveys, opinion polls)opinion polls)

Planning a studyPlanning a study

Suppose you were interested in Suppose you were interested in determining whether treatment X determining whether treatment X has an effect on outcome Y—has an effect on outcome Y—there are several issues that need there are several issues that need to be addressed so that a sound to be addressed so that a sound inference can be made from the inference can be made from the study resultstudy result


What is the population?What is the population? How will you select a sample that How will you select a sample that

is representative of that is representative of that population?population?– There are many ways to produce a There are many ways to produce a

sample, but not all of them will lead sample, but not all of them will lead to sound inferenceto sound inference

Sampling StrategiesSampling Strategies

Probability samples—result when Probability samples—result when subjects have a known probability subjects have a known probability of entering the sampleof entering the sample– Simple random samplingSimple random sampling– Stratified samplingStratified sampling– Cluster samplingCluster sampling


Non-probability samples—result Non-probability samples—result when subjects do not have a when subjects do not have a known probability of entering the known probability of entering the samplesample– Quota samplingQuota sampling– Convenience samplingConvenience sampling


Probability samples can be made to Probability samples can be made to be representative of a populationbe representative of a population

Non-probability samples may or Non-probability samples may or may not be representative of a may not be representative of a population—it may be difficult to population—it may be difficult to convince someone that the sample convince someone that the sample results apply to any larger results apply to any larger populationpopulation

Planning a study—Planning a study—Validity IssuesValidity Issues

Internal validityInternal validity– The extent to which the observed The extent to which the observed

effect on the dependent variable is effect on the dependent variable is actually caused by the independent actually caused by the independent variablevariable

– Depends on carefully controlling Depends on carefully controlling other potential causes of an effectother potential causes of an effect

– Excessive control may result in Excessive control may result in artificial circumstances artificial circumstances

Planning a study—Planning a study—Validity IssuesValidity Issues External validityExternal validity

– The extent to which one would The extent to which one would expect the results from a study to expect the results from a study to be duplicated in the real world—in be duplicated in the real world—in the larger populationthe larger population

– Depends on the representativeness Depends on the representativeness of the sample of the sample

– Also depends on artificiality of the Also depends on artificiality of the studystudy

Planning a study—Planning a study—Validity IssuesValidity Issues Always a tension between maximizing Always a tension between maximizing

internal vs. external validityinternal vs. external validity Efficacy studiesEfficacy studies

– Studies designed to determine the Studies designed to determine the maximum effectiveness of a treatment maximum effectiveness of a treatment under ideal conditions—internal validityunder ideal conditions—internal validity

Effectiveness studiesEffectiveness studies– Studies designed to determine the likely Studies designed to determine the likely

effect of a treatment in the real world—effect of a treatment in the real world—external validityexternal validity


Clinical trials are generally Clinical trials are generally designed to be efficacy trials—designed to be efficacy trials—highly controlled situations that highly controlled situations that maximize internal validitymaximize internal validity

We want to design a study to test We want to design a study to test the effect of treatment X on the effect of treatment X on outcome Y, and try to make sure outcome Y, and try to make sure that any difference in Y is due to Xthat any difference in Y is due to X


The simplest design would involve two groupsThe simplest design would involve two groups—an experimental group and a control group——an experimental group and a control group—that are created through random assignment. that are created through random assignment. In addition, neither the subjects nor the In addition, neither the subjects nor the experimenter knows the group assignment experimenter knows the group assignment (double blind)(double blind)

Two groups to address the possibility of change Two groups to address the possibility of change in Y occurring regardless of treatment Xin Y occurring regardless of treatment X

Random assignment to address the possibility Random assignment to address the possibility that the two groups were different to begin that the two groups were different to begin withwith

Blinding to address the possibility that patient Blinding to address the possibility that patient or experimenter expectations play a role in the or experimenter expectations play a role in the outcomeoutcome


At the end of this study you observe a At the end of this study you observe a difference in outcome Y between the difference in outcome Y between the experimental group and the control group.experimental group and the control group.

All of the effort in designing the study with All of the effort in designing the study with strict control is for one reason—at the end strict control is for one reason—at the end of the study you want only two plausible of the study you want only two plausible explanations for the observed outcome explanations for the observed outcome – ChanceChance– Real effect of treatment XReal effect of treatment X


The reason you want only these two explanations The reason you want only these two explanations is because if you can rule out chance, you can is because if you can rule out chance, you can conclude that treatment X must have been the conclude that treatment X must have been the reason for the difference in outcome Yreason for the difference in outcome Y

All inferential statistical tests are used to All inferential statistical tests are used to estimate the probability of the observed outcome estimate the probability of the observed outcome assuming chance alone is the reason for the assuming chance alone is the reason for the difference.difference.

If there are multiple competing explanations for If there are multiple competing explanations for the observed result, then ruling out chance offers the observed result, then ruling out chance offers little information about the effectiveness of little information about the effectiveness of treatment Xtreatment X

Two ways of using Two ways of using Inferential statisticsInferential statistics Hypothesis testing—answering Hypothesis testing—answering

the question of whether or not the question of whether or not treatment X may have no effect treatment X may have no effect on outcome Yon outcome Y

Point estimation—determining Point estimation—determining what the likely effect of treatment what the likely effect of treatment X is on outcome Y X is on outcome Y

Hypothesis TestingHypothesis Testing

The goal of hypothesis testing is The goal of hypothesis testing is somewhat twisted—it is to somewhat twisted—it is to disprove something you don’t disprove something you don’t believebelieve

In this case you are trying to In this case you are trying to disprove that treatment X has no disprove that treatment X has no effect on outcome Yeffect on outcome Y

You start out with two hypothesesYou start out with two hypotheses


Null Hypothesis (HNull Hypothesis (HOO))– Treatment X has no effect on Treatment X has no effect on

outcome Youtcome Y

Alternative Hypothesis (HAlternative Hypothesis (HAA))– Treatment X has an effect on Treatment X has an effect on

outcome Youtcome Y


If the trial has been carefully controlled, If the trial has been carefully controlled, there are only two explanations for a there are only two explanations for a difference between treatment groups—difference between treatment groups—efficacy of X, and chanceefficacy of X, and chance

Assuming that the null hypothesis is Assuming that the null hypothesis is correct, we can use a statistical test to correct, we can use a statistical test to calculate that the observed difference calculate that the observed difference would have occurred. This is known as would have occurred. This is known as the significance level, or p-value of the the significance level, or p-value of the test.test.


P-valueP-value– The probability of the observed The probability of the observed

outcome, assuming that chance alone outcome, assuming that chance alone was involved in creating the outcome. was involved in creating the outcome. In other words, assuming the null In other words, assuming the null hypothesis is correct, what is the hypothesis is correct, what is the probability that we would have seen probability that we would have seen the observed outcome.the observed outcome.

– This is only meaningful if chance is the This is only meaningful if chance is the only competing plausible explanation.only competing plausible explanation.


If the p-value is small, meaning the If the p-value is small, meaning the observed outcome would have been observed outcome would have been unlikely, we will reject that chance unlikely, we will reject that chance played the only role in the observed played the only role in the observed difference between groups and difference between groups and conclude that treatment X does in conclude that treatment X does in fact have an effect on outcome Yfact have an effect on outcome Y

How small is small?How small is small?


Reality ->Reality ->

DecisionDecision

HHOO is true is true HHOO is false is false

Retain HRetain HOO Correct Correct DecisionDecision

Type II Error Type II Error (())

(.2, .1)(.2, .1)

Reject HReject HOO Type I Error Type I Error (())

(.05, .01)(.05, .01)

Correct Correct DecisionDecision


Rules of thumb for effect sizes:Rules of thumb for effect sizes:– Small=.2Small=.2– Medium=.5Medium=.5– Large=.8Large=.8

So, if you want 80% chance of detecting So, if you want 80% chance of detecting a medium effect, using a .05 a medium effect, using a .05 value, value,

N= 4(1.96+.84)N= 4(1.96+.84)22/.5/.52 =2 = about 126, or 63 about 126, or 63 in each groupin each group

Point EstimationPoint Estimation

Hypothesis testing can only tell you Hypothesis testing can only tell you whether or not the effect of X is zero, it whether or not the effect of X is zero, it does not tell you how large or small does not tell you how large or small the effect is.the effect is.

Important—a p-value is not an Important—a p-value is not an indication of the size of an effect, it indication of the size of an effect, it depends greatly on sample sizedepends greatly on sample size

If you want an estimate of the actual If you want an estimate of the actual effect, you need confidence intervalseffect, you need confidence intervals


Confidence intervals give you an idea of Confidence intervals give you an idea of what the actual effect is likely to be in what the actual effect is likely to be in the population of interestthe population of interest

The most common confidence interval is The most common confidence interval is 95% and gives an upper and lower 95% and gives an upper and lower bound on what the effect is likely to be.bound on what the effect is likely to be.

The size of the interval depends on the The size of the interval depends on the sample size, variability of the measure, sample size, variability of the measure, and the degree of confidence you want and the degree of confidence you want that the interval contains the true effect.that the interval contains the true effect.


Many people prefer confidence intervals Many people prefer confidence intervals to hypothesis testing, because to hypothesis testing, because confidence intervals contain more confidence intervals contain more informationinformation

Not only can you tell whether the effect Not only can you tell whether the effect could be zero (is zero contained in the could be zero (is zero contained in the interval of possible effect values?) but interval of possible effect values?) but you also have the entire range of you also have the entire range of possible values the effect could bepossible values the effect could be

So, a confidence interval gives you all So, a confidence interval gives you all the information of a hypothesis test and the information of a hypothesis test and a whole lot more.a whole lot more.

Choosing the right testChoosing the right test

Typically one is interested in Typically one is interested in comparing group means.comparing group means.

If the outcome is continuous, and If the outcome is continuous, and one independent variable:one independent variable:– Two groups—t-testTwo groups—t-test– Three or more groups--ANOVAThree or more groups--ANOVA


If the outcome is continuous and If the outcome is continuous and there is more than one there is more than one independent variable:independent variable:– ANOVA, if all independent variables ANOVA, if all independent variables

are categoricalare categorical– ANCOVA or multiple linear ANCOVA or multiple linear

regression, if some independent regression, if some independent variables are continuousvariables are continuous


If the outcome is binary:If the outcome is binary:– Logistic regressionLogistic regression

If outcome is time until a If outcome is time until a specified outcome:specified outcome:– Survival analysis—Cox proportional Survival analysis—Cox proportional

hazards regressionhazards regression

Parametric vs. Non-Parametric vs. Non-parametric testsparametric tests Parametric tests are tests that use a known Parametric tests are tests that use a known

probability distribution to assess the p-value of probability distribution to assess the p-value of the outcome.the outcome.

Most outcomes do fairly closely follow a known Most outcomes do fairly closely follow a known probability distribution, and many tests are probability distribution, and many tests are robust to violations of distributional robust to violations of distributional assumptions, so the assigned p-value will be assumptions, so the assigned p-value will be fairly accurate in many situationsfairly accurate in many situations

For unique situations, such as specialized For unique situations, such as specialized outcomes or very skewed distributions, one outcomes or very skewed distributions, one can generate their own probability distribution can generate their own probability distribution to calculate a p-value. (jacknife, bootstrap, to calculate a p-value. (jacknife, bootstrap, etc.) Computers make these techniques very etc.) Computers make these techniques very fast and easyfast and easy

understanding inferential statistics

Technology