resampling: power analysis
DESCRIPTION
Austin Hilliard Feb 19 2009. Resampling: power analysis. We've done an experiment. Data:. boxplot([ctrl exp], 'labels' ,{ 'ctrl' 'exp' }). ctrl = 8 13 14 13 16 13 15 3 9 9. exp = 4 13 12 18 12 6 11 - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/1.jpg)
Resampling: power analysis
Austin HilliardFeb 19 2009
![Page 2: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/2.jpg)
We've done an experiment...
Data:
ctrl =
8 13 14 13 16 13 15 3 9 9
exp =
4 13 12 18 12 6 11 4 9 9
boxplot([ctrl exp],'labels',{'ctrl' 'exp'})
![Page 3: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/3.jpg)
...we've looked at the data...figuresubplot(1,2,1), hist(ctrl,5), xlabel('ctrl')subplot(1,2,2), hist(exp,5), xlabel('exp')
![Page 4: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/4.jpg)
...we've done some stats>>out = bootstrap2groups(@median,ctrl,exp) ans: gp1desc: 13 gp2desc: 10 testStat: 3 distMean: -0.0081 reflectLeft: -3.0162 pvalRight: 0.0955 pvalLeft: 0.0991 pval: 0.1946 groups: [10x2 double]
But there was no effect!
![Page 5: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/5.jpg)
Did we have sufficient power?
power = chance we have of detecting a difference of a particular size (δ), using a particular statistic, investigating a particular n
What can increase power? increased n increased across group variation decreased within group variation
Probably a cheat, but does increase power: increase α → relax significance threshold
![Page 6: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/6.jpg)
Our experiment
What was our n? → 10 What statistic?
difference of medians What was δ? →
~23% decrease What was α? → 0.05 Across group variation? Within group variation?
ctrl =
8 13 14 13 16 13 15 3 9 9
median = 13
stdev = 3.97
exp =
4 13 12 18 12 6 11 4 9 9
median = 10
stdev = 4.37
![Page 7: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/7.jpg)
Retrospective power
We got a null result, is it “real”? Maybe we didn't have large enough n Maybe we chose poor descriptors/statistic,
resulting in small δ Maybe we really do have a significant
difference (i.e. p < α), but study design prevented us from detecting it
study design = n, statistic, significance level (α)
![Page 8: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/8.jpg)
Retrospective power
Basic strategy Decide what δ to test → use actual (-23%), or can generate
exp groups w/ different δ by scaling ctrl group Generate null distribution of test stat Compute 95% confidence interval for null dist (assuming α =
.05) Generate alternate dist of test stat using δ (preserve
difference between groups when resampling) See how many test stats in alt dist are outside confidence
interval of null dist, divide by size of dist → power
![Page 9: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/9.jpg)
What will this tell us?
Considering a 23% decrease from ctrl to exp as significant w/ α = .05, with 10 subjects/group, using the difference between medians as our test statistic, the power is the percentage of times that we will actually detect this difference
Have created null distribution as per normal, then, instead of testing just our actual data against this null dist, test permutations of actual data to see how often we get a significant result
![Page 10: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/10.jpg)
Generate null distributionctrl = 8 13 14 13 16 13 15 3 9 9
exp = 4 13 12 18 12 6 11 4 9 9
boxNULL = [ctrl; exp]; % combine datasetsstatsNULL = []; % initialize vec for pseudo stats for trials = 1:10000 % run 10,000 times pseudo_ctrl = sample(length(ctrl), boxNULL); % create pseudo groups pseudo_exp = sample(length(exp), boxNULL); pseudo_stat_null = ... % compute pseudo test stat median(pseudo_ctrl) - median(pseudo_exp); statsNULL = [statsNULL pseudo_stat_null]; % store pseudo test stat end NULLconfint = ... % compute 95% confidence interval percentile(statsNULL, [.025 .975]
![Page 11: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/11.jpg)
Null distribution 95th percentileNULLconfint = -3.9700 3.9900
![Page 12: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/12.jpg)
Generate alternate distribution
% Don't combine groups here, want to preserve % difference since this is distribution reflecting% values of test stat when there IS a difference statsALT = []; % initialize vec for pseudo stats for trials = 1:10000 % run 10,000 times pseudo_ctrlALT = sample(length(ctrl), ctrl); % create pseudo groups pseudo_expALT = sample(length(exp), exp); % only sample from ctrl for % pseudo ctrl, only % sample from exp % for pseudo exp pseudo_stat_alt = ... % compute pseudo test stat median(pseudo_ctrlALT) - median(pseudo_expALT);
statsALT = [statsALT pseudo_stat_alt]; % store pseudo test stat end
![Page 13: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/13.jpg)
Alternate distribution
![Page 14: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/14.jpg)
Compute power
% Count test stats from alternate dist that% fall outside of significance threshold% (defined here as 95th percentile) of null dist
right_power = count(statsALT > NULLconfint(2)); % count values of test stat in % alt
dist % greater than 97.5 % percentile of null dist left_power = count(statsALT < NULLconfint(1)); % count values of test stat in
% alt dist % less than .025 % percentile of null dist power = (right_power + left_power) / trials % sum counts and divide by size
% of dist to compute power
![Page 15: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/15.jpg)
mean(statsNULL) = -.02 mean(statsALT) = 2.78 std(statsNULL) = 1.99 std(statsALT) = 2.11
power = 0.2963 → NOT GOOD
![Page 16: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/16.jpg)
Test different values of δ
our experiment
![Page 17: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/17.jpg)
Try different test statdifference of means, instead of difference of mediansbut remember what our data looked like? mean is a poor descriptor → likely over-estimating power
![Page 18: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/18.jpg)
Try different test statabsolute value of difference of medians
![Page 19: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/19.jpg)
function out = power_EXAMPLE(ctrl, change, Func) % ctrl is control group, should be column vector% change is factor to scale ctrl group by to create phantom exp% Func is method for computing descriptors exp = change * ctrl; % --- null hypothesis ---boxNULL = [ctrl; exp]; % combine datasetsstatsNULL = []; % initialize vec for pseudo stats for trials = 1:10000 % run 10,000 times pseudo_ctrl = sample(length(ctrl), boxNULL); % create pseudo groups pseudo_exp = sample(length(exp), boxNULL); pseudo_stat_null = ... % compute pseudo test stat feval(Func, pseudo_ctrl) - feval(Func, pseudo_exp); statsNULL = [statsNULL pseudo_stat_null]; % store pseudo test stat end out.NULLconfint = ... % compute 95% confidence interval percentile(statsNULL, [.025 .975]); % --- alternate hypothesis ---% Don't combine groups here, want to preserve % difference since this is distribution reflecting% values of test stat when there IS a difference statsALT = []; % intialize vec for pseudo stats for trials = 1:10000 % run 10,000 times pseudo_ctrlALT = sample(length(ctrl), ctrl); % create pseudo groups pseudo_expALT = sample(length(exp), exp); % only sample from ctrl for % pseudo ctrl, only % sample from exp % for pseudo exp pseudo_stat_alt = ... % compute pseudo test stat feval(Func,pseudo_ctrlALT) - feval(Func,pseudo_expALT); statsALT = [statsALT pseudo_stat_alt]; % store pseudo test stat end% Count test stats from alternate dist that% fall outside of significance threshold% (defined here as 95th percentile) of null dist out.right_count = count(statsALT > out.NULLconfint(2)); % count values of test stat for alt dist % greater than 97.5 % percentile of null dist out.left_count = count(statsALT < out.NULLconfint(1)); % count values of test stat for alt dist % less than .025 % percentile of null dist out.power = (out.right_count + out.left_count) / trials; % sum counts and divide by size of dist % to compute power figuresubplot(1,2,1), hist(statsNULL), title('Null distribution')subplot(1,2,2), hist(statsALT), title('Alternate distribution')
![Page 20: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/20.jpg)
% use this script to try different values of 'change'% using power_EXAMPLE.m% 'change' is scalar to create phantom exp group by scaling ctrl group% example: change = .9 will create phantom exp group that shows 10%% decrease compared to ctrl, since ctrl will be multiplied by .9 clear powersclear conf_intsclear runsclear more powers=[];conf_ints=[]; runs = [.99 .95 .9 .85 .8 .75 .7 .65 .6 .55 .5 .45 .4 .35 .3 .25 .2 .15 .1 .01];more = length(runs); for it = runs out = power_EXAMPLE(ctrl,it,@median); powers = [powers out.power]; conf_ints = [conf_ints; out.NULLconfint]; more more = more - 1;end figure subplot(1,2,1)plot(100.*(1-runs),powers,'-o','MarkerEdgeColor','b','MarkerSize',12,'Color','k')xlabel('Percent decrease from ctrl -> exp', 'fontsize', 14)ylabel('Power', 'fontsize', 14) subplot(1,2,2)holdplot(100.*(1-runs),conf_ints(:,1),'x','Color','r'),text(1,conf_ints(1,1)-2,'lower bound')plot(100.*(1-runs),conf_ints(:,2),'o','Color','b'),text(1,conf_ints(1,2)+2,'upper bound')xlabel('Percent decrease from ctrl -> exp', 'fontsize', 14)ylabel('Null distribution 95% confidence interval bounds', 'fontsize', 14) hold
![Page 21: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/21.jpg)
Best way to increase power?
Increase n Our power seemed to plateau around .7 (when
using medians) Maybe we just need to get more subjects How many more?
→ Prospective power analysis
![Page 22: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/22.jpg)
Prospective power
Basic strategy Choose n and δ to test Generate phantom ctrl group
sample from normal distribution w/ mean and standard deviation of real ctrl group n times
Generate phantom exp group scale phantom ctrl by δ, or use pilot data to generate
phantom exp with bigger n, as with ctrl group above Proceed w/ “retrospective” power analysis
![Page 23: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/23.jpg)
Generate groups
function power = power_prospectEXAMPLE(ctrl, change, n) % ctrl is control group% change is factor to scale ctrl by to create exp% n is sample size to test u = mean(ctrl); % compute mean and stdev of actual ctrl groupsig = std(ctrl); ctrl = normrnd(u, sig, n, 1); % generate phantom ctrl group by sampling from % norm dist w/ same mean and stdev as real % ctrl group, sample n times exp = change * ctrl; % generate exp group by scaling ctrls
...Now just run the retrospective analysis on these groups
![Page 24: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/24.jpg)
VariabilityProspective power values can fluctuate with each run, suggest running at least 10x
Example: n = 20, run 100x
mean power = .47median power = .43
![Page 25: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/25.jpg)
Try different values of nUse δ from actual data: 23% decrease from ctrl → expEach point is mean of 10 runs for that n
![Page 26: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/26.jpg)
![Page 27: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/27.jpg)
![Page 28: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/28.jpg)
![Page 29: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/29.jpg)
![Page 30: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/30.jpg)
![Page 31: Resampling: power analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062519/56814ed0550346895dbc6daf/html5/thumbnails/31.jpg)