will your study have enough power? c yrus s amii n ew y ork u niversity iegovern impact evaluation...
TRANSCRIPT
Will your study have enough power?
CYRUS SAMIINEW YORK UNIVERSITY
ieGovern Impact Evaluation WorkshopIstanbul, TurkeyJanuary 27-30, 2015
: #ieGovern
Definition
THE POWER OF YOUR STUDY MEASURES THE CONFIDENCE YOU CAN HAVE THAT IT WILL LEAD YOU TO DRAW THE RIGHT CONCLUSION.
What we will not do today
What we will not do today
Definition
THE POWER OF YOUR STUDY MEASURES THE CONFIDENCE YOU CAN HAVE THAT IT WILL ALLOW YOU TO DRAW THE RIGHT CONCLUSION.
BUT WHY MIGHT YOU DRAW THE WRONG CONCLUSION?
Definition
THE POWER OF YOUR STUDY MEASURES THE CONFIDENCE YOU CAN HAVE THAT IT WILL ALLOW YOU TO DRAW THE RIGHT CONCLUSION.
BUT WHY MIGHT YOU DRAW THE WRONG CONCLUSION?
THE WORLD IS MESSY.
Definition
POWER = EFFECT SIZE + NUMBERS - MESSY
Messy
“WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?”
Messy
“WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?”
EFFECTS ARE MESSY.
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
DEPENDS ON THE KIND OF PERSON: “EAGER BEAVER”
“NORMAL BUREAUCRAT”
“LAZY SOB”
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
Type of person With incentive Without incentive Effect of incentive
Eager beaver 10 cases/day 10 cases/day 0
Normal bureaucrat 10 cases/day 2 cases/day 8
Lazy SOB 2 cases/day 2 cases/day 0
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
IF INCENTIVE SCHEME APPLIED TO WHOLE DEPARTMENT:
EFFECT = (8 X 0) + (8 X 8) + (8 X 0) = 64 MORE CASES/DAY
Type of person With incentive Without incentive
Effect of incentive
Number of these types?
Eager beaver 10 cases/day 10 cases/day 0 8
Normal bureaucrat
10 cases/day 2 cases/day 8 8
Lazy SOB 2 cases/day 2 cases/day 0 8
Numbers and messy
SUPPOSE OUR STUDY WORKS WITH 2 OFFICES.
1 GETS THE INCENTIVE SCHEME AND 1 DOESN’T.
THEN WE MEASURE THE PRODUCTIVITY EFFECT.
WHAT COULD HAPPEN?
32 16
16 X 6 = 96
40 16
24 X 6 = 144
16 8
8 X 6 = 48
16
40
-24 X 6 = -144
Numbers and messy
IF WE ONLY WORK WITH 2 OFFICES, WE HAVE A STRONG POSSIBLY OF DRAWING WILDLY MISTAKEN CONCLUSIONS.
Numbers and messy
WHAT IF WE WORK WITH ALL SIX OFFICES?
32 16
40
16
16
8
2 X 48 = 96
24 32
40
8
40
16
2 X 16 = 32
24 32
40
8
16
16
2 X 40 = 80
32 32
40
8
16
16
2 X 16 = 32
Numbers and messy
COMPARE:
Numbers and messy
WITH EVEN MORE:
POWER = EFFECT SIZE + NUMBERS - MESSY
POWER = EFFECT SIZE + NUMBERS - MESSY
Numbers help to compensate for messy.
POWER = EFFECT SIZE + NUMBERS - MESSY
Numbers help to compensate for messy.
Sometimes increasing numbers is straighforward.
Sometimes you need to be creative:• For example, information campaigns experiments for
universal policies.
POWER = EFFECT SIZE + NUMBERS - MESSY
Other things you can do to reduce the problem of messy:• Stratification techniques• Analytical techniques (regression adjustment)
POWER = EFFECT SIZE +
NUMBERS - MESSY
POWER = EFFECT SIZE +
NUMBERS - MESSY
If the effect is REALLY large or small, then even with small numbers and lots of messy you can draw reasonable conclusions.
But if the effect is small, you need lots of numbers and ways to reduce messy to draw good conclusions.
POWER = EFFECT SIZE +
NUMBERS - MESSYOf course, you do not know the effect size.
So, what assumption should you make about it?
Define a minimal desirable effect that would lead you to adopt the intervention you are studying.
You want to have power to detect this minimal desirable effect.
Formally…
Minimal detectable effect
Test critical value Power critical value
Effect estimator standard deviation (“messy”) as a function of sample size (numbers)
Conclusion
THE WORLD IS MESSY. EFFECTS ARE MESSY.
RELIABLE EVIDENCE REQUIRES THAT WE HAVE ADEQUATE NUMBERS. SOMETIMES WE NEED TO BE CREATIVE WITH THIS.
WE WANT ADEQUATE POWER SO WE DON’T WASTE $$$ ON UNRELIABLE OR INCONCLUSIVE STUDIES.
RESEARCH DESIGN COURSE MATERIALS:
HTTP://CYRUSSAMII.COM/?PAGE_ID=1603