1

Hester van EerenErasmus Medical Centre, Rotterdam

Halsteren, August 23, 2010

2

The propensity score (1)The propensity score is “…the conditional probability

of assignment to a particular treatment given a vector of observed covariates.” (Rosenbaum en Rubin, 1983: 41).• Used in non-randomized studies to control for selection

bias• Balance observed pretreatment variables among

patient• Find an estimate of the average treatment effect

But, treatment effect can be different within subgroups

3

The propensity score (2)Univariate propensity score Multivariate propensity scoreBartak and colleagues (2009) Spreeuwenberg and colleagues

(2010)Used for 2 treatment categories Used for > 2 treatment

categoriesPropensity score used in:

• Matching• Stratification• Regression• Inverse probability weight• Combinations of …

Methods in this studyTo find a treatment effect within subgroups, if the

propensity score is applied:• Method 1: Regression analysis with propensity score,

subgroups and interaction with treatment assignment;

• Method 2: Weighted regression analysis with inverse of the propensity score (to weight observations), subgroups and interaction with treatment assignment;

• Method 3: Propensity score applied for groups defined on treatment assignment and subgroups; then, regression analysis with propensity score and dummies for groups

Two treatment categories and two subgroups are used in this study

4

Variable selection for propensity scoreDoes the variable for subgroups has to be included?Discussion about variable selection for propensity score;

• Only variables related to outcome?• Only variables related to treatment assignment?• Both variables…?

In this study; 8 different propensity scores (PS) formulated, based

on: • Variables related to outcome, with and without subgroup• Variables related to treatment assigment, with and without

subgroup• Both variables…, with and without subgroup• Only variables related to both outcome and treatment

assignment, with and without subgroup5

How to test? (1)Real dataset not useful because effects

unknown beforehand; You cannot test whether the effect found is

accurate

Monte Carlo simulation study to test methods and different propensity scores:Simulate data with known treatment effectsEstimate different propensity scores for this dataApply different methods for different propensity scores, for

this data Repeat this process 1000 times

6

How to test? (2)What do you want to know?

If the treatment effect estimated is (almost) equal to the treatment effect you used to simulate the data

Bias of estimator: difference between estimated treatment effect and the true value of parameter

Want to have an unbiased estimate; Less bias indicates a more accurate estimate of the

treatment effect

Bias is estimated for overall treatment effect and for the treatment effect within subgroups

7

Results

8

N=250;ρ=0 N=250;ρ=0.3 N=250; ρ=0.7 N=500;ρ=0 N=500;ρ=0.3 N=500; ρ=0.7 N=1000;ρ=0 N=1000;ρ=0.3 N=1000; ρ=0.7

Row Method Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSERegression1 PS1, func 1 .0736 .0243 .0706 .0265 .0797 .0304 .0844 .0160 .0763 .0159 .0803 0.172 .0812 .0109 .0786 .0113 .0834 .01352 PS1, func 2 .0728 .0237 .0724 .0258 .0785 .0288 .0853 .0156 .0774 .0156 .0807 .0169 .0801 .0104 .0779 .0110 .0825 .01303 PS1, func 3 -.0060 .0312 -.0050 .0336 -.0014 .0345 .0042 .0149 -.0025 .0162 .0006 .0168 -.0001 .0071 -.0053 .0080 .0029 .00924 PS1, subgr. -.0035 .0820 -.0056 .0762 -.0008 .0719 .0030 .0414 .0007 .0382 .0000 .0398 .0006 .0218 .0085 .0193 -.0014 .01785 PS2, func 1 .0746 .0262 .0710 .0290 .0798 .0317 .0860 .0175 .0745 .0171 .0804 .0180 .0813 .0116 .0787 .0119 .0838 .01396 PS2, func 2 .0740 .0257 .0730 .0281 .0785 .0298 .0864 .0171 .0755 .0166 .0807 .0175 .0804 .0112 .0778 .0116 .0830 .01357 PS2, func 3 -.0049 .0343 -.0055 .0371 -.0023 .0368 .0070 .0169 -.0053 .0183 .0012 .0178 .0002 .0083 -.0061 .0088 .0033 .00988 PS2, subgr. -.0030 .0901 -.0025 .0892 .0020 .0765 -.0015 .0467 .0028 .0448 -.0014 .0433 .0006 .0243 .0105 .0220 -.0009 .01969 PS3, func 1 .0727 .0237 .0723 .0258 .0787 .0289 .0852 .0156 .0774 .0156 .0807 .0169 .0802 .0104 .0779 .0110 .0824 .013010 PS3, func 2 .0740 .0237 0730 .0258 .0785 .0289 .0864 .0156 .0755 .0156 .0807 .0169 .0804 .0104 .0778 .0110 .0830 .013011 PS3, func 3 -.0062 .0312 -.0053 .0337 -.0017 .0346 .0039 .0149 -.0028 .0162 .0005 .0168 -.0001 .0071 -.0053 .0080 .0027 .009112 PS3, subgr. -.0030 .0819 -.0047 .0763 .0001 .0719 .0034 .0414 .0012 .0383 .0003 .0399 .0008 .0218 .0088 .0190 -.0011 .017813 PS4, func 1 .0738 .0257 .0731 .0281 .0782 .0299 .0863 .0170 .0756 .0166 .0808 .0175 .0805 .0112 .0778 .0116 .0830 .013514 PS4, func 2 .0739 .0257 .0730 .0281 .0782 .0299 .0862 .0170 .0756 .0166 .0807 .0175 .0805 .0112 .0779 .0116 .0830 .013515 PS4, func 3 -.0051 .0343 -.0056 .0372 -.0029 .0370 .0068 .0169 -.0053 .0183 .0011 .0178 .0002 .0083 -.0061 .0088 .0030 .009816 PS4, subgr. -.0027 .0901 -.0020 .0894 .0027 .0766 -.0012 .0466 .0033 .0449 -.0012 .0433 .0007 .0243 .0107 .0220 -.0006 .019717 PS5, func 1 .0838 .0345 .0855 .0383 .0859 .0357 .0809 .0201 .0785 .0213 .0779 .0196 .0827 .0138 .0768 ..0126 .0812 .013018 PS5, func 2 .0836 .0332 .0850 .0370 .0872 .0345 .0806 .0193 .0769 .0202 .0778 .0189 .0824 .0136 .0766 .0123 .0800 .012619 PS5, func 3 .0047 .0407 .0027 .0440 .0129 .0396 -.0018 .0209 -.0056 .0209 -.0026 .0183 -.0002 .0104 -.0002 .0100 .0000 .008520 PS5, subgr. -.0023 .0898 .0061 .0895 -.0135 .0868 .0064 .0481 .0066 .0421 .0013 .0381 .0067 .0224 -.0081 .0220 -.0006 .018421 PS6, func 1 .0836 .0332 .0849 .0369 .0871 .0346 .0805 .0193 .0769 .0202 .0778 .0188 .0823 .0136 .0765 .0123 .0800 .012622 PS6, func 2 .0836 .0332 .0850 .0370 .0871 .0345 .0806 .0193 .0768 .0202 .0778 .0189 .0823 .0136 .0765 .0123 .0801 .012623 PS6, func 3 .0046 .0407 .0023 .0439 .0126 .0395 -.0019 .0210 -.0059 .0209 -.0027 .0183 -.0003 .0104 -.0004 .0100 -.0001 .008524 PS6, subgr. -.0021 .0898 .0069 .0834 -.0127 .0869 .0067 .0481 .0070 .0421 .0016 .0381 .0068 .0224 -.0079 .0220 -.0003 .018425 PS7, func 1 .0839 .0320 .1611 .0528 .1663 .0544 .0791 .0180 .1561 .0380 .1584 .0377 .0829 .0130 .1561 .0303 .1632 .032826 PS7, func 2 .0836 .0307 .1603 .0518 .1678 .0536 .0784 .0173 .1550 .0369 .1587 .0373 .0827 .0129 .1560 .0301 .1621 .032227 PS7, func 3 .0045 .0362 .0790 .0460 .0948 .0459 -.0037 .0180 .0724 .0242 .0786 .0235 .0017 .0093 .0794 .0154 .0827 .015228 PS7, subgr. -.0013 .0819 .0037 .0849 -.0162 .0828 .0055 .0435 .0068 .0402 .0005 .0379 .0029 .0207 -.0085 .0205 -.0022 .018529 PS8, func 1 .0837 .0307 .1602 .0517 .1679 .0537 .0783 .0173 .1551 .0369 .1587 .0373 .0827 .0129 .1560 .0301 .1621 .032330 PS8, func 2 .0836 .0307 .1603 .0517 .1680 .0538 .0784 .0173 .1550 .0369 .1587 .0373 .0827 .0129 .1560 .0301 .1621 .032331 PS8, func 3 .0043 .0362 .0787 .0458 .0947 .0459 -.0039 .0181 .0722 .0242 .0785 .0235 .0016 .0093 .0793 .0154 .0826 .015232 PS8, subgr. -.0008 .0819 .0046 .0848 .0059 .0436 .0073 .0402 .0009 .0379 .0009 .0379 .0031 .0207 -.0083 .0205 -.0019 .0185

N=250;ρ=0 N=250;ρ=0.3 N=250; ρ=0.7 N=500;ρ=0 N=500;ρ=0.3 N=500; ρ=0.7 N=1000;ρ=0 N=1000;ρ=0.3 N=1000; ρ=0.7

Row Method Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSERegression1 PS1, func 1 .0812 .0272 .1017 .0301 .0987 .0321 .0874 .0173 .1043 .0208 .0960 .0208 .0796 .0113 .1043 .0158 .1019 .01602 PS1, func 2 .0780 .0258 .0735 .0247 .0715 .0274 .0851 .0163 .0746 .0149 .0687 .0156 .0796 .0110 .0751 .0104 .0763 .01143 PS1, func 3 -.0011 .0322 -.0022 .0296 -.0131 .0358 .0094 .0157 -.0040 .0148 -.0138 .0175 -.0002 .0080 -.0041 .0075 -.0109 .00874 PS1, subgr. .0025 .0806 -.0087 .0797 .0253 .0913 -.0103 .0426 -.0009 .0375 .0186 .0441 -.0005 .0202 .0000 .0184 .0306 .02185 PS2, func 1 .0824 .0289 .0792 .0280 .0762 .0289 .0866 .0185 .0799 .0175 .0741 .0180 .0781 .0118 .0780 .0119 .0802 .01256 PS2, func 2 .0805 .0273 .0813 .0280 .0763 .0293 .0846 .0175 .0767 .0170 .0726 .0169 .0780 .0115 .0802 .0117 .0801 .01247 PS2, func 3 -.0009 .0341 .0035 .0347 -.0086 .0380 .0094 .0179 .0011 .0170 -.0102 .0185 -.0006 .0092 .0000 .0086 -.0072 .00988 PS2, subgr. .0030 .0863 -.0047 .0968 .0248 .0989 -.0112 .0475 -.0012 .0451 .0183 .0477 -.0031 .0224 .0020 .0208 .0294 .02469 PS3, func 1 .0798 .0258 .0790 .0254 .0769 .0280 .0852 .0163 .0733 .0158 .0735 .0163 .0795 .0110 .0803 .0112 .0807 .012110 PS3, func 2 .0805 .0258 .0813 .0253 .0763 .0279 .0846 .0163 .0767 .0158 .0726 .0162 .0780 .0110 .0802 .0112 .0801 .012011 PS3, func 3 -.0014 .0322 .0018 .0296 -.0117 .0359 .0093 .0157 -.0002 .0148 -.0129 .0175 -.0003 .0080 -.0003 .0075 -.0105 .008812 PS3, subgr. .0033 .0806 -.0050 .0806 .0352 .0930 -.0098 .0426 .0027 .0377 .0286 .0451 -.0002 .0202 .0034 .0186 .0400 .022813 PS4, func 1 .0804 .0273 .0810 .0277 .0764 .0289 .0847 .0175 .0798 .0170 .0727 .0170 .0780 .0115 .0801 .0117 .0801 .012414 PS4, func 2 .0805 .0273 .0810 .0276 .0760 .0287 .0847 .0175 .0797 .0170 .0728 .0169 .0780 .0115 .0800 .0117 .0800 .012415 PS4, func 3 -.0010 .0342 .0030 .0347 -.0088 .0376 .0093 .0179 .0010 .0170 -.0102 .0185 -0007 .0092 -.0003 .0085 -.0073 .009716 PS4, subgr. .0033 .0863 -.0042 .0972 .0252 .0992 -.0109 .0476 -.0009 .0450 .0189 .0478 -.0029 .0224 .0021 .0209 .0294 .024717 PS5, func 1 .0851 .0343 .0769 .0326 .0800 .0330 .0875 .0213 .0784 .0201 .0752 .0191 .0830 .0139 .0787 .0134 .0729 .011818 PS5, func 2 .0845 .0329 .0758 .0317 .0786 .0321 .0873 .0204 .0789 .0195 .0744 .0184 .0830 .0136 .0779 .0129 .0743 .011619 PS5, func 3 .0062 .0410 -.0024 .0430 -.0029 .0403 .0078 .0201 -.0016 .0208 -.0123 .0185 .0009 .0097 -.0038 .0105 -.0092 .009520 PS5, subgr. -.0046 .0945 -.0029 .0899 .0144 .0955 -.0010 .0465 .0031 .0442 .0288 .0450 .0054 .0243 .0063 .0252 .0189 .023621 PS6, func 1 .0844 .0329 .0756 .0317 .0786 .0322 .0872 .0204 .0791 .0196 .0743 .0184 .0830 .0136 .0781 .0129 .0742 .011622 PS6, func 2 .0844 .0330 .0756 .0317 .0783 .0321 .0872 .0204 .0790 .0195 .0741 .0183 .0830 .0136 .0780 .0129 .0741 .011523 PS6, func 3 .0059 .0410 -.0030 .0430 -.0034 .0405 .0075 .0201 -.0016 .0208 -.0128 .0185 .0009 .0098 -.0039 .0105 -.0091 .009524 PS6, subgr. -.0041 .0934 -.0019 .0899 .0153 .0958 -.0006 .0465 .0035 .0442 .0296 .0451 .0056 .0243 .0065 .0252 .0185 .023425 PS7, func 1 .0831 .0303 .0187 .0592 .1980 .0650 .0875 .0199 .1881 .0478 .1941 .0505 .0816 .0128 .1882 .0420 .1939 .043926 PS7, func 2 .0819 .0289 .1426 .0437 .1426 .0447 .0875 .0192 .1461 .0332 .1389 .0318 .0815 .0125 .1448 .0272 .1406 .025627 PS7, func 3 .0046 .0359 .0677 .0433 .0672 .0412 .0080 .0185 .0677 .0231 .0585 .0210 -.0006 .0087 .0645 .0137 .0640 .013128 PS7, subgr. -.0068 .0852 -.0101 .0820 .0002 .0883 -.0010 .0437 -.0018 .0410 .0136 .0434 .0053 .0220 .0031 .0229 .0026 .022029 PS8, func 1 .0818 .0289 .1428 .0438 .1437 .0450 .0876 .0192 .1464 .0333 .1402 .0322 .0815 .0124 .1451 .0273 .1419 .025930 PS8, func 2 .0818 .0289 .1429 .0438 .1434 .0450 .0876 .0192 .1464 .0333 .1398 .0320 .0815 .0125 .1451 .0273 .1416 .025831 PS8, func 3 .0043 .0358 .0660 .0433 .0626 .0409 .0078 .0185 .0665 .0230 .0537 .0206 -.0006 .0087 .0631 .0135 .0595 .012632 PS8, subgr. -.0062 .0853 -.0051 .0821 .0146 .0892 -.0005 .0437 .0022 .0411 .0284 .0444 .0055 .0220 .0071 .0230 .0165 .0224

N=250;ρ=0 N=250;ρ=0.3 N=250; ρ=0.7 N=500;ρ=0 N=500;ρ=0.3 N=500; ρ=0.7 N=1000;ρ=0 N=1000;ρ=0.3 N=1000; ρ=0.7

Row Method Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSEInverse PS1 PS1, func 1 .0831 .0298 .1116 0377 .1402 .0712 .0892 .0181 .1054 .0244 .1127 .0409 .0810 .0118 .1041 .0173 .1102 .03112 PS1, func 2 .0835 .0283 .0714 .0295 .0738 .0490 .0881 .0173 .0611 .0165 .0478 .0257 .0816 .0117 .0597 .0101 .0483 .01753 PS1, func 3 .0049 .0380 -.0027 .0492 -.0034 .0781 .0120 .0198 -.0171 .0267 -.0311 .0473 .0027 .0099 -.0191 .0136 -.0387 .03134 PS1, subgr. .0015 .1177 -.0018 .1667 .0181 .2239 -.0068 .0634 .0032 .0881 .0136 .1394 -.0016 .0307 .0008 .0411 .0279 .08115 PS2, func 1 .0921 .0380 .1012 .0524 .1384 .0801 .0940 .0236 .0872 .0356 .1012 .0504 .0831 .0147 .0838 .0207 .0933 .04716 PS2, func 2 .0922 .0354 .1058 .0467 .1239 .0646 .0940 .0227 .0889 .0303 .0959 .0376 .0837 .0143 .0856 .0193 .0917 .03387 PS2, func 3 .0120 .0494 .0362 .0667 .0450 .0866 .0182 .0282 .0120 .0376 .0154 .0556 .0032 .0140 .0044 .0225 .0011 .04748 PS2, subgr. .0076 .1496 -.0095 .2144 .0219 .2375 -.0058 .0818 .0038 .1280 .0181 .1561 .0038 .0418 .0086 .0596 .0391 .11249 PS3, func 1 .0818 .0285 .0903 .0329 .1257 .0649 .0871 .0171 .0821 .0196 .1008 .0360 .0811 .0116 .0810 .0130 .1007 .026910 PS3, func 2 .0820 .0283 .0905 .0313 .1157 .0496 .0875 .0172 .0823 .0187 .0963 .0286 .0812 .0116 .0812 .0127 .0961 .021411 PS3, func 3 .0033 .0384 .0159 .0476 .0391 .0701 .0111 .0198 .0043 .0251 .0120 .0392 .0023 .0099 .0029 .0124 .0117 .024612 PS3, subgr. .0014 .1187 -.0026 .1638 .0108 .2112 -.0064 .0635 .0016 .0855 .0024 .1268 -.0015 .0307 -.0016 .0399 .0177 .071613 PS4, func 1 .0895 .0368 .1025 .0526 .1384 .0821 .0920 .0229 .0868 .0348 .1006 .0498 .0830 .0143 .0842 .0207 .0931 .046714 PS4, func 2 .0903 .0359 .1026 .0455 .1225 .0592 .0932 .0227 .0879 .0293 .0981 .0360 .0833 .0143 .0849 .0187 .0914 .032115 PS4, func 3 .0100 .0510 .0327 .0669 .0429 .0815 .0171 .0288 .0109 .0376 .0180 .0525 .0027 .0141 .0035 .0218 .0013 .044916 PS4, subgr. .0076 .1534 -.0092 .2170 .0230 .2368 -.0052 .0828 .0038 .1285 .0162 .1545 .0040 .0421 .0087 .0595 .0377 .112517 PS5, func 1 .0957 .0458 .0994 .0595 .1335 .0816 .0918 .0273 .0914 .0350 .1102 .0612 .0823 .0162 .0830 .0219 .0972 .031618 PS5, func 2 .0983 .0440 .0985 .0523 .1210 .0651 .0936 .0254 .0930 .0312 .1025 .0417 .0830 .0158 .0843 .0199 .0960 .024919 PS5, func 3 .0298 .0619 .0213 .0845 .0455 .0861 .0182 .0295 .0075 .0407 .0211 .0487 .0012 .0147 .0027 .0225 .0172 .025620 PS5, subgr. -.0204 .1722 .0117 .2172 .0125 .2358 -.0063 .0885 .0235 .1119 .0204 .1491 .0074 .0437 .0092 .0614 .0047 .074521 PS6, func 1 .0959 .0446 .0970 .0587 .1326 .0812 .0914 .0262 .0912 .0350 .1097 .0599 .0823 .0160 .0822 .0216 .0974 .030822 PS6, func 2 .0972 .0438 .0963 .0523 .1221 .0604 .0925 .0254 .0914 .0307 .1034 .0391 .0826 .0158 .0839 .0198 .0934 .023123 PS6, func 3 .0287 .0621 .0191 .0861 .0469 .0813 .0173 .0296 .0058 .0407 .0230 .0450 .0006 .0148 .0024 .0223 .0142 .024524 PS6, subgr. -.0208 .1731 .0115 .2208 .0111 .2352 -.0068 .0894 .0237 .1130 .0176 .1505 .0076 .0439 .0088 .0621 .0059 .074925 PS7, func 1 .0843 .0318 .1923 .0677 .2242 .1002 .0889 .0204 .1925 .0520 .2136 .0765 .0809 .0131 .1889 .0436 .2088 .058526 PS7, func 2 .0851 .0306 .1346 .0473 .1166 .0568 .0901 .0198 .1359 .0325 .1082 .0370 .0813 .0128 .1301 .0245 .1063 .023327 PS7, func 3 .0116 .0434 .0576 .0604 .0425 .0760 .0114 .0219 .0520 .0302 .0337 .0395 -.0012 .0107 .0502 .0160 .0332 .019928 PS7, subgr. -.0114 .1236 .0041 .1558 .0087 .2113 -.0007 .0648 .0165 .0792 .0022 .1262 .0076 .0325 .0032 .0407 -.0091 .060329 PS8, func 1 .0836 .0304 .1499 .0528 .1819 .0787 .0889 .0197 .1519 .0376 .1730 .0571 .0809 .0127 .1466 .0292 .1706 .041830 PS8, func 2 .0840 .0304 .1502 .0510 .1746 .0658 .0832 .0197 .1515 .0366 .1670 .0479 .0810 .0127 .1469 .0290 .1646 .036831 PS8, func 3 .0102 .0435 .0729 .0610 .1027 .0722 .0103 .0220 .0674 .0313 .0939 .0400 -.0016 .0107 .0672 .0176 .0932 .024332 PS8, subgr. -.0111 .1241 .0026 .1530 -.0049 .1924 -.0003 .0651 .0154 .0782 -.0076 .1122 .0079 .0325 .0015 .400 -.0174 .0522

Results (1)Which propensity score is the most accurate within

each method tested (tested with ANOVA):

But, some values for bias per propensity score where not very different from each other…

9

General treatment effect

Treatment effect within subgroups

Method 1

PS with variables related to outcome

PS with variables related to outcome

Method 2

PS with variables related to outcome and variable for subgroups

PS with variables only related to outcome and treatment assignment and variables for subgroups

Method 3

PS with variables related to outcome

NA

Results (2)Which method is most accurate when the most accurate

propensity scores are compared?Decide on partial effect size of method in ANOVA*

For general treatment effect, the partial effect size is 0,028, where method 1 gives the lowest bias (followed by method 3)

For treatment effect within subgroups, the partial effect size is 0,051, where method 1 gives the lowest bias too

Although the effect sizes for method are not very large, regression analysis with treatment assignment, subgroup, interaction between these and the propensity score, which is estimated with variables related to outcome, seems to be the most accurate method to find treatment effects within subgroups

*Effect size – 0,010 = small; 0,059 = medium; 0,138 = large (Cohen, 1988) 10

Discussion (1)Data simulation is done for different settings:

Sample size, correlation between covariates and correlation with covariate for subgroups are changed over simulations

Results for most accurate propensity score are based on sum of bias over all these settings; comparisons between methods for all propensity scores could give more in depth results

The overall bias for different propensity scores was sometimes not very different

Model for simulation of data was simple, linear; the relation between variables and outcome in practice can be more complicated

….11

Discussion (2)

Questions?

12