11

Building and Using Building and Using Disease Prediction Models Disease Prediction Models

in the Real Worldin the Real World

Discussion leader: Heejung Bang, Ph.D.Discussion leader: Heejung Bang, Ph.D.Weill Medical College of Cornell UniversityWeill Medical College of Cornell University

At Joint Statistical MeetingsAt Joint Statistical MeetingsSalt Lake Cit, UT, 2007Salt Lake Cit, UT, 2007

22

What are risk score/prediction What are risk score/prediction models?models?

A “prediction” is a statement or claim that a A “prediction” is a statement or claim that a particular event will occur in the future (current particular event will occur in the future (current or past event is also sensible).or past event is also sensible).

Response is often binary (event/non-event) or Response is often binary (event/non-event) or censored.censored.

Mathematical equation can be used to model the Mathematical equation can be used to model the rate (or probability or likelihood) of event.rate (or probability or likelihood) of event.

Scoring system (e.g., integer) can be derived to Scoring system (e.g., integer) can be derived to grade the risk, often by simplifying the grade the risk, often by simplifying the mathematical model (e.g., regression mathematical model (e.g., regression coefficients).coefficients).

Mathematical equation and/or scoring system can Mathematical equation and/or scoring system can be used to stratify subjects (e.g., high vs. low be used to stratify subjects (e.g., high vs. low risk )risk )

33

Why important?Why important?

Evidence-based medicine = Science Evidence-based medicine = Science (theory) + Data + Statistics(theory) + Data + Statistics

Risk score = Statistics + Art + RealityRisk score = Statistics + Art + Reality One of real practical solutions to reduce One of real practical solutions to reduce

the burden/incidence of some diseases.the burden/incidence of some diseases. People use it in real world (esp., lay and People use it in real world (esp., lay and

underserved people)underserved people)-- used in clinical setting, community -- used in clinical setting, community

setting, or self-use for pre-screening, setting, or self-use for pre-screening, screening or risk screening or risk assessment/prediction.assessment/prediction.

44

ButBut

Prediction is very hard, especially about the future - Yogi Berra

55

Prediction via multiple Prediction via multiple regression regression (Stat 101)(Stat 101)

Two general applications for multiple regression: Two general applications for multiple regression: explanation & prediction,explanation & prediction, two differing goals in two differing goals in research research

-- attempting to understand a phenomenon by -- attempting to understand a phenomenon by examining a variable's correlates on a group level examining a variable's correlates on a group level (explanation)(explanation)

-- being able to make valid projections concerning -- being able to make valid projections concerning an outcome for a particular individual (prediction)an outcome for a particular individual (prediction)

RefsRefs: 1. Osborne (2000). Prediction in multiple : 1. Osborne (2000). Prediction in multiple regression. regression. Pract Assessment, Res & EvalPract Assessment, Res & Eval . .

2. Neter, Kutner, Nachtsheim & Wasserman 2. Neter, Kutner, Nachtsheim & Wasserman (2003). Applied Linear Statistical Models.(2003). Applied Linear Statistical Models.

66

Regression Regression vs. vs.

Prediction/risk score Prediction/risk score

what make these two tasks what make these two tasks different?different?

77

1.1. Simple and easy: Simple and easy: Don’t let the Don’t let the perfect be the enemy of the perfect be the enemy of the

good!good! User-friendliness and easy use are User-friendliness and easy use are

important! important! -- a perfect model can be a jewel in your -- a perfect model can be a jewel in your

closet or a journal.closet or a journal.-- if biostatisticians can not use it, how lay -- if biostatisticians can not use it, how lay

persons can?persons can? Interactions or nonlinear function may make Interactions or nonlinear function may make

prediction model/risk score more complex.prediction model/risk score more complex.-- worth it? -- worth it? -- proper detection and modeling require -- proper detection and modeling require

larger N (more later). larger N (more later).

88

2. Variable categorization2. Variable categorization Most statisticians agree with Royston et al. (2005). Most statisticians agree with Royston et al. (2005).

“Dichotomizing continuous predictors in multiple “Dichotomizing continuous predictors in multiple regression: a bad idea”.regression: a bad idea”.

However, filling in continuous information (e.g., blood However, filling in continuous information (e.g., blood pressure (BP), BMI, CRP) can be hard for many people. pressure (BP), BMI, CRP) can be hard for many people.

-- Q: do you know your BP? Which BP? In what unit?-- Q: do you know your BP? Which BP? In what unit? A wrong unit is can be worse than nothing! (pound vs. A wrong unit is can be worse than nothing! (pound vs.

kg, mg vs. g) Unit is more complex than you ever kg, mg vs. g) Unit is more complex than you ever imagine.imagine.

Prediction model that includes continuous variables Prediction model that includes continuous variables may not be converted to a simple questionnaire.may not be converted to a simple questionnaire.

Prediction model solely based on categorical variables Prediction model solely based on categorical variables is still usable with a few missing inputs. is still usable with a few missing inputs.

It may be safe, informative, and instructive to develop It may be safe, informative, and instructive to develop “continuous models” and “categorical models” together “continuous models” and “categorical models” together and present both and let users decide.and present both and let users decide.

Intuitive cutpoints (optimal cutpoint may or may not Intuitive cutpoints (optimal cutpoint may or may not help)help)

99

3. Variable selection3. Variable selection >5-10 variables can be too many. >5-10 variables can be too many. -- hard to believe but it is true. Everyone is busy and -- hard to believe but it is true. Everyone is busy and

lazy! lazy! -- there are too many risk scores (or medical -- there are too many risk scores (or medical

calculators) in the world.calculators) in the world. Not all significant predictors should be included in Not all significant predictors should be included in

the final model: this is not a sin or cheating!the final model: this is not a sin or cheating! There are difficult variables and easy variables.There are difficult variables and easy variables. Statistical techniques (e.g., backward elimination, Statistical techniques (e.g., backward elimination,

data mining) can guide variable search but data mining) can guide variable search but subjectivity can come into play (e.g., some variables subjectivity can come into play (e.g., some variables such as SES may be intentionally excluded. Race such as SES may be intentionally excluded. Race may be excluded not to limit generalizability).may be excluded not to limit generalizability).

>1 model can be developed to accommodate >1 model can be developed to accommodate different data availabilities, say, with or without different data availabilities, say, with or without CRP.CRP.

Even clinicians don’t agree on variables (e.g., Even clinicians don’t agree on variables (e.g., internist vs. nephrologist, serum vs. urine).internist vs. nephrologist, serum vs. urine).

1010

4. Sample size (N) and data 4. Sample size (N) and data qualityquality Power/N calculation is less relevant (no specific Power/N calculation is less relevant (no specific

test involved) or can be complicated because it is test involved) or can be complicated because it is multivariables-setting. multivariables-setting.

No absolute consensus on N requirement. As the No absolute consensus on N requirement. As the goal is a stable regression equation, more is better. goal is a stable regression equation, more is better.

Creating a prediction equation involves gathering Creating a prediction equation involves gathering relevant data from a “relevant data from a “large, representativelarge, representative” ” sample from the population (if not, less sample from the population (if not, less reproducible risk score!)reproducible risk score!)

We may need to save some N for internal We may need to save some N for internal validation (e.g., split-sample method).validation (e.g., split-sample method).

Therefore, a large database is often used, e.g., Therefore, a large database is often used, e.g., NHANES, NHS, SEER, Framingham, ARIC, etc.NHANES, NHS, SEER, Framingham, ARIC, etc.

““No fancy statistical analysis is better than the No fancy statistical analysis is better than the quality of the data. Garbage in, garbage out, as quality of the data. Garbage in, garbage out, as they say.”they say.” (Robins) (Robins)

1111

5. Population characteristics5. Population characteristics Universal model may not exist.Universal model may not exist. Separate models may be needed:Separate models may be needed:-- by sex -- by sex -- by race or country (e.g., many countries have their -- by race or country (e.g., many countries have their

own diabetes risk score)own diabetes risk score)-- by age-- by age-- high risk (e.g., clinical setting) vs. general population-- high risk (e.g., clinical setting) vs. general population-- first vs. recurrent event. Before vs. after surgery-- first vs. recurrent event. Before vs. after surgery Often, not under investigator’s control (e.g., Often, not under investigator’s control (e.g., ≥ ≥ 65 65

old subjects only in Medicare database, no minority old subjects only in Medicare database, no minority in Framingham study).in Framingham study).

May need to be mentioned in the title of your paper.May need to be mentioned in the title of your paper.

RemarkRemark: More chances for publications!: More chances for publications! Not just a Not just a repeat but each effort can be meaningful. (good repeat but each effort can be meaningful. (good news!) news!)

1212

6. Databases6. DatabasesAdministrative dataAdministrative data Generally HUGEGenerally HUGE No lab data, not No lab data, not

many variablesmany variables Represent the target Represent the target

population wellpopulation well Generally well Generally well

maintained by maintained by reliable organization reliable organization

Data check well Data check well donedone

Clinical or Epi dataClinical or Epi data Small or mid-sizeSmall or mid-size Lab and clinical Lab and clinical

data and many data and many more variables more variables

Reduced Reduced generalizability or generalizability or representivenessrepresentiveness

Data quality not Data quality not always guaranteed.always guaranteed.

1313

Statistical tools for model Statistical tools for model developmentdevelopment

Standard regression: Logistic and CoxStandard regression: Logistic and Cox -- most popular-- most popular -- explicit mathematical formula and numeric scoring system -- explicit mathematical formula and numeric scoring system

can be derived (e.g., by converting regression coefficients)can be derived (e.g., by converting regression coefficients) Advanced regression: Fractional Polynomials regression Advanced regression: Fractional Polynomials regression

(Royston & Altman 1999; Sauerbrei et al. 2006).(Royston & Altman 1999; Sauerbrei et al. 2006). -- combining variable selection with determination of -- combining variable selection with determination of

functional relationships for predictorsfunctional relationships for predictors Tree-based methods: CART (by Breiman), Recursive Tree-based methods: CART (by Breiman), Recursive

Partitioning (by Hawkins & Kass), Optimal Discriminant Partitioning (by Hawkins & Kass), Optimal Discriminant Analysis/Classification Tree Analysis (by Yarnold), Logical Analysis/Classification Tree Analysis (by Yarnold), Logical Analysis of Data (by Hammer), Bayesian CART Analysis of Data (by Hammer), Bayesian CART

-- complex interactions can be revealed.-- complex interactions can be revealed. -- cutpoints identified-- cutpoints identified Neural networkNeural network Data mining techniquesData mining techniques

1414

Statistical measures for model Statistical measures for model evaluation/diagnosticsevaluation/diagnostics

Sensitivity & Specificity — most popularSensitivity & Specificity — most popular Discrimination (ROC/AUC) – most popularDiscrimination (ROC/AUC) – most popular Predictive values, positive or negative (PPV, NPV)Predictive values, positive or negative (PPV, NPV) Likelihood ratio (LR)Likelihood ratio (LR) Accuracy (e.g., Youden index, Brier score)Accuracy (e.g., Youden index, Brier score) Yield, number needed to treat (NNT), number needed Yield, number needed to treat (NNT), number needed

to screen (NNS)to screen (NNS) Model fit (e.g., AIC, BIC)Model fit (e.g., AIC, BIC) Lack of fit (e.g., Hosmer-Lemeshow test)Lack of fit (e.g., Hosmer-Lemeshow test) RR22 (coefficient of determination) (coefficient of determination) P-value (significance of association)P-value (significance of association) Predictiveness curve (based on RPredictiveness curve (based on R22, by Pepe), by Pepe) Calibration/re-calibrationCalibration/re-calibration Decision curve analysis (Vickers 2006)Decision curve analysis (Vickers 2006)RemarkRemark: LR, Youden index and ROC/AUC are functions of : LR, Youden index and ROC/AUC are functions of

sensitivity and specificity.sensitivity and specificity.

1515

Noted limitations of some Noted limitations of some methodsmethods P-valueP-value: significant association is often not enough : significant association is often not enough

for good prediction. for good prediction. -- can have small p-value but poor values for -- can have small p-value but poor values for

everything else (e.g., low p and low Reverything else (e.g., low p and low R22 is a well- is a well-known phenomenon).known phenomenon).

-- some adopt strict p-value thresholds (e.g., -- some adopt strict p-value thresholds (e.g., p<0.001) or multiple testing adjustment.p<0.001) or multiple testing adjustment.

AUCAUC: triply robust. Once it is high, it is extremely : triply robust. Once it is high, it is extremely difficult to increase. Odds ratio alone can be difficult to increase. Odds ratio alone can be problematic (Pepe et al. 2004; Cook 2007)problematic (Pepe et al. 2004; Cook 2007)

RR22:: oftentimes, hard to increase. oftentimes, hard to increase. Sensitivity/SpecificitySensitivity/Specificity: do not address the problems : do not address the problems

of the prevalence of disease in different of the prevalence of disease in different populations, e.g., if prevalence=0.01, sensitivity populations, e.g., if prevalence=0.01, sensitivity =0.95, specificity=0.95 then PPV=0.16.=0.95, specificity=0.95 then PPV=0.16.

Hosmer-Lemeshow testHosmer-Lemeshow test: different software can : different software can produce different test statistics/p-values.produce different test statistics/p-values.

RemarkRemark: For novel markers, relying on one statistical : For novel markers, relying on one statistical measure may not be wise.measure may not be wise.

1616

1717

Good referencesGood references

Ridgeway G. (2003). Strategies and Ridgeway G. (2003). Strategies and Methods for Prediction” Methods for Prediction” In The Handbook of In The Handbook of Data MiningData Mining (N. Ye, ed.). (N. Ye, ed.).

Harrell FE Jr, Lee KL, Mark DB. (1996). Harrell FE Jr, Lee KL, Mark DB. (1996). Multivariable prognostic models: issues in Multivariable prognostic models: issues in developing models, evaluating assumptions developing models, evaluating assumptions and adequacy, and measuring and reducing and adequacy, and measuring and reducing errors. Statistics in Medicine. 15(4): 361-87. errors. Statistics in Medicine. 15(4): 361-87.

1818

Prevalent vs. Incident diseasePrevalent vs. Incident disease Prevalent/concurrent diseasePrevalent/concurrent disease: : --cross-sectional study is needed. --cross-sectional study is needed. --useful for --useful for asymptomaticasymptomatic disease for detecting disease for detecting

undiagnosed cases (e.g., diabetes mellitus undiagnosed cases (e.g., diabetes mellitus (DM), kidney disease), not for all diseases.(DM), kidney disease), not for all diseases.

--simplicity in prediction model/risk score is --simplicity in prediction model/risk score is important.important.

Incident diseaseIncident disease: : --prospective study of event-free cohort is --prospective study of event-free cohort is

needed. needed. --simplicity is less important because prediction --simplicity is less important because prediction

of new cases is not as urgent as diagnosis of of new cases is not as urgent as diagnosis of (hidden) concurrent cases.(hidden) concurrent cases.

1919

Models evolveModels evolve Gail et al (1989, 1999, 2001)’s original and improved Gail et al (1989, 1999, 2001)’s original and improved

prediction models for breast cancer, called Gail et al. prediction models for breast cancer, called Gail et al. model1, model2, etc.model1, model2, etc.

& Barlow et al. and Chen et al. (2006) improved Gail et & Barlow et al. and Chen et al. (2006) improved Gail et al. with novel predictors.al. with novel predictors.

Similarly, Stroke Prognosis Instrument (SPI) I & II, Similarly, Stroke Prognosis Instrument (SPI) I & II, Acute Physiology and Chronic Health Evaluation Acute Physiology and Chronic Health Evaluation

(APACHE) I, II, III.(APACHE) I, II, III. Multiple efforts to simplify and improve the Multiple efforts to simplify and improve the

Framingham risk score.Framingham risk score. Many risk scores exist for incident and prevalent DM.Many risk scores exist for incident and prevalent DM.

RemarkRemark: act on the best available evidence, as : act on the best available evidence, as opposed to waiting for the best possible evidence opposed to waiting for the best possible evidence (Institute of Medicine)(Institute of Medicine)

2020

Other highly cited Other highly cited scores/equationsscores/equations

Not necessarily for specific disease predictionNot necessarily for specific disease prediction Charlson’s comorbidity indexCharlson’s comorbidity index: to give a 10 : to give a 10

year survival estimate for a patient year survival estimate for a patient (Charlson et al. 1987): (Charlson et al. 1987):

MDRD-GFRMDRD-GFR: kidney function measure (Levey : kidney function measure (Levey et al. 2000)et al. 2000)

APACHEAPACHE: a severity of disease classification : a severity of disease classification system at intensive care unit (Knaus et al., system at intensive care unit (Knaus et al., 1981). 1981).

--most were developed by clinicians (not --most were developed by clinicians (not statisticians) and more use for cliniciansstatisticians) and more use for clinicians

2121

Individual vs. population-level Individual vs. population-level riskrisk

Still unsolved issuesStill unsolved issues Started from Rose’s legendary paper, “Sick individuals Started from Rose’s legendary paper, “Sick individuals

and sick population” (1985, republished in 2001).and sick population” (1985, republished in 2001). Population-based model is a poor model in individual Population-based model is a poor model in individual

level.level.-- think about Winston Churchill!-- think about Winston Churchill!-- a large number of people at a small risk may give rise -- a large number of people at a small risk may give rise

to more cases of disease than the small number who to more cases of disease than the small number who are at a high risk.are at a high risk.

-- a preventive measure that brings large benefits to the -- a preventive measure that brings large benefits to the community offers little to each individual.community offers little to each individual.

-- advantages/disadvantages of “high-risk strategy” vs. -- advantages/disadvantages of “high-risk strategy” vs. “population strategy”. Not competing. Both are “population strategy”. Not competing. Both are necessary.necessary.

Individual prediction ever possible? Individual prediction ever possible? -- Genetic studies may be helpful.-- Genetic studies may be helpful.-- LAD answers better but a more complex algorithm is -- LAD answers better but a more complex algorithm is

needed—nothing is free!needed—nothing is free!

2222Copyright restrictions may apply.

Rose, G. Int. J. Epidemiol. 2001 30:427-432; doi:10.1093/ije/30.3.427

Percentage distribution of serum cholesterol levels (mg/dl) in men aged 50-62 who did or did not subsequently develop coronary heart disease (Framingham Study5)

2323Copyright restrictions may apply.

Elmore, J. G. et al. J. Natl. Cancer Inst. 2006 98:1673-1675; doi:10.1093/jnci/djj501

Ability of the Gail et al. breast cancer risk prediction model to discriminate between women who were diagnosed with breast cancer and women who were not diagnosed in the Nurses' Health

Study

2424

What diseases can be What diseases can be predicted?predicted? Breast cancerBreast cancer

--Gail et al. (1989), Rosner and Colditz (1996, 2000), --Gail et al. (1989), Rosner and Colditz (1996, 2000), Tyrer et al. (2004), Barlow et al. (2006) Tyrer et al. (2004), Barlow et al. (2006)

Other cancersOther cancers: cervical (2006), ovarian (2006), : cervical (2006), ovarian (2006), prostate (2005), lung (2007), colorectal (2007) ---all prostate (2005), lung (2007), colorectal (2007) ---all are recent!are recent!

Coronary heart diseaseCoronary heart disease---Framingham, ARIC, SCORE (Europe), Reynolds score---Framingham, ARIC, SCORE (Europe), Reynolds score StrokeStroke---SPI, Stroke-Thrombolytic Predictve Instrument, ARIC---SPI, Stroke-Thrombolytic Predictve Instrument, ARIC DiabetesDiabetes---Herman et al., San Antonio (Stern et al.), ARIC ---Herman et al., San Antonio (Stern et al.), ARIC

(Schmidt et al.)(Schmidt et al.) Kidney diseaseKidney disease---SCORED (Bang et al. 2007)---SCORED (Bang et al. 2007) Numerous other specific diseases/eventsNumerous other specific diseases/events

2525

Not all diseases/events are well Not all diseases/events are well predicted predicted

Some mental disorders (screening is more Some mental disorders (screening is more common than prediction)common than prediction)

HIVHIV How about car accident, divorce, bankruptcy, How about car accident, divorce, bankruptcy,

suicide, lay-off?suicide, lay-off? Cancer again (AUC can be as low as 0.56)Cancer again (AUC can be as low as 0.56) Too many “Don’t Do” (i.e., risk factors) are not Too many “Don’t Do” (i.e., risk factors) are not

more helpful than “Do Nothing”.more helpful than “Do Nothing”. Poor PPVPoor PPV

A good readingA good reading: Begg (2001). The search for cancer : Begg (2001). The search for cancer risk factors: when can we stop looking? AJPH.risk factors: when can we stop looking? AJPH.

2626

Can we predict low risk or Can we predict low risk or health?health?

It is hard for a risk model based on clinical factors It is hard for a risk model based on clinical factors to identify a group at very low risk that does need to identify a group at very low risk that does need to worry.  In other words, a group with very low to worry.  In other words, a group with very low BMI, high exercise levels, good genes, usually is BMI, high exercise levels, good genes, usually is not well captured by screening questions. not well captured by screening questions. 

This is true virtually in all disease prediction This is true virtually in all disease prediction problems. Discrimination at the low end may not problems. Discrimination at the low end may not be good but can be good at the high end of the be good but can be good at the high end of the risk spectrum. risk spectrum.

‘‘We know exactly why certain people commit We know exactly why certain people commit suicide. We don’t know, within the ordinary suicide. We don’t know, within the ordinary concepts of causality, why certain others don't concepts of causality, why certain others don't commit suicide. .............. We know a great deal commit suicide. .............. We know a great deal more about the causes of physical disease than more about the causes of physical disease than we do about the causes of physical health.’we do about the causes of physical health.’’ ("The "The Road Less Travelled" by Peck, 1978) Road Less Travelled" by Peck, 1978)

2727

Sample risk scores

1. Framingham score1. Framingham score

2828

2929

2. Reynolds score2. Reynolds score

3030Copyright restrictions may apply.

Bang, H. et al. Arch Intern Med 2007;167:374-381.

33. SCreening Occult REnal Disease . SCreening Occult REnal Disease (SCORED)(SCORED)

3131

3232

4. Prostate cancer 4. Prostate cancer nomogramnomogram

3333

Issues to consider:Issues to consider:before/during developmentbefore/during development

Predictable and meaningful disease?Predictable and meaningful disease? 11stst model: always thrilled! No need to compare with model: always thrilled! No need to compare with

other models.other models. Best model: also great. But one should show a new Best model: also great. But one should show a new

model improves the existing models/guidelines in model improves the existing models/guidelines in important aspects.important aspects.

External validation using independent dataset External validation using independent dataset within the same publicationwithin the same publication is a great strength is a great strength (editors seem to give brownie points).(editors seem to give brownie points).

-- advanced validation techniques (e.g., cross--- advanced validation techniques (e.g., cross-validation, bootstrap) are not popularly used in validation, bootstrap) are not popularly used in clinical publications. Split-Sample method is widely clinical publications. Split-Sample method is widely used.). However, it still utilizes the sample data used.). However, it still utilizes the sample data (“internal validation”).(“internal validation”).

Ask yourself “Will this model be reproducible?” Ask yourself “Will this model be reproducible?”

3434

After developmentAfter development

VALIDATE or Perish!VALIDATE or Perish! Validations will be done by you and others Validations will be done by you and others

(be careful and prepared to write Response.)(be careful and prepared to write Response.) People will compare your model and others’ People will compare your model and others’

once they are developed (not always in a fair once they are developed (not always in a fair manner. Usually, you don’t have a chance to manner. Usually, you don’t have a chance to review and give comments on others’ review and give comments on others’ publications that criticize your method)publications that criticize your method)

Do not publish “model not to be replicated” Do not publish “model not to be replicated” or “type-I-error” and run!or “type-I-error” and run!

Everybody loves external validation. Esp. Everybody loves external validation. Esp. EditorsEditors

3535

After publication, how to After publication, how to disseminate?disseminate? Power of marketing: good method deserves Power of marketing: good method deserves

good marketing.good marketing. Computer-system (e.g., web-based or Computer-system (e.g., web-based or

handheld) vs. paper-pencil method.handheld) vs. paper-pencil method. Work with Public Affair department in your Work with Public Affair department in your

institute or contact Media directly.institute or contact Media directly.-- authors may need to write Press Release -- authors may need to write Press Release

(PR).(PR).-- no one reads/understands your paper as -- no one reads/understands your paper as

well as you do. Highlight the main findings well as you do. Highlight the main findings clearly. clearly.

-- no p-value in PR!-- no p-value in PR!-- ready for interviews (esp., for 1-- ready for interviews (esp., for 1stst study) study) Work with authority and practitioners to Work with authority and practitioners to

implement/distribute your method implement/distribute your method (preferably after Validation).(preferably after Validation).

3636

Statistician’s role in risk Statistician’s role in risk prediction/scoreprediction/score

Statistician’s involvement is absolutely necessary.Statistician’s involvement is absolutely necessary. Statisticians (or epidemiologists) can be a leader/first Statisticians (or epidemiologists) can be a leader/first

author of clinical research/publication.author of clinical research/publication. Statisticians who develop risk score should be highly Statisticians who develop risk score should be highly

familiar with the current literature of the relevant familiar with the current literature of the relevant disease.disease.

Your title beyond “statistician” or “faculty of Your title beyond “statistician” or “faculty of (bio)statistics” may be helpful for PR and interview (bio)statistics” may be helpful for PR and interview purposes (sadly true! some reporters search for MD purposes (sadly true! some reporters search for MD authors).authors).

At times, clinical communities call for the development At times, clinical communities call for the development of a (new or improved) risk score. For example, of a (new or improved) risk score. For example,

-- Multiple editorials in 2006 called for renal score. -- Multiple editorials in 2006 called for renal score. -- Aitkins (1994) wrote “It would be a shame if -- Aitkins (1994) wrote “It would be a shame if

Spiegelman et al. were to stop short of presenting a Spiegelman et al. were to stop short of presenting a new model based on their substantially more powerful new model based on their substantially more powerful tool, the cohort study."tool, the cohort study."

-- Beyond Framingham.-- Beyond Framingham.

3737

Screen or not screen?Screen or not screen? Not all prediction models/pre-screening/screening are Not all prediction models/pre-screening/screening are

beneficial.beneficial.e.g., ADA recommended “Do screen” for DM (in one year) and “Do e.g., ADA recommended “Do screen” for DM (in one year) and “Do

not screen” (in the following year), in the same journal.not screen” (in the following year), in the same journal.e.g., a lot of controversies in breast cancer screeninge.g., a lot of controversies in breast cancer screening Freedman DA, Petitti DB, and Robins JM. (2004). On the efficacy of screening for

breast cancer. International Journal of Epidemiology, 33:43-55. Comment by Gotzsche, P.C. On the benefits and harms of screening for breast cancer,pp. 56-64. Comment by Miller, A.B. Commentary: A defence of the Health Insurance Plan (HIP) study and the Canadian National Breast Screening Study (CNBSS) pp. 64-65. Comment by Baum, M. Commentary: False premises, false promises and false positives - the case against mammographic screening for breast cancer. pp. 66-67. Comment by Berry, D. Commentary: Screening mammography: a decision analysis. pp. 68. Rejoinder by Freedman, D.A.,Petitti, D.B., and Robins, J.M.Rejoinder. pp. 69-73.

WHO announced 10 principles for national screening programs WHO announced 10 principles for national screening programs (1968).(1968).

False negatives can cause a serious problem. False negatives can cause a serious problem. False positives can create too much anxiety and scare people.False positives can create too much anxiety and scare people. Problem of high vs. low risk. What does “low” means?Problem of high vs. low risk. What does “low” means? Effectiveness of screening should ultimately be tested in RCTs.Effectiveness of screening should ultimately be tested in RCTs. Cost-effectiveness should also be evaluated.Cost-effectiveness should also be evaluated.

3838

US Preventive Services Task US Preventive Services Task ForceForce

The entity with the most rigorous evidence-based The entity with the most rigorous evidence-based approachapproach

An independent panel of experts in primary care An independent panel of experts in primary care and prevention that systematically reviews the and prevention that systematically reviews the evidence of effectiveness and develops evidence of effectiveness and develops recommendations for clinical preventive services.recommendations for clinical preventive services.

http://www.ahrq.gov/clinic/uspstf/uspstopics.htmhttp://www.ahrq.gov/clinic/uspstf/uspstopics.htm Many specific disease authorities (e.g., CDC, ADA, Many specific disease authorities (e.g., CDC, ADA,

NKF) have their own screening recommendations. NKF) have their own screening recommendations. Often considerably different.Often considerably different.

Some agencies (e.g., NCI, NHLBI) hold a workshop Some agencies (e.g., NCI, NHLBI) hold a workshop to review various risk models.to review various risk models.

A recent refA recent ref: Campos-Outcalt (2007). Screening: : Campos-Outcalt (2007). Screening: New guidance on what and what not to do. J of New guidance on what and what not to do. J of Family Practice.Family Practice.

3939

Risk score primerRisk score primer Simplicity, user-friendliness and accuracy are key Simplicity, user-friendliness and accuracy are key

issues in success. issues in success. Can have real impacts on people’s lives (esp., for Can have real impacts on people’s lives (esp., for

underserved)underserved) Useful for educating people about risk factors and Useful for educating people about risk factors and

increasing low-awareness of some diseases.increasing low-awareness of some diseases. Great collaboration area for clinicians and Great collaboration area for clinicians and

statisticians.statisticians. Name can be important (e.g., ABCD, APACHE, Name can be important (e.g., ABCD, APACHE,

SCORED, Framingham, Reynolds, Gail et al., Take SCORED, Framingham, Reynolds, Gail et al., Take the test and know your score, Indian diabetes the test and know your score, Indian diabetes score), Googlable?score), Googlable?

Nothing causal, all about association or correlation!Nothing causal, all about association or correlation!-- If causes can be removed, susceptibility ceases to -- If causes can be removed, susceptibility ceases to

matter (Rose 1985)matter (Rose 1985)

4040

Sample risk scores on Sample risk scores on internetinternet

Cancer: Cancer: http://riskfactor.cancer.gov/cancer_risk_prediction/http://riskfactor.cancer.gov/cancer_risk_prediction/ http://www.mskcc.org/mskcc/html/5794.cfmhttp://www.mskcc.org/mskcc/html/5794.cfm http://www4.utsouthwestern.edu/breasthealth/cagene/http://www4.utsouthwestern.edu/breasthealth/cagene/ APACHE: http://www.sfar.org/scores2/apache22.htmlAPACHE: http://www.sfar.org/scores2/apache22.html http://www.apache-web.com/public/pub_main.htmlhttp://www.apache-web.com/public/pub_main.html Charlson comorbity index: Charlson comorbity index:

http://www.medalreg.com/qhc/medal/ch1/1_13/01-13-01-ver9.php3http://www.medalreg.com/qhc/medal/ch1/1_13/01-13-01-ver9.php3 Framingham score: Framingham score:

http://hp2010.nhlbihin.net/atpiii/calculator.asp?usertype=profhttp://hp2010.nhlbihin.net/atpiii/calculator.asp?usertype=prof & & http://http://www.nhlbi.nih.gov/about/framingham/riskabs.htmwww.nhlbi.nih.gov/about/framingham/riskabs.htm UK CVD score: UK CVD score: http://www.riskscore.org.uk/http://www.riskscore.org.uk/ PROCAM score: PROCAM score: http://www.chd-taskforce.de/http://www.chd-taskforce.de/ Reynolds score: Reynolds score: http://http://www.reynoldsriskscore.orgwww.reynoldsriskscore.org// Herman et al.’s diabetes risk score: http://www.diabetes.org/risk-test.jspHerman et al.’s diabetes risk score: http://www.diabetes.org/risk-test.jsp German diabetes risk score: http://www.dife.de/German diabetes risk score: http://www.dife.de/ Angina score: http://www.anginarisk.org/Angina score: http://www.anginarisk.org/ Pneumonia score: http://www.ahrq.gov/clinic/pneuclin.htm#head1Pneumonia score: http://www.ahrq.gov/clinic/pneuclin.htm#head1 SCORED: http://kidneydiseases.about.com/od/diagnostictests/a/scored.htmSCORED: http://kidneydiseases.about.com/od/diagnostictests/a/scored.htm Depression: http://www.psycom.net/depression.central.screening.htmlDepression: http://www.psycom.net/depression.central.screening.html Medical calculator: http://medcalc3000.com/ (some are commercial) Medical calculator: http://medcalc3000.com/ (some are commercial)

In general, google can find these.In general, google can find these.