When Cherished Beliefs Clash When Cherished Beliefs Clash with Evidence, Don’t Expect to with Evidence, Don’t Expect to Be Thanked for What Science Be Thanked for What Science

Does BestDoes Best

James C. Coyne, Ph.D. University of Pennsylvania School of Medicine and

University Medical Center, Groningen jcoyne@mail.med.upenn.edu

Unpicking dodgy claims, unpacking the evidence behind dodgy claims, isn’t a kind of nasty carping activity; it socially useful, but it’s also extremely valuable explanatory tool. Because real science is all about critically appraising the evidence for someone else’s position.

--Ben Goldacre

MuckrakersMuckrakers

 Investigative journalists who reported on corruption, harmful social conditions, pollution, food and product safety standards, sexual harassment, unfair labor practices, and fraud.  

Came into prominence and named as such in Progressive Era of American Politics (1890-1920).

The top journals are filled with noble lies and fairy tales, promoting myths and claiming great success in addressing important patient needs and improving health outcomes, often where

there is none.

Articles appearing in top journals are not reliable guides to the best evidence.

James C. Coyne 1,2 , Brett Thombs3 ,Mariët Hagedoorn2

1University of Pennsylvania, School of Medicine, USA 2University Medical Center Groningen, the Netherlands3McGill University and Jewish General Hospital, Montreal, Canada

Ain’t Necessarily So: Review and Ain’t Necessarily So: Review and Critique of Recent Meta-Analyses Critique of Recent Meta-Analyses

of Behavioral Medicine of Behavioral Medicine Interventions in Interventions in Health PsychologyHealth Psychology

Storyline in Cancer CareStoryline in Cancer Care

Much of what we know about psychosocial aspects of cancer is mythical, even if the myths are widespread, contrary to fact, taken for granted as background assumptions, and resistant to evidence.

Myth or Storyline in Myth or Storyline in Cancer CareCancer Care

• Stress, personality, and emotion play substantial roles in incidence, progression, and outcome of cancer and psychosocial interventions can improve survival by strengthening the immune system.

• Storyline is highly consistent with entrenched cultural beliefs and fables about the triumph of human will over adversity and mind over body and its frailities.

Operation of Storylines in Scientific Discourse

Storlines often have the quality of a promissory note, giving favorable data more credence than is yet justified.

Storylines tend to exclude, minimize, or incorporate evidence with a confirmatory distortion.

Yet storylines are the basis by which evidence enters into clinical and public policy decisions and media depictions of issues.

Myths carry sentimentality, important emotional baggage that must be dealt with in any effort to counter the myth.

Sentimentality of this cancer storyline countered by articulation of concern that it hurts patients, not only by squandering resources, but by blaming patients for their morbidity and ultimate mortality.

We must not allow a shared commitment to improving the wellbeing of cancer patients to be

exploited with exaggerated claims and poorly conceived, poorly conducted, and poorly

reported clinical trials.

We Thought We Were We Thought We Were Done with Claims That Done with Claims That

Psychotherapy Promotes Psychotherapy Promotes the Survival of Cancer the Survival of Cancer

Patients….Patients….

No trial has ever found that psychotherapy improved the median survival time of women with metastatic breast cancer.

No trial in which survival was chosen as the outcome of interest ahead of time has demonstrated a survival effect for patients with any type of cancer, when psychotherapy was not confounded with improved medical surveillance or treatment.

  Coyne JC, Stefanek M, Palmer SC. Coyne JC, Stefanek M, Palmer SC. Psychotherapy and survival in cancer: the Psychotherapy and survival in cancer: the

conflict between hope and evidence. conflict between hope and evidence. Psychol Bull. Psychol Bull. 2007;133:367-394.2007;133:367-394.

Now classic 1989 Lancet Now classic 1989 Lancet study of Spiegel and study of Spiegel and

colleagues: Was there colleagues: Was there ever an effect there?ever an effect there?

No one can replicate odd survival curve of control group in intervention or observational studies, suggesting something went wrong.

Curve of Spiegel’s intervention group approximated by both intervention and control groups in subsequent studies, suggesting it was inert.

“The results suggest that we can help breast cancer patients make positive steps that may help them live longer and make recurrence less likely. We already knew a psychological intervention program could help breast cancer patients to handle their stress, function more effectively, and improve their health. Now we know it does even more.”

Andersen, B. L., H. C. Yang, et al. (2008). Andersen, B. L., H. C. Yang, et al. (2008). "Psychologic Intervention Improves "Psychologic Intervention Improves

Survival for Breast Cancer Patients A Survival for Breast Cancer Patients A Randomized Clinical Trial." Randomized Clinical Trial." CancerCancer

113(12): 3450-3458.113(12): 3450-3458.

•No survival effect found in simple analyses, claims depend on inappropriate multivariate analyses.

•No differences between intervention and control groups in recurrence or survival.

•Psychosocial intervention consisting of a mixture of relaxation training, problem solving and health behavior promotion.

•Null and weak results across 8 measures of mood (No effects on mood), 15 measures of immune function, and 4 measures of adherence.

A Closer look at Andersen, et al.A Closer look at Andersen, et al.(2008). (2008). CancerCancer 113(12): 3450-3458. 113(12): 3450-3458.

Psychosocial Psychosocial Intervention, Immune Intervention, Immune

Function, and Function, and Progression of Cancer: Progression of Cancer:

Unproven MedicineUnproven Medicine

Cameron LD, et al. Cognitive and Cameron LD, et al. Cognitive and affective determinants of decisions to affective determinants of decisions to attend a group psychosocial support attend a group psychosocial support

program for women with breast cancer. program for women with breast cancer. Psychosomatic MedicinePsychosomatic Medicine 2005;67:584-589. 2005;67:584-589.

Many breast cancer patients are attending group therapy with the belief that they are fighting their cancer by strengthening their immune system.

Weak or No Effects, But Positive Results Emphasized, Strong Confirmatory Bias in Reporting New Results and Recounting of Past Studies.

Doubtful Clinical Significance Even If Results Were Obtained.

Confused, Simplistic View of Role of Immune System in Cancer Progression.

Claims That Psychosocial Claims That Psychosocial Intervention Strengthens the Intervention Strengthens the

Immune SystemImmune System

Each of the measures used in this literature represents only a small facet of a complex, highly redundant system. It would therefore be inappropriate to conclude that intervention-related changes in any specific immune parameter signal a state of "immune enhancement" or altered susceptibility to immune-mediated disease. The normal functioning range for most immune measures is very broad, and psychological interventions typically do not induce changes of sufficient magnitude to move people outside of these boundaries (p. 48).

Miller GE, Cohen S. Psychological Miller GE, Cohen S. Psychological interventions and the immune system: A interventions and the immune system: A meta-analytic review and critique. meta-analytic review and critique. Health Health

Psychology. Psychology. Jan 2001;20(1):47-63.Jan 2001;20(1):47-63.

How Does Sticking to a How Does Sticking to a Storyline Get Enforced?Storyline Get Enforced?No Conspiracy Theory No Conspiracy Theory

Needed.Needed.

“Confirmatory Bias Has Existed Even in Historical Periods When There Were No

Incentives, in Terms of Fame or Fortune.”

-Robert K. Merton

Copyright ©2005 American Heart Association

Loscalzo, J. Circulation 2005;112:3026-3029

Incentives and Pressures to Fit Incentives and Pressures to Fit Research to a StorylineResearch to a Storyline

‘‘By the time many young people figure out the system, they are so much a part of it, so obsessed with keeping their grants, that their imagination and instincts have been so muted (or corrupted) that their best work is already behind them. This is made much worse by the US system in which assistant professors in medical schools will soon have to raise their own salaries. Who would dare to pursue risky ideas under these circumstances? Who could dare change their research field, ever?”

-Ted Cox, Director of the Program on Biophysics,Princeton University

Do Americans Do It Do Americans Do It Bigger and Better?Bigger and Better?

Scandinavian Data SetsScandinavian Data Sets

Democratic socialist countries with integrated health system and uniform medical and death records

Despite concern with social inequalities and disparities, Scandinavia lacks the gross differences found in the US due to race/ethnic status, income, and insurance status.

Nakaya et al (2010)Nakaya et al (2010) The [Finnish] cohort consists of all …same-sex twin pairs born

before 1958, of which both members were alive in 1975.

The Swedish twin cohort was identified from the population-based Swedish Twin Registry, the largest of its kind in the world, which has information on more than 140,000 twins.

Information on cancer diagnoses was obtained by record linkage to the national cancer registries …using the unique identification numbers assigned to everyone residing in those countries.

Data on emigration and death were obtained in Finland by record linkage to the Population Register Centre and in Sweden by record linkage to the National Population Register.

The Danish StudiesThe Danish Studies

Ross et al (2009): Between September 1996 and May 1999, we sought to include all patients aged 18 or over who were being treated for a primary colorectal cancer by abdominal surgery at the surgical departments of eight hospitals in the eastern part of Denmark.

Boesen, et al (2007): In this population-based, randomized trial conducted in a society with free hospital treatment, psychoeducation for patients treated for cutaneous malignant melanoma did not influence time to recurrence or survival.

Why Don’t the Danes Get the Why Don’t the Danes Get the Attention They Deserve for What Attention They Deserve for What is the Best-of-Its-Kind Research?is the Best-of-Its-Kind Research?

Danish Psycho-Oncology Research Danish Psycho-Oncology Research Lacks Hype, Confirmatory Bias Lacks Hype, Confirmatory Bias

of American Researchof American Research

Telling It Like It Ain’t: How to Telling It Like It Ain’t: How to Succeed in Psycho-OncologySucceed in Psycho-Oncology

Have Lots of Endpoints and Ignore Negative Results in Main Analyses of Primary Endpoints.

Favor Secondary Analyses, Subgroup Analyses, and Endpoints Developed Post Hoc Over Negative Findings for Primary Analyses.

Ignore Methodological Shortcomings That Would Make Trial or Meta Analyses Invalid.

Telling It Like It Ain’t: How to Telling It Like It Ain’t: How to Succeed in Psycho-OncologySucceed in Psycho-Oncology

• Present Negative Findings as if Positive in Subsequent Publications and Exaggerate Findings That Are Positive

Assess Multiple Endpoints and Treat Any Significant Finding as if it were a Replication of Past Findings

Create a False Consensus and Seeming Unanimity in the Literature by Cherrypicking Findings That Can be Construed as Positive and Ignoring the Rest

A credible scientific journal should publish all studies with ‘‘null’’ results provided they acknowledge their limitations. Conversely, such a journal should be cautious about publishing ‘‘positive’’ results, most of which are false. Independent replication is important and should be done by different teams, preferably by competitors. ‘‘Null’’ results should be published promptly in print in short versions, with more extensive details in web-based files.

Ioannidis, J. P. A. (2006). Journals Ioannidis, J. P. A. (2006). Journals should publish all "null" results and should publish all "null" results and should sparingly publish "positive" should sparingly publish "positive"

results. results. Cancer Epidemiology Cancer Epidemiology Biomarkers & PreventionBiomarkers & Prevention 15(1): 185- 15(1): 185-

186.186.

Ioannidis, J. P. A. (2006). Journals Ioannidis, J. P. A. (2006). Journals should publish all "null" results and should publish all "null" results and should sparingly publish "positive" should sparingly publish "positive"

results.results.

“Positive” results should be published equally promptly, but only on the web, pending independent replication; once refuted, the original article and the refutation could be printed as a single nice null report; the rare validated findings should appear in print with full details.

The GoalThe Goal

To Create a Climate in Which Honest Reporting Of Well Conducted, But Negative Trials is More Valued than Hyping and Hiding of Results.

To Value the Weight of Evidence, Whatever That May Be, Over Noble Lies and Fairy Tales That Make Health Psychology Look Good.

Taking Heat…Taking Heat…

Resistance to criticism of what is Resistance to criticism of what is published.published.

“[The authors’] frustration with the work of others is not enough to appoint themselves as the Supreme Judges of the work of others - however flawed this work might be.” [Anonymous reviewer, Tell It Like It Ain’t..article].

It is with a heavy heart that I have concluded that Dr. Coyne suffers from an incurable illness: narcissistic myopia. He is a depression researcher, so successful treatment of mere anxiety symptoms appears meaningless to his limited vision. It … seems to permit him to dismiss the results of 25 years of my and many other colleagues' research demonstrating positive effects of group support for cancer patients. I can live with his distortion of the published data…but when he insults my patients by informing us that our supportive/expressive groups are a "waste of seriously ill patients' potentially short remaining lives," my tolerance for his obvious impairment vanishes. One of my patients who attended our supportive/expressive group for six years said: "This group is the least superficial thing I do in my life." She and her family thanked me for it when I visited her home shortly before she died.

-David Spiegel, M.D., Stanford University (Listserv post)

Response to David Spiegel:Response to David Spiegel:

Well, David, seeing that no one, not even you, has been able to replicate the findings for survival reported in your 1989 supportive-expressive therapy article in Lancet, might you now consider the possibility that the 1989 findings were spurious? What further data would it take for you to consider that possibility?

Barbara Andersen on CoyneBarbara Andersen on Coyne

“Dr. Coyne’s writing strategy is, in the first portion of a sentence, to mischaracterize an aspect of a study and then, in the second part, assert why the aspect is methodologically flawed.

“Dr. Coyne uses wording such as “the authors’ claim” or sarcasm that are [sic] not appropriate to scientific discourse.”

Response to Barbara Andersen:Response to Barbara Andersen:

Well, Barbara, even if you wouldn’t respond to our peer-reviewed article suggesting that you did not get an effect on survival, might you consider publishing standard, unadjusted outcomes, such as a Kaplan-Meier estimate of the survival function so that readers can decide for themselves? After all, strong claims require strong data.

When an When an investigator, well investigator, well armed with a armed with a hypothesis hypothesis consistent with consistent with entrenched cultural entrenched cultural beliefs...beliefs...

encounters a gang of unruly data...encounters a gang of unruly data...

only the investigator walks away, only the investigator walks away, with his hypothesis unscathed.with his hypothesis unscathed.