SPP  D.  Mayo   1  

Replication Research Under an Error Statistical Philosophy

Deborah Mayo

Around a year ago on my blog: “There are some ironic twists in the way psychology is dealing with its replication crisis that may well threaten even the most sincere efforts to put the field on firmer scientific footing”

Philosopher’s talk: I see a rich source of problems that cry out for ministrations of philosophers of science and of statistics

SPP  D.  Mayo   2  

Three main philosophical tasks: #1 Clarify concepts and presuppositions #2 Reveal inconsistencies, puzzles, tensions (“ironies”) #3 Solve problems, improve on methodology

• Philosophers usually stop with the first two, but I think

going on to solve problems is important. This presentation is ‘programmatic’- what might replication research under an error statistical philosophy be? My interest grew thanks to Caitlin Parker whose MA thesis was on the topic

SPP  D.  Mayo   3  

Example of a conceptual clarification (#1)

Editors of a journal, Basic and Applied Social Psychology, announced they are banning statistical hypothesis testing because it is “invalid” It’s invalid because it does not supply “the probability of the null hypothesis, given the finding” (the posterior probability of H0) (2015 Trafimow and Marks) • Since the methodology of testing explicitly rejects the mode

of inference they don’t supply, it would be incorrect to claim the methods were invalid.

• Simple conceptual job that philosophers are good at

SPP  D.  Mayo   4  

Example of revealing inconsistencies and tensions (#2) Critic: It’s too easy to satisfy standard significance thresholds You: Why do replicationists find it so hard to achieve significance thresholds? Critic: Obviously the initial studies were guilty of p-hacking, cherry-picking, significance seeking, QRPs You: So, the replication researchers want methods that pick up on and block these biasing selection effects. Critic: Actually the “reforms” recommend methods where selection effects and data dredging make no difference

SPP  D.  Mayo   5  

Whether this can be resolved or not is separate. • We are constantly hearing of how the “reward structure”

leads to taking advantage of researcher flexibility • As philosophers, we can at least show how to hold their

feet to the fire, and warn of the perils of accounts that bury the finagling

The philosopher is the curmudgeon (takes chutzpah!) I’ll give examples of #1 clarifying terms #2 inconsistencies #3 proposed solutions (though I won’t always number them) .

SPP  D.  Mayo   6  

Demarcation: Bad Methodology/Bad Statistics

• A lot of the recent attention grew out of the case of Diederik

Stapel, the social psychologist who fabricated his data. • Kahneman  in  2012  “I  see  a  train-­‐wreck  looming,”  setting  up  a  “daisy  chain”  of  replication.  

• The Stapel investigators: 2012 Tilberg Report, “Flawed Science” do a good job of characterizing pseudoscience.

• Philosophers tend to have cold feet when it comes to saying anything general about science versus pseudoscience.

SPP  D.  Mayo   7  

Items in their list of “dirty laundry” include: “An experiment fails to yield the expected statistically significant results. The experimenters try and try again until they find something (multiple testing, multiple modeling, post-data search of endpoint or subgroups), and the only experiment subsequently reported is the one that did yield the expected results.”

… continuing an experiment until it works as desired, or excluding unwelcome experimental subjects or results, inevitably tends to confirm the researcher’s research hypotheses, and essentially render the hypotheses immune to the facts”. (Report, 48)

--they walked into a “culture of verification bias”  

SPP  D.  Mayo   8  

Bad Statistics Severity Requirement: If data x0 agree with a hypothesis H, but the test procedure had little or no capability, i.e., little or no probability of finding flaws with H (even if H is incorrect), then x0 provide poor evidence for H.

Such a test we would say fails a minimal requirement for a stringent or severe test.

• This seems utterly uncontroversial.

SPP  D.  Mayo   9  

• Methods that scrutinize a test’s capabilities, according to their severity, I call error statistical.

• Existing error probabilities (confidence levels, significance

levels) may but need not provide severity assessments.

• New name: frequentist, sampling theory, Fisherian, Neyman-Pearsonian—are too associated with hard line views and personality conflicts (“It’s the methods, stupid”)

(example of new solutions #3)

SPP  D.  Mayo   10  

Are philosophies about science relevant? One of the final recommendations in the Report is this:

In the training program for PhD students, the relevant basic principles of philosophy of science, methodology, ethics and statistics that enable the responsible practice of science must be covered. (p. 57)  

SPP  D.  Mayo   11  

A critic might protest: “There’s nothing philosophical about my criticism of significance tests: a small p-value is invariably, and erroneously, interpreted as giving a small probability to the null hypothesis that the observed difference is mere chance.” Really? P-values are not intended to be used this way; presupposing they should be stems from a conception of the role of probability in statistical inference—this conception is philosophical. (of course criticizing them because they might be misinterpreted is just silly)

SPP  D.  Mayo   12  

Two  main  views  of  the  role  of  probability  in  inference  Probabilism.  To  provide  a  post-­‐data  assignment  of  degree  of  probability,  confirmation,  support  or  belief  in  a  hypothesis,  absolute  or  comparative,  given  data  x0.      Performance.  To  ensure  long-­‐run  reliability  of  methods,  coverage  probabilities,  control  the relative frequency of erroneous inferences in a long-run series of trials.    What happened to the goal of scrutinizing bad science by the severity criterion?

SPP  D.  Mayo   13  

• Neither “probabilism” nor “performance” directly captures

it. • Good long-run performance is a necessary not a sufficient

condition for avoiding insevere tests.  

• The problems with selective reporting, multiple testing, stopping when the data look good are not problems about long-runs—

• It’s that we cannot say about the case at hand that it has

done a good job of avoiding the sources of misinterpretation.

 

SPP  D.  Mayo   14  

• Probabilism  says  H  is  not  justified  unless  it’s  true  or  probable  (made  firmer).  

• Error  statistics  (probativism)  says  H  is  not  justified  unless  something  (a  good  job)  has  been  done  to  probe  ways  we  can  be  wrong  about  H.  

• If  it’s  assumed  probabilism  is  required  for  inference,  error  probabilities  could  be  relevant  only  by  misinterpretation.  False!  

• Error  probabilities  have  a  crucial  role  in  appraising  well-­‐testedness  (new  philosophy  for  probability  #3)    

• Both  H  and  not-­‐H  be  can  be  poorly  tested,  so  a  severe  testing  assessment  violates  probability  

SPP  D.  Mayo   15  

Understanding  the  Replication  Crisis  Requires  Understanding  How  it  Intermingles  with  PhilStat  Controversies      

• It’s not that I’m keen to defend many common uses of significance tests

• It’s just that the criticisms (in psychology and elsewhere)

are based on serious misunderstandings of the nature and role of these methods; consequently so are many “reforms”

• How can you be clear the reforms are better if you might be

mistaken about existing methods?

SPP  D.  Mayo   16  

Criticisms  concern  a  kind  of  Fisherian  Significance  Test (i) Sample  space:  Let  the  sample  be  X  =  (X1,  …,Xn),  be  n  iid  (independent  and  identically  distributed)  outcomes  from  a  Normal  distribution  with  standard  deviation    σ      (ii)  A  null  hypothesis  H0:  µ  =    0    (Δ: µΤ − µC = 0)

   (iii)  Test  statistic:  A  function  of  the  sample,  d(X)  reflecting  the  difference  between  the  data  x0  =  (x1,  …,xn),  and  H0:    

The  larger  d(x0)  the  further  the  outcome  from  what’s  expected  under  H0,  with  respect  to  the  particular  question.    

   (iv)  Sampling  distribution  of  test  statistic:  d(X)  

SPP  D.  Mayo   17  

The  p-­‐value  is  the  probability  of  a  difference  larger  than  d(x0),  under  the  assumption  that  H0  is  true:  

p(x0)=Pr(d(X)  >  d(x0);  H0).    If p(x0)  is  sufficiently  small,  there’s  an  indication  of  discrepancy  from  the  null.    (Even  Fisher  had  implicit  alternatives,  by  the  way)

SPP  D.  Mayo   18  

P-­‐value  reasoning:  from  high  capacity  to  curb  enthusiasm  

 

If  the  hypothesis  H0  is  correct  then,  with  high  probability,  1-­‐p,  the  data  would  not  be  statistically  significant  at  level  p.  

x0  is  statistically  significant  at  level  p.  

____________________________  

Thus,  x0  indicates  a  discrepancy  from  H0.  

 That  merely  indicates  some  discrepancy!  

SPP  D.  Mayo   19  

A genuine experimental effect is needed “[W]e need, not an isolated record, but a reliable method of procedure. In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result.” (Fisher 1935, 14)

(low P-value ≠> H: statistical effect) “[A]ccording  to  Fisher,  rejecting  the  null  hypothesis  is  not  equivalent  to  accepting  the  efficacy  of  the  cause  in  question.  The  latter...requires  obtaining  more  significant  results  when  the  experiment,  or  an  improvement  of  it,  is  repeated  at  other  laboratories  or  under  other  conditions.”  (Gigerentzer  1989,  95-­‐6)  (H ≠> H*)

SPP  D.  Mayo   20  

 Still,  simple  Fisherian  Tests  have  Important  Uses      

• Testing  assumptions  • Fraudbusting  and  forensics:  Finding  Data  too  good  to  be  true  (Simonsohn)  

• Finding  if  data  are  consistent  with  a  model   Gelman and Shalizi (meeting of minds between a Bayesian and an error statistician) “What we are advocating, then, is what Cox and Hinkley (1974) call ‘pure significance testing’, in which certain of the model’s implications are compared directly to the data, rather than entering into a contest with some alternative model.” (p.20)  

SPP  D.  Mayo   21  

Fallacy  of  Rejection:  H  –  >  H*  :  Erroneously  take  statistical  significance  as  evidence  of  research  hypothesis  H*      The  fallacy  is  explicated  by  severity:  flaws  in  alternative  H*  have  not  been  probed  by  the  test,  the  inference  from  a  statistically  significant  result  to  H*  fails  to  pass  with  severity    Merely refuting the null hypothesis is too weak to corroborate substantive H*, “we have to have ‘Popperian risk’, ‘severe test’ [as in Mayo], or what philosopher Wesley Salmon called ‘a highly improbable coincidence.’” (Meehl and Waller 2002, 184)  

(Meehl  was  wrong  to  blame  Fisher)  

SPP  D.  Mayo   22  

NHST  are  pseudostatistical:    Why  do  psychologists  speak  of  NHSTs  –tests  that  supposedly  allow  moving  from  statistical  to  substantive?    So  defined,  they  exist  only  as  abuses  of  tests:  they  exist as something you’re never supposed to do      Psychologists  tend  to  ignore  Neyman-­‐Pearson  (N-­‐P)  tests:  N-­‐P  supplemented  Fisher’s  tests  with  explicit  alternatives        

SPP  D.  Mayo   23  

Neyman-­‐Pearson  (N-­‐P)  Tests:  A  null  and  alternative  hypotheses  H0,  H1  that  exhaust  the  parameter  space    So  the  fallacy  of  rejection  H  –  >  H*  is  impossible  (rejecting  the  null  only  indicates  statistical  alternatives)    Scotches  criticisms  that  P-­‐values  are  only  under  the  null    Example:  Test  T+:    sampling  distribution  of  d(x)  under  null  and  alternatives.  H0:  µ  ≤  µ0    vs.      H1:  µ  >  µ0        if  d(x0)  >    cα,  "reject"  H0,      if  d(x0)  <    cα,  "do  not  reject”  or  “accept"  H0,    e.g.  cα=1.96  for  α=.025  

SPP  D.  Mayo   24  

   The  sampling  distribution  yields  Error  Probabilities    Probability  of  a  Type  I  error  =  P(d(X)  >    cα;  H0)  ≤    α.  Probability  of  a  Type  II  error:  =  P(d(X)  <  cα;  H0)  =  ß(µ1),  for  any  µ1  >  µ0.  

The  complement  of  the  Type  II  error  probability=  power  against  (µ1)  

POW(µ1)=  P(d(X)  >  cα;  µ1)  

Even  without  “best”  tests,  there  are  “good”  tests      

SPP  D.  Mayo   25  

N-­‐P  test  in  terms  of  the  P-­‐value:  reject  H0  iff  P-­‐value  <  .025    

• Even  N-­‐P  report  the  attained  significance  level  or  P-­‐value  (Lehmann)  

 • “reject/do  not  reject”  uninterpreted  parts  of  the  mathematical  apparatus  

 Reject  could  be:  “Declare  statistically  significant  at  the  p-­‐level”    

• “The  tests…  must  be  used  with  discretion  and  understanding”  (N-­‐P,  1928,  p.  58)  

(“it’s  the  methods,  stupid”)        

SPP  D.  Mayo   26  

Why  Inductive  behavior?  

N-­‐P  justify  tests  (and  confidence  intervals)  by  performance,  control  of  long-­‐run  error  coverage  probabilities    

They  called  this  inductive  behavior,  why?  

• They  were  reaching  conclusions  beyond  the  data  (inductive)  

• If  inductive  inference  is  probabilist,  then  they  needed  a  new  term.  

In  Popperian  spirit,  they  (mostly  Neyman)  called  it  inductive  behavior-­‐-­‐  adjust  how  we’d  act  rather  than  beliefs  

(I’m  not  knocking  performance,  but  error  probabilities  also  serve  for  particular  inferences—evidential)  

SPP  D.  Mayo   27  

N-­‐P  tests  can  still  commit  a  type  of  fallacy  of  rejection:  Infer  a  discrepancy  beyond  what’s  warranted:    ––especially  with n sufficiently large:  large  n  problem.  

• Severity  tells  us:  an  α-­‐significant  difference  is  indicative  of  less  of  a  discrepancy  from  the  null  if  it  results  from  larger  (n1)  rather  than  a  smaller  (n2)  sample  size  (n1  >  n2  )

What’s  more  indicative  of  a  large  effect  (fire),  a  fire  alarm  that  goes  off  with  burnt  toast  or  one  so  insensitive  that  it  doesn’t  go  off  unless  the  house  is  fully  ablaze?  [The  larger  sample  size  is  like  the  one  that  goes  off  with  burnt  toast.)    

   

SPP  D.  Mayo   28  

Fallacy  of  Non-­‐Significant  results:  Insensitive  tests    • Negative  results  may  not  warrant  0  discrepancy  from  the  null,  but  we  can  use  severity  to  rule  out  discrepancies  that,  with  high  probability,  would  have  resulted  in  a  larger  difference  than  observed    

Similar  to  Cohen’s  power  analysis  but  sensitive  to  the  outcome—P-­‐value  distribution  (#3)  

 • I  hear  some  replicationists  say  negative  results  are  uninformative:  not  so  (#2  ironies)  

No  point  in  running  replication  research  if  your  account  views  negative  results  as  uninformative  

SPP  D.  Mayo   29  

Error  statistics  gives  evidential  interpretation  to  tests  (#3)    Use  results  to  infer  discrepancies  from  a  null  that  are  well  ruled-­‐out,  and  those  which  are  not      I’d  never  just  report  a  P-­‐value    Mayo  (1996);    Mayo  and  Cox  (2010):  Frequentist    Principle  of  Evidence:  FEV    

Mayo  and  Spanos  (2006):  SEV    

SPP  D.  Mayo   30  

One-­‐sided  Test  T+:    H0:  µ  <  µ0    vs.      H1:  µ  >  µ0      d(x)  is  statistically  significant  (set  lower  bounds)    (i)  If  the  test  had  high  capacity  to  warn  us  (by  producing  a  less  significant  result)  if  µ  ≤  µ0  +  γ.  then  d(x)  is  a  good  indication  of  µ  >  µ0  +  γ.    (ii)  If  the  test  had  little  (or  even  moderate)  capacity  (e.g.  <  .5)  to  produce  a  less  significant  result  even  if  µ  ≤  µ0  +  γ,  then  d(x)  is  a  poor  indication  of  µ  >  µ0  +  γ    (If  an  even  more  impressive  result  is  probable,  due  to  guppies,  it’s  not  a  good  indication  of  a  great  whale)    

SPP  D.  Mayo   31  

d(x)  is  not  statistically  significant  (set  upper  bounds)      (i)If  the  test  had  a  high  probability  of  producing  a  more  statistically  significant  difference  if  µ  >  µ0  +  γ,  then  d(x)  is  a  good  indication  that  µ  ≤  µ0  +  γ.  

 (ii)  If  the  test  had  a  low  probability  of  a  more  statistically  significant  difference  if  µ  >  µ0  +  γ,  then  d(x)  is  poor  indication  that  µ  ≤  µ0  +  γ.  (too  insensitive  to  rule  out  discrepancy  γ)    If  you  set  an  overly  stringent  significance  level  in  order  to  block  rejecting  a  null,  we  can  determine  the  discrepancies  you  can’t  detect  (e.g.,  risks  of  concern)  

SPP  D.  Mayo   32  

Confidence  Intervals  also  require  supplementing      Duality  between  tests  and  intervals:  values  within  the  (1  -­‐  α)  CI  are  non-­‐rejectable  at  the  α  level    • Still  too  dichotomous:  in  /out,  plausible/not  plausible  (Permit  fallacies  of  rejection/non-­‐rejection).  

• Justified  in  terms  of  long-­‐run  coverage  (performance).  • All  members  of  the  CI  treated  on  par.  • Fixed  confidence  level  (SEV  needs  several  benchmarks).  • Estimation  is  important  but  we  need  tests  for  distinguishing  real  and  spurious  effects,  and  checking  assumptions  of  statistical  models.  

 

SPP  D.  Mayo   33  

The  evidential  interpretation  is  crucial  but  error  probabilities  can  be  violated  by  selection  effects  (also  violated  model  assumptions)   One  function  of  severity  is  to  identify  which  selection  effects  are  problematic  (not  all  are)  (#3).      Biasing  selection  effects:  when  data  or  hypotheses  are  selected  or  generated  (or  a  test  criterion  is  specified),  in  such  a  way  that  the  minimal  severity  requirement  is  violated,  seriously  altered  or  incapable  of  being  assessed.      

   

SPP  D.  Mayo   34  

Nominal vs actual significance levels Suppose  that  twenty  sets  of  differences  have  been  examined,  that  one  difference  seems  large  enough  to  test  and  that  this  difference  turns  out  to  be  ‘significant  at  the  5  percent  level.’  ….The  actual  level  of  significance  is  not  5  percent,  but  64  percent!  (Selvin,  1970,  p.  104)    • They  were  clear  on  the  fallacy:  blurring  the  “computed”  or  “nominal”  significance  level,  and  the  “actual”  level  

 • There  are  many  more  ways  you  can  be  wrong  with  hunting  (different  sample  space)  

     

SPP  D.  Mayo   35  

This is a genuine example of an invalid or unsound method  You report: Such  results  would  be  difficult  to  achieve  under  the  assumption  of  H0 When  in  fact  such  results  are  common  under  the  assumption  of  H0

(formally): You say Pr(P-value < Pobs; H0) ~ α (small)   but in fact Pr(P-value < Pobs; H0) = high, if not guaranteed • Nowadays,  we’re  likely  to  see  the  tests  blamed  for  permitting  such  misuses  (instead  of  the  testers).  

 • Worse  are  those  accounts  where  the  abuse  vanishes!  

SPP  D.  Mayo   36  

What  defies  scientific  sense?    On  some  views,  biasing  selection  effects  are  irrelevant….  

Stephen  Goodman  (epidemiologist):    Two  problems  that  plague  frequentist  inference:  multiple  comparisons  and  multiple  looks,  or,  as  they  are  more  commonly  called,  data  dredging  and  peeking  at  the  data.  The  frequentist  solution  to  both  problems  involves  adjusting  the  P-­‐value…But  adjusting  the  measure  of  evidence  because  of  considerations  that  have  nothing  to  do  with  the  data  defies  scientific  sense,  belies  the  claim  of  ‘objectivity’  that  is  often  made  for  the  P-­‐value.”  (1999,  p.  1010).    

SPP  D.  Mayo   37  

Likelihood  Principle  (LP)  The  vanishing  act  takes  us  to  the  pivot  point  around  which  much  debate  in  philosophy  of  statistics  revolves:   In probabilisms, the import of the data is via the ratios of likelihoods of hypotheses:

P(x0;H1)/P(x0;H0)    

Different  forms:  posterior  probabilities,  Bayes  factor  (inference  is  comparative,  data  favors  this  over  that–is  that  even  inference?)      

SPP  D.  Mayo   38  

All  error  probabilities  violate  the  LP  (even  without  selection  effects):    “Sampling  distributions,  significance  levels,  power,  all  depend  on  something  more  [than  the  likelihood  function]–something  that  is  irrelevant  in  Bayesian  inference–namely  the  sample  space”.  (Lindley  1971,  p.  436)    The  information  is  just  a  matter  of  our  “intentions”   “The  LP  implies…the  irrelevance  of  predesignation,  of  whether  a  hypothesis  was  thought  of  before  hand  or  was  introduced  to  explain  known  effects  (Rosenkrantz,  1977,  122)  

SPP  D.  Mayo   39  

Many current Reforms are Probabilist Probabilist reforms to replace tests (and CIs) with likelihood ratios, Bayes factors, HPD intervals, or just lower the P-value (so that the maximal likely alternative gets .95 posterior)

while ignoring biasing selection effects, will fail.  The same p-hacked hypothesis can occur in Bayes factors; optional stopping can exclude true nulls from HPD intervals.

With one big difference: Your direct basis for criticism and possible adjustments has just vanished. (lots of #2 inconsistencies)  

SPP  D.  Mayo   40  

How  might  probabilists  block  intuitively  unwarranted  inferences?  (Consider  first  subjective) When we hear there’s statistical evidence of some unbelievable claim (distinguishing shades of grey and being politically moderate, ovulation and voting preferences), some probabilists claim—you see, if our beliefs were mixed into the interpretation of the evidence, we wouldn’t be fooled

We know these things are unbelievable, a subjective Bayesian might say That could work in some cases (though it still wouldn’t show what researchers had done wrong)—battle of beliefs.

SPP  D.  Mayo   41  

It wouldn’t help with our most important problem:

• How to distinguish the warrant for a single hypothesis H with different methods (e.g., one has biasing selection effects, another, registered results and precautions)?

So now you’ve got two sources of flexibility, priors and biasing selection effects (which can no longer be criticized). Besides, researchers really do believe their hypotheses.

SPP  D.  Mayo   42  

Diederik Stapel says he always read the research literature extensively to generate his hypotheses.

“So that it was believable and could be argued that this was the only logical thing you would find.” (E.g., eating meat causes aggression.) (In “The Mind of a Con Man,” NY Times, April 26, 2013[4])

SPP  D.  Mayo   43  

Conventional  Bayesians  

The most popular probabilisms these days are “non-subjective” (reference, default) or conventional designed  to  prevent  prior  beliefs  from  influencing  the  posteriors:   “The  priors  are  not  to  be  considered  expressions  of  uncertainty,  ignorance,  or  degree  of  belief.  Conventional  priors  may  not  even  be  probabilities…  .”  (Cox  and  Mayo  2010,  p.  299)    How  might  they  avoid  too-­‐easy  rejections  of  a  null?      

SPP  D.  Mayo   44  

Cult  of  the  Holy  Spike    

Give  a  spike  prior  of  .5  to  H0  the  remaining  .5  probability  being  spread  out  over  the  alternative  parameter  space,  Jeffreys.      This  “spiked  concentration  of  belief  in  the  null”  is  at  odds  with  the  prevailing  view  “we  know  all  nulls  are  false”  (#2)      Bottom line: By convenient choices of priors and alternatives statistically significant differences can be evidence for the null

 The  conflict  often  considers  the  two  sided  test    H0:  µ  =  0  versus  H1:  µ  ≠  0        

SPP  D.  Mayo   45  

   Posterior  Probabilities  in  H0  

                                                                             n  (sample  size)                                                                ____________________________      p                        z                        n=50          n=100            n=1000    .10              1.645                    .65                        .72                          .89  .05              1.960                    .52                        .60                          .82  .01              2.576                    .22                        .27                          .53  .001          3.291                    .034                    .045                    .124  

 If  n  =  1000,  a  result  statistically  significant  at  the  .05  level  leads  to  a  posterior  to  the  null  of  .82!    From  Berger  and  Sellke  (1987)  based  on  a  Jeffreys  pror  

SPP  D.  Mayo   46  

 • With  a  z  =  1.96  difference,  the  95%  CI  (2-­‐sided)  or  the  .975  CI  one  sided  excludes  the  null  (0)  from  the  interval    

• Severity reasoning: Were H0 true, the probability of getting

d(x) < dobs is high (~.975), so SEV  (µ  >  0) ∼ .975

• But they give P(H0 | z = 1.96 ) = .82

• Error statistical critique: there’s a high probability that they give posterior probability of .82 to H0:µ = 0 erroneously

• The onus is on probabilists to show a high posterior for H

constitutes having passed a good test.

SPP  D.  Mayo   47  

Informal  and  Quasi-­‐Formal  Severity  :  H  -­‐>  H*    

• Error  statisticians  avoid  the  fallacy  of  going  directly  from  statistical  to  research  hypothesis  H*    

• Can  we  say  nothing  about  this  link?  • I  think  we  can  and  must,  and  informal  severity  assessments  are  relevant  (#3)  

 I  will  not  discuss  straw  man  studies  (“chump  effects”).    

This is believable: Men react more negatively to success of their partners than to their failures (compared to women)?

Studies have shown: H: partner’s success lowers self-esteem in men

SPP  D.  Mayo   48  

Macho  Men  H*: partner’s success lowers self-esteem in men

 I  have  no  doubts  that  certain  types  of  men  feel  threatened  by  the  success  of  their  female  partners,  wives  or  girlfriends      I’ve  even  known  a  few.    

Can  this  be  studied  in  the  lab?  Ratliff  and  Oishi  (2013)  did:  .    

H*:  “men’s  implicit  self-­‐esteem  is  lower  when  a  partner  succeeds  than  when  a  partner  fails.”    

Not so for women Their example does a good job, given the standards in place.

SPP  D.  Mayo   49  

Treatments: Subjects are randomly assigned to five  “treatments”:  think,  write  about  a  time  your  partner  succeeded,  failed,  succeeded  when  you  failed  (partner  beats  me),  failed  when  you  succeeded  (I  beat  partner),  and  a  typical  day  (control).    

 Effects:  a  measure  of  “self-­‐esteem”  Explicit:  “How  do  you  feel  about  yourself?”  Implicit:  a test of word associations with “me” versus “other”. None showed statistical significance in explicit self-esteem, so consider just implicit measures

   

SPP  D.  Mayo   50  

 Some null hypotheses: The average self-esteem score is no different (these are statistical hypotheses) a) when partner succeeds (rather than failing) b) when partner beats (surpasses) me or I beat her c) control: when she succeeds, fails, or it’s a regular day There are at least double this, given self-esteem could be “explicit” or “implicit” (others too, e.g., the area of success)

 Only  null  (a)  was  rejected  statistically!  

Should  they  have  taken  the  research  hypothesis  as  

disconfirmed  by  negative  cases?    Or  as  casting  doubt  on  their  test?    

SPP  D.  Mayo   51  

Or  should  they  just  focus  on  the  null  hypotheses  that  were  rejected,  in  particular  null  (a),  for  implicit  self-­‐esteem.    

They  opt  for  the  third.    It’s not that they should have regarded their research

hypothesis H* as disconfirmed much less falsified.  This is precisely the nub of the problem! I’m saying the hypothesis that the study isn’t well-run needs to be considered

• Is the artificial writing assignment sufficiently relevant to the phenomenon of interest? (look at proxy variables)

• Is the measure of implicit self esteem (word associations) a

valid measure of the effect? (measurements of effects)

SPP  D.  Mayo   52  

Take,  null  hypothesis  b):  The average self-esteem score is no different when partner beats (surpasses) me or I beat her

 Clearly  they  expected  “she  beat  me  in  X”  to  have  a  greater  negative  impact  on  self-­‐esteem  than  “she  succeeded  at  X”.    

 Still,  they  could  view  it  as  lending  “some  support  to  the  idea  that  men  interpret  ‘my  partner  is  successful’  as  ‘my  partner  is  more  successful  than  me”  (p.  698),    

….as  do  the  authors.        That  is,  any  success  of  hers  is  always  construed  by  Macho  man  as,  she  beat  me.    

SPP  D.  Mayo   53  

Bending  over  Backwards  For  the  stringent  self-­‐critic,  this  skirts  too  close  to  viewing  the  data  through  the  theory,  a  kind  of  “self-­‐sealing  fallacy”.      I want to be clear that this is not a criticism of them given existing standards “I'm talking about a specific, extra type of integrity...bending over backwards to show how you're maybe wrong, that you ought to have when acting as a scientist.”   (R. Feynman 1974)  

I’m  describing  what’s  needed  to  show  “sincerely  trying  to  find  flaws”  under  the  austere  account  I  recommend  

 The  most  interesting  information  was  never  reported!  Perhaps  it  was  never  even  looked  at:  what  they  wrote  about.    

SPP  D.  Mayo   54  

Conclusion: Replication Research in Psychology Under an Error Statistical Philosophy

Replication problems can’t be solved without correctly understanding their sources  Biggest  sources  of  problems  in  replication  crises  (a) Stat  H  -­‐>research  H*  and  (b)  biasing  selection  effects:    

Reasons for (a): focus on P-values and Fisherian tests ignoring N-P tests (and the illicit NHST that goes directly H–> H*)

SPP  D.  Mayo   55  

Another reason, false dilemma: probabilism or long-run performance

plus assuming that N-P can only give the latter I argue for a third use of probability: Rather than report on believability researchers need to report the properties of the methods they used:

What was their capacity to have identified, avoided, admitted bias?

What’s  wanted  is  not  a  high  posterior  probability  in  H  (however  construed)  but  a  high  probability  the  procedure  would  have  unearthed  flaws  in  H  (reinterpretation  of  N-­‐P  methods)  

SPP  D.  Mayo   56  

What’s  replicable?  Discrepancies  that  are  severely  warranted  

Reasons  for  (b)  [embracing  accounts  that  formally  ignore  selection  effects]:  accepting  probabilisms  that  embrace  the  likelihood  principle  LP    There’s  no  point  in  raising  thresholds  for  significance  if  your  methodology  does  not  pick  up  on  biasing  selection  effects.      

SPP  D.  Mayo   57  

 Informal assessments of probativeness are needed to scrutinize statistical inferences in relation to research hypotheses H –> H* One  hypothesis  must  always  be:  our  results  point  to  the  inability  of  our  study  to  severely  probe  the  phenomenon  of  interest  (problem  with  proxy  variables,  measurements,  etc.)     The scientific status of an inquiry is questionable if it cannot or will not distinguish the correctness of inferences from problems stemming from a poorly run study If ordinary research reports adopted the Feynman “bending over backwards” scrutiny, the interpretation of replication efforts would be more informative (or perhaps not needed)

SPP  D.  Mayo   58  

REFERENCES    Baggerly,  K.  A.,  Coombes,  K.  R.  &  Neeley,  E.  S.  (2008).  “Run  Batch  Effects  

Potentially  Compromise  the  Usefulness  of  Genomic  Signatures  for  Ovarian  Cancer.”  Journal  of  Clinical  Oncology.  26(7):  1186-­‐1187.  

Bartless,  T.  (2012).  “Daniel  Kahneman  Sees  ‘Train-­‐Wreck  Looming’  for  Social  Psychology”.  Chronicle  of  Higher  Education  Blog  (Oct.  4,  2012)  article  w/links  to  email  D.  Kahneman  sent  to  several  social  psychologists.    http://chronicle.com/blogs/percolator/daniel-­‐kahneman-­‐sees-­‐train-­‐wreck-­‐looming-­‐for-­‐social-­‐psychology/31338.  

Berger,  J.  O.  (2006).  “The  Case  for  Objective  Bayesian  Analysis.”  Bayesian  Analysis  1  (3):  385–402.  

Berger,  J.  O.  &  Sellke,  T.  (1987).  “Testing  a  Point  Null  Hypothesis:  The  Irreconcilability  of  P  Values  and  Evidence  (with  Discussion).”  Journal  of  the  American  Statistical  Association  82  (397)  (March  1):  112–122.  

Bhattacharjee,  Y.  (2013).  “The  Mind  of  a  Con  Man”.  The  New  York  Times  Magazine  (4/28/2013),  p.  44.  

Cohen,  J.  1988.  Statistical  Power  Analysis  for  the  Behavioral  Sciences.  2nd  ed.  Hillsdale,  NJ:  Erlbaum.  

SPP  D.  Mayo   59  

 Coombes,  K.  R.,  Wang,  J.  &  Baggerly,  K.  A.  (2007).  “Microrrays:  retracing  steps.”  

Nature  Medicine.  13(11):1276-­‐7.  Cox,  D.  R.  &  D.  V.  Hinkley.  (1974).  Theoretical  Statistics.  London:  Chapman  and  

Hall.  Cox,  D.  R.  &  Mayo,  D.  G.  (2010).  “Objectivity  and  Conditionality  in  Frequentist  

Inference.”  In  Error  and  Inference:  Recent  Exchanges  on  Experimental  Reasoning,  Reliability,  and  the  Objectivity  and  Rationality  of  Science,  edited  by  Deborah  G.  Mayo  and  Aris  Spanos,  276–304.  Cambridge:  Cambridge  University  Press.  

Diaconis,  P.  (1978).  “Statistical  Problems  in  ESP  Research”.  Science  201  (4351):  131-­‐136.  (Letters  in  response  can  be  found  in  the  Dec.  15,  1978  issue  pp.  1145-­‐6.)  

Dienes,  Z.  (2011)  “Bayesian  versus  Orthodox  Statistics:  Which  Side  Are  You  On?”  Perspectives  on  Psychological  Science  6(3):  274-­‐290.  

Feynman,  R.    (1974).  “Cargo  Cult  Science.”  Caltech  Commencement  Speech.  Fisher,  R.  A.  (1947).  The  Design  of  Experiments,  4th  ed.  Edinburgh:  Oliver  and  

Boyd.  

SPP  D.  Mayo   60  

Gelman,  A.  (2011).  “Induction  and  Deduction  in  Bayesian  Data  Analysis.”  Edited  by  Deborah  G.  Mayo,  Aris  Spanos,  and  Kent  W.  Staley.  Rationality,  Markets  and  Morals:  Studies  at  the  Intersection  of  Philosophy  and  Economics  2  (Special  Topic:  Statistical  Science  and  Philosophy  of  Science):  67–78.  

Gelman,  A.  &  Shalizi,  C.  (2013).  “Philosophy  and  the  Practice  of  Bayesian  Statistics.”  British  Journal  of  Mathematical  and  Statistical  Psychology  66  (1):  8–38.  

Gigerenzer,  G.  (2000).  “The  Superego,  the  Ego,  and  the  Id  in  Statistical  Reasoning.  “  Adaptive  Thinking,  Rationality  in  the  Real  World,  OUP.  

Goodman,  S.  N.  (1999).  Toward  evidence-­‐based  medical  statistics.  2:  The  Bayes  factor.”  Annals  of  Internal  Medicine,  130:1005  –1013.  

Howson,  C.  &  Urbach,  P.  (1993).  Scientific  Reasoning:  The  Bayesian  Approach.  2nd  ed.  La  Salle,  IL:  Open  Court.  

Johansson  T.  (2010)  “Hail  the  impossible:  p-­‐values,  evidence,  and  likelihood.”  Scandinavian  Journal  of  Psychology  52:113-­‐125.  

Kruschke,  J.  K.  (2010).  “What  to  believe:  Bayesian  methods  for  data  analysis”.  Trends  in  Cognitive  Science,  14(7):  297-­‐300.  

Lehmann,  E.  L.  (1993).  “The  Fisher,  Neyman-­‐Pearson  Theories  of  Testing  

SPP  D.  Mayo   61  

Hypotheses:  One  Theory  or  Two?”  Journal  of  the  American  Statistical  Association  88  (424):  1242–1249.  

Levelt  Committee,  Noort  Committee,  Drenth  Committee.  (2012).  “Flawed  science:  The  fraudulent  research  practices  of  social  psychologist  Diederik  Stapel”.  Stapel  Investigation:  Joint  Tilburg/Groningen/Amsterdam  investigation  of  the  publications  by  Mr.  Stapel.  https://www.commissielevelt.nl/  

Lindley,  D.  V.  (1971).  “The  Estimation  of  Many  Parameters.”  In  Foundations  of  Statistical  Inference,  edited  by  V.  P.  Godambe  and  D.  A.  Sprott,  435–455.  Toronto:  Holt,  Rinehart  and  Winston.  

Mayo,  D.  G.  (1996).  Error  and  the  Growth  of  Experimental  Knowledge.  Science  and  Its  Conceptual  Foundation.  Chicago:  University  of  Chicago  Press.  

Mayo,  D.  G.  &  Cox,  D.  R.  (2010).  "Frequentist  Statistics  as  a  Theory  of  Inductive  Inference"  in  Error  and  Inference:  Recent  Exchanges  on  Experimental  Reasoning,  Reliability  and  the  Objectivity  and  Rationality  of  Science  (D.  Mayo  and  A.  Spanos  eds.),  Cambridge:  Cambridge  University  Press:  1-­‐27.  This  paper  appeared  in  The  Second  Erich  L.  Lehmann  Symposium:  Optimality,  2006,  Lecture  Notes-­‐Monograph  Series,  Volume  49,  Institute  of  Mathematical  Statistics,  pp.  247-­‐275.  

SPP  D.  Mayo   62  

Mayo,  D.  G.,  and  A.  Spanos.  (2006).  “Severe  Testing  as  a  Basic  Concept  in  a  Neyman–Pearson  Philosophy  of  Induction.”  British  Journal  for  the  Philosophy  of  Science  57  (2)  (June  1):  323–357.    

Mayo,  D.  G.,  and  A.  Spanos.    (2011).  “Error  Statistics.”  In  Philosophy  of  Statistics,  edited  by  Prasanta  S.  Bandyopadhyay  and  Malcom  R.  Forster,  7:152–198.  Handbook  of  the  Philosophy  of  Science.  The  Netherlands:  Elsevier.  

Meehl,  P.  E.  &  Waller,  N.  G.  (2002).  “The  Path  Analysis  Controversy:  A  New  Statistical  Approach  to  Strong  Appraisal  of  Verisimilitude.”  Psychological  Methods  7(3):  283–300.  

Morrison,  D.  E.  &  Henkel,  R.  E.  (eEds).  (1970).  The  Significance  Test  Controversy:  A  Reader.  Chicago:  Aldine  De  Gruyter.  

Micheel,  C.  M.,  Nass,  S.  J.  &  Omenn  G.  S.  (Eds)  Committee  on  the  Review  of  Omics-­‐Based  Tests  for  Predicting  Patient  Outcomes  in  Clinical  Trials;  Board  on  Health  Care  Services;  Board  on  Health  Sciences  Policy;  Institute  of  Medicine  (2012).  Evolution  of  Translational  Omics:  Lessons  Learned  and  the  Path  Forward.  Nat.  Acad.  Press.    

Neyman,  J.  (1957).  “‘Inductive  Behavior’”  as  a  Basic  Concept  of  Science.”  Revue  de  l'Institut  International  de  Statistique/Review  of  the  International  Statistical  Institute,  25  (1/3):  7-­‐22.  

SPP  D.  Mayo   63  

Neyman,  J.  &  Pearson,  E.  S.  (1928).  “On  the  Use  and  Interpretation  of  Certain  Test  Criteria  for  Purposes  of  Statistical  Inference.  Part  I,”  Biometrica  20A:  175-­‐240  (reprinted  in  Joint  Statistical  Papers,  University  of  California  Press,  Berkeley,  1967,  pp.  1-­‐66.)  

Popper,  K.  (1962).  Conjectures  and  Refutations:  The  Growth  of  Scientific  Knowledge.  New  York:  Basic  Books.  

Potti,  A.,  Dressman  H.  K.,  Bild,  A.,  Riedel,  R.  F.,  Chan,  G.,  Sayer,  R.,  Cragun,  J.,  Cottrill,  H.,  Kelley,  M.  J.,  Petersen,  R.,  Harpole,  D.,  Marks,  J.,  Berchuck,  A.,  Ginsburg,  G.  S.,  Febbo,  P.,  Lancaster,  J.    &  Nevins,  J.  R.    (2006).  “Genomic  signatures  to  guide  the  use  of  chemotherapeutics.”  Nature  Medicine.  Nov  12(11):1294-­‐300.  Epub  2006  Oct  22.    

Potti,  A.  &  Nevins,  J.  R.  (2007)  “Reply  to  Coombes,  Wang  &  Baggerly.”  Nature  Medicine  Nov  13(11):1277-­‐8.    

Ratliff,  K.  A.  &  Oishi,  S.  (2013).  “Gender  Differences  in  Implicit  Self-­‐Esteem  Following  a  Romantic  Partner’s  Success  or  Failure”.    Journal  of  Personality  and  Social  Psychology  105(4):  688–702.  

Rosenkrantz,  R.  (1977).  Inference,  Method  and  Decision:  Towards  a  Bayesian  Philosophy  of  Science.  Dordrecht,  The  Netherlands:  D.  Reidel.  

SPP  D.  Mayo   64  

Savage,  L.  J.  (1962).  The  Foundations  of  Statistical  Inference:  A  Discussion.  London:  Methuen.  

Savage,  L.  J.  (1964).  “The  Foundations  of  Statistics  Reconsidered.”  In  Studies  in  Subjective  Probability,  H.  Kyburg  &  H.  Smokler  (eds.),  173-­‐188.  New  York:  John  Wiley  &  Sons.  

Selvin,  H.  (1970).  “A  Critique  of  Tests  of  Significance  in  Survey  Research.”  In  The  Significance  Test  Controversy,  edited  by  D.  Morrison  and  R.  Henkel,  94-­‐106.  Chicago:  Aldine  De  Gruyter.  

Trafimow,  D.  &  Marks  M.  (2015).  “Editorial”.  Basic  and  Applied  Social  Psychology,  37(1),  pp.  1-­‐2.  

Wagenmakers,  E.-­‐J.  (2007).  “A  Practical  Solution  to  the  Pervasive  Problems  of  P  Values”.  Psychonomic  Bulletin  &  Review  14  (5),  779-­‐804.