HOME RADIO PEOPLE ABOUT USArt & Design Film & Music Sc ience Soc iety Words World

Facebook Google+ T witter T umblr

The maths of the paper disprovingconspiracy theories don't add upBy Martin Robbins

R e s e a r c h c l a i m i n g t o h a v e c r a c k e d t h e n u m b e r s b e h i n dm y s t e r i o u s p l o t s i s f l a w e dThere are several ways to tell when youve written a paper thats not quite up to scratch.Lingering doubts perhaps, or stinging comments from the reviewers. A definite clue is when youturn on Thought for the Day and hear it cited as evidence for the resurrection of Christ.

646646 384384

Listen:

"Conspiracy theories abound about what happened to Jesus ofNazareth." Anne Atkins - 28/01/16Thought for the Day

Thought for the DayThought for the Day

Download Flash Player now

You need to install Flash Player toplay this content.

The paper in question, On the Viability of Conspiratorial Beliefs, was published in open-accesspeer-review journal PLOS One a few days ago. It starts out with quite a good premise. There area lot of widely held beliefs that, if true, would require an extraordinary number of people to behiding something. If you think that man made climate change is a hoax, for example, then at aminimum tens of thousands of climate researchers and other scientists must be in on it in someway. If you think that NASA faked the moon landings then you have to deal with the fact that over400,000 NASA employees would have been in some way connected to it.

Given those kinds of numbers, it seems almost inevitable that leaks would occur. Can wepredict how long a conspiracy of a given size will last? Thats the question asked by Dr DavidGrimes - the researcher, skeptic and writer who published the paper. If we can provemathematically that a conspiracy involving 400,000 people cant last more than a few months oryears, then we can easily dismiss a number of popular beliefs.

Its a nice idea. Unfortunately the answer is a resounding "no", and the resulting paper ends upbeing a sort of case study in how not to do statistics. Inevitably media outlets loved it, and so nownews feeds are full of headlines like: Most conspiracy theories are mathematically impossible,The maths equation threatening to disprove conspiracy theories, Maths study shows

conspiracies prone to unraveling and so on and on.

Do you want to know a secret?Lets imagine we want to predict how long a conspiracy will last. One approach would be tocollect a number of examples of existing theories of different sizes and durations, and plot themon a chart. It might look something like the following graph, though ideally with a lot more datapoints. If were really lucky, we might even see a pattern.

Grimes takes a different approach, except he has a grand total of, well, three examples toextrapolate from. Its not exactly a data goldmine.

The examples are three real life conspiracies that were eventually exposed the NSAs PRISMprogramme, the Tuskegee syphilis experiment, and a scandal at the FBI about the accuracy oftheir forensics tests. In each case the number of people involved is complete guesswork. For theNSA Grimes uses a figure of 30,000, which assumes the entire workforce from director down tojanitor were involved in one project. In reality it could have been hundreds or even just tens ofpeople. The data is basically nonsense you could put the points almost anywhere to makewhatever pattern you liked.

So there arent enough examples, and the few listed arent very reliable. Theyre also the wrongkind. Were missing any data on conspiracies that have remained secret for obvious reasons.Two of the three predate the Internet era, and youd expect a revolutionary globalcommunications system to have some impact on communications. There arent any smallconspiracies of say 10 or 20 people. All of the examples are big, institutional or communityaffairs. All three are based in the United States, two in law enforcement or security serviceswhere secrecy is part of the job description and the cost of breaking it is extreme.

From these examples, Grimes calculates that the likelihood of a typical person blabbing abouttheir secret conspiracy in a given year is roughly five ten thousandths of a percent. If this weretrue, there wouldnt be a word for gossip in the dictionary. Wed walk around in stony silence allday tapping our noses at people. The standard response to How are you? would be Never youmind. Its the sort of number you accept without thinking when youve forgotten that the numbersyoure playing with relate to actual human beings.

Do you promise not to tell?We could get deeper into the basic premises of the paper. How do you even define a conspiracy,and how is it different to a secret? What does it mean for a conspiracy to leak, and can peopleeven agree on whether a leak has occurred? After all, climate skeptics would point to theUniversity of East Anglia e-mail leak as an example of a leak occurring and large numbers ofpeople becoming aware of the truth.

For that matter, why should dying prevent someone being the source of a leak (history is litteredwith examples of lost papers found in attics), and why would his or her probability of leakingremain constant over time, when some conspiracies by their nature are less important to concealyears after the fact?

As it happens though, all thats irrelevant because theres a much bigger issue. As far as I cantell the maths underpinning his model is wrong. Several statisticians who I spoke to or whochecked the paper independently agree.

The root of the problem lies in this graph. Forget the jargon for a moment. All this shows is thecumulative probability over time of a conspiracy being discovered.

cumulative probability over time of a conspiracy being discovered.

The assumption is that the average person has a 0.0005% chance of leaking in any given year.The blue line shows how the probability grows year after year if 5000 people are involved, allthose individual chances adding up with time. This is exactly the same type of curve youd get ifyou plotted the odds of rolling a six from a growing number of dice your odds get closer andcloser to 1 but never quite reach it.

The other lines show what happens if you assume that people die off, using two different rates.The orange line assumes half of the people involved die every 10 years, so after a decade thereare 2500 people left, 1250 after two decades and so on. The pink line uses a more sophisticatedequation for mortality.

Got that? If so, youre doing better than the author. Heres the problem: if the probability ofexposure is accumulating over time then why does it start going down? You cant make aconspiracy secret again! The increase each year should fall toward 0 as the population dies off,leaving the plots to trail off as flat lines.

In fact they dont account for a changing population at all, as I discovered when I wrote a simplecomputer simulation to test his model. Thanks to a very basic calculus error, the lines actuallyshow what the cumulative probability would have been if the population had started at the current

level and stayed constant. So if we take the orange line at 10 years, instead of telling us thecumulative probability if the population had started at 5000 and fallen to 2500 over time, it givesus the value if the population had started at 2500 and stayed constant. Since this error occurs inequation 1, which the rest of Grimess equations are derived from well, you get the idea.

In truth it doesnt affect the conclusions too much, because the timelines for the conspiraciesGrimes looked at were so short that mortality didnt really kick in. The conclusion still makessense if you have a lot of people, hiding a conspiracy becomes incredibly unlikely. But weknew this already. Even if we corrected his mistake, the model Grimes presents would just be astandard probability curve with some improbable assumptions plugged into it. It doesnt get usany closer to a prediction for the longevity of any real world conspiracy. As equations go, its upthere with the infamous Blue Monday a plausible sounding story for the press that lacks anyscientific rigour.

It would be easy to blame Grimes for all this, but the bigger failure here is in PLOS ONEs peerreview process. Its easy to screw up calculus. Whats less excusable is that expert reviewerslooked at this paper ahead of publication, and none of them spotted an elementary mistake thatmyself and others saw almost immediately. Numerous other helpers are cited in theacknowledgements, but none of them seem to have glanced at the math or challenged somereally odd assumptions. Grimes made a mistake we all do but he was also severely let downby his peers and colleagues.

And its frustrating. Its frustrating because skeptics and rationalists ended up retweeting newsstories that were just as bogus as the bad science they claim to stand against. Its frustratingbecause a paper that lashes out against the idea that scientists might be engaged in covering upbad research turns out to be an example of bad research that slipped through peer review. Itsfrustrating because the model was so ill thought that Anne Atkins on Radio 4's Thought for theDay was able to use it as evidence for the resurrection of Christ after all, if so many peoplebore witness to it then the maths "proves" that a conspiracy to conceal the truth could not havelasted two millennia.

Which leaves perhaps the biggest question of all: was this really just a bad paper, or was theresome deeper purpose behind it? Is Doctor Grimes engaged in some kind of charade, runninginterference on behalf of a master or masters unknown? Is he still the real Grimes, or has hebeen replaced by a foppish-haired lizard impersonator? The truth is out there

(Many thanks to Adam Jacobs of statsguy.co.uk for double-checking my late-night maths)

Science mat hemat ics conspiracy t heoriesScience

Jan 29, 2016

646646 384384

R E L A T E D P O S TNOW READ

Anti-Semitism is in the very marrow of conspiracy theory

[X]

Corbyn, Trump and Farage: The Rise of the Paranoid Style

Blue Mondays not real, but the happiness industry can still get you down

M A R T I N R O B B I N S

Martin Robbins is a writer and talker at the messy border of science and culture. He is acolumnist at VICE, and blogs for The Guardian and the New Statesman.

Little Atoms magazine Issue 1 on sale now

Feb 8, 2016

Feb 6, 2016

Feb 2, 2016

Newest | Oldest | Top Comments

ChadWeber

nmrqip

Conspiracy_X

Sign in 22 people listening

Grimes is a space lizard who is trying to make his own work seem shoddy so that people willsuspect him of being a space lizard because Grimes knows that everyone else will laugh at thepeople who believe in space lizards. That's the only logical explanation. And I'm pretty sure theauthor of this article is also a space lizard, who mentions the possibility of Grimes being a lizardin jest in an attempt to make the entire premise of space lizards seem laughable.

Like Reply

I think I have worked out the underlying physical model implied by his equations. Of course thisisn't what he actually did: he worked out an incorrect mathematical description of a slightly moresensible physical model. But it is possible to back out the implicit physical model for which hismaths is correct.

Grimes assumes that a conspiracy fails on a single defection. This is a hopelessly naive model(note that the mafia and drug cartels continue to exist despite multiple defections), but it is atleast well defined. He then correctly notes that if a conspiracy member dies before defecting thenthe number of potential defectors is reduced, and so is the probability of new defections. However his mathematical description implies that if a conspirator dies after defecting then theeffects of his previous defection are mysteriously undone, and if he as the only defector at thatstage then the conspiracy leaps back into secrecy again. This is why his failure probabilities fallto zero once everyone has died and all possible defections have been reversed.

The more I think about this paper the worse it gets.

Like Reply

This was naive! Ever heard of compartmentalization? Just an example, or two: NASA admitsmultiple times Moon landings couldn't have happened. We admittedly haven't figured out yet howto get man through the Van Allen Radiation Belt. Technology isn't there yet:https://youtu.be/51DED8dcNkA ------- Or, how about the admission that all of the Middle-Easternwars were planned before 9/11 and before Bush Jr. even stepped undeserved foot into office:

14 comments

+ Follow+ Follow Share Post comment as...

Feb 1, 2016

Jan 31, 2016

greybuscat

cbauch

wars were planned before 9/11 and before Bush Jr. even stepped undeserved foot into office:https://youtu.be/9RC1Mepk_Sw ------- How about the MLK Jr. family winning a lawsuit againstthe US government in 1999 for being directly behind his assassination:http://www.washingtonsblog.com/2015/01/martin-luther-king-assassinated-us-govt-king-family-civil-trial-verdict.html ------- Or, maybe you'd like to hear dozens of Politicians and thelike admitting to an agenda called the New World Order: https://youtu.be/uIH4jUIK5zs ------- But,just as important are the psychologists explaining why it's so hard for morons like yourself toaccept reality: https://youtu.be/f462ya0DC0g

NASA Admits They Can't Send Humans Through The VanAllen Radiation Belts

Like Reply

Feb 5, 2016TylerAndrewMaire@Conspiracy_X Nah man, you can get through it if you keep wearing your tinfoil hat

Like Reply

I wonder why he only picked three. The Tonkin Gulf incident is a great example of a real-lifeconspiracy. In the US, it's the probably THE best conspiracy one could choose.

Like Reply

I was the handling editor and the point about the choice of population size being overlooked inpeer review is wrong.In fact, this issue was flagged in the review process, but in most cases thereare no data to decide on what the relevant subpopulation size should be.As the handling editor, Iagreed it was better to use a known data point and to discuss the implications of it being too largein the papers Discussion section, rather than guess what the data should be, thus providing onlythe illusion of accuracy.In other cases, the population size the author used might have been toosmall, as in the case of vaccines where academic researchers conducting the clinical trials wouldalso have to have been in on the conspiracy.A univariate sensitivity analysis could have beenconducted, but again, we have no data to decide on suitable bounds for the sensitivity analysis.As it stands, someone who actually reads the entire paper will be able note the limitation of usingthe entire workforce size and understand how that might influence results.As I doubt NASA has

the entire workforce size and understand how that might influence results.As I doubt NASA hasgiven serious thought to how many employees would have to have been involved in a MoonLanding hoax 40 years ago, we need to leave the problem of subpopulation size to future work.Fornow, the paper has the nontrivial value of illustrating an interesting application of failure theory.

Questioning the validity of the study because evangelical Christians use it as evidence for theresurrection of Christ is also wrong, for the same reason that we cant discard the Big BangTheory just because the Catholic Church likes it, or that we cant discard legitimate researchexploring why some individuals (as it so happens, 10% of individuals of European descent) areresistant to HIV because white supremacists cite this as evidence of white superiority. MaybeRobbins was being facetious, but its hard to tell for sure.

The point about changing population size is valid and the author should have used a different typeof statistical model called a non-homogeneous Poisson model, but this should not change theconclusions and in fact should make them stronger, so the current analysis is actuallyconservative (although I have recommended the author be given a chance to respond through anerratum).The homogeneous Poisson model is in fact a simplifying approximation in the Gompertzcase for the first 30 years or so.Thank you to Jonathan Jones and colleagues for pointing out theissue of changing population size on the PLOS ONE website in a professional and balancedmanner.All scientists make mistakes at one point or another, and this is one way in whichscience corrects its mistakes and moves forward. Its good to see the process working and to seenon-scientists taking interest in it, although the confusion and distortion of the papers objectives,merits and peer review process are unfortunate.

Chris Bauch

Like Reply

Feb 1, 2016MartinRobbins@cbauch This is a truly surprising response. For the sake of brevity I'm going to focuson just one statement, which I think highlights just how poorly the handling editor of thepaper appears to understand the work being handled. You said: " this should notchange the conclusions and in fact should make them stronger, so the current analysisis actually conservative."

As I said over at the comments on PLOS, "If a major mathematical error in a modelfails to change the conclusion then one has to ask whether the methodology properlyevaluated the model. In this case, we can see that it did not." The only reason theconclusions stand is because the model was not tested on any examples with lessthan tens of thousands of participants, so that none of the conspiracies investigatedwould have collapsed in a long enough time period for mortality to be a factor.

To quote myself at PLOS again, "This is equivalent to Newton suggesting a law ofgravity that all objects fall to the ground within two seconds, but only ever testing it upto heights of one metre." The idea that somehow the conclusions are strengthened bythis failure to test them on a viable set of data is beyond parody. On what possiblebasis are they strengthened?! If it's simply that they conform more closely to theconstant population model then of course they do - if you select examples wheremortality isn't an issue then it will naturally lead to the constant population model beinga better fit!

Then let's take population. We have three examples where the populations are out bypotentially a factor of 100 or more, and even then one candidate value for theprobability per person per year is 50 times larger than the other two. You can't just takethe smallest value and say that you're being conservative when the value of 'p' chosencould be 100s of times out! Can you not see that this would have a dramatic impact on

Jan 31, 2016

Jan 30, 2016

Jan 30, 2016

Twundit

anarcissie

nmrqip

could be 100s of times out! Can you not see that this would have a dramatic impact onthe viability of any conspiracy theory?

Yes, this is hand-waved a little in the conclusion, but I'm lost as to what exactly thepaper achieves when 2 of the 3 models are incorrect, the data used to test them is toosmall and seriously flawed, and the third model is simply as standard Poissondistribution. You refer to "the confusion and distortion of the papers objectives."

Can you please explain what you believe those objectives to be, since they don'tapparently involve either developing a novel model of conspiracy theory viability ortesting such a model?

Like Reply

Feb 1, 2016BarryJWoods@MartinRobbins @cbauchI imagine you will be shown the red flag soon..

Nature: Research Integrity: Don't let transparency damage sciencehttp://www.nature.com/news/research-integrity-don-t-let-transparency-damage-science-1.19219

Like Reply

Well.. this was an interesting experiment to see how long it took for people to notice and come outwith the obvious error. All in line with the subject matter.

Like Reply

Jan 31, 2016anarcissie@Twundit -- Many people -- dozens or hundreds -- may have noticed the errors (that'sa plural), but not had time or inclination to write to the media which passed it along. There are at least two major social problems here. One is the very low quality ofreporting on science; it is evidently done in many cases by people who know very littleabout any science. The other is the deterioration of supposedly real science intopseudoscience at the source, as in the present case. Scientists are not supposed tojust make stuff up. I am still not sure the theory is not some kind of joke.

Like Reply

When I first saw this story, I thought it was a hoax or a satire of some sort. Regardless of themathematics used, the author seems to be pulling most of the numbers out of his nether regions. You don't need to be any sort of mathematician or sociologist to see that.

Like Reply

Jan 29, 2016

Jan 29, 2016

BarryJWoods

BarryJWoods

Martin Robbins is entirely correct in what he says here. The two of us detected the flawindependently, by very much the same reasoning (cumulative failure curves just can't turn down!).

I posted a comment on the PLOS article pageat http://www.plosone.org/annotation/listThread.action?root=88142 first thing on Fridaymorning explaining the error. The original author, David Grimes, has replied, implicitly acceptingthat I have correctly identified an error, but attempting to play down its significance. In reality theerror is fundamental: the incorrect formula is the only novel element of the paper, and everythingthat remains is essentially trivial. The paper should probably be withdrawn, but at the very least aformal correction to the published article should be made.

Like Reply

oops - 19% thought gov covered up existence of aliens..

Like Reply

Slightly off topic, but a survey of 137 2nd year pyschology undergraduates, showed 19% of them'believed in the moon landings were faked, and 54% thought Dianas death not an accident, and14% that governments covering up the existence of aliens. That is what the data says, didn'tmake it into the paper though..

Dead and Alive- Wood et al - Michael wood quickly sent out the datasets, so if curious just ask. http://www.winchester.ac.uk/academicdepartments/psychology/staff/Pages/DrMichaelWood.aspx

funny thing, he used stats to come up with the headline title, and not a single person actuallyagreed with that hypothesis, in the dataset.

my point, doing tickbox surveys in psychology, not very useful!2nd point, why survey a group of 2nd yr, 20.3 av yr ld, 83% female psychology undergrads, andthink it has any relevance with respect to the public.

Like Reply

R E C E N T P O S T S W h y d i d A m i r S i m a n - T o v d i e i n a B r i t i s h

d e t e n t i o n c e n t r e ?

Read more

T h e s e v e r y j u n i o r d o c t o r s h a v e am e s s a g e f o r J e r e m y H u n t

Read more

P o l a n d s h o u l d f a c e u p t o i t s a n t i -S e m i t i c p a s t

Read more

I n p r a i s e o f H a r p e r L e e

Read more

B o u t r o s B o u t r o s - G h a l i w r e c k e d t h eU n i t e d N a t i o n s

Read more

Tweets by @little_atoms

C O N T R I B U T O R S

Maruxa Ruiz del rbol

Adam Rutherford

Ade Adepitan

Ade Adepitan

Alex Masters

Alex Mik

Alison Rooper

Andrew Copson

Andrew Mueller

Anthony Nolan

Bill Thompson

1 of 8 next

The maths of the paper disproving conspiracy theories don't add upResearch claiming to have cracked the numbers behind mysterious plots is flawed

Listen:Do you want to know a secret?Do you promise not to tell?MARTIN ROBBINSRELATED POSTNOW READAnti-Semitism is in the very marrow of conspiracy theoryCorbyn, Trump and Farage: The Rise of the Paranoid StyleBlue Mondays not real, but the happiness industry can still get you downLittle Atoms magazine Issue 1 on sale nowRECENT POSTSWhy did Amir Siman-Tov die in a British detention centre?These very junior doctors have a message for Jeremy HuntPoland should face up to its anti-Semitic pastIn praise of Harper LeeBoutros Boutros-Ghali wrecked the United NationsCONTRIBUTORS