SE Myths on Replication

Natalia Juristo

Universidad Politecnica de Madrid Spain

www.grise.upm.es/miembros/natalia/

Myths

In SE we have a mythical view about replication in other sciences

Specially regarding replication in the hard sciences, Physics as paradigm

Replication is Not a Myth

Replication does exist in other disciplines To see replication and its absence in

practice Nature Magazine tracked the fate of 19 papers in issue 6893 to check whether the results had been reproduced 2 years later

In a large majority of cases they had. Papers on fields like Neurology, Paleontology, etc.

Myth 1

Others are able to give ALL the information to replicators

What Is All the information?

Myht Information that other researchers (physicist

as paradigm) interchange for replication is complete and perfect

Reality No, it is not. They face, as we do, two

problems Tacit knowledge Unknown relevant variables

Inspiring quotes “There is always something unspoken even in the middle

of the exact sciences” “The Logic of Tacit Inference”

Physicist-Chemist Michael Polanyi

“It’s very difficult to make a carbon copy. You can make a near one, but if it turns out that what is critical is the way he glued his transducers, and he forgets to tell you that the technicians always puts a copy of Physical Review on top of them for weight, well, it could make all the difference”

A physicist quoted in “Changing Order”

Harry Collins

Inspiring quotes “If I am trying to explain how to build the TEA laser I will not tell you that the

inductance of the top is important, because I assume you already know...

…Finally, you do not know what you need to know and I do not know what I know...

…In fact, the principle of TEA lasers, scientists did not know necessarily that the inductance of the top was important”

A physicist quoted in “Tacit knowledge: you don't know how much you know”

New Scientist Magazine N. 2762 Harry Collins

2010

Inspiring quotes

“It is often hard to tell whether an inability to replicate a result is due to a group’s failing or a flaw in the original paper.

The reason is often the countless tiny details of experimental method that are omitted from the paper but can influence results”

Gillian Murphy Cell Biologist

University of Cambridge Quoted in “The Trouble with Replication”

Nature 2006

Myth 2

Independent replications means 0 interaction

Independence = No Interaction

Myth Other researchers (physicist as paradigm) just

interchange materials and never interact that is why it is an independent replication

Independent replication means do not interact with the previous experimenters at all except interchanging the protocol

Reality They interact, talk, discuss why my replication does

not get the same results than yours, they visit each other to see how you run your replication,…

They Do Interact

“A paper can never be a foolproof recipe for a replication of its results because this sort of information can never be entirely captured in a scientific paper… Researchers compensate by exchanging tips by e-mail and at conferences. Because this social interactions are not recorded anywhere, it is hard to consult or build on them”

“The Trouble with Replication” Jim Giles

Nature 2006

Inspiring quotes “When I started in Science I was told that I

should be able to repeat an experiment by reading the paper, but that is almost never the case”

Michael Ronemus Molecular and Cell Biologist

Cold Spring Harbor Laboratory Editor of the Journal Cold Spring Harbor Protocols

Cold Spring Harbor Protocols, PLoS ONE and Nature

Protocols include discussion facilities that allow researchers to talk to the original authors and others users of the protocol. These could prove ideal places to resolve problems that crop up during attempts to reproduce a previous result

New ways of interchanging the critical Info

PLoS ONE takes the comment model further than anyone else with various functions for promoting discussion

“The aim is to recreate the kind of discussion that takes place in front of conference posters”

Surridge PLoS ONE Manager

They Even Visit Each Other

“The story will be a familiar one to many biologist. After publication, rival labs fell over themselves to reproduce the results. Many contacted Verfaillie with requests for the cells and the reagents used to create them, or to ask for more details of the experimental protocol involved. Several high-profile groups sent researchers to Minnesota to learn how to extracted and culture the cells…the procedure takes up to six weeks to master…”

“The Trouble with Replication”

Jim Giles Nature 2006

…Cold fusion (NYT May 1989)

Nathan Lewis (Caltech) “Pons would never answer any of our questions. So we asked Los Alamos National Lab to put our questions to him instead, since they were in touch with him”

Edward Redish (Univ of Maryland) “We have invited Fleischmann to participate in the Baltimore sessions“

“Many speakers at the meeting reported failure in their effort to elicit information or comments from Pons”

Myth 3

Other sciences get equal results

Hard Sciences Get Same Results

Myth If results are truth (not flaw) they get the same

results Variability in Physics experiments results is

very low Reality

Variation of experimental results is also very common in Physics

“Those of us […] know intuitively that there is something ‘softer’ and less cumulative about our research than about those of physical sciences. […] distinguished researchers have cited the pervasive presence of interactions or historical influences as reasons not to expect a cumulative […] science. Still others have cited the low quality of data […] as a barrier.”

“What is surprising is that the research results in the physical sciences are not

markedly more consistent […] data do suggest that results from replicated experiments do not always tend to be consistent”

“The data […] are striking. When all studies are included in the quantitative reviews,

the average Birge ratio is over 2.00, which is 100% larger than expected when studies yield consistent results. Moreover, 6 of the 13 reviews (46.2%) show statistically significant disagreement among studies.”

L. Hedges

How hard is hard science, how soft is soft science? The empirical cumulativeness of research

American Psychologist 1987

Myth 4

They know the relevant variables to control &

Know how to measures them

Myth Mature experimental sciences are able to

perfectly describe experiment variables, know how to perfectly measure response variables, etc.

Reality Even in Physics they can discuss about

interpretation of results or adequate ways of measuring response variables

The Cold Fusion Case After several trials of reproducing Pons&Fleischmann experiment and fail the

assertion was that cold fusion observations were based on experimental errors What type of errors can be done in a Physics experiment?

Using more sensitive equipment. Failing to install a stirring device in the test cell, temperature differences in the cell led to false estimates of its overall heat. This may have suggested that its cell was producing fusion energy So measuring response variables are sometimes not direct or well-defined

Regarding helium traces in P&F experiment: helium is a trace component of air, and the amount of helium in the cell corresponded to what normally enters from the atmosphere. Results can be interpreted differently

Regarding the burst of heat that was observed: Someone turned the current off for a while. When that happens hydrogen naturally bubbles out of the palladium cathode, and creates a hazard of fire or explosion. It is a simple chemical reaction that has nothing to do with fusion. Again results interpretation is not clear

The experiment was flawed because of the system used to measure heat. There were error on temperature measurement. Meyerhof of Stanford University said: “Some scientist put a thermometer at one place and not another”

About Jones replication of P&F had used relatively crude neutron-detecting equipment, and had measured only a very small excess of

neutrons over what could be expected from natural sources without any fusion

Myth 5

Learning vs. verifying

Maturity & Changes

Depending on the maturity of the experiment, the level of similarity required can vary

When an experimental study is young there are many aspects that are unknown. Any change in replication might cause a variation in the results of the experiment impossible to be interpreted

In immature experimental studies begin replication as similar as possible can know the conditions to be controlled

Problems with Identical Replications

SE has tried to make identical repetition of experiments, but no exact replications have yet been achieved

The complexity of the software development setting prevents the many experimental conditions from being reproduced identically

Learning vs. Checking A replication designed to verify results should be exact,

whereas a replication for the purposes of knowledge discovery and learning should be different

Both are necessary. At the beginning of experimental research, equality is not even an option The critical variables to be controlled are now known!

Later on, both knowledge discovery and testing can be more systematic

In the early stages, failure to get the expected results should not be construed as falsification, but as a step towards the discovery of some new factor

Evidence Structure Result

A is more effective than B

Constructs A applied as follows: xxx B applied… Effectiveness measured …

Population

For inexperienced subjects For small programs For these types of faults: omission, …

SE Myths on Replication

Natalia Juristo

Universidad Politecnica de Madrid Spain

www.grise.upm.es/miembros/natalia/