ISERN 1

Witch Doctors and Sugar Pills

Barbara Kitchenham

ISERN 2

Subject & Experimenter Impacts

• Human-centered disciplines subject to experimenter & subject induced distortions

– Accepted in medicine, sociology, psychology etc.• Placebos “cure” illnesses

• Patients get better because doctors say they will

• Virtually ignored in Software Engineering Although cause of problems

– Criticism of experiment on formal methods• Published in TSE

• Identified motivation effects as a serious problem for the validity of the experiment

ISERN 3

Distortion & Bias

• Distortion Additional variance due to impact of experimenter

and subject behavior and expectations– May not be able to detect difference among treatments

– May underestimate importance of differences

• Bias Systematic distortion

– Distortions that affect treatments differentially

– May lead to completely wrong results

ISERN 4

Standard Theory – 1/2

• Rosnow and Rosenthal Categorise experiment bias

– Observer Bias• We see what we expect to see

– Interpreter Bias• We code/mark results in favor of our own preconceptions

– Intentional Bias• We might even fudge the results

– Glass ”Some people, I think, are making up numbers”– Conte et al. “Indeed the smooth exponential drop in productivity

exhibited in Table 5.18 and the model in Equation (5.14) give us some cause for concern. It is simply too smooth to represent real-world projects”

ISERN 5

Standard Theory – 2/2

• Experimental artifacts that cause distortions Biosocial, Psychological, Situational, & Modeling

effects– Effects due to attributes of the experimenters, interactions

with subjects and the research setting

Experimenter Expectancy Bias– Have a real problem if we evaluate our own technologies

• Have a responsibility to evaluate our own work

• Cannot avoid possibility of bias

ISERN 6

Experimenter expectancy

• Rosnow & Rosenthal suggest 6 approaches to minimise this problem Most important are:

– Replication

– Blinding when ever possible

– Reducing contact between subject & experimenters

ISERN 7

Replication

• Cannot afford replication to be too exact Source of systematic bias can be replicated

• Good replication practice Use the same hypotheses Use different experimental protocols

– Parallel v. sequential v. mixed Use different materials and tasks

– Artificial v. Industrially derived Use different subject types

– Students v. Practitioners Undertaken by independent research groups

• See Miller, 2004

ISERN 8

Assist Replication

• Identify the “real” validity issues Problems not resolved by experimental protocol Remaining limitations e.g.

– Are the treatment and control conditions appropriate?• Represent the way that the technique is used/will be used in

industry– Are other treatment conditions important?

• Ensure that infrastructure issues are clearly reported

– Are the training requirements and pre-requisites clear?– Are any necessary tools/ development environments

available to other researchers?

ISERN 9

Discussion Issues

• How do we improve experimental protocols to address subject-experimenter expectations?

• How do we improve experimental reporting to assist replication?

• How do we make replication a part of our research culture?

• How do we perform high quality replication studies? What are the most appropriate protocols?

• What other disciplines should we look to for advice?

ISERN 10

References

• Conte, S.D., Dunsmore, H.E. and Shen, V.Y. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company Inc., 1986.

• Robert L. Glass. Good numbers – And bad. HSS 44, 1998, pp85-86.

• Ralph L. Rosnow and Robert Rosenthal. People Studying People Artifacts and Ethics in Behavioural Research. W.H. Freeman & Co, New York, 1997.

• James Miller. Replicating software engineering experiments: a poisoned chalice or the Holy Grail  Information and Software Technology, Volume 47, Issue 4, 15 March 2005, Pages 233-244