Researching medical educationImperial School of Medicine

Faculty Teaching Forum

2 May 2012

Dylan Wiliam

Pasteur’s quadrant

Educational research“An elusive science” (Lagemann)A search for disciplinary foundations

Making social science matter (Flyvbjerg, 2001) Contrast between analytic rationality and value-rationalityPhysical science succeeds when it focuses on analytic rationalitySocial science

fails when it focuses on analytic rationality, but succeeds when it focuses on value-rationality reasonableness, rather than rationality (Toulmin) as the key criterion

Research methods 101: causalityDoes X cause Y?

Given X, Y happened (factual) Problem: post hoc ergo propter hoc

If X had not happened, Y would not have happened (counterfactual) Problem: X did happen

So we need to create a parallel world where X did not happen Same group different time (baseline measurement) Need to assume stability over time Different group same time (control group) Need to assume groups are equivalent Randomized contolled trial

Plausible rival hypothesesExample: Smoking cigarettes causes lung cancer

Randomized controlled trial not possible Have to rely on other methods Logic of inference-making Establish the warrant for chosen inferences Establish that plausible rival interpretations are less warranted

Criteria for causal inferencesThe environment and disease: association or causation? (Hill, 1967)Criteria for determining a causal association:

1. strength

2. consistency

3. specificity

4. temporality

5. biological gradient

6. plausibility

7. coherence

8. experimental evidence

9. analogy.

KnowledgeNot justified-true-belief

Discriminability (Goldman, 1976)

Elimination of plausible rival hypotheses

Building knowledge involves:marshalling evidence to support the desired inferenceeliminating plausible rival interpretations

‘Plausible’ determined by reference to a theory, a community of practice, or a dominant discourse

Inquiry systems (Churchman, 1971)System Evidence

Leibnizian Rationality

Lockean Observation

Kantian Representation

Hegelian Dialectic

Singerian Values, ethics and practical consequences

The Lockean inquirer displays the ‘fundamental’ data that all experts agree are accurate and relevant, and then builds a consistent story out of these. The Kantian inquirer displays the same story from different points of view, emphasising thereby that what is put into the story by the internal mode of representation is not given from the outside. But the Hegelian inquirer, using the same data, tells two stories, one supporting the most prominent policy on one side, the other supporting the most promising story on the other side (Churchman, 1971 p. 177).

Inquiry systems

The ‘is taken to be’ is a self-imposed imperative of the community. Taken in the context of the whole Singerian theory of inquiry and progress, the imperative has the status of an ethical judgment. That is, the community judges that to accept its instruction is to bring about a suitable tactic or strategy [...]. The acceptance may lead to social actions outside of inquiry, or to new kinds of inquiry, or whatever. Part of the community’s judgement is concerned with the appropriateness of these actions from an ethical point of view. Hence the linguistic puzzle which bothered some empiricists—how the inquiring system can pass linguistically from “is” statements to “ought” statements— is no puzzle at all in the Singerian inquirer: the inquiring system speaks exclusively in the “ought,” the “is” being only a convenient façon de parler when one wants to block out the uncertainty in the discourse. (Churchman, 1971: 202).

Singerian inquiry systems

Educational research…can be characterised as a never-ending process of assembling evidence that:particular inferences are warranted on the basis of the available evidence;such inferences are more warranted than plausible rival inferences; the consequences of such inferences are ethically defensible.

The basis for warrants, the other plausible interpretations, and the ethical bases for defending the consequences, are themselves constantly open to scrutiny and question.

Effects of feedbackKluger & DeNisi (1996)

Review of 3000 research reports

Excluding those:without adequate controlswith poor designwith fewer than 10 participantswhere performance was not measuredwithout details of effect sizes

left 131 reports, 607 effect sizes, involving 12652 individuals

On average feedback does improve performance, butEffect sizes very different in different studies38% (50 out of 131) of effect sizes were negative

Getting feedback right is hardResponse type Feedback indicates performance…

exceeds goal falls short of goal

Change behavior Exert less effort Increase effort

Change goal Increase aspiration Reduce aspiration

Abandon goal Decide goal is too easy Decide goal is too hard

Reject feedback Feedback is ignored Feedback is ignored

Kinds of feedback (Nyquist, 2003)Weaker feedback onlyKnowledge or results (KoR)

Feedback onlyKoR + clear goals or knowledge of correct results (KCR)

Weak formative assessmentKCR+ explanation (KCR+e)

Moderate formative assessment(KCR+e) + specific actions for gap reduction

Strong formative assessment(KCR+e) + activity

Effects of formative assessment (HE)Kind of feedback Count Effect/sd

Weaker feedback only 31 0.14

Feedback only 48 0.36

Weaker formative assessment 49 0.26

Moderate formative assessment 41 0.39

Strong formative assessment 16 0.56