AS 3.12, Externally assessed, 4 creditsExperiments and Observational Studies

Evaluation of Statistical Reports

Claims made in Newspapers every

day – do you believe them?

Do drugs have an adverse effect on teenage brains?

Does listening to music help students study?

What acne treatments are the most effective?

What information would you need to answer these questions?

Who, What, Where, Why and How ? Identify in an article

Article Structure1. Introduction – Broad summary of story or

main finding. Slightly more detail than provided in the heading.

2. Main angle developed – further details provided

3. More on main angle4. Strongest Quote5. More details such as results of other

studies or quotes from other experts or witnesses

6. Background information

What is an Experiment?

• Participants randomly allocated to treatments

• Variable of interest is measured and results compared between treatment groups

Identify which are EXPERIMENTS ?• Study which compares pulse rates of students with and

without exercise• Study that compares performance in an IQ test with and

without caffeinated drinks• Study of teenage binge-drinkers’ brain activity• Impact of smoking during pregnancy on infant performance

ability• Effect of birth weight on adult academic performance• Effect of acupuncture or physiotherapy on back pain• Effect of exercises on age babies start walking• Study of breakfast habits and study results• Study of massage as a cure for migraines• Are food colours linked to hyperactivity in children?• Are defensive drivers better drivers?• Effect of sleep deprivation on driving ( in a simulator)

KEY TERMINOLOGY

• Response Variables – the variable measured as the outcome of an experiment. Measure blood pressure to assess effect of new drug.

• Explanatory Variables – the variable that attempts to explain differences in the response variable

• Treatment- one or a combination of explanatory variables assigned by the researcher

• Confounding Variables – variables connected to the explanatory variable that may be the actual cause of the differences in the response variable.

CONFOUNDING VARIABLES

“ Imported lemons reduces highway fatalities – study reveals”

Identify Explanatory, Response Variables and Treatment in following studies• Study which compares pulse rates of students with and

without exercise• Study that compares performance in an IQ test with and

without caffeinated drinks• Study of teenage binge-drinkers’ brain activity• Impact of smoking during pregnancy on infant performance

ability• Effect of birth weight on adult academic performance• Effect of acupuncture or physiotherapy on back pain• Effect of exercises on age babies start walking• Study of breakfast habits and study results• Study of massage as a cure for migraines• Are food colours linked to hyperactivity in children?• Are defensive drivers better drivers?• Effect of sleep deprivation on driving ( in a simulator)

How to design a good experiment• Random Allocation – participants randomly allocated

to treatments and random order of treatment• Use of a Control Group – use to establish base line

effect• Use of a Placebo – dummy, existing or no treatment• Use of Blinding – single and double blinding

What is an Observational Study?A study which observes differences in the explanatory variable and then assesses whether these differences are related to differences in the response variable

Famous NZ Observational Study is the Dunedin Multidisciplinary Health and Development Study (DMHDS) .A study of 1037 children who were born in Dunedin, in 1972-73. Where they now live shown below.

Potential Problems with Observational StudiesConfounding Variables

“Happy People Live Longer”

PROBLEM – Causal Claim cannot be made by Observational Study

Possible confounding variables?

Why is this an observational study?

Potential Problems with Observational Studies

Extending Results inappropriately

“WOMEN HIT HARDER BY HANGOVERS”

The study observed only undergraduate students at one University so results cannot be extended to cover ALL WOMEN

Potential Problems with Observational StudiesUsing the past as a source of data

“LEFT HANDED PEOPLE DIE YOUNGER”

In early part of twentieth century children forced to write with right hands, hence greater numbers of right-handers.Confounding variables can change over time.Recalled information often unreliable.

COMPARISON OF EXPERIMENTS AND OBSERVATIONAL STUDIES

Experiments Observational StudiesExplanatory Variable specified Explanatory variable specifiedResponse variable specified Response variable specifiedParticipants randomly allocated to treatments

Participants NOT randomly allocated to treatments, sometimes because it is unethical to do so

Treatment is the only factor that is varied.  

Changes in response variable measured Changes in response variable observedCAUSAL CLAIM CAN BE MADE CAUSAL CLAIM CANNOT BE MADE  Claim can be made about study participants or

similar group ONLY

Good experiments include random allocation to treatments, control groups, placebos and use blinding

 Good observational studies acknowledge & account for all potential problems

EVALUATION OF STATISTICALLY BASED REPORTSStep One – Is report based on sound research?

Step Two – Identify type of study – experiment or observational?

Step Three – Who funded research? Did they have vested interest in outcome?

Step Four – Where reliable data collection methods used?

Step Five – Who or what was studied? How were they selected?

EVALUATION OF STATISTICALLY BASED REPORTS

Step Six – Are measurements provided?Step Seven – Familiar or unfamiliar settings?Step Eight – Size of observed effect quantified or just described?Step Nine – Confounding variables?Step Ten – Results extended appropriately?Step Eleven – Any missing data?Step Twelve – Do findings make sense?

EVALUATION OF STATISTICALLY BASED REPORTS

Does your evaluation persuade you or someone you know to modify their behaviour? What is your overall conclusion?