Today:

• Assignment 2 back on Friday• True Experiments: Single-Factor Design• Today’s readings:– The base paper– What did you find in your domain of interest

• Research paper

It’s a matter of controlTrue Experiment

• Random assignment of subjects to condition

• Manipulate the IV

• Control allows ruling out of alternative hypotheses

Quasi Experiment• Selection of subjects for

the conditions• Observe categories of

subjects– If the subject variable

is the IV, it’s a quasi experiment

• Don’t know whether differences are caused by the IV or differences in the subjects

Other features

• In some instances cannot completely control the what, when, where, and how– Need to collect data at a certain time or not at all– Practical limitations to data collection,

experimental protocol

http://www.socialresearchmethods.net/kb/destypes.php

Validity

• Internal validity is reduced due to the presence of controlled/confounded variables– But not necessarily invalid

• It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences– Need to convince self and audience of the validity

External validity

• If the experimental setting more closely replicates the setting of interest, external validity can be higher than a true experiment run in a controlled lab setting

• Often comes down to what is most important for the research question– Control or ecological validity?

Terminology• Factors: Independent Variables (IVs) of an

experiment• Level: particular value of an IV• Condition: a group or treatment (technique)– e.g., Condition 1: old system, Condition 2: new

system

• Treatment: a condition of an experiment• Subject: participant (can also think more broadly

of data sets that are ‘subjected’ to a treatment)

Factors to Treatments• At least 1 Factor (IV) has to vary to have an experiment– Effect of screen size and input technique on performance

(speed, accuracy)• An IV must always have at least 2 levels• Condition refers to a particular way that subjects are

treated– Between subject: experimental conditions are the same as the

groups– Within subjects: only 1 group, that experiences every condition

(can be many conditions in an experiment)– Mixed: some variables are between, some within

Experimental designs Between subjects: Different participants - single

group of participants is allocated randomly to the experimental conditions.

Within subjects: Same participants - all participants appear in both conditions. Takes care of individual differences

Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc. Compromise – groups not likely to be equal, but can

match on the factors you think might most impact results

www.id-book.com 9

Within-subjects• It solves the individual differences issues• But raises other problems:– Need to look at the impact of experiencing the two

conditions– Will they get tired? Gain practice? Learn what is

expected?

• Need to control for order and sequence effects?

Order Effects

• Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?)

• Arises from warm-up, learning, fatigue, etc.• Effect can be averaged and removed if all possible

orders are presented in the experiment and there has been random assignment to orders

Sequence effects

• Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2)

• Effects viewed may not be main effects of the IV, but interaction effects

• Can be controlled by arranging each condition to follow every other condition equally often

Counterbalancing• Controlling order and sequence effects by arranging subjects to

experience the various conditions (levels of the IV) in different orders

• Self-directed learning: investigate the different counterbalancing methods– Randomization– Block Randomization– Reverse counter-balancing– Latin squares and Greco squares (when you can’t fully counterbalance)– http://www.experiment-resources.com/counterbalanced-measures-desig

n.html

Between, within, matched participant design

www.id-book.com 14

True Experiment – Single Factor Design

Images & additional notes text from: http://www.nationaltechcenter.org/index.php/products/at-research-matters/quasi-experimental-study/

Experimental Design: spot the flaw

• One-Group Post-Test-Only Design– Group of subjects are given a treatment (x)• Single factor – only one IV

– Then tested on the dependent variables (observation – o)

• What’s the problem?

Experimental Design: spot the flaw

• Post-Test-Only, non-equivalent control groups– Non-random (N) allocation of subjects into groups– One group is given the treatment, one doesn’t

receive it (different levels to each group)– Post-test: measure the DV

• What’s the problem?

N X O

N O

Experimental Design: spot the flaw• One-Group Pre-Test-Post-Test Design– Single group (within subjects)– Pre-test: measure the DVs– Give the treatment– Post-test: Re-measure the DVs

• What’s the problem?

Two-Group, Pre &Post-Test Design

• Two groups:– Between subjects: random allocation

• Treatment• Pre-test and Post-test: measure the DV

Within-subjects (repeated measures)

• Similar to the one-group pre-test-post-test design• It solves the individual differences issues• But raises other problems:– Need to look at the impact of experiencing the two

conditions– Will they get tired? Gain practice? Learn what is

expected?• Need to control for order and sequence effects

Counter balanced

Determining effect of IV on the DV

• Advantage of single factor design:– Easy to analyze (only one IV)– Fewer conditions (as many as number of levels)

• Simple experimental design

• BUT– Do not know how results would change for other

levels of the controlled variables• Only measuring at one level of the controlled variable• Will the results hold for other levels? Or are there

interactions between the IV and other variables?

Solutions?

• Is repeating the experiment for another level of the controlled variable a valid solution? Why or why not?