Today:

• Our process• Assignment 3 Q&A• Concept of Control• Reading: Framework for Hybrid Experiments• Sampling• If time, get a start on True Experiments:

Single-Factor Design

Assignment 2 problems?• Did you have trouble with IV/DV identification?• Did you provide enough detail to really answer the

questions?• Were you able to fully operationalizing the concept of the

DV into the actual data that will be measured?• Discrete/continuous (for the measurement)• Scaling types• What are descriptive statistics• Which descriptive statistics are appropriate for the

different types of data • Estimation of reliability/validity of measurement (not the

experiment)

Where we are in the process

• Have defined broad research question and identified hypotheses or very refined research questions that must be evaluated to answer the question (Ass 1)

• Have identified the variable that need to be manipulated/studied in order to evaluate the hypotheses and have thought about what kind of data will need to be measured (Ass 2)

Assignment 3• Start thinking about threats to validity – what will you have

to justify in order to convince readers that you have a valid study that addresses the research question– Note: as of yet, you don’t have a study designed. But you

should be thinking about possible study designs that might be appropriate

• Ass. 2 was descriptive stats (individual variables) – in Ass. 3, you need to think about which comparisons you need to make– Different measurements, single variable?

• Satisfaction before/after treatment?– Correlations between different variables?

• Satisfaction as it relates to a participant’s age?– Focus on describing the data – what tables/graphs would you

use to show the relationships

Assignment 3: Q&AFor five of the threats to EXPERIMENTAL validity covered in class, give an example of a possible threat to validity for your research question. •Note: as you have not yet designed your experiment, this exercise is to identify possible threats under a possible experimental design.• You can use different possible experimental designs for each threat. What comparisons should you make to explore the answers to your research question? •That is, which of the variables you identified in Assignment 2 should be analyzed individually and which variables should be compared against each other (which do you think might have interactions)? •For each variable, specify what tabular and graphical methods you would use to describe the data.

Later (starting to think about a study design):

• Assignment 4: Hypothesis testing & Control– Starting to think about statistical evaluation of the

hypotheses:• Stating alternate and null hypotheses• Is each hypothesis one tailed or two tailed?• If there was a type I error, what would be the

conclusion statement based on the error? Type II? • Where you will get the data• How you will rule out alternate explanations for the

results (give an alternate & show why it’s not viable)– The limitations to your study

Term paper: Ongoing action items

• Look at related work:– What papers will you cite when writing your

research paper – Papers to motivate the problem– Papers that have used similar methodologies to

answer similar types of questions– Papers that establish baselines in your area– Papers about methodologies– Papers that you should compare your results with

(if you actually end up running the study)

Term paper: Ongoing action items

• Think about what kind of study is needed to be able to answer the research questions/hypothesis that you have identified– How many factors are you looking at?– How much control do you need?– What are your biggest validity concerns that need

to be addressed?– What is the usual approach in your discipline?– …

The research paper

• We’ll review the full details next week• Proposing a study to evaluate your research

questions• The assignments are the setup to get you

thinking about the details• In the paper, you will be writing them formally• In the presentation, you will be presenting them• Your ability to discuss alternatives and justify

your chosen approach will be key to a good mark

Control

• Basic idea: isolate the effect of the independent variable on the dependent variable– Ensure there are no reasonable alternative causes

True Experiments

True Experiment

• Experimental control• Control as many potential threats to validity

as possible• Random assignment of participants/data to

conditions• Could be within-subjects or between-subjects

Control

• True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects– Control over the who, what, when, where, how

• Control of the who => random assignment to conditions– Only by chance can other variables be confounded

with IV• Control of the what/when/where/how => control

over the way the experiment is conducted

Quasi-Experiment

• When you can’t achieve complete control– Lack of complete control over conditions– Participants/elements for different conditions

come from potentially non-random pre-existing groups

Control

• Basic idea: isolate the effect of the treatment on the dependent variable– Ensure there are no reasonable alternative causes

• Basic design: control vs. experimental group• Setting: – Field vs. lab– Experiment vs. non-experiment

Today’s reading

A taxonomy and evaluation for developing 802.11-based wireless mesh network testbeds. •Discussion of what is being controlled (and how)•What have you found in your own research domain? How much control is required? How clearly are populations, experimental conditions, and sampling methods and other experimental decisions defined?

Controlled studies

Controlled Experiments

• Predict the relationship between two or more variables.

• Independent variable(s) manipulated by the researcher

• Dependent variable(s) depends on the independent variable(s)

• Typical experimental designs have one or two independent variables.

• Validated statistically & replicable.

www.id-book.com 18

Control• True experiment = complete control over the

subject assignment to conditions and the presentation of conditions to subjects– Control over the who, what, when, where, how

• Control of the who => random assignment to conditions– Only by chance can other variables be confounded with

IV

• Control of the what/when/where/how => control over the way the experiment is conducted

Sampling

Allocation (sampling)

• Probabilistic:– Simple random sample, systematic sample,

stratified random sample, cluster sample– Video describing and contrasting– http://www.youtube.com/watch?v=be9e-Q-jC-0

• Non-Probabilistic– convenience sample– purposive sample (choosing to meet an identified

need)

Sampling in the broader sense• Simple random sample, systematic sample,

stratified random sample, cluster sample• Convenience sample, purposive

• Discuss:– Elements: Data… Participants…. Tasks… Times…– Will you be able to work with the whole population/all

possible scenarios? Or will you have to sample?– What kind of populations will you be dealing with?