Study Design

Research Methods Professional Development Institute

Kali TrzesniewskiDecember 4, 2015

Study

DesignData

Collection Method

Study

DesignData

AnalysisMethod

Study

Design

A Blueprint

Before you start planningNeed to know where you are going

If you were going to build a hotel

BEFOREdeciding the look and function of the hotel or making a blueprint

buy

materialsMake an action plan

So, don’t start working on

• Measures• Sample• Design (e.g., cross-sectional, experimental)• Tasks (e.g., activities)• Etc

Until you have a clear goal

What type of evidence will allow

me to reach my goal in a

convincing way?

Study Design Function

The function of a study design is to ensure that the evidence obtained

enables us to answer the initial question as unambiguously as

possible

So, when designing a study, ask:

Given my research question, what type of evidence is needed to

answer the question in a convincing way?

Keep in Mind

• Part of answering your research question in a convincing way…

• is addressing alternative, rival explanations

Types of Research Designs

• Descriptiveo Documents

frequencies or levels

• Correlationo Tests relations

between two variables

o Focused on individual differences

o May be cross-sectional or longitudinal

• Experimento Controlled

context to test causal impact

o Focused on group differences

o Requires random assignment to control for individual differences

• Quasi-Experimento Naturally

occurring experiment

o Weaker causal impact

o Focused on group differences

o Requires control group, but not random assignment

Types of Research DesignsTime

in EFNE

PAmount of

Vegetables

Consumed

Re-enroll in 4-H

Participation in County

Fair

Better

Goal Settin

gRandomly

assigned to 4-H Thrive!

Bad Okay Great!02468

1012

How was the Research Methods Institute?

Research Design Strengths and Limitations

• Descriptiveo Quick and easyo Interpretation not always clear

• Correlationo Quick and easyo Cannot inform causation

Research Design Strengths and Limitations

• Experiment o Can inform causation o Harder to doo No one wants to be the controlo Might not reflect real life

• Quasi-experimento Easier than experimento Cannot make strong causation conclusionso With statistical adjustments (e.g., propensity matching)

causation can be suggested

The design you use will determine the conclusions you

can make

Research Question:Is my program effective?

Hypothesis:Youth who participate in

my program will make better food choices

• Do youth increase on self-reported food choices pre to post?

• Do youth who have participated in my program make better food choices than youth who have not participated in the program?

• Does my program change the food choices youth make?

Operationalizing your hypothesis

• Research design: correlation• Conclusion: program resulted in better food

choices• Why people might not trust the conclusion:

change might reflect naturally-occurring growtho Need a control sample to show change isn’t

happening in a similar population of non-participants

Do youth increase on self-reported food choices pre to post?

• Research design: quasi-experiment & descriptive

• Conclusion: program resulted in better food choices

• Why people might not trust the conclusion: youth who ended up in program might have always had better food choiceso Need random assignment or statistical controls

Do youth who have participated in my program make better food choices than youth who have

not participated in the program?

• Research design: experiment• Conclusion: program resulted in better food

choices• Why people might not trust the conclusion:

people generally trust experimentso Questions usually revolve around generalizability

and scaling up

Does my program change the food choices youth make?

Choosing Which Design to Use

What do I want to learn?What will I want to report?• Participants, leaders, community members,

funders, etc.What might I want to know for program refinement?What will my take-home message be?What figures or tables will I want?

Jump to the End

What can I feasibly do?

Common Mistakes in Research

• Insufficiently motivated research questionso Is this something people other than me care

about?o Will it generate new knowledge?o Does it address a real problem?o Does it impact a sufficient portion of the

population?

Common Mistakes in Research

• Unresearchable problemso Can this question be answered through ethically

and valid research practices?o Is this feasible for me to do?

• Favored research methodso Don’t design a study around a research method

• Blind data miningo Don’t collect data then figure out what to do with

it

Go forth and research!