Evaluation Research Step by StepStep 1 – Formulate Research Question

Step 2 – Operationalize Concepts◦Valid and Reliable Indicators

Step 3 – Decide on Sampling Technique◦Draw Sample

Step 4 – Select Data Collection Technique◦Collect Data

Step 5 – Analyze Data

Step 6 – Write up the Report

Evaluation Research Step by StepSteps 1 and 2 have already been

covered in previous power points.

Step 3 - Drawing A Sample Identify Population

Decide on either Random (probability) or Nonrandom (nonprobability) Sampling Technique

SamplingTerms to Know

◦Population: Total group to be studied◦Case: Units to be sampled

(individual, county, etc.)◦Sample: Group to be interviewed◦Sampling Frame: List from which we

obtain the sample (a roster from a class, or a directory from

an organization, etc.)

SamplingTypes of Sampling Designs

◦Simple Random Sampling Single number assigned to each case Table of random numbers is used Assumes a complete list of the population Laborious

◦Systematic Sampling Every “x”th element from a sample frame is selected

Determine size needed Take random start Select every “x”th element

◦Probability Proportionate to Size (PPS) Larger ones (clusters) have greater probability of being

selected

SamplingTypes of Sampling Designs

◦Stratified Sampling Sample separately from different key

categoriesExample:Determine number needed from each racial/ethnic group (this number should be approximately the same for each group).

Total # # NeededAfrican Americans 300 100Hispanics 200 100Asian Americans 100 100Anglo Americans 500 100

Total Population 1000 400

Sample separately from each group taking proportion needed to get approximately equal numbers from each racial/ethnic group

African Americans Need 50% sampleHispanics Need 50% sampleAsian Americans Need 100% sampleAnglo Americans Need 20% sample

SamplingTypes of Sampling Designs

◦ Multistage Cluster Sampling Used when don’t have a complete list Sample in stages (units within units)

Example of multistage random sample:

(1) Universities

USU U of U BYU

(2) Colleges

Engineering Agriculture

(3) Departments

Landscape, Architecture and Environmental Planning (LAEP)

Plants, Soils and Climate Department (PSC)

(4) Classes

Theory of Design

Landscape Materials

(5) Students

MollyTrish

Jim

Determining Sample SizeAppendix 2 Recommended sample sizes for two different precision levels

Population Size

10 10 275 163 7415 14 300 172 7620 19 325 180 7725 24 350 187 7830 28 375 194 8035 32 400 201 8140 36 425 207 8245 40 450 212 8250 44 475 218 8355 48 500 222 8360 52 1000 286 9165 56 2000 333 9570 59 3000 353 9775 63 4000 364 9880 66 5000 370 9885 70 6000 375 9890 73 7000 378 9995 76 8000 381 99100 81 51 9000 383 99125 96 56 10,000 385 99150 110 61 15,000 390 99175 122 64 20,000 392 100200 134 67 25,000 394 100225 144 70 50,000 397 100250 154 72 100,000 398 100

Sample Size

+/- 5 % +/- 10%Population Size

Sample Size

+/- 5 % +/- 10%

N

D2

.9604N + 1

=

Nonrandom Sampling Availability or Convenience Samples

Select those who are available or convenient Quota Samples - select representative size of different types of

respondents

Purposive Samples Need specific types of information – select those who will have

that information

Theoretical Samples Theory specifies characteristics of your units of analysis. For

instance, a theory of inequality may require you to have respondents at different levels of income.

Snowball Samples Hard to Reach – Hard to identify populations

Nonrandom – Nonprobability SamplingAvailability or Convenience Samples

Sample classrooms of students here on campus because they are available and convenient

Availability or Convenience Using QuotaSample classrooms of students here on campus

because they are available and convenient, BUT

Select classes with approximately equal numbers of men and women because there are approximately equal numbers in university

Nonrandom - Nonprobability Sampling Purposive Samples

Select Key Informants who can provide information you need If you want to look at community leader’s attitudes

toward homeless then you would selectLaw enforcementCivic leaders/policy makersHeads of social service agencies

Three GuidelinesKnowledge about the culture/situation or experienceWilling to talkRepresentative of the range of points of view

Two TestsCompleteness – overall sense of what they thinkSaturation – not hearing anything new

Nonrandom – Nonprobability Sampling Theoretical Samples

Theory specifies characteristics of your units of analysis

Example – Research on parent/teacher interaction using Expectation States Theory.

Research Questions - How do parents and teachers interact? How is interaction influenced by race and social class of parent? How are expectations influenced by socioeconomic characteristics of parents.

Theory specifies that as inequality between parent and teacher increase, so does “expert power” of teacher – and conformity of parent.

NonRandom- Nonprobability Sampling Snowball Samples

Key Informant or GATEKEEPER who then refers you to someone else

Sample size “snowballs” or increases rapidly.

Used when you are working with groups that are reluctant to be studied, and you need to gain confidence of group members.

Example of groups where snowball samples are used– Gang members, Alcoholics anonymous members, etc.

Ways that People Generally GatherInformation – Data Collection Techniques

Participant Observation Content Analysis Surveys/Intakes/Questionnaires

This excludes non-empirical ways of collecting information (intuition, inspiration, 6th sense etc.)

Types of Data Collection Techniques Paired with Most Likely Sampling Technique

Participant Observation – generally uses nonrandom sample.

Content Analysis – generally uses nonrandom sample.

Surveys/Intakes/Questionnaires – generally uses random sample.

This excludes non-empirical ways of collecting information (intuition, inspiration, 6th sense etc.)

Contact InformationDr. Carol AlbrechtAssessment Specialist979-777-2421Carol.albrecht@usu.edu