Cost Effective Evaluation Strategies

Robin Kipke

Tobacco Control Evaluation Center

rakipke@ucdavis.edu

November 19, 2009

Topics We’ll Cover

Adapting existing instruments Saving time in data collection Tapping into volunteer labor Using TCEC resources

Caveat

TCEC is not advocating for cutting evaluation corners that result in a decline of rigor or quality of data!

Cost-cutting strategies should only be used as the context of other factors allow

Always think first about what data will be convincing to your target audience

Adapt Existing Instruments

Not always necessary to reinvent the wheel

Adapt other data collection instruments for your project’s purposes

Start with high quality instrument Add questions you need data on Delete unnecessary questions Make sure it will collect what you need

Instrument Sources

TCEC website has model instruments on limited number of objectives

Contact TCEC to access repository of instruments created/used by other projects

Ask other projects working on similar objectives to share their instruments

Statewide Sites

Download instruments from statewide sites CA Health Interview Survey –

http://www.chis.ucla.edu/questionnaires.html CA Healthy Kids Survey –

http://www.wested.org/cs/chks/view/chks_s/17?x-layout=surveys

CDC – http://cdc.gov/tobacco/data_statistics/surveys/

YTPS on STORE Website – http://www.tcsstore.org/stages/1_document/t_yps.pdf

Saving Time on Data Collection

Ask to have your questions added to data collection activities being conducted by other departments Pay attention to what’s going on within

your county or agency Find out when other projects/divisions will

be collecting data Learn which population groups will be

targeted by data collection and see how that fits your needs

Adding Your Questions

Obtain a copy of the instrument to see what questions are already being asked

Insert your most important questions Add transitions as needed to preface your

question topics Examples:

County’s annual Healthy Kids survey Maternal & Child Health survey (every 5 years) Needs assessments (use of Prop. 63 funds, adult

health survey)

Some Limitations

Think about data sources – who will be reached/excluded?

Will not be a pre-/post measure of same people, so cannot measure intervention effect

Will be limited to space allowed Access to analysis – may need raw

data

Saving Time on Data Collection

Distribute questionnaires/materials via high traffic locations Back-to-school packets Church service programs Grocery store flyers Classroom teachers Entertainment tickets (e.g. movie theaters) Organizational newsletters

Other Distribution Avenues

Online surveys (Survey Monkey) Invite participation via email, social

networking media (Facebook, Twitter) Your organization’s website (and those

of coalition members, agencies that serve target population)

Other Distribution – Limitations

With each distribution outlet, think about who will it reach/not reach?

How to address representativeness Use screening questions to avoid

duplication Loss of control over administration Potentially lower response rate

Saving Time on Data Collection

When low population density makes collecting a sufficient POS sample difficult, a series of focus groups (with purposive samples to ensure representativeness) may suffice

Caveat: First find out if the data will be convincing enough for your purposes

Tapping into Volunteer Labor

Develop relationships with local universities, high schools, programs or clubs

Recruit students looking for research experience to do data collection

Offer training, internships, course credit, other incentives

Tapping into Volunteer Labor

Make use of community volunteers and coalition members To field test instruments/materials for

appropriateness to data sources To translate instruments/materials into

other languages To collect data To provide access to other distribution

points

Utilize TCEC Resources

Online coalition satisfaction survey administration and analysis service

Additional model data collection instruments on a variety of topics will be developed in coming months

Model instruments will be translated into other languages (Spanish, Vietnamese, Hmong and Russian) depending on need

We’re working with other statewide projects to develop tools that will help you use and share your data more effectively for your policy work

Contact TCEC

For individualized assistance with: evaluation planning development of data collection

instruments advice on analyzing & interpreting data guidelines for reporting & sharing info