and the survey says…. putting your healthy youth survey results to work
DESCRIPTION
And the Survey Says…. Putting YOUR Healthy Youth Survey Results to Work. Spring 2003. Purpose: Help people understand & use their HYS reports. Welcoming picture drawn by a Healthy Youth. Welcome!. Introduction of presenters How many agencies are here today? Review Agenda - PowerPoint PPT PresentationTRANSCRIPT
And the Survey Says….Putting YOUR Healthy Youth Survey
Results to Work
Spring 2003
Purpose: Help people understand & use
their HYS reports
Welcome!
• Introduction of presenters
• How many agencies are here today?
• Review Agenda
• What we will have accomplished by the end of the day?
Welcoming picture drawn by a Healthy Youth
Ice breaker
Ice Breaker
• In small groups, each person shares – Name
– Where you work
– Share one interesting or obscure fact about yourself
Background of HYS 2002
History of WA survey efforts
Content of HYS 2002
Sources of HYS 2002 items
Data processing and quality control steps
Past Washington Surveys: Primary Content
‘02‘01‘99‘97‘96‘95‘94‘93‘89‘88 ‘00‘92‘90 ‘91 ‘98
Health-risk Focused
Risk & Protective Factor Focused
Combination
Joint Survey Planning Committee
• Office of the Superintendent of Public Instruction
• Division of Alcohol & Substance Abuse, DSHS
• Department of Health
• Office of Community Development
• Governor’s Family Policy Council
• RMC Research (survey contractor)
HYS 2002: Collaborative Effort of OSPI, DOH, DSHS, and CTED
• Simple random sample of schools recruited at state level– County samples drawn as appropriate
• Survey consistently administered in the Fall of even years
• Survey given to 6th, 8th, 10th, and 12th graders
• Survey booklets will have one-page tear-off answer sheet
• Survey for 8th, 10th, and 12th graders will use 2 form “interleaved” administration
• The Healthy Youth Survey 2002 was completed…
HYS 2002: Participation
By 137,335 students…
In 752 schools…
In 203 school districts…
In 39 counties.
Risk & Protective Factor DataConsistent with CTC/MTF SurveyDevelopment led by DASA/DSHS & OSPI
WSSAHB Questionnaire• Demographics• Alcohol, Tobacco, other drugs• School risk/protective factors• Community risk/protective factors• Peer-Individual-Family risk/ protective factors
Youth Health Behavior DataConsistent with National YRBS & YTSDevelopment led by DOH
YRBS Questionnaire• Demographics• Alcohol, Tobacco, Other drugs• Nutrition• Physical Activity• Safety Behaviors (Helmets, Seatbelts)• Mental Health/Depression/Suicide• Additional Tobacco Indicators
Getting the Best of Both
Form AWSSAHB-
like
Form BYRBS-likeSurvey
Core
6th Grade Survey
One Form
SurveyCore
+ Mutually Agreed Items
Core Survey Items
• Demographics• Alcohol, Tobacco and Other Drug Use Items• Key Violence-Related Items• School-Specific Asset Items• Depression
Sources of HYS 2002 Items
• Monitoring the Future (MTF)
• Youth Risk Behavior Survey (YRBS)
• Communities that Care (CTC)
• Youth Tobacco Survey (YTS)
• PRIDE Survey
Data Processing and Quality Control
• Administration procedures• Data cleaning procedures
Statistical Issues
Validity and reliability
Confidence intervals
Comparing state and local results
Generalizability
Validity and Reliability
• Definitions• How we assure validity: Items from established
instruments, validity checks• How we assure reliability: Standardized
administration procedures
Confidence Intervals
• What are they?• How to interpret the numbers • How to interpret the graphs• Practice activity (Ex. 1–2)
What is a Confidence Interval?
• The reported value is probably a little different than the “true” value for all your students
• We are “95% confident” that the “true” value is within the +/- range, called the “confidence interval”
Interpreting CIs Numerically
Smoked cigarettes: 9.2% (± 1.1%)
9.2 – 1.1 = 8.1
9.2 + 1.1 = 10.3
Between 8.1% and 10.3% smoked cigarettes
Interpreting CIs Graphically
0 5 10 15 20
Smokedcigarettes
Percent of students
Practice Activity
• Exercises 1 and 2
Why Are Confidence Intervals Different Sizes?
• Number of students• Inherent variability• Level of confidence (All 95% for HYS)• Sampling design
If the data are valid, why do we need confidence intervals?
• Confidence intervals account for variability among students, NOT validity of the data
• Variability is inherent in any population worth studying
• Variability causes uncertainty in the results• “Doubt is not a pleasant condition, but certainty is
absurd” – Voltaire
Comparing State and Local Results
• Looking at differences numerically• Looking at differences graphically• Practice activity (Ex. 3–4)
Significant Differences: Numerically (Ex. 1)
• Smoked cigarettes in the state: 9.2% (± 1.1%)
Between 8.1% and 10.3%
• Smoked cigarettes at my school: 14.1% (± 2.22.2%)
Between 11.911.9% to 16.316.3%
• Conclusion: Difference IS statistically significant
Significant Differences: Numerically (Ex. 2)
• Smoked cigarettes in the state: 9.2% (± 1.1%)
Between 8.1% and 10.3%
• Smoked cigarettes at my school: 14.1% (± 9.69.6%)
Between 4.54.5% to 23.723.7%
• Conclusion: Difference is NOT statistically significant
Significant Differences: Graphically (Ex. 1)
0 5 10 15 20 25
Smokedcigarettes
Percent of students
Local State
Significant Differences: Graphically (Ex. 2)
0 5 10 15 20 25
Smokedcigarettes
Percent of students
Local State
Practice Activity
• Exercises 3 and 4
Generalizability
• What is generalizability?• To whom can we generalize results?
• State & County sample
• Non-sampled counties, districts, and ESDs
• Schools
• Why 70% participation is required
Quotations From Teachers Who Administered the Survey
• About 25 students missed part of the session because of club photos
• Absent students are taking a PSAT test• Many students had too much homework. Chose to
do that rather than take the survey.• Seven students came too late to take the survey• The perfectionists, poor direction followers and
slow readers could not finish
Challenges to Generalizability
• Brainstorm challenges to generalizability – School level
– ESD, County, or District level
• How do these challenges affect data interpretation?
What if I don’t want to generalize?
• Confidence intervals matter if you want to generalize to a larger population
• You can describe the students surveyed, in that moment, without confidence intervals:– “Elmo Elementary School students who took the survey
said…”
Final Note About Significance
• Even if a difference is statistically significant, it might not be practically significant
• Example– State alcohol use: 12.8 (±0.2) %
– Local alcohol use: 14.4 (±1.2) %
Statistically significant difference, but should it influence program planning?
Comparing data over time
Things to think about
• Did the questions change?• Were the challenges to generalizability similar?• Is there a reason to think that things would have
changed? (a prevention program)
Surveys over time
• WSSAHB 2000 reports did not report confidence intervals – but the variability was still there!
• Future HYS reports will have confidence intervals, so 2002 vs. 2004 comparisons will be easier
What should I know about comparing 2000 to 2002 data?
• If you think you have comparable data, then:– If the 2000 percentage is INSIDE the 2002 confidence
interval, then there is not a “statistically significant” difference
– If the 2000 percentage is OUTSIDE the 2002 confidence interval, then you can’t be sure if the difference is “statistically significant” but the further away it is the more likely that a difference is significant
– If you really, really need to know whether it’s significant, you can contact DOH for a specialized data run
What if I am in a small school, and we have huge confidence intervals?• Having a confidence interval protects you (and
your program) from appearing to be ineffective when just a few students can make big changes
• Consider the input of teachers & staff from small-school environments when interpreting data – the data should be used to complement what they already know about their kids
Break
Local Report Overview
Guide to the Local Report
FAQ
How to use your local report
Local Report Contents
• Introduction and Overview– Key to the Notes
– Highlights of the Local Results
• Graphical summary of selected results• Selected results disaggregated by gender• Responses to all items
Guide to the Local Report, cont.
Healthy Youth Survey 2002Survey Results
Sample Middle School, Grade 8
Number of students surveyed:
Number of valid responses:
64
59
The impact of adolescent health risk behaviors remains a primary concern of citizens throughout the country. Many health problems experienced by adolescents are caused by a very few preventable behaviors….
The number of students who submitted survey forms
Guide to the Local Report, cont.
Healthy Youth Survey 2002Survey Results
Sample Middle School, Grade 8
Number of students surveyed:
Number of valid responses:
64
59
The impact of adolescent health risk behaviors remains a primary concern of citizens throughout the country. Many health problems experienced by adolescents are caused by a very few preventable behaviors….
The number of forms that were considered usable.
Guide to the Local Report, cont.
Selected Results Disaggregated by Gender
25. During the past 30 days, on how manydays did you smoke cigarettes?
Local(n = 59)
State(n = 4,960)
Female Male Female MaleNone 70.4% 93.8% 89.9% 84.9%1 or more 29.6 6.2 10.1 15.1
Note. Results are suppressed if any cell represents fewer than 15 students;p = .021 from a chi-square test.
To protect anonymity when data are broken down into identifiable groups.
Guide to the Local Report, cont.Item Results 42. How old were you the first time you smoked a whole
cigarette? Local
(n = 20) State
(n =4,687)
a. Never have 60.0% (± 21.5%) 74.7% (± 1.2%) b. 10 or younger 10.0 (± 13.1) 6.4 (± 0.7) c. 11 15.0 (±15.6) 6.5 (± 0.7) d. 12 5.0 (± 9.6) 6.8 (± 0.7) e. 13 10.0 (± 13.1) 4.8 (± 0.6) f. 14 0.0 (± 0.0) 0.8 (± 0.3) g. 15 0.0 (± 0.0) 0.0 (± 0.0) h. 16 0.0 (± 0.0) 0.0 (± 0.0) i. 17 or older 0.0 (± 0.0) 0.1 (± 0.1)
[Item 43 appears only on the elementary version of the survey.] 45. Do you think young people risk harming themselves if they
smoke from 1–5 cigarettes per day?B (one to five cigarettes a day?) C
Local (n = 8)
State (n =4,877)
a. Definitely no 12.5% (± 22.9%) 4.9% (± 0.6%) b. Probably no 12.5 (± 22.9) 11.5 (± 0.9) c. Probably yes 0.0 (± 0.0) 38.7 (± 1.4) d. Definitely yes 75.0 (± 30.0) 44.9 (± 1.4)
Number of students who responded to this item
Where Do Find Information About . . . ?
• General Information (e.g. demographics)• Alcohol, Tobacco, and Other Drug Use• Other Health Concerns (e.g. health/safety)• School climate (e.g., bullying)• Quality of Life (e.g., student outlook)• Risk and Protective Factors:
– Scale Results
– Item Results
Frequently Asked Questions
• Review handout• Note about optional items
How to use your local report
• Example• Practice Activity
Local Report Example
168. If you wanted to get some marijuana, how easy would it be for you to get some?
Local (n = 84)
State (n = 3,738)
a. Very hard 52.4% (± 10.7%) 56.6% (± 2.7%) b. Sort of hard 19.1 (± 8.4) 16.9 (± 1.5) c. Sort of easy 15.5 (± 7.7) 13.9 (± 1.3) d. Very easy 13.1 (± 7.2) 12.6 (± 1.4)
State Agencies Using the HYS Data
Federal Initiatives
• Drug-Free Schools and Communities Act • Department of Education Safe and Drug-Free
Schools Program • Principles of Effectiveness • Healthy People 2010: National Health Promotion
and Disease Prevention Objectives • National Drug Control Strategy • No Child Left Behind
Washington State Initiatives
• Washington Education Reform Act • Omnibus Alcohol and Controlled Substances Act• Violence Reduction Programs Act• Department of Health Priority Health Goals• Governor's Council on Substance Abuse
Why Do You Keep Changing Your Terminology?
Or…When is an Impact an Outcome?
Putting Your Data to Work
Program Planning and Evaluation
What’s the Problem?
• What is “a problem”?– Getting worse
– Worse than average
– Just plain bad
• Combine HYS data with other information, including knowledge of community stakeholders– For example, HYS data may show that the highest rates
of tobacco use are in a particular area of the county, where police know there is a problem with retailers who provide tobacco to minors
What’s the Problem?
Exercise 1: What’s the Problem?
Lunch
• HYS data helped to identify a set of problems• Need to prioritize among them• Identify available resources
– Who is already working on the issue, or who has a stake in the issue
– Gaps in services or programs
– Modify existing programs, instead of creating all new ones
– Figure out what the community is “ready” for
What’s the Priority?
What’s the Priority?
• Target Efforts– Describe a target audience (grade level, gender,
universal or selective approaches)– Write “SMART” goals and objectives to get
FOCUSED and CLEAR with partners, and evaluation• Specific• Measurable• Achievable• Relevant• Time-bound
Setting a Good Objective
• You have 1,000 students• Current smoking prevalence is 25% (250 kids)• SMART Objective: By 2004, we will implement a
comprehensive tobacco prevention program and reduce current smoking among our school’s students by 10%
• This translates into about 25 fewer smokers• This is a change from 25% to 22.5%
Keeping it Real
• Will your program reach enough kids to affect the overall prevalence?
• Remember the difference between a “percentage point” goal and a “percent change” goal!
Population Group Changes - Gender
Current Smoking Trends, by Gender
0%
20%
40%
girls boys girls boys girls boys girls boys
6th 8th 10th 12th
Grade
Cu
rren
t S
mo
kers
2000
2002
Source: WSSAHB 2000, HYS 2002, state sample data
Exercise 2: What’s the Priority?
What Should I Do About It?
• Find “Best Practice” programs– Search for proven activities– The “Community Guide to Preventive Services”– CSAP “Science-based Prevention Programs”– Agency recommendations
• for example, DOH recommends conducting a retailer compliance program + public awareness about sources of tobacco
How Do I Know If It Worked?
Evaluate • Identify and collect process indicators, outcomes,
and impacts relative to your program– Process measures may include compliance checks &
compliance rates
– Outcomes may include perceived availability of tobacco & ‘usual sources’ of tobacco
– Impacts may include lifetime and current use of tobacco
Evaluation made easy
• What do I do when I go to work every day?• ….and then what happens?• ….and then what happens?• ….so that the WHY of going to work every day
happens.
Exercise 3: How Do I Know If It Worked?
Just in case you need more money…
• Healthy Youth Survey data can help– Provide justification of need for funding– Provide means for evaluating funded activities
• Find out about grant scoring• You will need to justify that HYS data are
– Valid– Reliable
Exercise 4: Grant Writing
Next Steps
Communicating With Your Data
Communications
• Nuts and bolts• Communicating about your program• Communicating to the media• Using data to make change
The Nuts and Bolts
• Round percents to whole numbers– 17.0% to 17.4% becomes 17%
– 17.5% to 17.9% becomes 18%
• Use language to acknowledge that these percents are estimates
• “About 17%”
• “14 to 20 percent”
• “plus or minus three percent”
Different Ways to Say ItInstead of saying: • “17.2% +/- 3.2% youth said that they ‘mostly’ or
‘definitely’ felt unsafe at school”
You could say:• “About 17% of our youth feel unsafe in school”• “About one in five youth feel unsafe in school”
Remember you can present it positively:• “About 83% of our youth feel SAFE in school”• “About four out of five youth feel SAFE in school”
Additional Tips
• If you had a low response rate for your survey, it would be best to say:
– “Elmo Elementary School students who took the survey said…”
Instead of
– “Elmo Elementary School students said…”
Communicating About Your Program
What data are appropriate for each type of communication?
• Your co-workers• Your neighbor at a cocktail party• Your grandmother
Communicating to the Media• What’s your SOCO? (Single Overriding
Communication Objective)– For broadcast media: 10-12 word “sound bite”
– For print media: 1-3 line quotation
• Message mapSingle Overriding Communication Objective
Detail 1 Detail 3 Detail 3
Fact Fact Fact
Fact Fact Fact
Fact Fact Fact
For Example: Tobacco Program Results
• SOCO: Our program has been successful in reducing youth smoking– Detail 1: Fewer youth are currently smoking than prior
to the program• Fact 1: 53% fewer 6th graders
– Detail 2: Fewer youth have ever tried smoking• Fact: 29% fewer 10th graders have ever smoked a
whole cigarette– Detail 3: Washington is doing better than the nation
• Fact: there was a 22% decrease in the nation for current smoking, so Washington’s decline is greater
Potential Pitfalls• When communicating with data, it’s easy to make
these mistakes:– Include information that’s not totally relevant to your
SOCO
– Include information that is relevant, but goes beyond what you NEED (too much detail)
– Speculate beyond what you have in the data
– Treat non-significant or unimportant differences as significant
– Forget to double-check your results and/or the math
Keep “on message”
• The answer to every question is your SOCO.
“That’s an interesting point, but what I’d really like people to know is,
[ my SOCO]”
Message Mapping Exercise
In small groups• Choose 1 topic of interest from HYS• Develop SOCO and 3 supporting details• Use HYS local report to find 3 supporting facts for
each detail • Message map template in handouts
Using Data to Make Change
• The message should tell your audience what you want them to do:– Do you want them to be outraged? And then what…
– Do you want them to call a policymaker?
– Do you want them to change their beliefs or behaviors?
– Do you want them to support your program?
Choose and Describe Your Audience
• Potential Audiences– Co-workers– Area residents– Elected officials– Civic organizations– Health care providers– Media– Regulatory agencies– Activists
• Demographics• Characteristics
– Concerns– Attitudes– Levels of interests– Levels of involvement– Histories– Levels of knowledge– Opinions– Reasons for interest– Types of involvement
How Will You Reach Your Audience?
Area residents
• Community meetings
• Newspaper articles and ads
• Radio and TV talk shows
• Fliers
• Direct Mailings
Elected officials
• Frequent phone calls• Fact sheets• Personal visits• Invitations to
community meetings• News releases
Cardinal Rules of Communication with Data1. Accept and involve the public as a partner
– They’re going to comment on your stuff anyway– Know ahead of time what public opinion is around your topic– Anticipate (and prepare for) the “tough questions & smart answers” –
including criticism of the survey itself
2. Plan carefully and evaluate your efforts• Was your SOCO ‘heard’?• Did you reach your target audience?• Did they know what you wanted them to do?
3. Be honest, frank, and open• Be prepared to talk about the limitations of your data
4. Work with other credible sources• Have someone critique your findings or interpretation• …and double-check your math
Cardinal Rules of Communicating with Data (cont.)
5. Meet the needs of the media • Meet their deadlines• Help them understand what the data mean• Help them understand how the survey works, if needed
6. Speak clearly and with compassion• “data are just people with the tears wiped away”• Practice saying it out loud (numbers can trip your tongue)
7. Use AV or graphics with caution• Sometimes a graph hurts more than helps you• Visual aids might distract from your message• Don’t try to exaggerate with graphics
Thank You!! Sun setting on training session
More questions about the training?
• Department of HealthLauren [email protected]
• OSPIBob [email protected] Mona [email protected]
• DASA, DSHSLinda Becker
360-407-0640
• RMC Research CorporationEric Einspruch
Gwen Hyatt
1-800-788-1887
http://www.rmccorp.com/project/hys02.html
The Joint Survey Planning Committee
OSPI
• Mona Johnson
• Bob McArdle
• Martin Mueller
• Pam Tollefsen
OCD
• Susie Roberts
Family Policy Council
• Bill Hall
DASA
• Linda Becker
• Steve Smothers
DOH
• Lillian Bensley
• Julia Dilley
• Lauren Jenks
• Susan Richardson
• Judy Schoder
• Juliet VanEenwyk