assessing undergraduate sustainability knowledge campus wide: adam zwickle - osu tomas koontz - osu...
TRANSCRIPT
![Page 1: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/1.jpg)
Assessing Undergraduate Sustainability Knowledge Campus Wide:
Adam Zwickle - OSUTomas Koontz - OSUAndrew Bodine - OSU
Mark Stewart – UMDNicole Horvath - UMD
Environmental and Social Sustainability Lab
From design to implementation and analysis
![Page 2: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/2.jpg)
2
Overview How we developed our Assessment of
Sustainability Knowledge (ASK)
Why an ASK is important & how it can help
An aside on Knowledge and Literacy
Conducting an assessment
Thinking long term…
![Page 3: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/3.jpg)
3
Developing an assessment Built upon the “triple bottom line”, the
“three legged stool”, the “3 p’s” Environmental (planet) Economic (prosperity) Social (people)
Basic knowledge from each knowledge domain?
~OR~ Knowledge bridging the different spheres?
![Page 4: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/4.jpg)
4
Developing an assessment Replicated questions used in the past
Coyle, 2005. “Environmental Literacy in America.”
Solicited topics and questions from experts
Held expert focus groups Pilot tested among professors, graduate,
and undergraduate students Narrowed down to 30 questions
![Page 5: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/5.jpg)
5
Developing an assessment Distributed those 30 to OSU students Used IRT to throw out 14 Added UMD’s 16 Distributed those to OSU and UMD
students Used IRT to throw out 2 Current ASK has 28 items:
ess.osu.edu
![Page 6: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/6.jpg)
6
Conceptualizing sustainability knowledge
Social
Economic
Environmental
Sustainability
Which of the following is the most commonly used definition of sustainable development?
Meeting the needs of the present without compromising the ability of future generations to meet their own needs
What is the most common cause of pollution of streams and rivers?
Surface water running off yards, city streets, paved, lots, and farm fields
Many economists argue that electricity prices in the U.S. are too low because…
They do not reflect the costs of pollution from generating the electricity
![Page 7: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/7.jpg)
7
How is this helpful?
![Page 8: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/8.jpg)
8
Need for measuring knowledge University goals-
More along the lines of: “Become carbon neutral by 2050”
Less common: “Create sustainably minded citizens of tomorrow”
Can help track improvement over time
![Page 9: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/9.jpg)
9
Need for measuring knowledge Serves as an evaluation for specific academic
efforts: Interdisciplinary programs Sustainability majors/minors
STARS Credit: ER 6 (STARS 2.0) – 3 points available
![Page 10: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/10.jpg)
10
“Knowledge” vs. “Literacy” Knowledge
Can be objectively measured Can be used to evaluate academic programs
Literacy = Knowledge Knowledge + values, attitudes, and behaviors Can be used to evaluate outreach efforts,
campaigns
![Page 11: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/11.jpg)
11
Measuring Behaviors, Values, and Attitudes
Make sure you are asking the right questions
Collaborate with sustainability departments to target specific behaviors (e.g., leaving lights on) Include questions on behavioral barriers Other behaviors that may get at the same
concept
![Page 12: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/12.jpg)
12
Measuring Behaviors, Values, and Attitudes
Collaborate with academic departments to develop a good survey design No need to reinvent the wheel Each survey could be a master’s thesis
![Page 13: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/13.jpg)
13
Conducting an Assessment
![Page 14: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/14.jpg)
14
Conducting an Assessment Find your partners!
IRB approval Required for publication Exempt status
Registrar approval* Student’s emails, majors, and demographics
Survey software* We use Qualtrics, but there are others
*If a large scale assessment is planned
![Page 15: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/15.jpg)
15
Maximizing Response Rates Maximizing response rates is important to
reduce uncertainty about how well your completed sample matches the population of interest.
Dillman (2008) and others have long studied how to maximize response rates for surveys that were telephone, mailed, or in-person.
There is growing research on electronic survey response rates.
![Page 16: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/16.jpg)
16
Research Methods We compare response rates from five
different survey implementations: 1: 2012 spring OSU (n=10,000) 2: 2013 spring OSU sample A (n=10,000) 3: 2013 spring OSU sample B (n=10,000) 4: 2013 spring OSU School of ENR (n=538) 5: 2013 spring UMD (n=10,000)
We tried different treatments and tracked survey responses with survey software (Survey Monkey and Qualtrics)
![Page 17: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/17.jpg)
17
Key Variables Affecting Response Rates among College Students
Timing When to send the invitation and
reminders
Incentives
Questionnaire format Long list of questions vs. more page clicks
Email text Who it is from
![Page 18: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/18.jpg)
18
Timing Matters Time of Semester:
Last week of semester and into finals week
Middle of semester
Time of Day: 6:00 am 6:00 pm
Reminders are critical big spike in completed surveys after each
reminder, with a fast decay (see next slide)
![Page 19: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/19.jpg)
19
Timing Matters Overall, you want students sitting at
their computers… but wanting to be distracted
![Page 20: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/20.jpg)
20
Incentives Some disagreement in research on best
way to provide incentives ahead of time (Dillman 2008) randomly select winner of 1 big prize give more/all respondents smaller prizes.
May impact the validity of the data
![Page 21: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/21.jpg)
21
Survey Responses
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 160
200
400
600
800
1000
1200
136
30 6 0 3 5 2 0
10322 4 5
1132
7418 16
1 2 3 4 5 6 7 8 9 10 11 120
200
400
600
800
1000
1200
408
71
570
226
47
545
52
516
11844 22 2
UMDn=1,556
OSUn=2,621
Day
Day
![Page 22: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/22.jpg)
22
Email text
How the invitation emails are phrased affects response rates.
Besides making the invitation personal, clear, and as short as possible, prior research has found that who the invitation comes from matters. An appeal from a trusted authority
increases response rates.
![Page 23: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/23.jpg)
23
Email text: Appeal from Authority We compared an appeal from authority
versus an appeal from a peer (student).
Two surveys, A and B, had an appeal from a higher authority (School Director or University VP) for first 3 contacts.
For the 4th contact kept the higher authority for A, but switched B to have to an appeal from a grad student
![Page 24: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/24.jpg)
24
Email text: Appeal from Authority
Results:
= additional 538
= additional 348
over the next 65 hours
![Page 25: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/25.jpg)
25
Questionnaire Format Trade-off:
Long list of questions to scroll down Shorter lists with a page click to get to the
next page
We analyzed respondent drop-outs spots Highest spots were just after clicking to
the next page
We recommend finding a balance…
![Page 26: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/26.jpg)
26
Non-Respondent Bias Even after efforts to maximize response
rates, are the non-respondents different from the respondents?
We conducted a non-respondent short survey (5 questions + some demographics).
Results indicate that non-respondents are slightly but significantly less knowledgeable about sustainability topics, but no difference in GPA or pro-environmental behaviors.
![Page 27: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/27.jpg)
27
Planning ahead… For longitudinal studies:
Write a multiple year IRB Let registrar know this is a yearly survey Partnering up with an academic
department for help Sustainability office Academic departments with survey expertise
(interdisciplinary social science, communications, sociology, political science, psychology)
![Page 28: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/28.jpg)
28
AcknowledgementsFunded by:
The Ohio State Office of Sustainability http://sustainability.osu.edu/
OSU’s School of Environment
& Natural Resources http://senr.osu.edu/
![Page 29: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/29.jpg)
29
Thank You!
Environment and Social Sustainability Lab ess.osu.edu Contains:
This presentation The 28 question ASK Forthcoming article
Email [email protected] [email protected]
Questions?
![Page 30: Assessing Undergraduate Sustainability Knowledge Campus Wide: Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath](https://reader036.vdocuments.mx/reader036/viewer/2022062621/551bfdc9550346a84f8b4a73/html5/thumbnails/30.jpg)
30