casey langer tesfaye and susan white
DESCRIPTION
Can a survey of U.S. High Schools be Replaced or Reduced through Web Searches? The Successes and Complications of an Experimental Strategy. Casey Langer Tesfaye and Susan White. Nationwide Survey of High School Physics Multiphase survey Sample: public & private HS via NCES - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/1.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
Casey Langer Tesfaye and Susan White
Can a survey of U.S. High Schools be Replaced or Reduced through Web Searches? The Successes and Complications of an
Experimental Strategy
![Page 2: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/2.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
2
Background
•Nationwide Survey of High School Physics
•Multiphase survey Sample: public & private HS via NCES
Refreshed quadrennially, Redrawn periodically Phase 1: schools, Phase 2: teachers
![Page 3: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/3.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
3
Phase 1: Schools
•Does your school offer Physics?•Names & e-mail addresses of physics teachers
![Page 4: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/4.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
4
Background
•Multimode survey Mode: paper, web, phone
![Page 5: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/5.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
5
Problem 1
•Paper survey is really expensive
![Page 6: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/6.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
6
![Page 7: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/7.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
7
Problem 2
•Schools are oversurveyed•One IRB is not enough
More and more schools have custom research applications or processes
![Page 8: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/8.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
8
Problem 3
•Address lists out of date Some schools close or move
•Some schools difficult to reach by phone
![Page 9: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/9.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
9
Goal
•Eliminate paper survey altogether•Reduce contact with schools•Filter out closed schools faster
![Page 10: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/10.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
10
Proposed solution
•Begin with web searches•Call remaining schools
![Page 11: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/11.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
11
![Page 12: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/12.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
12
Strategy
•3 minute limit•Zoned form
![Page 13: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/13.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
13
![Page 14: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/14.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
14
Evaluating the effort
• Responses Response rate Data quality Research process requests
• $ savings
![Page 15: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/15.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
15
Outcome
•Overall response rate lower, but still over 90%
![Page 16: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/16.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
16
Outcome
Mode # of Responses % of total Responses
Web search 1061 29%
E-mail 894 25%
Phone call 1663 46%
Total 3618 100%
![Page 17: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/17.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
17
Data Quality
•Difficult to measure•Many issues to consider
![Page 18: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/18.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
18
Training and Streamlining
•Googling is a skill Google evolves
•Search strategies and techniques vary Evolve over time
There is a tradeoff between encouraging creative search strategies and evaluating the quality of the information yielded
![Page 19: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/19.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
19
Data Sources varied
•Sources varied Search results & search pages (e.g. Physics
Olympic teams, activities rosters) School websites, class websites, department
pages, grade-level pages, parental resources Great schools websites LinkedIn and other social networking sites
![Page 20: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/20.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
20
Quality Control
•Many schools share the same name Double check school location
•Many cities share the same name Use full/er address to verify
•Many teachers share the same name Listings with last names only are common
![Page 21: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/21.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
21
Information varied
• Very different configurations of information for each school Some easier to navigate, some had more or
less information or differently targeted information (e.g. sports teams)
Different definitions (e.g. Upper School)
![Page 22: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/22.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
22
Information varied
•Search results vary Date of updates varies Site content varies Websites go up & down, sometimes cached,
sometimes redesigned
![Page 23: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/23.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
23
What makes a complete entry?
•Goal of form was staged Stage 1:1. Is physics information available?2. Is physics teacher information available?
![Page 24: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/24.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
24
What makes a complete entry?•Goal of form was staged
Stage 1:1. Is physics information available?2. Is physics teacher information available? Stage 2:3. Then is there any helpful contact information?- What constitutes helpful contact information?
New classification: helpful, but not complete
![Page 25: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/25.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
25
Data Quality
•Every mode of survey has data quality issues Web searches yielded more inaccurate
information that in the past Inaccurate information leaks into teacher
survey E.g. “I don’t teach physics” “I teach physics, but
I have no students this year” More of a need for a streamlined system for
corrections
![Page 26: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/26.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
26
Cost
•Web search, email & phone call cheaper than Paper survey, email & phone call
•Web search & phone calls roughly equal (but web had the added benefit of filtering out closed schools faster and reducing contact with schools)
![Page 27: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/27.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
27
In Sum…•Web searches save $ •Web searches lead to less research process requests and potential rejections
But…•Web searches cannot be the only mode•A system that uses web searches:
Must have streamlined procedures for updating incorrect information
![Page 28: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/28.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
28
The web continually evolves
•More information online every day•Google continually improving and streamlining search results
![Page 29: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/29.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
29
The web continually evolves
![Page 30: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/30.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
30
The web continually evolves
• Training cannot be static Develop search and quality standards that can
evolve over time Develop a system that encourages web search
staff to discuss their strategies Use these interactions to understand what
information you are collecting and where you are collecting it from
![Page 31: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/31.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
31
In the future
•We will continue the web searches But we will strengthen and streamline the
feedback loop
![Page 32: Casey Langer Tesfaye and Susan White](https://reader036.vdocuments.mx/reader036/viewer/2022081502/568162d0550346895dd35d51/html5/thumbnails/32.jpg)
AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014
32
Thanks!
•Questions, comments?Casey Langer [email protected]