alfred d. tuttle rebecca l. morrison united states census bureau david h. galler
DESCRIPTION
From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment. Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler United States Bureau of Economic Analysis. Overview – Redesign of BEA Form BE-605. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/1.jpg)
From Start to Pilot: Comprehensive Redesign of a
Questionnaire Measuring Foreign Direct Investment
Alfred D. Tuttle
Rebecca L. Morrison
United States Census Bureau
David H. Galler
United States Bureau of
Economic Analysis
![Page 2: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/2.jpg)
2
Overview – Redesign of BEA Form BE-605
• Reimbursable project with BEA
• Multiple, intensive stages of research, design, testing, revision
• Qualitative and quantitative evaluations
• New BE-605 form mailed out in March 2007
![Page 3: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/3.jpg)
3
Outline
• Background – BEA Form BE-605
• Summary of stages of research
• Findings
• Next steps
![Page 4: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/4.jpg)
4
Background
• BE-605 (quarterly) and BE-15 (annual)
• Foreign Direct Investment (FDI) – >10% of voting rights in an incorporated entity (or equivalent interest in unincorporated entity)
• Sample – foreign-owned businesses in US
![Page 5: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/5.jpg)
5
Research stages and methods
• Background investigations– Expert review of form– Focus groups with survey analysts– Observations of callbacks– Respondent debriefings
![Page 6: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/6.jpg)
6
Research stages and methods
• Cognitive testing of mockups
![Page 7: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/7.jpg)
7
Research stages and methods
• Pilot test – Respondent debriefings – Evaluation questions
![Page 8: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/8.jpg)
8
Findings from background research
Data collection and design issues:
• Dense formatting
• Economic accounting concepts differ from US GAAP
• Separate instructions
![Page 9: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/9.jpg)
9
Recommendations
• Formatting recommendations– Letter size pages (not legal)– More “open” visual design– Incorporate instructions into form:
• within questions• adjacent to questions on facing page• key instructions become questions
– Diagrams to augment questions
![Page 10: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/10.jpg)
10
Original form
![Page 11: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/11.jpg)
11
Recommendation: More “open” design
![Page 12: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/12.jpg)
12
Recommendation: Embedded instructions
![Page 13: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/13.jpg)
13
Recommendation: Adjacent instructions
![Page 14: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/14.jpg)
14
Recommendations: Questions from instructions, and diagrams
![Page 15: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/15.jpg)
15
Cognitive testing
• Problematic sections mocked-up and tested (not entire forms)
• 5 rounds of cognitive interviews
• Iterative process: – mock-ups revised between rounds – design features modified & re-tested
![Page 16: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/16.jpg)
16
Findings from cognitive testing
• New design features were effective:– Majority of respondents liked the more
open format – Embedded instructions would likely
improve data quality– Some “InstructionsQuestions” and
diagrams required extensive modification, though most did not
![Page 17: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/17.jpg)
17
Developing 605 pilot
• Redesigned entire BE-605 form
• Close collaboration with BEA survey staff
![Page 18: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/18.jpg)
18
Pilot test
• Mailed 2nd quarter 2006
• Sent to sub-sample (n = 653; survey sample ~4,000)
• Completing pilot form optional
• Response rate: 53% (350 returned forms) during initial 8-week processing period
![Page 19: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/19.jpg)
19
Pilot form evaluation
• Respondent debriefings
• Evaluation questions in form
![Page 20: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/20.jpg)
20
Findings – R debriefings
• Favorable impressions of new form:– More pleasing to the eye– White reporting spaces easier to see
against colored background – Larger font, bold text, more space– Letter size pages easier to handle– Diagrams were helpful– Embedded instructions helpful
![Page 21: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/21.jpg)
21
Findings – R debriefings
• Drawbacks:– More pages– Booklet format makes it harder to fax– Took longer to complete than original form
(disruption of reporting routine)
![Page 22: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/22.jpg)
22
Limitations
• Stymied by experienced Rs, reporting routines…
• …though even experienced Rs were helped by new formatting and instructions.
![Page 23: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/23.jpg)
23
Findings – Evaluation questions
• Four questions appended to survey
• Each item received between 279 and 290 responses
![Page 24: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/24.jpg)
24
Findings – Evaluation questions
Instruction placement?
79.3% - Near questions
18.3% - In separate booklet
2.4% - Not applicable – I haven’t completed
the usual BE-605
![Page 25: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/25.jpg)
25
Findings – Evaluation questions
Open space and number of pages?
48.7% - less open space, fewer pages
47% - more open space, more pages
4.3% - Not applicable – I haven’t completed the usual BE-605
![Page 26: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/26.jpg)
26
Findings – Evaluation questions
Use of diagrams?
65.4% - very helpful or somewhat helpful
11.7% - only a little helpful
18% - did not need them
3.9% - confusing
![Page 27: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/27.jpg)
27
Findings – Evaluation questions
Easy/difficult compared to usual form?
50.5% - harder
42.5% - easier
7% - Not applicable – I haven’t completed the usual BE-605
![Page 28: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/28.jpg)
28
Findings – Evaluation questions
• Instructions preferred near questions
• Organization charts helpful
• Easier or more difficult?
• Preference for more/fewer pages?
![Page 29: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/29.jpg)
29
Summary
• Redesigned BE-605 went into production in March 2007– Recommended analysis of edit checks
during form processing
![Page 30: Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler](https://reader036.vdocuments.mx/reader036/viewer/2022081603/568139cf550346895da18127/html5/thumbnails/30.jpg)
30
Summary
• Multi-method approach to comprehensive survey redevelopment