1999 nsaf telephone survey methods

1999 NSAFTelephoneSurveyMethods

Prepared by:

Westat

Nancy Vaden-KiernanPat CunninghamPatricia Warren

1999 NSAFTelephoneSurveyMethods Report No. 9

An Urban InstituteProgram to AssessChanging Social Policies

Assessingthe NewFederalism

Meth

odology

Rep

ortsNSAF

Preface

1999 NSAF Telephone Survey Methods is the ninth report in a series describing themethodology of the 1999 National Survey of America's Families (NSAF). It is a companion tothe recently reissued 1997 NSAF report the same subject (also as No. 9 in that series). Twoclosely related reports are those for 1997 and 1999 on the nontelephone data collection done inboth rounds. These appear as Report No. 5 in each series.

About the National Survey of America’s Families (NSAF)

As discussed elsewhere (e.g., see especially Report No. 1 in the 1997 NSAF methodologyseries), NSAF is part of the Assessing the New Federalism Project at the Urban Institute, beingdone in partnership with Child Trends. Data collection for the NSAF was conducted by Westat.

In each rounds of NSAF, carried out so far, over 40,000 households were interviewed, yieldinginformation on over 100,000 people. NSAF has focused on the economic, health, and socialcharacteristics of children, adults under the age of 65, and their families. The sample isrepresentative of the nation as a whole and of 13 states. Because of its large state sample sizes,NSAF has an unprecedented ability to measure differences between the 13 states it targeted.

About the 1997 and 1999 NSAF Methodology Series

The 1997 and 1999 methodology series of reports have been developed to provide readers with adetailed description of the methods employed to conduct the 1997 NSAF. The two series arenearly parallel, except for the documentation of the public use files, where an on-line system isbeing used for the 1999 survey and we are planning to reissue the 1997 files on a similar basis.

Report No 1 in the 1997 series introduces NSAF. Report Nos.2 through 4 in both series—plusReport No. 14 in the 1997 series—describe the sample design, how survey results were estimatedand how variances were calculated. Report Nos. 5 and 9 in each series describe the interviewingdone in for the telephone (RDD) and in-person samples. Report Nos. 6 and 15 in the 1997 seriesand Report No. 6 in the 1999 series displays and discusses the comparisons we made to surveysthat partially overlapped NSAF in content—including the Current Population Survey and theNational Health Interview Survey, among others. Report Nos. 7 and 8 in both series cover whatwe know about nonresponse rates and nonresponse biases. Report No. 10 in both series coversthe details of the survey processing, after the fieldwork was completed, including the imputationdone for items that were missing. Report No. 11 in both series introduces the public use filesmade available.

In the 1997 series, there were additional reports on the public use files available in a PDF formatas Report No. 13, 17-22. These will all eventually be superceded by the on-line data filecodebook system that we are going to employ for the 1999 survey. The 1997 and 1999 NSAFquestionnaires are available respectively as Report No. 12 in the 1997 series and Report No. 1 inthe 1999 series. Report No. 16 for the 1997 series, the only report not so far mentioned contains

occasional papers of methodological interest given at professional meetings through 1999,regarding the NSAF work as it has progressed over the years since 1996 when the project began.

About this 1999 Report

Report No. 9 describes methods employed to complete the telephone component of the 1999NSAF including a list-assisted method to select the random digit dialing (RDD) sample oftelephone numbers and computer-assisted telephone interviewing (CATI) for screening andinterviewing. Subsampling of telephone households is described, as is how respondents wereselected, information on the topics covered during the interview, and data quality controlmethods used. The report concludes that there were few difficulties with the telephoneinterviewing component of the NSAF. Those that did exist, primarily related to withinhousehold coverage issues and family definition problems.

For More Information

For more information about the National Survey of America's Families, contact Assessing theNew Federalism, Urban Institute, 2100 M Street, NW, Washington, DC 20037, telephone: (202)261-5377, fax: (202) 293-1918, Website: http://newfederalism.urban.org. For moreinformation about this report, contact [email protected].

Kevin WangandFritz Scheuren

i

TABLE OF CONTENTS

Chapter Page

1 Introduction and Overview of the Telephone Protocol ................... 1-1

2 The Screening Interview................................................................... 2-1

2.1 Option A versus Option A and B Paths.................................... 2-12.2 Purpose of the Screener ........................................................... 2-12.3 Screener Content ..................................................................... 2-22.4 Emancipated Minors ............................................................... 2-32.5 Pretest ..................................................................................... 2-42.6 Changes in Screener Introductions During Field Period........... 2-62.7 Summary................................................................................. 2-6

3 Extended Interview........................................................................... 3-1

3.1 Process of Compiling Questions and MakingDesign Decisions..................................................................... 3-1

3.2 Flow of Extended Interviews................................................... 3-13.3 Questionnaire Content ............................................................. 3-23.4 Cognitive Interviews ............................................................... 3-73.5 Pretest ..................................................................................... 3-73.6 Special Procedures .................................................................. 3-7

3.6.1 Proxy and Facilitator Interviews ................................ 3-83.6.2 Movers ...................................................................... 3-93.6.3 Students .................................................................... 3-93.6.4 Missing Spouses/Partners .......................................... 3-103.6.5 Special “D Restarts” to Include Household Members

Left Off the Household List....................................... 3-10

3.7 Significant Changes to the Content of the Questionnaire.......... 3-113.8 Summary................................................................................. 3-14

4 Training............................................................................................. 4-1

4.1 Overview of Training Plan ...................................................... 4-14.2 Development of Training Materials ......................................... 4-24.3 Organization of Training Sessions and Teams ......................... 4-2

4.3.1 Training Sessions ...................................................... 4-24.3.2 Training Teams ......................................................... 4-8

ii

TABLE OF CONTENTS (CONTINUED)

Chapter Page

4 Training............................................................................................. 4-1

4.4 Stages of Interviewer Training................................................. 4-9

4.4.1 GIT ........................................................................... 4-94.4.2 CATI Training with Teltrain...................................... 4-104.4.3 NSAF Project Training.............................................. 4-10

4.5 Schedule and Number of Interviewers Trained ........................ 4-124.6 Training for Special Procedures............................................... 4-15

4.6.1 Refusal Avoidance and Conversion ........................... 4-16

4.7 Interviewer Performance ......................................................... 4-16

4.7.1 Practice Interviews .................................................... 4-174.7.2 Criteria to Bring Online and Continue Interviewing... 4-174.7.3 Supplemental Training .............................................. 4-174.7.4 Gaining Respondent Cooperation .............................. 4-184.7.5 Interviewer Meetings................................................. 4-19

4.8 Summary................................................................................. 4-19

5 Scheduling and Release of Work...................................................... 5-1

5.1 Processing the Sample of Telephone Numbers Calling ............ 5-15.2 Advance Mailings ................................................................... 5-25.3 Answering Machines............................................................... 5-35.4 Initial Release.......................................................................... 5-35.5 Time-Slice Strategy................................................................. 5-45.6 Maximum Call (MC) Limits.................................................... 5-45.7 Language Strategy................................................................... 5-55.8 Refusal Conversion Strategy ................................................... 5-5

5.8.1 Refusal Conversion Strategy for Telephone NumbersWithout Addresses .................................................... 5-6

5.8.2 Refusal Conversion Strategy for Households withAddresses .................................................................. 5-7

5.8.3 Special Incentives...................................................... 5-85.8.4 Results of Conversion Effort ..................................... 5-8

iii

TABLE OF CONTENTS (CONTINUED)

Chapter Page

5 Scheduling and Release of Work (continued)

5.9 Manual Treatment ................................................................... 5-85.10 Switching the Order of Interviews ........................................... 5-105.11 Call Attempt Results ............................................................... 5-115.12 Summary................................................................................. 5-12

6 Quality Control ................................................................................. 6-1

6.1 CATI Testing .......................................................................... 6-16.2 Online Range and Logic Checking .......................................... 6-16.3 Training .................................................................................. 6-26.4 Practice Interviews and Supplemental Training ....................... 6-26.5 Interviewer Memos ................................................................. 6-26.6 Interviewer Meetings............................................................... 6-26.7 Interviewer Monitoring ........................................................... 6-36.8 Triage...................................................................................... 6-36.9 Using Comments and Problem Sheets to Find Problems.......... 6-46.10 Checks on Data in Monthly Data Deliveries ............................ 6-46.11 Summary................................................................................. 6-4

References ........................................................................................................... R-1

LIST OF TABLES

Table Page

2-1 Detailed Result of CATI Pretest of Screener andExtended Interviews............................................................................ 2-5

3-1 Questionnaire Content by the Type of Person, Family, or Household .. 3-6

3-2 Special Case Results ........................................................................... 3-11

4-1 Training Dates, Locations, and Number of Interviewers 1999 ............. 4-12

5-1 Mailing Schedule ................................................................................ 5-2

5-2 Cooperation and Conversion Rates for the Screener andExtended Interview............................................................................. 5-9

5-3 Average Number of Call Attempts per Telephone Number, by Final Result at the Screener Level, Cycle1 and Cycle2 ................... 5-12

LIST OF EXHIBITS

Exhibit Page

1-1 Study Findings Mailout....................................................................... 1-4

1-2 Prenotification Letter .......................................................................... 1-19

1-3 Study Brochure................................................................................... 1-20

1-4 Letter Sent at First Screener Refusal ................................................... 1-22

1-5 Letter Sent at Second Screener Refusal ............................................... 1-24

1-6 Letter Sent at First Extended Interview Refusal................................... 1-26

1-7 Letter Sent at Second Extended Interview Refusal .............................. 1-27

4-1 Lead Trainer Agenda .......................................................................... 4--4

1-1

1. INTRODUCTION AND OVERVIEW OF THE TELEPHONE PROTOCOL

The 1999 National Survey of America's Families (NSAF) had both a telephone and an in-person protocol. This report focuses on the telephone protocol. The in-person protocol isdescribed in the 1999 NSAF In-Person Methods, Report No. 5 in this methodology series.

The NSAF has had two cycles of data collection. The first was in 1997 (Cycle 1) and thesecond was in 1999 (Cycle 2). This report describes Cycle 2. For information abouttelephone survey methods in Cycle 1, see Telephone Survey Methods, Report No. 9 in theCycle 1 methodology series.

The telephone component of the NSAF 1999 used a list-assisted method to select therandom-digit-dialing (RDD) sample of telephone numbers and computer-assistedtelephone interviewing (CATI) for screening and interviewing. The sample was selectedseparately for each of the study sites and for the balance of the nation. The sample in thebalance of the nation was expanded in Cycle 2 to improve national estimates.

The Cycle 2 sample included an overlap of the same numbers used in Cycle 1, although anew sample was also drawn. For details about the sample design, see the 1999 NSAFSample Design Report, Report No. 2 in this methodology series.

Telephone households were subsampled, with the subsampling rates depending on thepresence of children in the household and the respondent's answer to a simple incomescreening item. All households with children that classified as low-income householdswere selected, while higher-income households with children and all households withoutchildren (but with someone under 65) were subsampled.

From the sample of 383,653 telephone numbers, 147,623 households were screened, anddetailed extended telephone interviews were conducted in 44,499 households with 48,679persons under the age of 65.1 As noted above, there was also an in-person component tothe study. In the in-person component, 1,676 extended interviews were conducted in1,497 selected households. This meant that in all there were a total of 50,355 telephoneand in-person extended interviews in 45,996 households.

The longer interview (45 minutes), referred to as Option A, was administered inhouseholds with children under 18, and the shorter interview (27 minutes), called OptionB, to adults under 65 without children under 18 living in their household. There were32,767 Option A interviews and 17,739 Option B interviews. Option A interviews askedquestions about both children and their families, while Option B interviews containedonly the questions from Option A that were relevant to adults. The questionnaire wasdivided into several sections, including the following topics: education, health care 1 In Cycle 2, completed interviews were defined as those that were complete through section D of the interview. Also, in households

with multiple interviews, if the first and later interviews were partial completes, only the first interview was retained. In Cycle 1,completed interviews were those that were complete through section K. If the Cycle 1 definition of a complete had been used inCycle 2, there would be 45,040 completes. If the Cycle 2 definition had been applied in Cycle 1, there would have been 50,198extended interviews in Cycle 1 rather than the 46,640 extended interviews that were counted using the Cycle 1 definition.

1-2

coverage and access, child care, employment and earnings, family income, welfareparticipation, housing and economic hardship, social services, problems, race, ethnicity,and nativity.

Before the interview was administered, households were screened to determine eligibilityand subsampled as appropriate, and persons were selected for the interview. Inhouseholds with children, up to two children (called focal children) were selected, oneunder age 6, and the other between ages 6 and 17. Information about the child(ren) andthe household (the Option A interview) was obtained from the most knowledgeable adult(called the MKA). It was possible to have separate MKAs for each focal child, though itwas more common to have one MKA for both children. It was also possible to sampleadults from households with children. To be eligible for this selection protocol, the adulthad to be under 65 years old, not a spouse or partner of the respondent, and not have anychildren of his or her own under 18 years old living in the sample household. This groupis referred to as the Option B stragglers because their interview always followed theinterview for the focal children. Other adults under 65 were sampled from householdswithout children. In these households one or two adults were randomly selected for theOption B interview.

The interviews and screener were programmed into Westat's CATI system to facilitateadministration and data editing. Data collection for the screener and extended interviewstarted on February 15, 1999. The timing of data collection was planned to allowrespondents a chance to receive all of their 1998 tax information documents (W-2s,Forms 1099, etc.) before the interview and thus be able to answer questions about 1998income in a more informed way. The survey was completed on October 3, 1999.

In late January, before the initial contact, respondents who had also been called for the1997 survey were sent special letters telling them about the upcoming survey andproviding them with some of the study findings from 1997 (see exhibit 1-1). The findingswere geared specifically to the respondent's state if he or she lived in one of theintensively studied states and nationally for everyone else. In addition, before screening,all respondents with addresses were sent a prenotification letter and a brochure describingthe study in some detail (see exhibits 1-2 and 1-3). Refusal conversion letters were alsosent at both the screener and extended interview levels (see exhibits 1-4 through 1-7).The conversion protocol is discussed in detail in section 5.8.

All interviewing was conducted from one of seven interviewing facilities—Frederick,Rockville, and Chestertown, Maryland; Toms River, New Jersey; Toledo, Ohio; Sarasota,Florida; and Chambersburg, Pennsylvania. A total of 546 interviewers were trainedacross all sites, 142 in Frederick, 197 in Rockville (Twelve Oaks Telephone ResearchCenter TRC); 31 in Chestertown, 29 in Toms River, 52 in Toledo, 69 in Sarasota, and 26in Chambersburg. In-person interviewers used cell phones and called in interviews to theFrederick and Twelve Oaks TRCs. This procedure is described in 1999 NSAF In-PersonSurvey Methods, Report No. 5 in this methodology series.

1-3

A variety of quality-control methods were used in NSAF. Some of them, such as CATItesting and training, were used before the beginning of data collection as preventivemeasures. Others, such as supplemental interviewer training, monitoring, and commentand problem sheet review, were used during data collection to respond to issues withinterviewers or to make adjustments to the questionnaire.

One of the main issues that occurred in the survey in Cycle 1 was a problem with thefamily definition. Because changes were made to the definition in Cycle 1, it was not aproblem in Cycle 2. See Telephone Survey Methods, Report No. 9 in the Cycle 1methodology series for a discussion. The 1997 report also describes a small within-household coverage issue that is relevant to both cycles, but it is not discussed again here.The bottom line is that there were no major difficulties with the NSAF survey in Cycle 2.

In Cycle 2, there were three additional substudies conducted for the NSAF. These aredescribed in other reports and are not discussed here. In the "Milwaukee experiment,"nonrespondents from the telephone sample for whom addresses were available wereasked in person to participate in the study. The data of those who cooperated in theMilwaukee study after refusing in the telephone sample were compared with those whocooperated initially. The Milwaukee study is described in Cunningham et al. (2000). Inanother substudy, referred to as "Capture-Recapture," respondents and nonrespondents inthe telephone sample were asked a brief set of questions about their opinions anddemographic characteristics. This study is described in Brick et al. (2000). Finally, therewere also additional data collected in three counties in California. The "California Add-On" study is described in Vaden-Kiernan and Cunningham (forthcoming).

Details of the main study telephone component are described in the chapters that follow.The telephone screener is discussed in chapter 2 and the extended interview is describedin chapter 3. Chapter 4 focuses on training. Scheduling and the release of work aredescribed in chapter 5 and quality control is described in chapter 6.

Exhibit 1-1. Study Findings mailout

1-4

Exhibit 1-1. Study Findings mailout (continued)

When we finish the study in 1999, we Westat will report back to Alabama on 1650 Research Boulevard whether the situation has changed. 1-5 Rockville, MD 20850

Volume 1 / January 1999 National Survey of America's Families

In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided thecountry with baseline information on the well-being of America’s families and children just before state and Federal governmentsmade changes in policies.

About Adults….♦ 59% of adults age 18 to 64

live with a spouse; 6% live with an unmarried partner.

♦ 12% of adults age 18 to 64 lived in a family with income below poverty.

♦ 18% of adults received education or formaljob training in 1996.

♦ 88% of adults age 18 to 64 reported being in good or excellent health.

About Children….♦ 70% of children lived in two-

parent families. 27% lived inone-parent families and 3% livedin families without a biological,adoptive, or step-parent.

♦ 20.5% of children lived in a family with income below poverty.

♦ 41% of children were highly interested and involved in school.

♦ 95% of children under age 18 werereported as being in good or excellenthealth.

♦ 8% of children had parents who were notconfident that their children could getnecessary medical care.

About Health Care….♦ 83% of adults age 18 to 64

had health insurance.

♦ 88% of children under theage of 18 had healthinsurance.

♦ 8% of adults age 18 to 64 have healthinsurance paid for by the state or federalgovernment.

♦ 19.5% of children under the age of 18 havehealth insurance paid for by the state orfederal government.

♦ 18% of adults age 18 to 64 did not have aregular place or provider of health care.

♦ 6% of children under the age of 18 did nothave a regular place or provider of healthcare.




In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided Alabamawith baseline information on the well-being of Alabama's families and children just before Alabama and the Federal governmentmade changes in policies.

About Adults in Alabama

♦ 60% of adults age 18 to 64 live with a spouse; 3% live

with an unmarried partner.


♦ 16% of adults received education or formal job training in 1996.

♦ 84% of adults age 18 to 64 reported being ingood or excellent health.

About Children in Alabama

♦ 63% of children lived in two-parent families. 32% lived inone-parent families and 5% livedin families without a biological,adoptive, or step-parent.

♦ 27% of children lived in a familywith income below poverty.


♦ 95% of children under age 18 were reported as being in good or excellent health.


About Health Care in Alabama

♦ 82% of adults age 18 to 64had health insurance.

♦ 86.5% of children under theage of 18 had healthinsurance.


♦ 20% of children under the age of 18 havehealth insurance paid for by the state orfederal government.






In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided Californiawith baseline information on the well-being of California's families and children just before California and the Federal governmentmade changes in policies.

About Adults in California

♦ 54% of adults age 18 to 64 livewith a spouse; 7% live with anunmarried partner.

♦ 16% of adults age 18 to 64 lived in afamily with income below poverty.

♦ 19% of adults received education or formaljob training in 1996.

♦ 85% of adults age 18 to 64 reported beingin good or excellent health.

About Children in California

♦ 68% of children lived in two-parent families. 29% lived inone-parent families and 2.5%lived in families without abiological, adoptive, or step-parent.

♦ 28% of children lived in a family with income below poverty.


♦ 93% of children under age 18 were reportedas being in good or excellent health.

♦ 11.5% of children had parents who were notconfident that their children could getnecessary medical care.

About Health Care in California










In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided Coloradowith baseline information on the well-being of Colorado's families and children just before Colorado and the Federal governmentmade changes in policies.

About Adults in Colorado

♦ 59% of adults age 18 to 64 livewith a spouse; 6% live with anunmarried partner.




About Children in Colorado

♦ 75% of children lived in two-parent families. 22% lived inone-parent families and 3% livedin families without a biological,adoptive, or step-parent.

♦ 14% of children lived in a family withincome below poverty.


♦ 96% of children under age 18 werereported as being in good or excellenthealth.

♦ 8% of children had parents who were not confident that their children could get

necessary medical care.

About Health Care in Colorado










In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided Floridawith baseline information on the well-being of Florida's families and children just before Florida and the Federal government madechanges in policies.

About Adults in Florida

♦ 58% of adults age 18 to 64 livewith a spouse; 8% live


♦ 13% of adults age 18 to 64 lived in a familywith income below poverty.


♦ 87.5% of adults age 18 to 64 reported beingin good or excellent health.

About Children in Florida

♦ 63% of children lived in two-parent families. 32% lived inone-parent families and 5%lived in families without abiological, adoptive, or step-parent.



♦ 95.5% of children under age 18 were reported as being in good or excellent health.


About Health Care in Florida










In 1997 we asked people like you about their health, education and economic welfare in 1996. This information providedMassachusetts with baseline information on the well-being of Massachusetts' families and children just before Massachusetts andthe Federal government made changes in policies.

About Adults in Massachusetts

♦ 56% of adults age 18 to 64 livewith a spouse; 7% live





About Children inMassachusetts


♦ 15.5% of children lived in a family withincome below poverty.





About Health Care inMassachusetts






♦ 2.5% of children under the age of 18 did nothave a regular place or provider of healthcare.




In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided Michiganwith baseline information on the well-being of Michigan's families and children just before Michigan and the Federal governmentmade changes in policies.

About Adults in Michigan

♦ 59.5% of adults age 18 to 64live with a spouse; 6% livewith an unmarried partner.

♦ 9.5% of adults age 18 to 64 lived in afamily with income below poverty.



About Children in Michigan




♦ 97% of children under age 18 were reported asbeing in good or excellent health.


About Health Care in Michigan










In 1997 we asked people like you about their health, education and economic welfare in 1996. This information providedMinnesota with baseline information on the well-being of Minnesota's families and children just before Minnesota and the Federalgovernment made changes in policies.

About Adults in Minnesota

♦ 60% of adults age 18 to 64 livewith a spouse; 7% live with an unmarried partner.




About Children in Minnesota




♦ 97% of children under age 18 were reported asbeing in good or excellent health.


About Health Care in Minnesota










In 1997 we asked people like you about their health, education and economic welfare in 1996. This information providedMississippi with baseline information on the well-being of Mississippi's families and children just before Mississippi and the Federalgovernment made changes in policies.

About Adults in Mississippi

♦ 58% of adults age 18 to 64 live with a spouse; 4% live with anunmarried partner.


♦ 15% of adults received education orformal job training in 1996.


About Children in Mississippi

♦ 56% of children lived intwo-parent families. 37% livedin one-parent families and 7%lived in families without abiological, adoptive, or step-parent.






About Health Care in Mississippi










In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided NewJersey with baseline information on the well-being of New Jersey's families and children just before New Jersey and the Federalgovernment made changes in policies.

About Adults in New Jersey

♦ 58% of adults age 18 to 64 live with a spouse; 5% live with anunmarried partner.




About Children in New Jersey

♦ 75% of children livedin two-parent families. 22%lived in one-parent families and3% lived in families without abiological, adoptive, or step-parent.






About Health Care in New Jersey

♦ 87% of adults age 18 to 64 hadhealth insurance.

♦ 91% of children under the ageof 18 had health insurance.

♦ 5.5% of adults age 18 to 64 have healthinsurance paid for by the state or federalgovernment.

♦ 13% of children under the age of 18 havehealth insurance paid for by the state or federalgovernment.


♦ 5% of children under the age of 18 did nothave a regular place or provider of health care.




In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided New Yorkwith baseline information on the well-being of New York's families and children just before New York and the Federal governmentmade changes in policies.

About Adults in New York

♦ 52% of adults age 18 to 64 live with a spouse; 6% live with an unmarried partner.




About Children in New York

♦ 65% of children lived in two-parent families. 31.5% lived inone-parent families and 3% livedin families without a biological,adoptive, or step-parent.






About Health Care in New York



♦ 9.5% of adults age 18 to 64 have healthinsurance paid for by the state or federalgovernment.







In 1997 we asked people like you about their health, education and economic welfare in 1996. This information provided Texaswith baseline information on the well-being of Texas' families and children just before Texas and the Federal government madechanges in policies.

About Adults in Texas

♦ 61% of adults age 18 to 64live with a spouse; 4.5% livewith an unmarried partner.




About Children in Texas

♦ 68% of children lived in two-parent families. 27% lived in one-parent families and 4.5% lived infamilies without a biological,adoptive, or step-parent.






About Health Care in Texas










In 1997 we asked people like you about their health, education and economic welfare in 1996. This information providedWashington with baseline information on the well-being of Washington's families and children just before Washington and theFederal government made changes in policies.

About Adults in Washington

♦ 61% of adults age 18 to 64 live with a spouse; 7.5% live with an unmarried partner.




About Children in Washington

♦ 76% of children lived intwo-parent families. 22% lived inone-parent families and 2% lived infamilies without a biological,adoptive, or step-parent.


♦ 41% of children were highly interestedand involved in school.

♦ 96% of children under age 18 werereported as being in good or excellent health.

♦ 7% of children had parents who were not confident that their children could get necessary medical care.

About Health Care in Washington








When we finish the study in 1999, we Westat will report back to Alabama on 1650 Research Boulevard whether the situation has changed. 1- Rockville, MD 20850

18


In 1997 we asked people like you about their health, education and economic welfare in 1996. This information providedWisconsin with baseline information on the well-being of Wisconsin's families and children just before Wisconsin and the Federalgovernment made changes in policies.

About Adults in Wisconsin

♦ 60% of adults age 18 to 64 live with a spouse; 7% live with an unmarried partner.




About Children in Wisconsin

♦ 74% of children lived intwo-parent families. 23% livedin one-parent families and 2%lived in families without abiological, adoptive, or step-parent.






About Health Care in Wisconsin







Exhibit 1-2. Prenotification letter

When we finish the study in 1999, weWestat

will report back to Alabama on 1650 ResearchBoulevard whether the situation has changed. 1- Rockville, MD20850

19

Dear Potential Study Participant:

I am writing to strongly encourage your participation in an upcomingtelephone survey, the National Survey of America’s Families (NSAF).

The enclosed Brochure explains more about the study.

Westat, a research firm, will be conducting the interviews. Sometimeduring the next month, a Westat interviewer will call you. If you are called at aninconvenient time, please suggest a better time to call. If you would like to set upan appointment anytime before someone calls, contact Westat at their toll freenumber (1-800-649-0114).

Sincerely,

Joan MartinProject Coordinator

Exhibit 1-3. Study brochure

1-20

Exhibit 1-3. Study brochure (continued)

1-21

Exhibit 1-4. Letter sent at first screener refusal

1-22

Spring, 1999

Dear Potential Study Participant,

I am writing about the National Survey of America’s Families (NSAF).Recently, one of our telephone interviewers called and asked for your household’sparticipation. As of the time we sent out this letter, we had not completed aninterview with your household.

You Are Important to the Success of the Study!

Because we know your time is valuable, we have enclosed $5 as a token ofour appreciation for your participation in the study.

The success of the NSAF depends on how accurately we can describe theexperiences of people in your state and across the United States. To do this, we needto talk to a wide variety of people. Your participation is very important in achievingthis goal. The back of this letter lists answers to questions you may have about thesurvey.

Help Us Build on What We Learned in 1997.

In 1997 we did a survey and found out answers to questions such as:

• How many adults have health care coverage? How many children?

• How confident are people that they can receive adequate medical care?

• How many people are relying on government programs for support?

With your help, we will find out if the situation has changed since 1997.

What Will Happen Next?

In the next few days, a Westat interviewer will be calling to ask you if youwould agree to participate. If we happen to call at an inconvenient time, please suggesta time that is better for you. If you would like to set an appointment before we call oryou have questions about the study, contact us at the toll-free number 1-800-649-0114.

I hope you will take part in this very important study.

Sincerely,


Exhibit 1-4. Letter sent at first screener refusal (continued)

1-23

What kinds of questions will you be asking?

First, we will ask a few questions about your household, so thatwe can determine if we can include you in our study. If so, wewill ask about how people in your household are doing in someareas, like health care, education, and housing.

How long will it take?

The first few questions is a short interview to see if we caninclude your household in our study. This will only take about 3minutes. Depending on your answers, the rest of the questions weask you may take anywhere from 25 to 45 minutes.

Exhibit 1-5. Letter sent at second screener refusal

1-24

Spring 1999


I am writing about the National Survey of America’s Families (NSAF).Recently, one of our telephone interviewers called and asked for your household’sparticipation. As of the time we sent out this letter, we had not completed aninterview with your household.

The purpose of the NSAF is to understand how recent changes in Federalpolicies will affect people in communities around the country. Our understandingwill be based on the real experiences of people in the U.S. For example, the projectwill find out whether recent changes in Federal policies increase or decrease thenumber of people who buy health insurance or who find a job.

The large number of interviews that we have done is very encouraging. But,whether we will be able to collect accurate information for the whole countrydepends upon getting information from you and others who have not yet responded.

In the next few days, a Westat interviewer will be calling to ask you if youwould agree to participate. If we happen to call at an inconvenient time, please suggesta time that is better for you. If you would like to set an appointment anytime before wecall or if you have any questions, contact Westat at their toll-free number (1-800-649-0114).

I hope you will take part in this very important study.

Sincerely,


Exhibit 1-5. Letter sent at second screener refusal (continued)

1-25

What kinds of questions will you be asking?

First, we will ask a few questions about your household, so that we can determine ifwe can include you in our study. If so, we will ask about how people in yourhousehold are doing in some areas, like health care, education, and housing.

How long will it take?

The first few questions, a short interview to see if we can include your household inour study, will only take about 3 minutes. Depending on your answers, the rest ofthe questions we ask you may take anywhere from 25 to 45 minutes.

Exhibit 1-6. Letter sent at first extended interview refusal

1-26

Spring, 1999


Recently, a Westat interviewer called your home and completed part of an interviewwith a member of your household for the National Survey of America's Families(NSAF). I want to thank you very much for talking with us and would like toencourage you to finish the rest of the interview.

Your Time is Valuable to Us.

We know your time is valuable, so we have enclosed $5 to thank-you forparticipating in the study. Because your time is so important to us, we will send youan additional $20 if you agree to complete the remaining part of the interview.

The Results of the Study Are Important for Policies in Your Area

Policies that affect health care, education, employment and other serviceshave changed since we did the first study in 1997. The NSAF is an independentassessment of how these changes are working. For example, the study will find out ifinsurance coverage and access to medical care has changed since the new policieshave been in place.

Because of it’s importance in tracking recent government policies, the studyhas received national attention. It has been featured in over 80 newspapers andmagazines, including US News and World Report (February 1st), USA Today (January26) and the Los Angeles Times (January 26).

What Will Happen Next?

We have asked an interviewer to contact you again to see if you willparticipate. If we happen to call at an inconvenient time, please suggest a time that isbetter for you. If you would like to set an appointment anytime before we call,contact Westat at their toll-free number (1-800-649-0114).

Sincerely,


Exhibit 1-7. Letter sent at second extended interview refusal

1-27

Spring 1999


I am writing about the National Survey of America’s Families (NSAF).Recently, one of our telephone interviewers called and asked for yourparticipation. As of the time we sent out this letter, we had not completed aninterview with you.

The large number of interviews that we have done is very encouraging. But,whether we will be able to collect accurate information depends upon gettinginformation from you and others who have not yet responded.

Previously, we sent you $5 in recognition of your participation in the study.We will send you an additional $20 if you agree to complete the remaining part ofthe interview.

In the next few days, a Westat interviewer will be calling to ask you if youwould agree to participate. If we happen to call at an inconvenient time, pleasesuggest a time that is better for you. If you would like to set an appointment anytimebefore we call or if you have any questions, contact Westat at their toll-free number(1-800-649-0114).

On the back of this letter, there are answers to some questions you mayhave about the study. I hope you will take part in this very important study.

Sincerely,


Exhibit 1-7. Letter sent at second extended interview refusal (continued)

1-28

SOME FREQUENTLY ASKED QUESTIONS

Q. How will the study results be used? What will you do with thisinformation?

A. The information we collect will tell us about people’sexperiences with changes in health care, education, and human services inyour area. Findings should help researchers, policymakers, and the publicstay informed about changes happening in your state and other states in thenation. Results from this study will be widely distributed to researchers,government officials, national associations, community groups, health andsocial service program administrators, news organizations, and the generalpublic.

Q. What foundations are sponsoring this study?

A. The study is being sponsored by the Urban Institute and ChildTrends with support from the Annie E. Casey Foundation, the Henry J.Kaiser Family Foundation, the W.K. Kellogg Foundation, the John D. andCatherine T. MacArthur Foundation, the McKnight Foundation, the Fund forNew Jersey, the Joyce Foundation, the Robert Wood Johnson Foundation,and several other foundations. These foundations are private organizationsthat are interested in the well-being of children, adults, and families.

Q. Will you keep my information confidential?

A. Yes. All the information you give to the interviewer will be keptcompletely confidential. All employees who are working on this study arerequired to protect the confidentiality of respondents. Also, individualresponses are never published in reports; they are added to the responses ofothers and are published as combined information only.

2-1

2. THE SCREENING INTERVIEW

In this chapter, the screening interview is described, including the flow of the screener, the pathsof the interviews, the purpose of the screener, the screener content, the selection of respondents,a pretest used to test the screener along with the extended interview, and changes to the screenerintroduction during the field period.

The screening interview was used to screen households on the basis of age composition. If therewas someone in the household who was age-eligible, the household was then sampled on thebasis of household income. See Report No. 1, p. SC-1, for the final 1999 NSAF screener. Once ahousehold was sampled, the extended interview was administered to the selected respondent. Theextended interview is described in chapter 3.

2.1 Option A versus Option A and B Paths

The screening interview varied depending on whether the interview was in a path designated forchild interviews (also called the Option A path) or a path designated for both child and adultinterviews (called the Option A and B paths). Option A households were defined as thosecontaining at least one child between the ages 0 to 17. Option B households were those with nochildren under the age of 18 but at least one person who was 18 to 64 years old.

In the Option A group, screener questions were administered to locate households with eligiblefocal children. If a household did not have any children, it was considered ineligible. In theOption A and B group, screener questions were included that tried to locate either a householdwith focal children or a household with at least one member between the ages of 18 and 64.

2.2 Purpose of the Screener

The screener was developed by Westat and the Urban Institute during the 1997 NSAF Cycle 1. Aseries of pretests was conducted to determine the best screener for the study. This process isdescribed in Telephone Survey Methods, Report No. 9 in the 1997 methodology series.

The purpose of the screener was to perform the following:

1. Find out if there was an eligible subject (or subjects) in the household, based on therequirements for the Option A or Option A and B paths.

2. Select eligible subject(s) for the extended interview (i.e., either focal children or adults). Todo this, two pieces of information had to be collected. First, data had to be collected onwhether the total household income was above or below 200 percent of the poverty level(please refer to the appendix of the Snapshots of America’s Families). Those with incomesabove 200 percent of the poverty level were sampled, while all those below were included in

2-2

the study. Second, a list of eligible subjects in the household was collected. For Option Ahouseholds, this was a list of children 0 to 17 years old, along with their respective ages. Forthe Option B households, this was a list of all adults ages 18 to 64. Once the listing wasobtained, subjects were sampled for the extended interview. For Option A households, therecould be up to two subjects (one child 0 to 5 years old and one child 6 to 17 years old). ForOption B households, there could be up to two adults selected.

3. Identify the adult in the household who was most knowledgeable about the selected child (theMKA) in Option A households. Once a child was selected as a subject, data had to becollected to determine the MKA.

2.3 Screener Content

The screener did not change substantively between Cycle 1 and Cycle 2. In both cycles thescreener contained the following sequence of items:

1. An introduction that identified the interviewer and briefly explained the purpose of the call.

2. Questions that confirmed that the interviewer was speaking to a member of the householdwho was at least 18 years old. If all household members were less than 18, the interviewerasked to speak to the male or female head of household. Such households were categorizedas consisting of one or more “emancipated minors” (see section 2.4 below).

3. A question that determined whether the telephone was for residential use. If not, the screenerterminated with a closing statement for nonresidences, and the screener was finalized with anonresidential result code.

4. A question that determined whether there was anyone in the household who was under age65. If no one under 65 lived in the household, an ineligible result code was assigned.

5. A question that determined whether there were any children 17 or under in the household.The response to this question determined whether the household had any eligible children.

6. A question about the total number of people who lived in the household. This was used todetermine what the threshold for 200 percent of poverty would be.

7. A series of opinion questions on ways to improve education in the country. These questionswere placed immediately before the question about income in order to focus the respondent’sattention on a substantive area that would legitimize the study.

8. A question to determine whether the household was below 200 percent of poverty. This wasa single item that asked whether the total family income was above or below a particularincome level (e.g., $33,000 for a four-person household with children). The level wascalculated for household sizes of one through eight members and households with more thaneight members.

2-3

9. A question to identify the children (ages 0 to 17) or adults (ages 18 to 64) in the household.Once this list was compiled, the CATI program sampled up to two children or up to twoadults for subjects on the extended interview.

10. A series of questions to determine the name and relationship of the person mostknowledgeable about the selected child or children (the MKA). These questions were onlyasked if the household included children who had been sampled for the extended interview.

For 10 percent of the cases, questions were asked about (1) whether the home or apartment wasowned or being bought by someone in the household or whether the home or apartment wasrented by someone in the household; and (2) whether everyone in the household was covered byhealth insurance. These two questions were tested to determine their ability to act as proxies forincome in a future round of the survey. What was learned here is found in 1999 NSAF ResponseRates and Methods Evaluation, Report No. 7 in this methodology series.

At the end of the screener, the interviewer asked to speak to the first person selected to conductthe extended interview. This was either the MKA selected to talk about one of the focal children(see items 9 and 10 above) or an adult who had been selected.

2.4 Emancipated Minors

In all but a few cases, if a household respondent was eligible, the screener identified an adultrespondent to complete either an adult interview about himself/herself or to complete aninterview about a child or children. In 23 cases in the telephone sample,1 interviewers completedan interview with a youth under the age of 18 who was living on his/her own without a parent orguardian who was 18 years old or older. These youths were treated as “emancipated minors” andwere eligible to be interviewed using an adult interview if they were at least 16 years old. Inaddition, data were collected on selected emancipated minors through related-child interviews inthe household, conducted with either the emancipated minor (when he/she served as the MKAfor another child) or with the emancipated minor’s spouse or unmarried partner (also whenhe/she served as the MKA for another child). No interviews were administered to persons underthe age of 16.

Emancipated minors could potentially be found at different points in the process. One point waswhen the screener questions were initially confirming that the respondent was at least 18 yearsold. If the respondent was not at least 18 and stated there was no one in the household who wasat least 18, the screener was completed with the person that was considered the head of thehousehold. The household head was considered an emancipated minor and was eligible for theadult interview. A second point where emancipated minors could be found was when the MKAwas selected. If the focal child had no parent or guardian living in the household or if the

1 1999 NSAF Sample Design, Report No. 2, in the 1999 Methodology series reports 29 cases with emancipated minors. This report includes 6 cellphone cases in its count.

2-4

relationship of the MKA was a roommate, spouse/partner, or sibling, the selected child wasdetermined to be an emancipated minor and an adult interview was administered.

An emancipated minor could serve as an MKA (if appropriate) or as another adult in thehousehold. For example, if the emancipated minor was selected as both a child and as the MKAfor a younger child, the emancipated minor was administered the MKA interview for theyounger child. No other interviews were administered related to the emancipated minor’s statusas a child. Similarly, if the emancipated minor was the spouse/partner of the MKA for theyounger child, no other interviews were administered. If the emancipated minor was not theMKA or the spouse/partner of an MKA, the adult interview was administered (if the individualwas sampled as part of the study).

2.5 Pretest

The screener was pretested along with the extended interview. Although the content of thescreener had not changed since Cycle 1 (except for the addition of two new items), the screenerwas still pretested to make sure that there were no unforeseen problems with the CATI programsbefore data collection began. The screener was also needed in the pretest to select respondentsfor the extended interview, which had been modified for Cycle 2.

The NSAF Cycle 2 pretest was conducted in the Frederick TRC between October 17 and 23,1998. Sixteen interviewers were trained for and worked on the pretest, following a trainingsession held on Saturday, October 17. All of the interviewers had been involved in NSAF Cycle1 data collection.

The telephone numbers for the pretest were obtained from two sources. The first source was theCycle 1 practice account from which 1,806 numbers were used from households in Eastern andCentral time zones that had never refused and had addresses. The second source was 500telephone numbers ordered from Genesys that were known to be households with listedtelephone numbers and someone under the age of 65. The 2,306 phone numbers were“interleaved” in the sample load, so that practice account cases alternated with Genesys cases.Among practice account cases, households with incomes below 200 percent of poverty wereloaded first. The first 1,400 cases were sent a prenotification letter 10 days before the start of thepretest.

Of the 2,306 telephone numbers, 1,595 were called at the screener level. From this total, 326potential extended interviews were generated. The cooperation rates at both levels were verygood, with a screener cooperation rate of 68.4 percent and an extended interview rate of 96.2percent. Cooperation was notably higher among the practice account cases (screener 75.9 percentand extended 98.2 percent) than it was among the new telephone numbers from Genesys(screener 57.3 percent and extended 91.9 percent).

2-5

On October 23, 1998, 189 completed child interviews and 38 completed adult interviews weredelivered to the Urban Institute. The average length of the child interview was 38.34 minutes,and the average length of the adult interview was 23.32 minutes.

There were several differences between the CATI pretest and the procedures that were institutedfor the main study. First, no refusal conversion was conducted on the CATI pretest. Second, thesample used in the pretest was not the entire U.S. population, and most of it was made up ofcooperators from the practice account in 1997. Thus, the rates for this sample would not begenerally applicable to the entire country.

On Wednesday, October 28, a debriefing meeting was held with the pretest interviewers togather their reactions to the revised questionnaire. In addition, the pretest database was reviewed,including comments, frequencies, and problem cases, to look for evidence of possible difficultiesrelated to questionnaire changes.

No changes in question wording were made in the screener after the pretest. We felt that becausethe screener questions had been used before in Cycle 2, enough development had gone into thescreener that further changes were unnecessary.

The results of the pretest for the screener and extended interviews are shown in table 2-1.

Table 2-1.Detailed Result of CATI Pretest of Screener and Extended Interviews

Screener ExtendedCompletes* 391 229

Interim 791 72Refusals 181 9No contacts 191 10Other** 419 52

Nonresidential/nonworking 406 N/ALanguage problems 7 1

Total 1,595 302

*"Completes” included those screened as either eligible or ineligible for the study. They alsoinclude partial completes that made it at least through section D of the extended questionnaire.

**Included callbacks, answering machines, and problems. Problems included cases

that, for example, had selected respondents who would not be available during the

time of the pretest or who were not mentally capable of answering the questions.

2-6

2.6 Changes in Screener Introductions During Field Period

The introduction was changed once during the field period, based on comments from theinterviewers that a shorter introduction would be better. The shorter introduction contained thesame information as the longer one, but the purpose of the study did not have to be volunteered ifit was not needed to obtain the respondent’s participation. The new introduction went into effecton March 19, 1999. The first and second introductions are shown below.

Hello, this is (NAME) with the National Survey of America’s Families,a study to see how recent changes in federal laws affect people’s livesin your community. I am not asking for money—this is a study forprivate foundations on education, health care, and other services in thestate of (STATE).

Hello, this is (NAME) with the National Survey of America’s Families.I am not asking for money—this is a study for private foundations oneducation, health care, and other services in the state of (STATE).

[IF ASKED: This study is to see how recent changes in federal lawsaffect people’s lives in your community.]

2.7 Summary

In summary, the screening interview was used to determine whether any household memberswere eligible for the extended interview. Eligibility was based on the age of household membersand on the income level of the household. All screeners were conducted in one of two paths. Thefirst path was only for child interviews. The second path was for child or adult interviews.

The screener was developed by Westat, the Urban Institute, and Child Trends during Cycle 1 ofthe study. No substantive changes were made in Cycle 2, except that two questions were addedfor 10 percent of the sample to determine whether those questions could serve as a proxy forincome in a future round of the study.

A CATI pretest was conducted to test the screener along with the extended interview. Nochanges were made to the screener after this pretest.

During the field period, the screener introduction changed once, on March 19, in response tointerviewer suggestions to make the introduction shorter.

3-1

3. EXTENDED INTERVIEW

In this chapter, we describe the extended telephone interviews done for the 1999 NSAF. Outlinedhere is the process of compiling questions for the interviews and making design decisions, theflow of the extended interviews, the content of the questionnaire, the use of cognitive interviews,the pretest that was conducted, special procedures that were used in administering extendedinterviews, and significant changes in the questionnaire content during the course of the study.The final extended questionnaire instrument is available in 1999 NSAF Questionnaire, ReportNo. 1, p. B-1, in the 1999 NSAF methodology series.

3.1 Process of Compiling Questions and Making Design Decisions

The Urban Institute and Child Trends compiled the majority of questions in Cycle 1 of theNSAF. In Cycle 2, some questions were added and others were deleted so that the questionnairewould be approximately the same length as it was in Cycle 1. The changes made were donejointly, after considerable testing.

3.2 Flow of Extended Interviews

The flow of extended interviews within a household was based on rules that determined the orderand types of interviews that would be administered. Multiple extended interviews could beconducted in a household after the screener was completed. Child interviews were referred to asOption A interviews, and adult interviews were referred to as Option B interviews. For detailsabout who was included and excluded in the sample design, see the 1999 NSAF Sample DesignReport, Report No. 2 in this methodology series.

In each household that had multiple child or adult interviews, there were two different versionsof the questionnaire—a full and an abbreviated version. The full version was always conductedfirst, followed by one of two types of abbreviated versions. One version was for cases in whichthe respondent for the abbreviated questionnaire was in the same family, and the other versionwas for cases in which he/she was in a different family. If the respondent was in the same familyas the respondent to the full questionnaire, many questions about the household and family didnot need to be asked again. If the respondent to the second questionnaire was not in the samefamily, questions about the household did not have to be asked again, but some family questionswere re-asked.

For child interviews, there was an order rule based on the age of the focal child. If there weretwo selected children and two persons named as the MKAs, the MKA interview for the olderchild was conducted first. This is because it was believed that the MKA for the older child wouldusually be better able to provide some of the income and health care information than the MKAof the younger child. For example, if we were speaking to a mother and her daughter and bothwere MKAs (the mother as the MKA for her daughter and the daughter as the MKA for a

3-2

younger child), it seemed more appropriate to interview the mother before talking to herdaughter.

In addition, there were also order rules across types of interviews. For example, adult interviewscould follow child interviews. These were referred to as “Option B stragglers.” Eligible adultstragglers were those who were not the spouse/partner of an MKA or an emancipated minor andnot the parent of, or the spouse/partner of a parent of, any child under 18 in the household. Up totwo of these adults could be sampled from a household to be administered an adult extendedinterview. In adult-only households, an adult interview could also follow another adult interview.However, child interviews could never follow adult interviews.

Because of the many types of interviews that could be administered, interview types werenumbered within Option A child and Option B adult interviews. Each type is defined below:

n A1: Option A interview with the first MKA

n A2: Option A interview with a second MKA; both MKAs are in the same family

n A3: Option A interview with a second MKA; MKAs are not in the same family

n B1: Option B interview with the first adult

n B2: Option B interview with a second adult; both adults are in the same family

n B3: Option B interview with a second adult; adults are not in the same family

n B4: Option B interview with an emancipated minor

n B5: Option B interview with an emancipated minor; minor is in a different family.

The child interview (option A interview timings averaged together) required longer to administerthan the adult interview (option B interview timings averaged together). On average, the childinterview was 45.20 minutes long, and the adult interview was 26.55 minutes long.

3.3 Questionnaire Content

The 1999 questionnaire was very similar to the 1997 questionnaire, with a few additional itemsand some deletions (see 1999 NSAF Questionnaire, Report No. 1 in the 1999 methodologyseries). As in 1997, the questionnaire content was designed to progress from less sensitive tomore sensitive topics (see 1997 NSAF Questionnaire, Report No. 12 in the 1997 methodologyseries). In designing the survey, we also assumed that parents would be more likely to talk abouttheir children first rather than themselves, so questions about children’s health and educationwere asked before questions about household membership and relationships.

3-3

The 1999 NSAF extended interview was divided into 16 sections, labeled A through P. Listedbelow is a brief description of the content areas of each section along with an indication of the“unit” (in bold) upon which the section was focused. Sometimes the unit of a section waseveryone in the household, sometimes it was everyone in the family, sometimes it was therespondent (and his/her spouse/partner), and sometimes it was the focal child(ren). The sectionsare described below:

A. Student Status. This section contained two questions—one that asked whether therespondent was a student and one that asked whether this was the respondent’s usualresidence. The questions in section A were asked of respondents who were 16 to 24 yearsold. If the household was not the respondent’s usual residence, the CATI system woulddetermine that the respondent was ineligible to continue.

B. Health Status and Satisfaction. In section B, questions were asked about the respondent’ssatisfaction with health care, access to health care, the health status of focal children, and,in some interviews, the health status of the respondent. Questions were also asked aboutwhether the respondent had heard of health care programs specific to his/her state.

C. Parent/Child/Family Interaction and Education. This series of questions asked abouteducation for focal children. Questions included the focal child’s current grade (or the lastgrade completed) and the child’s attitudes toward school and schoolwork, skipping school,suspensions, and changing schools. For children over 11 years old, there were alsoquestions about working for pay during the last four weeks.

D. Household Roster. In this section, the name, age, and sex of all persons living in thehousehold were recorded. Questions were asked to identify the relationships between allhousehold members. The lease or mortgage holder for the household is identified.

E. Health Care Coverage. For the respondent, the respondent’s spouse/partner (ifapplicable), and the focal children (if any), information was gathered about currenthealth insurance coverage (or lack thereof). Questions were also asked about coverage forthe past 12 months and about periods in which family members had no insurance. Forfamily members with particular types of coverage, questions were asked about thecharacteristics of their health plans.

F. Health Care Use and Access. This section gathered information about health status,health care services received, and necessary health care services that were postponedduring the last 12 months. This section included questions on routine care, overnight staysin hospitals, dental care, mental health care, women’s health care, well-child care, andprescription medicines. Questions were asked about the focal children and either therespondent or his/her spouse/partner in the child interview. During the adult interview,questions were asked about the adult and his/her spouse/partner.

G. Child Care. In this section, we asked about child care arrangements of focal children whowere under 13 years old. Child care arrangements included Head Start, child care centers,

3-4

before- or after-school care, and babysitters. Questions asked about the total number ofhours per week in each care situation, the typical number of children cared for, the typicalnumber of adult child care providers, and child care costs.

H. Nonresidential Parent/Father. These questions determined whether a focal child had anonresident parent, how often the child saw his/her nonresident parent, whether thenonresident parent provided financial support, and whether nonresident parents wererequired by child support orders to provide financial support.

I. Employment and Earnings. This section contained a series of questions about theemployment and earnings of the respondent and the spouse/partner this year and lastyear. Questions included those about current employment status, occupation, industry,employer-provided health insurance, hours worked, and earnings. Some questions werealso asked about the earnings of other family members.

J. Family Income. Family income from a wide variety of “other” sources (not includingearnings from employment) was identified. These sources included public assistance(Temporary Assistance for Needy Families [TANF], Aid to Families with DependentChildren [AFDC], General Assistance, emergency assistance, vouchers), food stamps,child support, foster care payments, financial assistance from friends or relatives,unemployment compensation, workers’ compensation or veteran’s payments,Supplemental Security Income, Social Security, pension or annuity income, interest ordividend income, income from rental property, and any other source.

K. Welfare Program Participation. This section gathered detailed information aboutTANF/AFDC and food stamp benefits that the respondent may have received in the pasttwo years. For both types of assistance, periods in which the respondent’s benefits werereduced or eliminated were identified, as were strategies for coping during such times.Current TANF/AFDC or food stamp recipients were asked about any requirements theyhad to fulfill (job search, training, etc.) in order to receive these benefits. For respondentswith children, questions were asked about benefits received in the previous year throughthe supplemental food program for Women, Infants and Children (WIC) and the free orreduced cost school breakfast and lunch programs. Questions were also asked about effortsto obtain government assistance in paying for childcare and to obtain Medicaid or SCHIPhealth insurance for the focal children. The section concludes with questions asked of allrespondents about experience with the Earned Income Tax Credit Program.

L. Education and Training. This series of questions was asked for the respondent andhis/her spouse/partner. Questions included those about the highest grade completed,highest degree earned, participation in job training programs during the past year, andclasses taken for credit during the past year.

M. Housing and Economic Hardship. Questions covered the respondent’s livingarrangement, the name(s) of person(s) holding the lease or mortgage in the household, andthe amount of rent or mortgage paid monthly. Information was collected about financial

3-5

contributions by the respondent or his/her spouse/partner to children under 18 livingoutside the household. The effect of economic hardship on the family’s food consumptionand ability to pay for housing costs was also assessed. The last questions in this sectioncovered household telephone service over the past year.

N. Issues, Problems, and Social Services. Questions in this section covered therespondent’s state of mind, feelings about his/her child(ren), constructive activities thechild(ren) may have been involved with (e.g., extracurricular activities, clubs, sports, etc.),child(ren)’s behavior, the availability of social services in their community, problems thechild(ren) may have had in the past year and efforts to obtain help for those problems, therespondent’s involvement in volunteer and religious activities, and the respondent’sattitude toward taking risks.

O. Race, Ethnicity, and Nativity. Race and ethnicity were asked for selected familymembers. For household members who were born outside the United States, country oforigin and citizenship questions were asked.

P. Closing. A short series of questions elicited the respondent’s opinions about welfare andworking and about raising children. Closing questions asked for ZIP code, whether thehousehold had the telephone number since the month that the screener was finalized duringthe Cycle 1 survey (or since two years ago, for respondents who did not participate inCycle 1), home address, and, for families who used welfare services since January 1997,tracing information for use in a followup survey.

3-6

Table 3-1.Questionnaire Content by Type of Person, Family, or Household

Focal child RespondentContent areas Household Family Less than 13

years old13 years old

and older15 years old

and older16 to 24years old

more than 24years old

Respondent’sspouse/partner

Otherfamily

membersA: Student Status XB: Health Statusand Satisfaction

X X X X X

C: Parent/Child/Family Interactionand Education

X X X

D: HouseholdRoster

X

E: Health CareCoverage

X X X X X X

F: Health Care Useand Access

X X X X X X

G: Child Care XH: NonresidentialParent/Father

X X X

I: Employment andEarnings

X X X X

J: Family Income XK: WelfareProgramParticipation

X X X X X

L: Education andTraining

X X X

M: Housing andEconomic Hardship

X X X X X

N: Issues,Problems, SocialServices

X X X X X

O: Race, Ethnicity,and Nativity

X X

P: Closing X X

3-7

3.4 Cognitive Interviews

Cognitive interviews were used to probe respondent understanding of questions that were addedsince Cycle 1. A total of nine interviews were completed by telephone by senior and mid-levelWestat staff. The interviews were recorded and, in most cases, monitored by a staff member atthe Urban Institute. Five of the interviews were Option A (households with children) and fourwere Option B (adult-only) households. All households were below 200 percent of poverty.Households were recruited by sending flyers to 400 households that had participated in the Cycle1 pretest and that were classified as being under 200 percent of poverty at that time. The flyerasked respondents to call an 800 telephone number if they were interested in participating in thestudy. Participants were paid $20 for completing an interview. A total of 13 individualsresponded to the ad, 9 of whom participated in the interviews.

The interview itself consisted of asking respondents to “think out loud” when answeringquestions. The interviewer specifically probed particular items that were of interest (e.g.,defining TANF to determine whether respondents recognized this as the program that used to becalled AFDC or checking definitions of other key terms and questions). Findings from thecognitive interviews were combined with those in the pretest to make changes to thequestionnaire.

3.5 Pretest

As discussed in section 2.6, a CATI pretest was conducted for the screener and extendedinterview. Several questions in the extended interview were changed in response to results fromthe Urban Institute and Westat’s monitoring of live interviews. An interviewer meeting was alsoheld to get feedback about the questions and flow of the interviews; interviewer comments wereused to make minor changes to the questionnaire.

The pretest training was conducted on October 17 and 18, 1998. Following this, interviewingstarted and lasted seven days. Sixteen interviewers were trained.

3.6 Special Procedures

Data collection started for the main study on February 15, 1999. Although most extendedinterviews were handled as planned, there were issues that arose that required differentprocedures by specially trained interviewers. Special procedures included the use of proxy andfacilitator interviews for respondents who could not do the interview without assistance, rules forconducting interviews with respondents who moved or who were students living outside thehousehold, and strategies for dealing with cases in which household members were erroneouslyleft off of the household roster. These issues are described below.

3-8

3.6.1 Proxy and Facilitator Interviews

Proxy interviews were conducted when the selected respondent was physically or mentallyincapable of answering the questions or when he/she was not in the household because ofhospitalization, moving out of the household, or living away at school as a student. The proxyrespondent could answer questions without necessarily consulting the subject. Proxy respondentswere asked most of the questions in the questionnaire, except for those that required opinionsfrom the respondent (e.g., the respondent’s satisfaction with medical care).

Facilitator interviewers were used when the subject or household member was mentallycompetent to answer the questions and was present in the household but could not respondbecause of communication problems. This communication problem could have been ahearing/speech problem or a problem created because the selected respondent spoke a languageother than English or Spanish. The facilitator respondent relayed questions and answers betweenthe interviewer and the subject or household member. Both proxies and facilitators had to be atleast 16 years old. Cases with proxy or facilitator interviews were marked with variables forquestions that asked interviewers whether the interview was a proxy or facilitator interview,whether a proxy or facilitator respondent was available, the reason for the proxy, and the qualityof information obtained. A total of 605 proxy or facilitator extended interviews were attemptedand 578 interviews were completed. (The 27 cases that were not obtained were distributed acrossresult codes for language problems, cases that had been called a maximum number of times,refusals, and other nonresponse.) The number of proxy cases in Cycle 1 was 841 attempted and543 completed.

Guidelines were developed to avoid proxy or facilitator interviews in some cases. For childinterviews, efforts were made to find another respondent in the household who wasknowledgeable about the child and could replace the MKA rather than use a proxy. However, ifthere was no other household member who could be an MKA for the child, the interview wascompleted with a proxy or facilitator.

In addition, if a respondent was hard of hearing or deaf, we offered to call back using TTY/TTDservices. If the respondent preferred this method of doing the interview rather than using afacilitator, we would conduct the interview with the respondent. Ten interviews were conductedusing this method.

When proxy or facilitator interviews were conducted, additional questions were asked thatdetermined the first name, age, and relationship of the proxy or facilitator to the respondent andasked why a proxy or facilitator was needed. Additionally, another question about the quality ofthe data collected was asked at the end of the interview for the interviewer to answer. Becausewe anticipated that some interviews would change from facilitator to proxy interviews (e.g., if itbecame too difficult for the facilitator to repeat questions or if the facilitator was moreknowledgeable about some issues than the respondent), we also asked interviewers if theinterview was completed mostly by proxy, with a facilitator, or both.

3-9

3.6.2 Movers

Another special procedure involved selected respondents who moved from the household sincethe time of their selection. Once respondents were selected for an extended interview, everyeffort was made to complete the interview with that respondent (however, for some childinterviews, movers were replaced with another respondent—see below). Approximately 189questionnaires were attempted with movers. A total of 118 respondents were interviewed. (The71 cases that were not obtained were distributed across result codes for cases that hadnonworking telephones, cases that received the maximum number of calls, and othernonresponse.) In Cycle 1, 300 questionnaires were attempted with movers and 108 respondentswere interviewed.

Because many questions in the interview focused on household members, movers who werefollowed were asked to answer questions for the original household before the move. This wasnecessary to make the household match the original household in the first screening or partiallycompleted extended interview.

There were several rules that dictated how interviews with movers were handled. For example, inchild interviews, we called an MKA at a different household if both the MKA and the focal childmoved. Otherwise, if the MKA moved but the child did not, we selected a new MKA for thechild. If the child moved, but the MKA did not, we interviewed the MKA about the child, eventhough the child no longer lived in the household. However, for cases in which there were twoMKAs in a household and the MKA for the second child moved but the MKA for the first childwas still in the household, the interview for the second child was done with a proxy respondentin the same household.

Other rules dictated the order of interviews with movers. For example, if a household had a childwith a mover and an adult interview with a nonmover, the child interview was completed firstand then the interviewer called back the original household to contact the selected adult.However, if it was a household with selected adults and one adult had moved, we completed thefirst adult interview with the nonmover and then contacted the mover.

3.6.3 Students

Students who were selected as adults in an eligible household but who lived away at school werealso followed under some conditions. If a student interview was the first interview to beconducted in a household, the interview was done with an appropriate proxy in the sampledhousehold. This was done because it was thought that a proxy in the household would generallyprovide better information about the household income and health insurance than would astudent living away at school. The only time that a student interview would be the first interviewin a household was when it was the only interview in the household.

3-10

If the student interview were a second interview, efforts were made to obtain a telephone numberor location information for the student from the household. If the household refused to provide atelephone number or locating information, Westat requested that they provide the student withthe 800 telephone number. Westat checked with the household a few days later to see if they hadbeen able to give the 800 telephone number to the student. A total of 858 cases were moved tothe manual queue because the selected respondent was a student living away at school. Of these,730 interviews were obtained. (The 128 cases that were not obtained were distributed acrossresult codes for cases that were ineligible because the student did not consider the household tobe his or her usual place of residence, cases that received a maximum number of calls, refusals,and other nonresponse.) In Cycle 1, 347 cases with students were moved to the manual queueand 258 interviews were obtained.

If the household refused to provide contact information and/or refused to give the 800 telephonenumber to the student, the interview was conducted with an appropriate proxy who lived in thesampled household. However, if the respondent provided the 800 telephone number to thestudent but the student did not call Westat, another request was made of the household. After tworequests of the household, the interview was conducted with an appropriate proxy who lived inthe sampled household.

3.6.4 Missing Spouses/Partners

In both rounds of the survey, there were problems with respondents not listing theirspouses/partners as household members. In Cycle 1, reinterviews were attempted in 183households for 209 interviews. Reinterviews had to be conducted in Cycle 1 for cases in whichthis issue was mentioned by the respondent because important questions about the spouse/partnerwere not asked (e.g., income and employment).

In Cycle 2, a special edit was added to ask respondents whether a spouse was living in thehousehold if the respondent indicated he/she was married, but did not list a spouse on thehousehold list. The edit was used for 490 cases. Of these, 475 interviews were completed. (The15 cases that were not obtained were distributed across response categories for cases that hadbeen called a maximum number of times, refusals, and other nonresponse.)

3.6.5 Special “D Restarts” to Include Household Members Left Off the Household List

There were also approximately 130 cases in which other household members were left off thehousehold list (in Cycle 1, there were 110 such cases). The interviews were stopped as soon asthe problem was realized. Out of these, 114 interviews were completed. (The 16 cases that werenot obtained were distributed across response categories for language problems, cases thatreceived a maximum number of calls, refusals, and other nonresponse.) A special procedure wasused to restart the case in the household list section (section D). When section D was asked thesecond time, only questions pertaining to the missing person were asked. The missing person’s

3-11

name was added or corrected on the household list, all questions asked about that person, andthen CATI continued with the rest of the interview.

Table 3-2.Special Cases Results

Item

Number ofinterviewsattempted(Cycle2)

Number ofinterviewscompleted(Cycle2)

% of interviewscompleted(Cycle2)

% of interviewscompleted(Cycle1)

Proxy andFacilitatorsInterview

605 578 95.5 64.5

Movers 189 178 94.2 36.0

Students 858 730 85.1 74.4

Missing Spouses 490 475 96.9 71.8

Special “DRestarts”

130 114 87.7 NA

3.7 Significant Changes to the Content of the Questionnaire

Several changes to the questionnaire were made in mid-March. Other changes were made at thebeginning of the summer in early June. These are described below.

Section K. The skips in section K were changed for three items (K38, K38a, and K38b) to savetime and make the sequence more appropriate to the age of the focal child. These items askedabout receiving government assistance in paying for child care. Instead of asking these questionsof all respondents, starting in mid-March they were asked only of MKAs of children 12 years oldor younger.

Section O. On March 19, the words “Native American” were removed from a response categoryfor item O3 (what is your/[person’s] race?). This term was erroneously being recorded by someinterviewers as meaning someone who was born in America, rather than an American Indian. Aprobe was also added on March 19 so that if a respondent said that he/she was Native American,it was confirmed that the respondent meant American Indian.

3-12

Section P. Several changes were made in section P during March in response to comments fromthe Urban Institute. First, on March 12, the skip for item P4a (“Has your household had thistelephone number since {Cycle 1 date}?”) was changed so that it would be asked of all RDDhouseholds, not just recycled Cycle 1 cases. New RDD households were asked about the timeperiod of the current interview month minus 2 years.

In addition, starting March 19, item P3 (“...may I have your ZIP Code?”) was collected once perhousehold instead of for every extended interview. In addition, on this same date, items P4B to Cwere deleted (“...has anyone moved into the household since Cycle 1?”).

Summer Administration. Another area of change in the questionnaire was that screeners andextended interviews were altered somewhat in order to be appropriate for administration duringthe summer months. A similar set of changes was made during the summer in Cycle 1. Questionsabout household membership had to be changed to include children who were temporarily awayfor the summer. In addition, questions about school and child care had to be changed becauseduring the summer many children are out of school or temporarily away from home. A specialrelease group of 30,000 cases was released for the new summer questions. A total of 9,385households had at least one completed or partially completed interview in which the specialsummer questions were asked.

Changes were made to the screener in section C about children’s schooling and section G aboutchild care. Changes to the screener affected the interviewer instructions written in brackets onseveral screens (in effect between June 4 and September 26, 1999).1 The new instructions in thescreener indicated that children who were temporarily away for part of the summer should stillbe included as household members if they usually live in the household. The summerinstructions said,

[ALSO INCLUDE CHILDREN WHO USUALLY LIVE THERE BUT ARETEMPORARILY STAYING WITH RELATIVES, FRIENDS, OR AT A SUMMERPROGRAM.]

Interviewers were also given cards with these instructions on them, so that they had them to referto for screens that did not have these words.

New questions in section C (in effect between June 14 and September 8, 1999) asked whetherthe child was attending school or summer school, and, if the child was out of school, what gradethe child attended at the end of the last school year. In addition to the questions, one question(C3) had a few extra words added to it so that it would refer to activities during the regular

1 These dates were chosen as an approximation of when children might be away temporarily during the summer. A somewhat later start date(June 14) was chosen for changes to sections C and G. This date corresponded to common school schedules during Cycle 1 and was maintainedin Cycle 2. The end date for section C (September 8) was chosen to correspond to when most children would be back at school. The end datefor summer questions in section G (September 26) was made somewhat later because the child care questions focus on child care arrangementsover the last four weeks.

3-13

school year. New question-by-question (QxQ) specifications were given to interviewers for thesequestions so that interviewers would have explanations available to them online, as they did forother questions in the extended interview.

Changes to another section, section G (child care), involved adding new time references that tookinto account that children are often away during the summer months (these changes were ineffect between June 14 and September 26, 1999). Two versions of section G were used in the1999 survey. In one version (7,514 completed interviews), questions about child care referred tochild care during the month of May. These were used with all sample release groups that werereleased before June 14. This version of the questions was not used in the 1997 survey.For the summer release group that started on June 14, another version of section G was used(1,976 completed interviews). This version was similar to that used in the 1997 survey.Questions were designed to get respondents to focus on the child care they used during the timethat the child was home in the last month, even if it was not for the whole month. For example, ifthe child was only home for two weeks in the last month, we only asked about child care duringthe two weeks the child was home.

Within the time period in which the child was home, we also distinguished between time spent inschool and out of school. This was done because parents use different kinds of child care whentheir children are in school than when they are out of school. Whether we asked about the time inschool or out of school depended on how the child spent the majority of the time he/she washome. If the majority of the time at home was spent in school, we asked about school. Otherrules about how questions were asked are shown in the CATI specifications for section G(summer version, versions 1 and 2). The exact wording of each question depended on theanswers given within the interview.

Because the time reference varied by interview responses, the QxQs for these questions were notchanged to show the “tailored” time periods. For example, the QxQ for the question about howmany hours the child was in Head Start indicated that we were interested in the usual number ofhours per week during the last month that the child was in Head Start. The question on the screenmight have said something like, “In the three weeks that {CHILD} was in school in the lastmonth, how many hours per week was {CHILD} usually cared for in a Head Start Center?” Thedifference between the time references in the QxQs and the actual displays in the questions wasexplained to interviewers and did not pose a problem.

One difficulty did occur with the summer questions before their administration. For two days,June 1 and 2, 1999, questions that were part of the program in Cycle 1 during the summer of1997 appeared in sections C and G of the 1999 questionnaire. This was the result of a dateproblem within the CATI program. Because questions during the summer administration ofCycle 1 differed somewhat from the Cycle 2 questions in these sections, the data from affectedcases did not completely match the new questions. In total, 285 cases were affected, and 281 ofthese were completes. These cases have flags set to 1 in the affected sections so that they can beidentified easily.

3-14

3.8 Summary

The extended interview for the 1999 NSAF was initially developed in Cycle 1 of the study andwas modified in Cycle 2. Just as in Cycle 1, the design of the study called for multiple interviewsin a household, with specific rules about what type of interview could follow another and whatquestions could be omitted in second interviews. The interviews covered a wide range of contentareas, including health care, child care, education, employment, income, welfare participation,and the availability of social services.

A pretest for the 1999 NSAF was conducted from October 17 to 23, 1998, to test the questionsand the CATI program. Questions were changed in response to monitoring interviews andfeedback from interviewers.

Data collection for the extended interview started on February 15, 1999. During data collection,most extended interviews were conducted according to plan; however, some required specialprocedures. Guidelines were developed to handle special cases, such as those in whichrespondents were not physically or mentally capable of answering questions on the telephone,respondents had moved, and/or persons were inadvertently left off of the household list by therespondent.

As data collection continued, minor changes were made to the questionnaire in mid-March 1999.Questions were also added for interviews conducted during the summer months.

4-1

4. TRAINING

4.1 Overview of Training Plan

In Cycle 2 of the NSAF training, Westat covered many of the same key concepts that werecovered in Cycle 1 training. However, extra efforts were made to make the Cycle 2 trainingpackage somewhat different than it was in Cycle 1 by varying the mode of some trainingtechniques (e.g., the interview and methods of gaining cooperation were demonstrated both onvideo and in other training exercises). In both rounds of the study, the appearance of all materialswas standardized so that all trainers could follow the format, regardless of the author, and delivera consistent training program across groups.

Training sessions were organized according to standardized Westat procedures, with training forlead trainers that served as a dress rehearsal and check on training materials before they wereused with supervisors and interviewers. Training teams were organized with staff who haddistinct responsibilities (e.g., a lead trainer who delivered the training script, a group leader whoevaluated trainees, runners who helped trainees during interactives and role plays, etc.) so thattraining sessions flowed smoothly.

Initial training was provided to all interviewers on general interviewing techniques and the use ofthe computer system. The interviewers then received project-specific training that focused on theNSAF screener and extended interviews.

Training was conducted February 6 through May 1, 1999. Training sessions were held at allsites—Frederick, Rockville, and Chestertown, Maryland; Toms River, New Jersey; Toledo,Ohio; Sarasota, Florida; and Chambersburg, Pennsylvania.

As a final stage in the training process, interviewers conducted practice interviews until theyreached the desired level of proficiency. Interviewers who were not proficient enough receivedcoaching sessions and were monitored until proficiency was achieved or until they were releasedfrom the study.

After all interviewers started production, they received supplemental training about specificquestionnaire issues that were new since the initial training. They also received more training ingaining respondent cooperation. Interviewers were monitored throughout data collection as amethod of quality control.

Some interviewers also received training in how to handle special procedures. These includedinterviews with proxy respondents and persons who had refused to participate during an earliercall to the household. These cases were placed in a special queue so that only interviewers whowere trained to handle the cases would be given them.

4-2

We required 26 hours of training for experienced interviewers with NSAF or other RDD studies,and 32 hours for new interviewers.

4.2 Development of Training Materials

Before training, key members of the study area staff, the TRC operations manager, and seniorTRC staff developed training materials. Guided by an outline of all the concepts relevant to thestudy, the staff produced a complete set of training materials that complemented one another.These materials included the following:

n A Training Program Agenda. The agenda identified the format of the sessions (lecture,interactive, dyad role play, etc.), the topics to be covered (overview of questionnaire,particular questionnaire sections, etc.), and the length of time the session was scheduled totake (see exhibit 4-1). This document was used during training by the lead trainer andothers assisting in training to establish what materials should be used by the lead trainer, aswell as the interviewer, during each session. An abbreviated agenda was produced for theinterviewers that showed the sessions but not the lead trainer materials.

n Interviewer Training Manuals for CATI Interviewing. Two manuals were created toserve as training tools and a reference source during data collection. The first manual,NSAF Interviewer Manual, Volume I (Westat 1999), included an introduction to the study,contact procedures, information about administering the extended interview, andspecifications for each question asked in the first portion of the interview, the screener. Thesecond manual, NSAF Interviewer Manual, Volume II (Westat 1999), includedspecifications for each question asked in the extended interview.

n Lead Trainer’s Manual. This manual, NSAF Trainer’s Manual, Cycle 2 (Westat 1999)contained all material presented by the lead trainer. They included interactive scripts andexercises that were designed to fully test the level of an interviewer’s comprehension ofsurvey materials and procedures.

n Dyad Role-Play Scripts. Role plays were produced that focused on contact proceduresand provided practice on administration of the extended interview.

4.3 Organization of Training Sessions and Teams

4.3.1 Training Sessions

Separate training sessions were conducted with lead trainers, supervisors, and interviewers. Thelead trainer training was considered a dress rehearsal for subsequent training. The goals of thetraining were to (1) familiarize the trainers with the standardized materials used in training; (2)

4-3

ensure that the scripts were accurate and complete; (3) determine the time required for eachelement of training; and (4) serve as a final test of the CATI training account. Modifications tothe training materials were made based on the results of this dress rehearsal.

4-4

Exhibit 4-1. Lead Trainer Agenda

Session Length Topic Interviewer/Trainee Materials

1 30 min. Introduction n Agenda

n Two Manuals

n Screener Manual

n QxQ Manual

2 45 min. Demonstration Interview (Video)/Preparing to Interview n Key Questions Card

3 2 hrs.

15 min.

Screener/Contact Scenarios

(Includes 15 min. BREAK)

n PC

n Key Screener Concepts

n Telephone Company Recording/NR Coding Card

n CATI Abbreviations

5 30 min. Exercise on Contact Procedures n Exercise on Contact Procedures

1 hr. LUNCH (Weekend Sessions)

7 15 min. Overview of Extended Interview n Handouts

n Subject Topic Areas

n Flow of Work

8 1 hr.

30 min.

Interactive 1:

Child Interview – Sally

n PC

Exhibit 4-1. Lead Trainer Agenda (continued)


(Includes 15 minute BREAK)

10 45 min. Strategies for Gaining Respondent Cooperation(Video)/Review of Commonly Asked Questions/ SensitivityIssues

n Key Questions Card

n Commonly Asked Questions

9 1 hr.

30 min.

Screens/Pathways, Part 1:

Industry and Occupation & Section I

Exercises

n Handouts

n Exercises

n PC

11 1 hr.

30 min.

Interactive 2:

Adult Interviews—Norma

Abbreviated Adult Interview—Christine

n PC

12 1 hr.

45 min.

Screens/Pathways, Part 2:

(Sections E, J, K)


n Handouts

n PC

13 30 min. Exercises on Screens/Pathways,

Part 2

n Exercises

14 15 min. Interviewing Respondents Under 18 Years of Age

4-5



1 hr. LUNCH (Weekend Sessions)

15 4 hrs. Interactive 3:

Child Interview—Michael and Alyssa

Abbreviated Adult Interview—Urie


n PC

16 4 hrs. ROLE PLAYS


n Role Plays (7)

n PC

17 4 hrs. LIVE INTERVIEWING (TRAINING)


18 2 hrs. Discussion

4-6


4-7

Lead trainer training was held on January 18–20, 1999, and lasted 22 hours. After modificationswere made to the training materials from lead trainer training and review, supervisor trainingswere held. Supervisors attended training sessions held on February 1 (a group of supervisors wasalso trained in sessions on February 2 and 3) using the revised standardized training scripts.

Supervisors trainings allowed for further comments on the training materials and provided anopportunity for the lead trainers to rehearse their roles and be completely prepared beforeinterviewer training. This approach resulted in a training program that ran smoothly, remained onschedule, and built the interviewers’ confidence in both the material and the people who trainedthem. All members of the training teams, as well as shift supervisors, attended the supervisortraining or one of the interviewer training sessions before their active involvement in the trainingprocess.

Interviewer training started on February 6. Trainees were divided according to their experiencelevel, and received different training programs based on their experience with the NSAF or otherRDD studies. Training required 26 hours for experienced interviewers and 32 hours for newinterviewers.

Interviewers attended training in group sessions. Each trainee was given an NSAF interviewer’smanual to use as a reference tool, and all training used the online CATI system developed for theNSAF.

4.3.2 Training Teams

The training team for each group consisted of a lead trainer, a data display operator, a groupleader, and two runners. The roles and responsibilities of the team members are described below.

Lead Trainer. Lead trainers were responsible for the overall presentation and the pace oftraining. All lead trainers for the NSAF had several years of training experience and were wellversed in training techniques and group control. It was the role of the lead trainers to concentrateon delivery of the material; trainee evaluation was the responsibility of the group leader.

Data Display Operator. The data display operator was responsible for following the leadtrainer’s script and making entries in the master terminal that displayed the CATI interview onlarge screens in the front of the training room. The data display operator was familiar with theCATI program and entered responses given by the lead trainer.

Group Leader. The group leader was responsible for taking attendance, coordinating traineeevaluations, troubleshooting, and making certain that all materials were available when needed.The group leader was also responsible for pairing trainees for role plays and for making sure that


4-8

each person was sufficiently monitored in role-play situations to evaluate performance. Mostimportantly, the group leader was responsible for coordinating an evaluation of each trainee.Information from each member of the training team was compiled and used to determine whethera trainee was ready for live interviewing. If not, a remedial training program was implemented orthe person was released. Remedial training typically involved more role play. If the additionalrole play did not result in sufficient performance improvement, the person was released. Onceinterviewing began, the group leader was responsible for ensuring that each trainee wasadequately monitored and provided feedback. The role of group leader was filled by shiftsupervisors with many years of experience working with interviewers.

Runners. As the name implies, runners moved around the training room making sure eachtrainee kept up with the script and assisting trainees who made entry errors that put them in aninappropriate place in the interview. Two runners were assigned to each group. Runners wereshift supervisors and senior interviewers who had direct experience working with interviewers ina one-on-one setting.

Before interviewer training, data display operators, group leaders, and runners attended ameeting during which roles and responsibilities of each position were discussed. The work of thetraining teams was coordinated and closely supervised by the operations manager, as well as bythe project director and the director of the TRC.

4.4 Stages of Interviewer Training

Interviewers were trained in three stages. The first two stages, General Interviewing Techniques(GIT) and Teltrain (CATI training), are standard for all CATI interviewers, while the last stage isspecific to the project.

4.4.1 GITEvery new interviewer participated in a four-hour GIT session; this training was supported byWestat and was not charged to the project. In GIT training, interviewers were introduced toWestat and to survey research, shown samples of types of survey questions and recordingconventions, and taught basic ways to obtain accurate data through listening and probing. Theylearned confidentiality procedures and methods for gaining respondent cooperation. The formatincluded a PowerPoint presentation that was interspersed with exercises, interactive lectures, roleplays, a question-and-answer period, and practice exercises. Each interviewer received a manual(the Westat GIT Interviewer’s Manual) that documented the material presented in the session.This session also allowed staff to identify those interviewers whose reading and speaking skillswere inappropriate for the study.


4-9

4.4.2 CATI Training with Teltrain

Before specific project training, each trainee participated in a two- to three-hour training sessionon using the CATI system. This session used an interactive, computer-assisted training programthat was supervised, but self-administered, and took each participant through the procedures forconducting interviews using CATI. The session instructed interviewers on the use of thecomputers, all Westat CATI recording functions, and special CATI commands. The scriptincluded practice with logging on to the computer and using the keyboard (particularly the keysthat control the flow of the CATI interview). This training also served as an opportunity toidentify trainees who could not use a keyboard skillfully. Those who could not learn to use akeyboard were released from the NSAF training program.

4.4.3 NSAF Project Training

After interviewers were trained in GIT and the use of the CATI system, they participated in atraining session devoted to the specific procedures and the administration of the NSAF CATIquestionnaire.

Because of the multiple skills interviewers needed, training focused on the techniques designedto cultivate these skills. This involved the active participation of all trainees by simulating theactual conditions of the interview. This approach required trainees to use the same proceduresand data collection instruments they used to conduct the survey. This approach is summarizedbelow.

Interactive Lectures. Interactive lectures were used to familiarize interviewers with thequestionnaire. They were conducted as mock interviews in which the trainer acted as therespondent and the interviewers asked the questions, using the computer to read the questiontext. In addition, the trainer took time to explain or define concepts pertinent to the NSAFinterviews or to ask the interviewer to read a definition or procedure from the interviewer’smanual.

The scripts used for interactive training were prepared using the Cheshire Automated TrainingScripts (CATS) System. CATS is a series of macros created in MS Word for Windows for study-area and TRC staff to develop scripted training materials. With this program, NSAF training staffcreated and saved training scripts. Standards of style have been developed so that each trainingscript looks uniform regardless of the author, and all training groups hear the same information,regardless of which trainer presents the material.


4-10

Dyad Role Plays. In dyad role plays, one trainee took the role of interviewer using the computerwhile the other played the respondent, both using a prepared script that was produced using theCATS system. Interviewers reversed roles after the end of each role play. Each interviewerparticipated in several dyads. Group leaders and other training team members monitored the roleplays.

Written Exercises. In addition, written exercises were given to the interviewers during trainingto reinforce what was learned during the interactive interviewing sessions. These exercises dealtwith contact procedures, defining household membership, and gaining respondent cooperation.

For the extended interview, trainers conducted intensive review of the QxQs of several sectionsof the questionnaire in short interactives. These QxQs were used to teach interviewers aboutquestions such as those on health care coverage, employment and earnings, family income,program participation, and industry and occupation. Exercises were included to reinforceconcepts presented. Interviewers were given time to complete the exercises independently, andthen each question was reviewed by the group as a whole. The lead trainer used an answer key sothat all interviewers heard consistent answers across training groups.

Practice Answering Key and Commonly Asked Questions. Frequently asked questions andanswers were discussed and reviewed throughout training. The questions dealt with both generalinterviewing issues and NSAF project-specific issues. There were two documents given tointerviewers to help them practice. One was a key questions card that contained the fivequestions most frequently asked by respondents (e.g., Why should I do this study? How did youget my number?) along with several suggested responses for each. Another document was a setof cards that showed other commonly asked questions and their answers. These questions wereanticipated to be asked less frequently than the key questions (e.g., Why don’t you just send methe questionnaire?). Cardstock copies of this document were designed as a stairstep arrangementthat allowed interviewers to go quickly to a tab for a particular category of questions and answers(e.g., sponsors of the study), reducing the time that the interviewer needed to look up responses.

Videotaped Introduction and Demonstration. In Cycle 2 training, two videotapes were createdto provide a different mode of training from the interactives and other activities. One videotapeintroduced the study and showed a mock interview with actors from beginning to end, includinga few questions from each section of the questionnaire. This videotape was used as part of a briefintroductory lecture that provided background about the purpose of the study, showed recentnewsclips about findings from the first round, and explained study concepts.

The other videotape used in training demonstrated techniques for gaining cooperation. Duringthis tape, interviewers watched actors, acting as interviewers and respondents, interact at thebeginning of a telephone call. Each vignette on the tape showed a different style for gainingrespondent cooperation and demonstrated techniques that did and did not work well. Trainees


4-11

discussed what the interviewer on tape did incorrectly and then watched the interviewer on tapetry again and be more successful at gaining cooperation.

Practice Account. The final element of training consisted of interviewers logging onto theNSAF practice account and interviewing respondents using the same instrument that would beused on the production account. The practice account is discussed in more detail in section 4.7.1.

4.5 Schedule and Number of Interviewers Trained

Training was conducted by Westat training staff at all sites. The training dates, locations, andnumber of interviewers are included in table 4-1. The number of interviewers shown are thosewho completed training and worked on the production account. As noted earlier, a total of 546interviewers successfully completed training.

Bilingual training was conducted on March 13, 1999. Thirty-one Westat interviewers completedtraining.

Table 4-1.Training Dates, Locations, and Number of Interviewers, 1999

SITE GROUP DATES TOTAL

Frederick 1 (cancelled)

2 2/10–2/15 39

3 2/15–2/21 18

4 3/1–3/7 33

5 3/15–3/21 15

6 3/15–3/21 25

7 3/29–4/10 12

Total 142


4-12

Twelve Oaks 1 2/6–2/10 30

2&3 2/6–2/13 19

4 2/10–2/16 13

5 2/22–2/28 25

6 2/22–3/2 24

7 3/1–3/7 30


4-13

Table 4-1.Training Dates, Locations, and Number of Interviewers, 1999 (continued)

SITE GROUP DATES TOTAL

Twelve Oaks 8 3/8–3/15 15

9 3/8–3/14 41

Total 197

Toledo 2/7–2/15 52

Toms River 2/21–3/6 29

Sarasota 1 2/28–3/6 36

2 4/5–4/11 33

Total 69

Chestertown 3/21–3/27 31

Chambersburg 4/25–5/1 26

TOTAL 546


4-14

4.6 Training for Special Procedures

Special trainings were held to teach interviewers how to handle the procedures described insection 3.5 (pretest). These trainings were conducted with interviewers assigned to work on themanual queue. Interviewers who were assigned to the manual queue were those who TRC staffjudged to be able to handle more complex cases. The CATI scheduler delivered only manualqueue cases to these interviewers.

Cases were moved to the manual queue by the CATI program when circumstances warranted areview before interviewing continued. This included cases with selected persons who had movedfrom the sampled household and sampled persons who were students living away at school.Cases were also moved to the manual queue by project staff resolving cases in the problemreview queues. These problems were much more varied but included cases requiring a proxyinterview, problems in completing the household verification section of the interview, cases inwhich the respondent claimed to have already completed the survey, cases in which the selectedrespondent had moved or should be called at a different telephone number, cases in which thetelephone number was now nonworking, and cases that needed special attention from project andprogramming staff.

When a case was moved to the manual queue, the program printed out a manual sheet fromwhich the trained interviewer would work. The manual sheet included important informationabout the case that was much more detailed than that available to the interviewer accessing casesfrom the scheduler. The manual sheet also included a call record upon which results of theattempt were recorded along with any comments from the interviewer who had made theattempt. It was also on the manual sheet that the supervisor recorded specific instructions, ifrequired.

Training for the manual queue took place on April 4, 1999, in Frederick, Maryland, with 28interviewers. Training took four hours. It included a comprehensive discussion of the conceptsbasic to the NSAF interview: interview types (adult, child, straggler), household membershipdefinitions, family relationships, the expected order of interviews, and result codes. Interviewerswere trained to read and record data on the manual sheet and to read and use a printout thatdescribed the family relationships in the household. Training also included a review of theguidelines designed to help the interviewers decide what action to take to resolve problem cases.Training was provided in each of the special areas for which procedures had been developed:selected persons who had moved from the household, selected persons who were students livingaway at school, and selected persons who were unable to complete without the use of a proxy.The remainder of the training session was spent in completing and discussing a written exercisethat focused on the NSAF’s basic concepts. Trainees also used a CATI role play to practiceproxy interviewing.


4-15

One manual interview involved in training was bilingual; interviewers began conductinginterviews in Spanish after training. Other bilingual interviewers were trained individually asthey were recruited to be part of the project. In total, five bilingual interviewers worked themanual queue.

During the first weeks of the study, an average of 25 interviewers worked on the manual queue.After the study was halfway completed, an average of 15 interviewers consistently worked onthe manual queue. The work was overseen by one full-time TRC supervisor and one part-timeTRC supervisor, with guidance from a project staff member. The supervisors followed specificguidelines in resolving and finalizing manual queue cases.

4.6.1 Refusal Avoidance and Conversion

During the regular project training, all interviewers received information in refusal avoidancemethods. Further strategies were reviewed at all sites in special refusal avoidance meetings.Included in the effort to improve respondent cooperation were special coaching sessions bysupervisors assigned to small groups of interviewers. In these meetings, the emphasis was on thereview of good interviewing techniques by direct observation and intervention. In addition,supervisors selected experienced interviewers with higher-than-average cooperation rates ineither the screener, the extended interview, or both for refusal conversion activities.

Refusal conversion focused on attempts to persuade respondents who had previously refused toparticipate or to complete an interview. Interviewers received special training in recontacting andencouraging participation by those respondents who had originally declined. The refusalconversion training sessions lasted between one to two hours and covered specific conversionstrategies. They explored common reasons for refusals, reasons specific to the NSAF, and theimportance of addressing respondent concerns with appropriate responses.

Refusal avoidance meetings began one week after the start of interviewer training at each site.Conversion sessions began the first week in March. By July 5, almost all interviewers whoremained on the study had received refusal conversion training. The only site that did not receiverefusal conversion training was Chambersburg, Pennsylvania. Training was planned for this site,but since fewer interviewer hours were needed as the study progressed, the Chambersburg sitewas no longer used for conversion interviews.

4.7 Interviewer Performance

Interviewer performance was evaluated in several ways—through practice interviews,examination of cooperation rates, and monitoring of skills needed for interviewing effectively.


4-16

4.7.1 Practice Interviews

To maximize performance when contacting NSAF respondents, interviewers were first assignedto the practice account. Because the practice account design was a mirror image of theproduction account, interviewers were able to gain experience in gaining cooperation and masterthe skills needed to administer the instrument. The sample of telephone numbers for the practiceaccount was drawn from the same geographic areas as the production account to simulateexperiences with households with similar characteristics.

The practice account was available as a training device from the beginning of the study throughthe first week of May 17, 1999. Interviewers were initially assigned to this account after training.Before beginning interviewing for production, each NSAF trainee worked on the practiceaccount until he/she had achieved a level of proficiency that met the needs of the study.

4.7.2 Criteria to Bring Online and Continue Interviewing

For NSAF interviewers, Westat established the minimal acceptable cooperation rate on thepractice account at 55 percent (with a sample of approximately 50 cases). Interviewers whoachieved this cooperation rate were moved to the production account. After interviewers beganworking on the production account, their work was closely tracked through monitoring andreview of various reports.

Interviewers whose performance fell below acceptable levels attended additional coachingsessions with an emphasis on gaining respondent cooperation and answering respondentquestions. Approximately 10 percent of all the interviewers who completed training fell into thiscategory. These sessions required an additional 1.5 hours to 2 hours per interviewer.Performance was monitored closely by a coaching coordinator to determine whether minimalperformance levels had been reached.

In addition to having their cooperation rates tracked, interviewers were monitored by TRCsupervisors and training staff to determine if the following skills were demonstrated: use of aconversational style; reading fluency; ability to answer respondent questions quickly, accurately,and completely; ability to gain respondent cooperation; reading screens verbatim; and use ofneutral probes. These skills were evaluated throughout the study.

4.7.3 Supplemental Training

Interviewer memos were distributed to the staff to clarify and reinforce issues, as well as toinform staff of procedural changes. A total of six memos were distributed to interviewers.


4-17

4.7.4 Gaining Respondent Cooperation

Approximately one week after each training session, interviewers began attending sessionsdesigned to maximize respondent cooperation. Two types of training sessions were conducted: amonitoring session designed to accommodate a small group of interviewers, and an interactivesession with a larger number of interviewers. The small group sessions involved monitoringinterviewers identified as highly effective, with discussion of the techniques used to gain andretain cooperation. These sessions ranged from 1.5 hours to 2 hours in length. The larger groupsessions included reinforcement of concepts introduced in project training, techniques observedduring monitoring sessions, and discussion and practice addressing specific objections orquestions. The large group sessions were also 1.5 hours to 2 hours long. All interviewersattended both the small and the large group sessions.

The following techniques were used to identify and reinforce behaviors effective in gainingrespondent cooperation:

n The operations manager sent a daily priority list to shift coordinators. It included lists ofinterviewers by name and/or category (practice, Spanish, etc.) targeted for heavymonitoring because of recent change in status, such as role play to practice, practice toproduction, special assignment; lower-than-average cooperation rates; and evaluation forspecialized tasks, refusal conversion, and manual queue. The issues that were to be focusedon during monitoring were also provided, such as the interviewer’s ability to answerrespondent questions/concerns quickly and accurately and to read all screens (in particular,the screener introduction) at the appropriate pace and tempo for the respondent; readscreens verbatim; and probe neutrally and appropriately. For refusal interviewers, theemphasis was on the ability to engage respondents and use appropriate techniques.

n Supervisors provided feedback to interviewers on an individual basis after monitoringsheets had been completed. This included feedback on poise aspects of the interview andsuggestions for improving performance.

n Shift coordinators sent daily reports regarding interviewer performance to the operationsmanager. Reports identified strengths and weaknesses as reported in monitoring sheets.Shift coordinators also provided input on interviewers recommended for special tasks.

n Shift coordinators’ reports were used in combination with cooperation rates to identifyinterviewers for refusal conversion and other specialized tasks.


4-18

4.7.5 Interviewer Meetings

Meetings were held with the interviewing and supervisory staff to reinforce procedures, reviewpoints of emphasis, provide updates on procedures, and inform staff of study progress.

4.8 Summary

In summary, the training plan for the NSAF was developed according to standard Westatprocedures and included a series of three training programs for interviewers: GIT, training onCATI, and project-specific training. Project-specific training included interactive lectures, roleplays, written exercises, videotapes, and frequent review of answers to questions thatrespondents commonly ask.

Training for interviewers was conducted from February 6 through May 1, 1999. After training,interviewers conducted live practice interviews until they had achieved the desired level ofproficiency. After production interviewing started, interviewers received supplemental trainingabout new issues in the questionnaire and methods of gaining respondent cooperation.Interviewers were monitored throughout the course of the study.

5-1

5. SCHEDULING AND RELEASE OF WORK

The strategy for scheduling and releasing telephone numbers to interviewers is described in thischapter. The chapter begins with a discussion of the steps that were taken to prepare the samplefor interviewing. The next several sections describe the initial release and followup strategies.The chapter ends with a description of the outcome of the calling strategies.

The scheduling and release of work were also affected by the sample’s division into releasegroups. Each release group contained 2,000 numbers per site, for a total of 30,000 numbers. Tworelease groups, one of which was used as a predictor sample, were scheduled so that they wouldbe completed before June 14, 1999. Another release group, the summer sample, was alsoreleased on June 14. These groups and other aspects of the sample are described in 1999 NSAFSample Design Report, Report No. 2 in this methodology series.

5.1 Processing the Sample of Telephone Numbers before Calling

There were 383,653 telephone numbers selected for the study. Of the 383,653 telephone numbersselected, approximately 14 percent (52,480) were removed before the numbers were turned overto interviewers for screening. Of these, nearly 4 percent (14,630) were eliminated because theywere listed only in the yellow pages, and 10 percent (37,230) were eliminated by a computersystem that dials numbers to eliminate nonworking numbers. This computer can detect thetritone signal for a nonworking number very quickly, usually without an audible ring of anytelephone number that is tested. In addition, 620 numbers were eliminated because they werenumbers drawn from the new portion of the Cycle 2 sample that were duplicates of the telephonenumbers being reused from Cycle 1.

The remaining 331,173 telephone numbers were sent to a reverse directory service for addresses.From this service, addresses were obtained for 146,835 (44 percent) households. Later, after datacollection started, another service was also used to obtain addresses (43,987 additionaladdresses) for households for which addresses could not be obtained by the first service.Addresses from both services were eventually used in different refusal mailings to try to ensurethat a contact with the household was made by mail. The second mailing service had someadvantages over the first because it made apartment numbers available. Because the secondservice was not being used at the beginning of the study, the first seven mailing groups used thefirst service for first refusal letters and the second service for second refusal letters. For the lastfive mailing groups, the second mailing service was used for first refusal letters.

On June 18, 1999, 10 percent of the cases in the summer release group (3,000 cases) weredeleted to save time and money. Thus, a total of 328,793 telephone numbers were called. Ofthese, addresses were found for 180,479, or 54.9 percent.

5-2

5.2 Advance Mailings

Two types of advance mailings were sent to respondents. One type was sent to the addresses fortelephone numbers that we called in 1997. On January 21, 1999, we sent special letters to thisgroup (101,379 letters were mailed) telling them about the upcoming survey, as well asproviding them with some of the findings (exhibit 1-1). The findings were geared specifically tothe respondent’s state if he/she lived in one of the 13 intensively studied states and nationally ifnot.

Before screening, another type of letter was also sent. A prenotification letter from the Westatproject coordinator that encouraged the household’s participation was sent to all households forwhich we had addresses (exhibit 1–2). Included in this mailing was a study brochure (exhibit 1–3). The letter was mailed to coincide as closely as possible to the initial release date of thetelephone number. The mailing schedule is shown in table 5-1.

Table 5-1.Mailing Schedule

Batch number Number ofaddresses

Approximatemailing date

1 23,313 2/5/99

2 3,728 2/12/99

3 11,043 2/19/99

4 7,946 3/5/99

5 11,221 3/12/99

6 13,488 3/19/99

7 14,750 3/26/99

8 14,392 4/2/99

9–11 (one batch) 35,370 5/14/99

12 11,584 6/4/99

5-3

5.3 Answering Machines

Studies indicate that leaving a message on an answering machine seems to increase cooperationrates (Xu et al. 1993). Apparently the message acts as an advance letter in that it legitimizes thestudy, allows the respondent time to make an informed decision, and distinguishes the surveytelephone call from telemarketing calls. Because of this, the message below was left the firsttime an answering machine was encountered at a telephone number.

Hello, I’m calling for the National Survey of America’s Families. This isa study to see how recent changes in federal laws affect people’s lives inyour community. I’m not asking for money—this is a study for privatefoundations on education, health care, and other services in the state of{STATE}. We will call you back in the next few days.

Messages were also left whenever an answering machine was encountered after the “strategy”changed (e.g., the case moved into a refusal or language treatment; for the latter, messages weretranslated into Spanish) (see sections 5.7 and 5.8 for a discussion of the treatments). There werealso several special situations in which messages were left. For example, during the manualoperation (described in section 5.9) an interviewer might leave a message geared to thecircumstances surrounding the call.

5.4 Initial Release

The initial calls for cases that had also been part of Cycle 1 were treated differently than casesthat were new in Cycle 2. To build on information obtained in Cycle 1 about when respondentswould be at home and perhaps be receptive to our call, complete and ineligible cases from Cycle1 were called within the time-slice queues (see section 5.5 below) in which they had beenfinalized two years earlier.

For all other cases, our initial release strategy was based on the premise that telephone numbersfor which an address was obtained were more likely to be residential. Residential numbers aremore likely to be answered in the evening than during the day, so they were first contacted in theevenings after 7 p.m. (within a time zone) or on weekends. Those without addresses wereassumed to be mostly business, nonworking, and unlisted numbers. Because of this, their releasewas heavily weighted toward daytime calling so that most of the nonworking and nonresidentialnumbers would be eliminated on the first call. This daytime release for unlisted residentialnumbers had the added advantage of allowing interviewers to leave an answering machinemessage before contact. We hoped this would act as an advance letter at those households.

5-4

5.5 Time-Slice Strategy

If the initial call attempt resulted in no answer, a busy signal, or an answering machine, theautomatic call scheduler would place the telephone number into time-slice queues so thatadditional calls would be made over several days at several different times of day. The goal herewas to find a time when someone would answer the telephone (Brick et al. 1996; Sebold 1988).

The time slices were defined as Saturday 10 a.m. to 6 p.m.; Sunday 2 p.m. to 9 p.m.; weekdays,9 a.m. to 2 p.m., 2 p.m. to 6 p.m., 6 p.m. to 7:30 p.m., 7:30 p.m. to 9 p.m., and 6 p.m. to 9 p.m.If, after seven calls, there was still no answer, the telephone number was temporarily retired.

Those with an answering machine contact in their history were re-released after a one-weekretirement for seven more calls within the time slices. This re-release protocol happened up tothree times for a total of 28 attempts. After 28 tries, the number was permanently retired andassigned the code “NM,” for no contact except for an answering machine. One percent (3,891) ofthe telephone numbers ended up as NM after this procedure.

Those without an answering machine in their history were retired for two weeks, at which pointthey were re-released for seven additional calls within the time slices for a total of 14 attempts.After 14 calls the telephone number was coded “NA,” for no answer.

On April 9, a new rule for NAs was put into effect to save time and money. Instead of callingnumbers that no one answered 14 times, nine calls were made, with the last two calls separatedby 10 days. The extra waiting period between each of the last two calls was expected to result ina greater chance that someone would answer the call. At the end of the survey there were 24,842NAs (6.4 percent of the sampled telephone numbers). At the end of the survey in Cycle 1, therewere 25,299 NAs (5.2 percent of the sampled telephone numbers).

5.6 Maximum Call (MC) Limits

When a person answered the telephone, the telephone number was removed from the time-slicestrategy described above. Once contact was made, all subsequent calls were based upon therespondent’s assessment of the best time to call or the interviewer’s suggestion of the best time(either in terms of an exact appointment or a general best time to call [e.g., day, evening, orweekend]). The maximum call counter for these cases for both the screener and the extendedinterview was set at 50 each. This high limit was set to allow enough calls for two refusalconversion efforts and calls in Spanish. As a result, only 0.6 percent of the calls ended as MC atthe screener level (2,445) and 2.3 percent ended up as maximum calls at the extended level(1,305).

5-5

5.7 Language Strategy

In households where there was no adult with the English communication skills necessary tocomplete the screener or where the selected respondent could not communicate enough tocomplete the extended, interviewers coded the telephone number as a “language problem.” Thiscategory combined two situations: The household might speak a language other than English orthe respondent might have a hearing and/or speech problem that hampered communication.These cases went to special “workclasses,” one for non-English problems and one forhearing/speech problems. The classification of the case being in the language strategy remainedmarked on the extended interview after it was complete.

A workclass defined a “group of work” which a special group of interviewers was especiallytrained to handle. Interviewers who were effective at dealing with elderly, infirm, or hard-of-hearing respondents were assigned the hearing and speech problems. If the specializedinterviewers were unable to complete interviews, they sent the cases to interviewers trained toconduct proxy interviews (see section 5.9).

Bilingual interviewers, who were trained to interview in both English and Spanish, worked onthe non-English cases. If the bilingual interviewer was unable to conduct the interview in eitherlanguage, the case went to interviewers trained to conduct proxy interviews (see section 5.9).Bilingual interviews received four more hours of training than traditional interviewers. Duringthis training, the Spanish language version of the questionnaire was practiced through interactivelectures and dyad role plays as described in chapter 4.

5.8 Refusal Conversion Strategy

When a respondent refused to be interviewed for either the screener or the extended interview,the CATI program asked the interviewer to complete a Non-Interview Report Form (NIRF). TheNIRF was designed to collect information to help interviewers convert refusals. It also providedsupervisors with sufficient information to determine whether a particular refusal should berefielded. Refusal conversion was not attempted in situations in which the respondent usedprofanity or was otherwise rude or threatening. This happened very rarely.

Information on the NIRF included the following:

n Where the respondent broke off1

n Why the respondent broke off

n What the respondent said that might help a converter anticipate questions or concerns

1 At the screener level, more than 83 percent of all breakoffs took place during the screener introduction screens. At the extended level, more than47 percent of the breakoffs occurred by the beginning of section B; 61 percent had occurred by the beginning of section E. The percentage ofbreakoffs in sections F through P varied from 4.00 percent to 0.01 percent.

5-6

n The respondent’s gender

n Whether the respondent sounded elderly or not

Refusal conversion training began the first week in March and continued through July 5 aspotential converters were identified.

Additional refusal workclasses were used in the first two months of the study to put specializedrefusal converters on cases in states that showed the lowest cooperation rates in Cycle 1: NewYork, New Jersey, and Massachusetts. These workclasses were formed to determine whetherhigher cooperation rates could be obtained with a specialized workforce. However, it was foundthat the cooperation and worked rates in the northeastern states were lower in the first twomonths of data collection than they were in other sites and proportionately lower than Cycle 1 aswell. On April 23, it was decided that these “northeastern” workclasses would be collapsed intonon-northeastern workclasses, meaning that these cases would no longer be assigned only toparticular interviewers.

A total of 85,211 households refused the screener. Of these, 59,139 (69 percent) had an addressthrough one of two address services used by the study. At the extended level, 13,568 sampledpersons initially refused to be interviewed, of whom 9,077 (67 percent) had an address throughone of the address services. The refusal treatments for households with addresses were based onthe results of a series of experiments conducted in Cycle 1 that were described in 1997 NSAFResponse Rates and Methods Evaluation, Report No. 8. (See also Report No. 7 in the 1999NSAF methodology series.) A new address service was employed in addition to the old onestarting May 5, 1999. Because these addresses were not available at the beginning of the study,they could only be used for refusal conversion and not for other letters, such as theprenotification letter. The strategies for converting respondents with and without addresses aredescribed in the following sections.

5.8.1 Refusal Conversion Strategy for Telephone Numbers without Addresses

Refusal conversion for telephone numbers without addresses started on March 10, 1999.Refusals at both the screener and extended levels were re-released a minimum of two weeks afterthe initial refusal was obtained. Specially trained interviewers then attempted conversion. If theywere unsuccessful, the case was held for an additional minimum of 10 days before a secondattempt at conversion was made. For the most part, refusal converters had to rely on theinformation on the NIRF and their interviewing skills when attempting to convert the household.However, at the extended level interviewers offered a $25 incentive at the first refusal. Refusalswith mailing addresses received both letters and incentives to aid in the conversion process.These procedures are described below.

5-7

5.8.2 Refusal Conversion Strategy for Households with Addresses

Several experiments were conducted in Cycle 1 to develop a strategy to optimize refusalconversion. These experiments are described in 1997 NSAF Response Rates and MethodsEvaluation, Report No. 8 in the Cycle 1 methodology series.

Beginning March 13, 1999, refusal cases with addresses were released for conversion attempts.The strategies for mailings before conversion attempts are outlined below.

Screener-First Refusals. In the first seven mailing groups, express mail letters containing $5were mailed to respondents. Mailings were timed to arrive on a Friday and the conversion callsbegan on a Saturday. Most weeks, each telephone number had been dialed at least once by thefollowing Monday. The same process was followed for mailing groups 8 through 11, butincluded in these groups were addresses obtained from a second address-matching firm. Thisfirm had addresses that the first matching firm did not.

Screener-Second Refusals. Second refusals in the first 11 mail groups were sent another letterto urge their cooperation. This was sent by regular mail and contained no money. In the firstseven mail groups, letters were sent to addresses identified by the second address-matching firmthat had not been available at the time of the first refusal mailing. These cases received anexpress mail letter containing $5.

Extended Interviews—First Refusals. First-refusal letters were sent by express mail andcontained $5. In the letters, an additional $20 was offered if an extended interview wascompleted.

Extended Interviews—Second Refusals. Second-refusal letters at the extended level were sentby regular mail. No additional incentives were offered at this level. In the first seven mailgroups, letters were sent to addresses identified by the second address-matching firm that had notbeen available at the time of the first refusal mailing. These cases received an express mail lettercontaining $5. In the letter, an additional $20 was offered if an extended interview wascompleted.

Subsampling of Refusals. On July 27, 1999, a total of 1,800 screener refusal cases weresubsampled and closed out to save time and money. The subsample for the refusals excluded thefollowing types of cases: hostile refusals, language problems, problem cases, those with specificor general appointments, and those that were being mailed letters at the time. The two strata (firstrefusals and second refusals) were sorted by site, listed status, and original sampling stratum.There were 6,810 first refusal cases eligible for sampling and 3,109 second refusals. We selected900 cases in the first-refusal category and 900 cases in the second-refusal category and excludedthem from all subsequent follow-up activities.

Summer Release Group Experiment. Starting June 14, 1999, an experiment was done to testthe effect of sending an incentive at the screener level by express mail at the second rather than

5-8

the first refusal. In the experiment, a regular letter was sent at the first refusal and an expressmail letter with $5 was sent at the second refusal. The results of this experiment are described in1999 NSAF Response Rates and Methods Evaluation, Report No. 7 in this methodology series.

5.8.3 Special Incentives

In general, interviewers were to follow instructions on the CATI screens and not offer extramoney to respondents; however, they were given some leeway to offer money to a secondrespondent in the household if the first person had received an incentive.

In addition, starting September 20, 1999, interviewers were asked to offer all respondents $50 tocomplete the extended interview (they were offered $45 if they had already received $5 after anearlier refusal). This offer was made to households both with and without addresses.

In total, 1,240 respondents were contacted in 1,170 households. There were 563 interviewscompleted, one ineligible case, and 407 refusals. Thus, the cooperation rate for extended contactsafter September 20 was 58.1 percent.

5.8.4 Results of Conversion Effort

The results of the conversion effort are shown in table 5-2. The cooperation rate is calculated as

Complete interviews + ineligible interviewsComplete interviews + ineligible interviews + all initial refusals

and the conversion rate as

Complete interviews + ineligible interviewsComplete interviews + ineligible interviews + all final refusals

In Cycle 2, response rates were also calculated at the household level, rather than just at theperson level as they were in Cycle 1. A more complete discussion of response rates andcomparisons to the rates from Cycle 1 is presented in the 1999 NSAF Response Rates andMethods Evaluation, Report No. 7 in this methodology series. Table 5-2 shows how conversionand cooperation rates for the household level differ from those at the person level.

5.9 Manual Treatment

Some of the work was so specialized, varied, and demanding that it required the use of a cadre ofhighly skilled, specially trained interviewers. These interviewers worked on the “manual queue”and carried out various tasks, including the closeout of one of the first release groups (thepredictor sample), which was done on paper in the final weeks to ensure that all cases were

5-9

worked thoroughly. Manual-queue interviewers also worked on cases that had issues whoseresolution was not straightforward. As described in section 3.6 about special procedures, thiswork consisted of problem cases (e.g., missing persons in the roster), sampled persons who hadmoved since the screener, students sampled at home who were away at school, interviews withproxy respondents, and so on. Often these cases had detailed histories that interviewers needed toread and understand in order to conduct the interview appropriately. Once read and absorbed, aninterviewer was able to “call up” the case “manually” by entering the identification number intothe computer system. Descriptions of the major categories of work in the manual queue follow.For more details about the procedures and the number of cases in each category, see section 3.6.

Table 5-2.Cooperation and Conversion Rates for the Screener and Extended Interview*

Screener Extended (Individual Level) Extended (Household Level)

Initial releaseComplete + ineligibles 105,122 40,380 36,624Refusals 85,207 13,563 11,577Total 190,329 53,943 48,201Cooperation rate 0.552 0.749 0.760

All conversionsComplete + ineligibles 42,479 7,827 6,992Refusals 42,728 5,736 4,585Total 85,207 13,563 11,577Conversion rate 0.498 0.577 0.604

* Several points should be noted about the table. First, the numbers shown are unweighted. Second, the screener numbers include 1,800subsampled refusal cases (900 first refusals and 900 second refusals) that were closed out of the sample (see section 5.8.2). Third, the extendedinterview numbers shown include both completed and partially completed interviews. Finally, it should be noted that a new partial completedefinition was used in Cycle 2. Interviews that were finished through section D were considered complete. In households with multipleinterviews, if the first and later interviews were partial completes, only the first interview was retained. In Cycle 1, interviews that were finishedthrough section K were counted as complete.

Proxy and Facilitator Interviews. As noted in section 3.6.1, proxy interviews were conductedwhen a respondent was unable to complete the interview because of a physical or mentalimpairment that limited his/her ability to communicate or when he/she was not in the householdfor a few select reasons (e.g., hospitalization). Facilitator interviews were used when therespondent was mentally competent to answer questions, but could not respond because ofcommunication problems (e.g., because of hearing/speech problems or because the respondentspoke a language other than English or Spanish, the two languages in which interviews wereconducted).

Proxy and facilitator interviewers tried to find someone else in the household who wasappropriate to answer questions for the respondent. Special interviewers were needed becausewords in the questionnaire that referred to the respondent as “you” or “your” had to be changedto the sampled person’s name by the interviewer conducting the proxy interview.

5-10

For hard-of-hearing respondents, we offered to conduct the interview using TTY/TDD texttelephone services. When this equipment was available to a respondent—which was rare—wedid not need a proxy.

Movers. Occasionally, a sampled respondent moved between the time of the screening interviewand the extended interview. In these situations we followed the mover if his/her telephonenumber was known. For details about the rules used for movers, see section 3.6.2. Because someof the questions in the interview were about the household, and, in these situations, the formerhousehold, it was necessary to remove these cases from the general interviewing population sothat the questions could be phrased appropriately.

Students Living Away at School. Occasionally students were sampled from their parent’s homeand were away at school. This required tact and finesse on the part of interviewers because thestudent’s telephone number had to be obtained from parents who were often uncertain about thepurpose of the study. Moreover, the information the students were asked about their householdreferred to the sampled household, not the student’s residence at school. For these reasons, thespecially trained manual-queue interviewers contacted students. For details about the proceduresfor these contacts, see section 3.6.3.

Persons Missing from the Roster and Other Problems. There were several problem situationsthat were so complicated that they were recontacted by manual interviewers (for details seesections 3.6.4 and 3.6.5). Typical of this group were households in which it was discoveredbeyond the roster (section D) that someone was missing. These interviews had to be set back tosection D so the household enumeration could be corrected, then the interview began again. Thisrestart (called the “Special D Restart” in section 3.6.5) was the work of the manual-queueinterviewers.

5.10 Switching the Order of Interviews

When there were multiple adult interviews in a household, the first interview that was attemptedwas normally finalized before attempts were made to complete the second interview in thehousehold. The move from a finalized first interview to a second adult interview was doneautomatically by the scheduler when neither interview had begun. But if the first adult interviewhad already started, the scheduler did not switch to the second interview. This was becauseimportant information was obtained in the first interview that was not collected again in secondor third interviews. Later interviews also used information from the first interview. Thesedependencies meant that interviews could not be switched automatically once the first interviewhad started. As the study came to a close, though, it became apparent that there was sometimes aneed to attempt the second interviews even though the first interviews were not yet finalized. Theorder of interviews in this type of household was intentionally switched if the first interview hadstarted but had not yet finished and it was considered a final refusal or the MC counter had beenreached. It was important to be certain that the first interview could not be obtained, sinceswitching the order of interviews closed off the first interview so that it was never rereleased

5-11

again. The order of interviews was not switched if there was still hope that the first interviewwould be completed.

Decisions about switching the order of interviews were made on a case-by-case basis. Generally,if the respondent had moved and could not be located, if messages indicated that the firstrespondent had not been providing accurate information, or if a first interview was a refusal andthere was reason to believe that the second respondent would cooperate, the order of interviewswas switched. The first interview data were deleted, and the second respondent in the householdbecame the B1 interview (see section 3 for definitions of interview types). Switches were notmade if the first respondent was simply hard to reach. The order of interviews for 20 adultinterviews was switched. No child interviews were switched.

If the decision was not to switch the order of interviews, the aid of the second respondent wasenlisted in getting the first interview completed. A cash incentive was offered to the secondrespondent if he/she could help to get the first respondent to complete and then also completehis/her own interview.

5.11 Call Attempt Results

It took an average of 6.64 call attempts, or more than 2 million telephone calls, to finalize all thenumbers that were released to interviewers for screening. As shown in table 5–3, the number ofattempts in Cycle 2 varied greatly by final result. This was also true in Cycle 1, as is also shownin the table for comparison. Among the cooperating households, 5.24 average call attempts weremade (5.16 in Cycle 1), with fewer calls needed among the ineligibles than among the eligiblehouseholds. Many more call attempts were made to nonresponding households, with refusalsrequiring an average of 15.5 calls (14.6 in Cycle 1). As mentioned in section 5.6, the MC counterwas set high, so that the average MC case received almost 40 calls before it was finalized (41.30in Cycle 1). Many calls were also made to telephone numbers whose residential status wasuncertain. “No answers” received an average of 10.63 call attempts (15.50 in Cycle 1) and“answering machines” received 28.39 attempts (29.10 in Cycle 1). Nonresidential andnonworking numbers received the fewest call attempts at 3.12 (3.30 in Cycle 1).

5-12

Table 5-3.Average Number of Call Attempts per Telephone Number, by Final Results at the Screener

Level, Cycle 2 and Cycle 1

Cycle 2 Cycle 1

Final result

TelephoneNumbers

AverageNumberof Calls

TelephoneNumbers

AverageNumberof Calls

Complete/ineligible households 147,623 5.24 179,281 5.16 Complete and sampled* 50,111 5.76 50,417 6.43 Complete not sampled 40,467 5.55 57,493 5.06 Ineligible 57,045 4.57 71,371 4.34

Nonresponding households 40,651 16.72 44,125 13.60 Refusals 36,088 15.50 39,395 14.60 Language problems 1,360 16.41 1,638 15.60 Maximum calls 2,445 39.18 2,854 41.30 Other nonresponse 758 3.02 238 8.49

Uncertain household status 28,373 13.07 29,249 17.30 Answering machines 3,891 28.39 3,950 29.10 No answers 24,482 10.63 25,299 15.50

Nonhouseholds 112,146 3.12 143,991 3.30 Nonresidential 34,723 4.31 49,186 3.60 Nonworking 77,423 2.70 94,805 3.20

Total 328,793 6.64 396,646 6.30

* The “complete and sampled” group includes emancipated minors. In 1997 NSAF Telephone Survey Methods, Report No. 9 in theCycle 1 methodology series, this group was included in the “complete and not sampled” group.

NOTE: Numbers eliminated prior to calling (e.g., business and nonworking numbers) are not included in this table.

5.12 Summary

The 383,653 telephone numbers that were sampled for the survey were processed to eliminatebusiness, nonworking, and duplicate numbers. After this, 331,173 numbers remained. Threethousand more cases were deleted from the summer release group to save time and money, so thetotal number of cases called was 328,793. In order to increase cooperation rates, two advancemailings were sent to households for which Westat had a mailing address. One mailing was torespondents who had been called in 1997 and the other mailing was a prenotification letter andbrochure for all respondents. Answering machine messages were also left the first time there wasno answer (and whenever the treatment changed for a particular case) to legitimize the study andallow respondents time to make an informed decision about participating.

5-13

The scheduling and release of work were affected by a variety of factors. For numbers that werealso part of Cycle 1, information on when a case was finalized in Cycle 1 was used to set thetime at which the case would initially be called in Cycle 2. For other numbers, those thought tobe residential because they had a mailing address were first contacted in the evenings, whenrespondents would most likely be home. Numbers thought to be nonresidential were first calledin the daytime to eliminate ineligible cases most quickly.

Cases that were not successfully contacted at the first attempt were placed in different time slicesfor future calls, so that calls could be made over several days at different times of day. Oncecontact was made, subsequent calls were made based on general or exact appointments with therespondent. A high limit was placed on the maximum number of calls that could be made inorder to allow enough calls for refusal conversion efforts and calls in Spanish.

Specialized interviewers were used for different types of cases. For example, some interviewersworked with cases that required bilingual interviewers, others specialized as refusal converters,and others worked in a manual queue that required procedures individualized to specific cases.

In March 1999, refusal conversion started. Strategies varied by whether an address was availablefor the household and whether it was the first or second refusal at the screener or extendedinterview level. Techniques involved the use of letters to respondents (both express and regularmail letters) and incentive money. Cooperation and conversion rates were higher whenever aletter was sent. Near the end of data collection, respondents were all offered a total of $50 tocomplete the extended interview.

Another procedure that affected the release of work was the switching the interview order inmultiple-interview households when a first adult interview was not finalized.

The average number of call attempts for numbers released to interviewers for screening was6.64. The number of attempts varied by final result (e.g., more call attempts were made tononresponding households).

6-1

6. QUALITY CONTROL

Many quality control procedures were in place throughout the study. Some of them, such asCATI testing and training, were used before the beginning of data collection as preventivequality controls. Others, such as supplemental interviewer training, monitoring, and commentand problem sheet review, were used during data collection to respond to issues withinterviewers or to make adjustments to the questionnaire. Each quality control method isdescribed briefly below.

6.1 CATI Testing

Quality control was begun with making sure that the CATI instrument was working properly.The questions and skip patterns were tested as soon as new items from Cycle 2 in thequestionnaire were programmed. This testing included review by project staff, TRC staff(including interviewers), data preparation staff, the statistical staff and programmers, and staff atthe Urban Institute The testing by staff members representing different aspects of the project wasdone to ensure that the system was working properly from all of these perspectives.

6.2 Online Range and Logic Checking

Another method of quality control involved the use of edits in the CATI system. Specifically,online range checks were programmed for several sections of the questionnaire to catch unlikelyor impossible responses and also to catch errors that might result from typographical errors byinterviewers. Each check had defined ranges with minimum and maximum values. For example,there were checks to ensure that a child’s reported grade level made sense in terms of his or herage. A message would appear, for instance, if an 8-year-old was reported to be in nursery school(below the minimum value) or the fifth grade (above the maximum value). However, becausesome 8-year-olds could actually be in those grades, the edit was a “soft-range” check thatallowed the interviewer to continue after confirming the response with the respondent.

Some questions had both soft and hard ranges. “Hard-range” checks did not allow theinterviewer to continue without entering an answer within the range programmed. For example,one question asked about the amount received in an TANF check. The soft range for a monthlyamount was $0 to $1,250. The hard range was $0 to $9,999. An answer above $1,250 could beentered after confirmation, but an answer of $10,000 was not allowed.

Other checks were used to check logic. For example, there was a check that did not allow arespondent with multiple jobs to report that the hours worked in all jobs combined were less thanor equal to the hours worked in one job. Another logic check was used when a respondentreported that he or she was married, but then reported that no one living in the household was aspouse. If this occurred, the respondent was asked if his or her spouse lived in the household.This edit was particularly helpful for preventing the problems that the study had had in 1997

6-2

when spouses were left off of the household roster and data retrieval had to be done later toobtain information about the spouse. In 1999, edits were also added to check when unlikely ages(based on their relationships to others) were given for household members. These were valuablefor ensuring that the characteristics of household members and their relationships with each otherwere clear.

6.3 Training

A good training program is another important quality-control measure. Westat has found thatthorough and intense training yields the best results in the collection of high-quality data.Training was standardized across sessions so that all interviewers received the same information.In addition, supervisors attended the same project-specific training sessions as the interviewersso that they would be well prepared to handle their duties. Supervisors were also preparedbecause of their previous experience. Many of our TRC supervisory staff occupy permanentpositions at Westat, have worked on many RDD surveys, and are very familiar with the specificquestions asked by interviewers and respondents and the common problems that occur.

6.4 Practice Interviews and Supplemental Training

As discussed in chapter 4, interviewers conducted live practice interviews until they reached anadequate level of proficiency to start real production interviews. This practice gave interviewersan opportunity to use the techniques learned in training. Practice interviews also gave TRC staffthe opportunity to monitor interviewers and determine their strengths and weaknesses before anyproblems affected data collection.

In addition, about one week after each training session, interviewers began attending sessionsdesigned to maximize respondent cooperation. Following this training, interviewers weremonitored further and feedback was provided about how well they were doing and what theymight do to improve their performance.

6.5 Interviewer Memos

As discussed in chapter 4, interviewer memos were given to the staff to clarify and reinforceissues, as well as to inform staff of procedural changes. A total of six memos were distributed tointerviewers (see Cycle 2 NSAF interviewer memos [Westat 1999]).

6.6 Interviewer Meetings

Interviewer meetings were also held as a quality-control procedure. These meetings wereconducted with the interviewing and supervisory staff as necessary to reinforce procedures,review points of emphasis, provide updates on procedures, and inform staff of study progress.

6-3

These were important to the interviewing process whenever minor changes were made duringdata collection.

6.7 Interviewer Monitoring

Westat monitored telephone interviewer performance throughout the field period. Monitoringforms for each interviewer were reviewed weekly, and any interviewers who were identified asin need of additional monitoring were monitored more heavily in the following week.Supervisors also performed additional monitoring if there was concern about an interviewer’sperformance.

Westat’s capacity to monitor telephone interviewers is based on our investment in highlysophisticated equipment and electronic linkages. From a remote location, our supervisors andmonitors intercepted calls and silently listened to both the interviewer and the respondent. At thesame time, the supervisor could see what appeared on the interviewer’s computer screen and theresponses that the interviewer entered. Supervisors simultaneously checked on interviewingtechnique and the interviewer’s ability to correctly capture data. Ten percent of the interviewingtime was monitored.

Westat supervisors and monitors selected 15-minute intervals of each interviewer’s working timeto monitor. Interviewers were monitored each shift, although this was not always possiblebecause resources had to be allocated to other activities (e.g., special coaching sessions withinterviewers, meetings, etc.). In these circumstances, interviewers were monitored within two orthree shifts. Supervisors performed extra monitoring if there was a concern about aninterviewer’s performance. An interview monitoring report form was completed each time aninterviewer was monitored. Interviewers who continued to have significant problems afterreceiving feedback or remedial training were released from the study.

During the first weeks following completion of training, the results of monitoring were discussedwith each interviewer immediately following the monitoring session. This discussion providedthe interviewer with feedback and suggestions on how to improve his/her techniques to gaincooperation, ask questions, and/or record responses. Subsequent reports were only reviewed withan interviewer if there was a specific problem, in which case the report was discussedimmediately. Supervisors reviewed the monitoring reports throughout the survey period toidentify any common problems that might have revealed the need for additional interviewer-widetraining.

6.8 Triage

Interviewing during all hours of TRC operation was supported by a specially trained “triage”supervisor. The triage supervisor was called whenever a problem interfered with an interviewer’sability to conduct CATI interviewing. When the triage supervisor received a problem report,he/she diagnosed the problem and called the appropriate personnel. Hardware, software, and

6-4

project-specific support were always available via home telephones or beeper numbers. Theappropriate support personnel were able to respond to problems within minutes of a problemreport, regardless of the time.

6.9 Using Comments and Problem Sheets to Find Problems

Interviewers made comments within the CATI questionnaire whenever a response did not fit acategory and/or when they perceived a problem with a question. During the first two weeks ofdata collection in each data collection site, comments were read thoroughly to determine whetherthere were any problems with items. If there were many comments about a particular item, itmight indicate that a question needed to be changed or reinforced with an interviewer memo ormeeting. After the two-week period, 10 percent of comments were read to monitor datacollection.

Problem sheets were also used for quality control. When interviewers or supervisors encountereda problem in conducting or monitoring an interview, they completed a CATI problem sheet. Forcases that were not complete, these sheets were reviewed by a triage supervisor and forwarded tothe appropriate staff member for resolution. For completed cases, problem sheets were sent to theUrban Institute.

6.10 Checks on Data in Monthly Data Deliveries

Westat sent the Urban Institute data files on a monthly basis. These were checked by the UrbanInstitute, and any problems that were found (e.g., differences in variable names between theCATI specifications and the data, an unusual sequence of interviews in a particular household,etc.) were checked by Westat and corrected, if needed. This allowed for a gradual monitoring ofthe data collected, so that there would be no unexpected problems at the end of data collection.

6.11 Summary

In summary, Westat had multiple quality-control procedures in place both before and after datacollection. Before the beginning of data collection, CATI testing was conducted to ensure thatthe system was working properly. In addition, several range and logic checks were programmedinto CATI so that erroneous data were not entered.

Training interviewers thoroughly and intensely was another step in quality control. All staffinvolved in training, monitoring, and interviewing attended training sessions to ensureconsistency in understanding of the study concepts. Part of training also involved live practiceinterviews. This gave interviewers the opportunity to use their new skills and staff the chance tomonitor interviewers before they started data collection.

6-5

After data collection started, supplemental training was provided, along with interviewermeetings to discuss changes and review procedures. Interviewers were also sent memos to clarifyissues and inform them about changes. Ongoing interviewer monitoring ensured thatinterviewers performed well and that issues discussed in supplemental training, memos, andmeetings were understood. In addition, data preparation staff reviewed comments and problemsheets to determine whether any data or questionnaire changes were needed, or whether aninterviewer memo was required to clarify an issue. Frequent data deliveries were made to theUrban Institute, and these data were checked to identify potential problems. Finally, Westat’striage system was used to handle any issues that arose with the CATI system.

R-1

REFERENCES

Methodology References

Brick, J. M., and D. Cantor. Forthcoming. No. 7: 1999 NSAF Response Rates and MethodsEvaluation.

Brick, J. M., I. Flores-Cervantes, and D. Cantor. 1999. No. 8: 1997 NSAF Response Rates andMethods Evaluation.

Cunningham, P., and J. Meader. 2000. No. 5: 1999 NSAF In-Person Methods.

Dipko, S., N. Vaden-Kiernan, and P. Cunningham. Forthcoming. No. 8: 1999 NSAF DataEditing and Imputation.

Judkins, D., J. M. Brick, P. Broene, D. Ferraro, and T. Strickler. 1999. No. 2: 1999 NSAF SampleDesign.

Vaden-Kiernan, N., D. Cantor, P. Cunningham, S. Dipko, K. Molloy, and P. Warren. 1999. No.9: 1997 NSAF Telephone Survey Methods.

Wang, K., D. Cantor, and N. Vaden-Kiernan. 2000. No. 1: 1999 NSAF Questionnaire.

Wang, K., S. Dipko, and N. Vaden-Kiernan. 1999. No. 12: 1997 NSAF Questionnaire.

General References

Brick, J. M., B. Allen, P. Cunningham, and D. Maklan. 1996. “Outcomes of a Calling Protocol ina Telephone Survey.” Proceedings of the Survey Research Methods Section of the AmericanStatistical Association, 142–149.

Brick, J. M., N. Vaden-Kiernan, Fowlkes, C., and Cunningham, P. 2000. Report on the NSAFCycle 2 Capture/Recapture Study.

Vaden-Kiernan, N., and P. Cunningham. Forthcoming. Report on the California Add-On Study.

Cunningham, P., D. Cantor, C. Fowlkes, and J. Meader. 2000. Milwaukee Report.

Sebold, J. 1988. “Survey Period Length, Unanswered Numbers, and Nonresponse in TelephoneSurveys.” In Telephone Survey Methodology, edited by R. M. Groves, P. P. Biemer, L. E.Lyberg, J. T. Massey, W. L. Nicholls II, and J. Waksberg (247–256). New York: John Wiley &Sons.

R-2

Westat. 1999. National Survey of America’s Families (NSAF) Interviewer Manual, Volume 1.(Prepared under contract to the Urban Institute). Rockville, MD: Westat.

Westat. 1999. National Survey of America’s Families (NSAF) Interviewer Manual, Volume II.(Prepared under contract to the Urban Institute). Rockville, MD: Westat.

Westat. 1999. National Survey of America’s Families (NSAF) Cycle 2 Trainer’s Manual.(Prepared under contract to the Urban Institute). Rockville, MD: Westat.

Westat. 1999. Cycle 2 National Survey of America’s Families (NSAF) Interviewer Memos.(Prepared under contract to the Urban Institute). Rockville, MD: Westat.

Westat. 1999. Westat General Interviewer Training Interviewer’s Manual. Rockville, MD:Westat.

Xu, M., B. Bates, and J. Schweitzer. 1993. “The Impact of Messages on Survey Participation inAnswering-Machine Households.” Public Opinion Quarterly 57: 232–237.

1999 nsaf telephone survey methods

Documents