the evolution of a phoenix

1

Understanding Event History Calendars(Update on Reengineering SIPP)

COPAFSMarch 5, 2010

David JohnsonJason Fields

US Census Bureau

2

The Evolution of a Phoenix

3

Congresswomen Maloney hails 25th Anniversary of SIPP, Census Bureau’s

Survey of Income and Participation "October, 2008 marks 25 years of SIPP data collection. The vital

data collected by career professionals at the Census Bureau allows for the evaluation of the efficiency and effectiveness of government programs and gives us a more robust picture of how well we are doing as a nation in helping families progress through tough economic challenges"

“The SIPP allows Congress to allocate scarce government resources and save tax dollars. It’s fitting that during this national economic crisis we draw attention to this important diagnostic tool which helps us understand how we can best provide assistance to families in need.”

4

The Unique Value of SIPP

• To provide a nationally representative sample for evaluating:– annual and sub-annual dynamics of income– movements into and out of government transfer

programs– family and social context of individuals and households– interactions between these items

5

National Academy of Sciences National Research Council

Committee on National Statistics

Panel ReportReengineering the Survey of Income and

Program Participation

Constance F. Citro and John Karl Scholz, Editors

July 2009

6

CNSTAT Report – Importance of SIPP

Conclusion 2-1:The Survey of Income and Program Participation is a

unique source of information for a representative sample of household members on the intrayear dynamics of income, employment, and program eligibility and participation, together with related demographic and socioeconomic characteristics. This information remains as vital today for evaluating and improving government programs addressed to social and economic needs of the U.S. population as it did when the survey began 25 years ago.

7

Our work on SIPP Improvements

• Improve Processing System and Collection Instrument

• Develop Event History Calendar (EHC) Instrument

• Examine use of administrative records data to supplement and evaluate survey data

• Continue meetings with stakeholders, development of survey content, and use of reimbursable supplements

8

Statements from the CNSTAT Report:

On EHC methodology• As discussed in Belli (1998), in an event history

calendar, “respondents are encouraged to consider various events that constitute their personal pasts as contained within broader thematic streams of events. Not only can respondents not the interrelationship of events within the same themes (top-down and sequential retrieval) but, depending on which themes are represented by the calendar, respondents can also not the interrelationships among events that exist with different themes (parallel retrieval).”

9

Timeline for SIPP Development

2008 paper EHC

2007 2008 2009 2010 2011 2012 Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan 2013

2009 SIPP Re-EngineeringInstrument Dev.

Processing andEvaluation

SIPP 2008 Panel – Waves 1 – 10 collection

Eval. Analysis

Systems Tests -Preparation

2009 Re-engineered SIPP automated

Prototype Reference Period

Field Activities

2012/13 SIPP Re-EngineeringInstrument Refinement

Systems Tests -Preparation

2013 Reengineered SIPP Reference Period

Field Activities

SIPP 2008 Panel – Waves 1 – 13 data release Data GapSIPP 2004

2nd automated prototype

Reference Period

Field Act.

SIPP 2004 Panel data release Waves 11 – 13

Re-engineered SIPP – Progress Update –

COPAFSMarch 5, 2010

Current SIPP BasicsNational panel survey – Since 1984 with sample size between

about 11,000 and 45,000 interviewed households

The duration of each panel varies from 2½ yrs to 4 yrs

The SIPP sample is a multistage-stratified sample of the U.S. civilian noninstitutionalized population

The survey uses a 4-month recall period – 3 interviews / year

The sample is divided into 4 rotation groups for monthly interviewing

Interviews are conducted by personal visit and by decentralized telephone

EHC Interviewing

Human Memory- structured/organized- links and associations

EHC Exploits Memory Structure- links between the occurrence and timing of events

EHC Encourages Active Assistance to Rs- flexible approach to help elicit an

autobiographical “story”

Evaluations of EHC Methods

Many EHC vs. “Q-List” Comparisons- various methods- in general: positive data quality results

BUT, Important Research Gaps- data quality for need-based programs?- comparison to 4-month reference period?

SIPP Re-engineering Field Test Plans

- Proof of concept test - - 2008 paper and pencil reinterview test

- EHC CAPI test- - 2010 Integrated Blaise and C# instrument prototype

- CAPI Revised test- - 2011 Test improvements to the wave 1 instrument, training,

and expand sample to all regional offices.

- - 2012 Test wave 2 concepts and instrument, examine movers and attrition issues, and refine training procedures.

2008 Paper Field Test Goals & Design

Basic Goal:Can an EHC interview collect data of comparablequality to standard SIPP?

- month-level data- one 12-month ref pd interview vs. three 4-month ref pd interviews- especially for need-based programs

Basic Design:EHC re-interview of SIPP sample HHsApproximately 2000 HHs in IL and TX

Results Summary•3 Patterns:

• 1. EHC = SIPP All Year • equivalent data quality•

0.0%

2.5%

5.0%

7.5%

10.0%

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

% Y

es

SIPP % yes

EHC % yes

SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary

- no “main effect” for method (SIPP = EHC)

- no significant method difference in any month

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

•

Results Summary

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year•

Results Summary

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year• Medicare; Social Security;

WIC (TX); Food Stamps (IL)

•

Results Summary

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year• reduced EHC data quality, but • not due to longer recall period

•

Results Summary

15.0%

17.5%

20.0%

22.5%

25.0%

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

% Y

es

SIPP % …EHC % …

SOCIAL SECURITY -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary

- significant “main effect” for method (SIPP > EHC)

- method difference is constant across months

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year• Medicare; Social Security;

WIC (TX); Food Stamps (IL)

•

Results Summary

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year• Medicare; Social Security;

WIC (TX); Food Stamps (IL)

• 3. EHC < SIPP, Early in the Year Only•

Results Summary

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year• Medicare; Social Security;

WIC (TX); Food Stamps (IL)

• 3. EHC < SIPP, Early in the Year Only• Food Stamps (TX); TANF (TX);• employment; school enrollment

Results Summary

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year• Medicare; Social Security;

WIC (TX); Food Stamps (IL)

• 3. EHC < SIPP, Early in the Year Only• EHC data quality may suffer due

– to longer recall period

Results Summary

0.0%

2.5%

5.0%

7.5%

10.0%

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

% Y

es

SIPP % yes

EHC % yes

FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports

Analysis Summary

- no significant “main effect” for method

- BUT significant variation by month --

JAN-MAY: SIPP > EHC

later months: no difference (reversal?)

0.0%

2.5%

5.0%

7.5%

10.0%

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

% Y

es

SIPP % yesEHC % yes

FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports and ADRECS

•3 Patterns:

• 1. EHC = SIPP All Year • SSI; WIC (IL)

• 2. EHC < SIPP All Year• Medicare; Social Security;

WIC (TX); Food Stamps (IL)

• 3. EHC < SIPP, Early in the Year Only• Food Stamps (TX); TANF (TX);• employment; school enrollment

Results Summary

2008 Paper Field Test Overall Summary

Successful “Proof of Concept”

Overwhelming Finding: SIPP-EHC Agreement

Valuable Lessons to Inform Next Test- larger, broader sample- “correct” timing of field period- automated questionnaire

Specific Data Comparisons are Instructive

2010 EHC CAPI Test Goals & DesignBasic Goal:Can a CAPI Event History Calendar interview be:

- developed in-house (new/unknown requirements)- integrated with survey management systems- administered by regular field staff interviewers

Develop and test new processing system

Determine the comparability of data collected:- month-level data- (1) 12-month ref pd intvw vs. (3) 4-month ref pd intvws- especially for need-based programs

Basic Design:8000 HHs interviewed in 10 states limited to high poverty strata that

can be matched to areas in 2008 Panel SIPP data.Recording 400-500 of the completed interviews

2010 EHC CAPI Instrument

Current StatusInterviewing in 6 of 12 regional office areas (10 States)

Interviews continue through March 13.

Recording 400-500 of the completed interviews – transcription will begin shortly.

Reviewing and making changes to content and design.

Reviewing and revising training materials and methods

Planning for 2013 Production implementation

LANDMARKSRESIDENCES

PRESENCE OF PARENT - MOMPRESENCE OF PARENT - DAD

REFERENCE YEAR 20XX INTERVIEW YEAR 20XX+1

INTV

SCHOOL ENROLLMENT – (CURRENTLY ENROLLED fill)JOBS/BUSINESSES – (fill name of emp/bus from ehc as entered)+ Job/Bus 2 – (fill name of emp/bus from ehc as entered)+ Job/Bus 3 – (fill name of emp/bus from ehc as entered)+ Job/Bus 4 – (fill name of emp/bus from ehc as entered)+ Job/Bus 5 – (fill name of emp/bus from ehc as entered)ALL OTHER WORK FOR PAYTIME NOT WORKING

MARITAL STATUS – (CURRENT MARITAL STATUS fill)

SSI - SUPPLEMENTAL SECURITY INCOMESNAP - FOOD STAMPS/SUPPLEMENTAL NUTRITION ASSISTANCE PRGMTANF - TEMPORARY ASSISTANCE FOR NEEDY FAMILIESGA – GENERAL ASSISTANCE PROGRAMWIC – WOMEN, INFANTS AND CHILDREN PROGRAMHEALTH INSURANCE – PRIVATE COVERAGE+ ADDITIONAL PRIVATE COVERAGEMEDICARE COVERAGEMEDICAID COVERAGE – (fill interview state program name)MILITARY COVERAGE – (VA, CHAMPUS, CHAMPVA)OTHER HEALTH INSURANCE COVERAGETIME WITHOUT HEALTH INSURANCE COVERAGE

Topic – (FILL TOPIC LABEL)* INTRO TO TOPIC QUESTION FILL – ASKS ABOUT STATUS NOW OR DIRECTS FR TO PROCEED [ ] (1) YES/PROCEED (2) NO (3) REFUSED

* FOLLOW-UP ASKS ABOUT EVER DURING REFERENCE YEAR OR DIRECTS FR TO PROCEED [ ] (1) YES/PROCEED (2) NO (3) REFUSED

* IDENTIFY NEW PERIOD OF TIME [ ] 0-NEW [F4 – EDIT SPELL] [CTRL-X DELETE SPELL]

* WHEN DID THIS PERIOD OF (TOPIC FILL) END? [ ] CHOOSE MONTH * WHEN DID THIS PERIOD OF (TOPIC FILL) BEGIN? [ ] CHOOSE MONTH

ANY MORE PERIODS OF (TOPIC FILL) DURING REFERENCE YEAR? [ ] (1) YES (2) NO

RECORDING ‘NO’ ADDS STRIKEOUT TO ANY UNASSIGNED MONTHS IN THIS TOPIC – ANSWERING ‘YES’ CYCLES BACK TO NEW PERIOD – UPON SPELL COMPLETION YOU EXIT THE DETAILED QUESTIONS TO THE ‘ANY MORE’ ITEM.

DEPENDENT DATA FILL UP TO LAST WAVES ‘INTV’ MONTH

FILL’S ARE USED TO LABEL INTERVIEW MONTH STATUS OR NAMES FROM JOBS/PROGRAM NAMES

(FILL NAME OF PERSON FOR THIS EHC)

Mockup (example of possible changes) for 2011 EHC CAPI Instrument

DATA ARE RECORDED UP TO THE MONTH OF INTERVIEW, WE ALLOW UP TO 6 MONTHS FROM THE END OF THE CALENDAR YEAR TO CONDUCT INTERVIEWS – MONTHS BEYOND THE INTERVIEW ARE GREY’ED OUT.

IF YES TO ‘NOW’ INITIALIZE FIRST PERIOD AND FILL INTERVIEW MONTH IN THE ‘TO’ BOX AND SET FOCUS ON THE ‘FROM’ BOX

Assessing Users’ Needs

URL: http://www.census.gov/sipp

Comments:

Jason Fields – Jason.M.Fields@Census.Gov