Comparability of categorical variables in longitudinal survey research

Paul Lambert*, Alison Bowes*, Vernon Gayle*, Guy Warner*, and Tom Doherty**

(*University of Stirling; ** University of Glasgow)

Presented to the Society for Longitudinal & Life Course StudiesClare College, Cambridge22-24 September 2010

Comparability of categorical variables in longitudinal survey research

1) Comparability strategies and selected problems

2) Some analytical prescriptions

3) The relevance of the ‘GESDE’ services

..Underlying motivation to study patterns/changes over time in distributions measured by categorical instruments..

“..Comparisons Are the Essence..” (Treiman, 2009: 382)

2

1) Comparability strategies

Harmonisation/standardisation is used to aid in the comparison of data across contextsMeasurement v’s meaning/functional equivalence

[i.e. same absolute qualities of position, v’s same relative meaning of position within the distribution]

Ex post v’s pre-harmonisation [e.g. Harkness et al. 2003; Hoffmeyer-Zlotnik & Harkness 2005]

Comparability problem: when comparison (over time) is muddled by a lack of equivalence in absolute and/or relative position

3

1994 1998 2002 2006

Income in £

1994 1998 2002 2006

Income zscore by year

1994 1998 2002 2006

% in EGP-2 ('salariat')

1994 1998 2002 2006

CSM zscore by year

Source: BHPS, adults aged 30-50, x-sectional weighting

Returns to education, 1991-2008

1a/b Incompl./Elementary 1c Basic voc. 2a Intermed. voc.

2b Intermediate general 2c General 2d Vocational

3a Lower tertiary 3b Higher tertiary

4

Categorical variables: coding membership of groups

Most social survey data is categorical! Importance of: fine levels of detail; boundaries;

recoding strategies; cognitive issues5

Interest in this talk High Medium Low [n.a.] ordinal

Country of birth UK India Poland ….. nominal

..Please describe your occupation...

Secretary Lecturer Shop assistant

No job nominal

NS-SEC 3 1.2 6 8 (recode of 3)

CAMSIS(F) 62.4 94.2 42.8 [n.a.] (scaling of 3)

Sports you enjoy Football Cricket Darts …. nominal+MR

Three important categorical variables

Standards Key correlates (r2)* Reference

Educational qualifications

ISCED; GHS; years of study

year of birth (0.25); gender (0.04);

Schneider (2010); Treiman (2007)

Occupations NS-SEC; RGSC gender (0.41); age (0.05); region (0.02);

Rose and Harrison (2010)

Ethnicity ONS; Years since immigr.

year of birth (0.08); immigrant status (0.26); region (0.09)

Bosveld et al. (2006)

Longitudinal problems of comparison include: Changing structural contexts (distributions; correlations; sparsity) Changing measurement practices (e.g. decennial revisions; admin data) Changing international comparisons (e.g. ISCO88-SOC; ISCED-ONS)

Existing measurement instruments are generally outside the domain of concept formation research (cf. Jowell, 2007)

* 2008 BHPS 20+yrs, qfedhi, jbsoc, ‘xeth’ using race{l} + 0.01 downwt for Wh

Selected comparability problems for categorical measures in longitudinal analysis

7

Interpretation problems (same relative meaning over time?)

Operationalisation problems

Changing categories

Changing distribution of categories

Sparse categories

Parsimony – multivariate models and interactions

Hidden change in administrative practices (e.g. recoding)

Changing relevant correlates (e.g. with yob)

Timing issues

Documentation

Common practice with longitudinal surveys

Comparative model for categorical data is overwhelmingly one of measurement equivalence

• ‘Nominal equivalence’: offer, or locate, categories in the same scheme over time (often suppressing further detail in released data)

• Distribute explanatory metadata (e.g. www.ipums.org)

• Validity studies may be used to test correlation with expectations (e.g. Rose and Harrison 2010)

This is commonly visible as coding to the ‘lowest common denominator’

8

9

A measurement equivalence example – comparing ethnicity categories in context of demographic change

Are the compatible categories equivalent? Who does the work (data distributors and/or analysts)…? …& who records it (e.g. Mohler et al., 2008)?

10

Here, measurement equivalence is compromised by administrative errors, & meaning equivalence is doubtful due to industrial restructuring (orig. occ. codes not available)

Unskilled

Skilled manual

Petty-bourg.

Non-manual

Salariat

Source: Females from LFS/GHS, using data from Li and Heath (2008)

percent of year category

Goldthorpe class scheme harmonised over time

[Changing correlates] Occupational measures change over time in total distribution and this interacts with occupational gender segregation

11

0.1

.2.3

.4F

itted

va

lues

1990 1995 2000 2005 2010year

RGSC % advantaged EGP % advantaged

RGSC gender difference EGP gender difference

Source: BHPS waves 1-18, x-sectional respondent weighting, young adults 16-40 only

Gender trends 16-40yrs, social advantage in 2-category EGP and RGSC

…Meaning equivalence is preferable in longitudinal survey research…!!

Collect and preserve the data in its original detailPreserves information; easier for respondent; better documentation

Allow analysts to harmonise/standardise using flexible/tailored equivalence strategies

Same relative meaning may=diff. absolute properties (e.g. Educ-by-yob)?

Ideally, trying and comparing more than one strategy

Pressures for measurement equivalence from statistics agencies, analysts, consumers “..goal of standardisation is to enhance comparability; inappropriate

standardisation may do just the opposite” (Harkness, 2008: 61)

“..when nominal equivalence is not enough..” (Schneider, 2010) 12

2) Some analytical prescriptions

Measurement equivalence • ..is what we already do• Appealing because we have existing

standards/literature on most measures

Improved practice: • Use the published standards when they exist(!)

cf. temptation to recode/collapse recommended measures

• Documentation/replication of derivations• Comparison between a few different options• Recognise parsimony/multivariate interactions

13

14

.25

.3.3

5.4

.45

1990 2000 2010 2020 2030

Observed

Educ4 Educ4++Educ2a Educ2a++Educ2b Educ2b++

ISCED ISCED++

Source: Simulations on the BHPS balanced panel subsample (weighted by xlwght). Data from 1991-2007; simulations from 2008-2025. ++ lines show impact of educational upgradings.

Father-Child CAMSIS r, all adultsIntergenerational correlation simulations

(Own analysis, for the NeISS project, www.neiss.org.uk)

15

Meaning equivalence

For categorical data, equivalence for comparisons is often best approached in terms of meaning equivalence

(because of non-linear relations between categories and shifting underlying distributions)

(even if measurement equivalence seems possible)

Arithmetic standardisation offers a convenient form of meaning equivalence by indicating relative position with the structure defined by the current context

For categorical data, this can be achieved/approximated by scaling categories in one or more dimension of difference

16

None/low

Interm

FE/Voc

Univ

None/low

IntermFE/Voc

Univ 12

3

4567

89

1

2

3

4

567

89

3040

5060

70

FCS E L E L E L 60 70 80 90 00

Educ-4 scale scores Region averagesEGP class averages Occ major groups/orders

Individual FCS values (rescaled mean50, sd6)

Source: 'Slow Degrees' dataset 1963-2005 (Lambert et al., 2007), all adults 20+, N~=164k, categories are scaled by father's occ advantage (mean 50, sd15), E=1963-1989, L=1990-2005.

Effect proportional scaling using father's occupational advantage

UK born Not UK born

Whi Iri 0 (base)

Whi Other -5.4 -3.4

Black Car. -9.6 -1.0

Black Afr -12.7 -5.9

Indian -16.1 -1.0

Pak/Bang. -16.8 -3.2

Chinese -15.8 -6.5

Other -11.0 -3.917

educ6_6 -.2346746 .0869549 -2.70 0.007 -.4051031 -.0642461 educ6_5 -.3082588 .1122945 -2.75 0.006 -.528352 -.0881655 educ6_4 -.2971853 .1083325 -2.74 0.006 -.5095131 -.0848574 educ6_3 -.4110193 .1492615 -2.75 0.006 -.7035665 -.1184722 educ6_1 -.0771515 .0341444 -2.26 0.024 -.1440733 -.0102298 yob -.0298194 .0107592 -2.77 0.006 -.0509071 -.0087317 ethim2 Coef. Std. Err. z P>|z| [95% Conf. Interval] ( 1) [phi1_1]_cons = 1

Log likelihood = -47960.017 Prob > chi2 = 0.2616 Wald chi2(6) = 7.69Stereotype logistic regression Number of obs = 21999

Is scaling useful?

..sometimes.. Revealing exploratory exercise Parsimonious functional form

If complex categorical measure is a control variable If interaction effects are considered If story told by (transformation of) a linear functional

form is persuasive (e.g. exponential increase) Good for Multiple Response data

(e.g. take arithmetic function of related cases)

18

3) The GESDE services

DAMES (www.dames.org.uk) node seeking to support data management in social research activities

‘Data management’ = tasks involved in organising/ manipulating/enhancing data in preparation for research analysis Making tasks easier and more consistent

Pre-analysis tasks conducted by researchers and data distributors Re-coding; harmonising; standardising; cleaning; linking Recording and distributing metadata for replication

19

20

Contributions: Preserving/facilitating data management

Count

323 0 0 0 0 323

982 0 0 0 0 982

0 425 0 0 0 425

0 1597 0 0 0 1597

0 0 340 0 0 340

0 0 3434 0 0 3434

0 0 161 0 0 161

0 0 0 1811 0 1811

0 0 0 0 2518 2518

0 0 0 331 0 331

0 0 0 0 421 421

0 0 0 257 0 257

102 0 0 0 0 102

0 0 0 0 2787 2787

138 0 0 0 0 138

1545 2022 3935 2399 5726 15627

-9 Missing or wild

-7 Proxy respondent

1 Higher Degree

2 First Degree

3 Teaching QF

4 Other Higher QF

5 Nursing QF

6 GCE A Levels

7 GCE O Levels or Equiv

8 Commercial QF, No OLevels

9 CSE Grade 2-5,ScotGrade 4-5

10 Apprenticeship

11 Other QF

12 No QF

13 Still At School No QF

Highesteducationalqualification

Total

-9.001.00

Degree2.00

Diploma

3.00 Higherschool orvocational

4.00 Schoollevel orbelow

educ4

Total

DAMES provides online services for data coordination/organisation

Tools for handing variables in social science data

Recoding measures; standardisation / harmonisation; Linking; Curating

17/MAR/2010 DIR workshop: Handling Social Science Data

21

GESDE – Search and browse supplementary data on occupations; educational qualifications; ethnicity

17/MAR/2010 DIR workshop: Handling Social Science Data

22

23

Conclusions: What we do and what we ought to do

Research writers routinely select single simplifying sub-optimal collinear categorisations of concepts Due to coordinated instructions [e.g. Blossfeld et al. 2006] Due to perceived lack of available alternatives Due to perceived convenience

To make longitudinal comparative analyses more scientific (cumulative & open to cross-examination) we should… Acknowledge and discuss our equivalence strategies Operationalise and deploy various categorisations (measurement

equivalence) and scalings/arithmetic measures (meaning equivalence), and explore their distributional properties

… and keep a replicable trail of all these activities.. …ideally by using GESDE…!!

References Blossfeld, H. P., Mills, M., & Bernardi, F. (Eds.). (2006). Globalization, Uncertainty and Men's Careers: An

International Comparison. Cheltenham: Edward Elgar. Bosveld, K., Connolly, H., Rendall, M. S., & (2006). A guide to comparing 1991 and 2001 Census ethnic group data.

London: Office for National Statistics. Harkness, J. (2008). Comparative survey research: goals and challenges. In E. De Leeuw, J. Hox & D. A. Dillman

(Eds.), International Handbook of Survey Methodology. London: Psychology Press. Harkness, J., van de Vijver, F. J. R., & Mohler, P. P. (Eds.). (2003). Cross-Cultural Survey Methods. New York: Wiley. Hoffmeyer-Zlotnik, J. H. P., & Harkness, J. (Eds.). (2005). Methodological Aspects in Cross-National Research.

Mannheim: GESIS - ZUMA Zentrum fur Umfragen, Methoden und Analysen. Jowell, R., Roberts, C., Fitzgerald, R., & Eva, G. (2007). Measuring Attitudes Cross-Nationally. London: Sage. Lambert, P. S., Prandy, K., & Bottero, W. (2007). By Slow Degrees: Two Centuries of Social Reproduction and

Mobility in Britain. Sociological Research Online, 12(1). Li, Y., & Heath, A. F. (2008). Socio-Economic Position and Political Support of Black and Ethnic Minority Groups in

the United Kingdom, 1972-2005 [computer file]. 2nd Edition. Colchester: UK Data Archive [distributor], SN: 5666. Mohler, P. P., Pennell, B.-E., & Hubbard, F. (2008). Survey Documentation: Toward professional knowledge

management in sample surveys. In E. De Leeuw, J. Hox & D. A. Dillman (Eds.), International Handbook of Survey Methodology (pp. 403-420). Hove: Psychology Press.

Rose, D., & Harrison, E. (Eds.). (2010). Social Class in Europe: An Introduction to the European Socio-economic Classification London: Routledge.

Schneider, S. L. (2010). Nominal comparability is not enough: (In-)Equivalence of construct validity of cross-national measures of educational attainment in the European Social Survey. Research in Social Stratification and Mobility.

Simpson, L., & Akinwale, B. (2006). Quantifying Stablity and Change in Ethnic Group. Manchester: University of Manchester, CCSR Working Paper 2006-05.

Treiman, D. J. (2007). The Legacy of Apartheid: Racial Inequalities in the New South Africa. In A. F. Heath & S. Y. Cheung (Eds.), Unequal Chances: Ethnic Minorities in Western Labour Markets. Oxford: Oxford University Press, for the British Academy.

Treiman, D. J. (2009). Quantitative Data Analysis: Doing Social Research to Test Ideas. New York: Jossey Bass.

Abstract In working with longitudinal social surveys, we sometimes ignore, or at the very least over-

simplify, challenges of measurement comparability in variables over time. Often, our approaches to harmonization involve searching for categories with consistent names over time, frequently achieving such ‘nominal equivalence’ by reducing the level of detail of measures to the so-called lowest common denominator.

In this paper we highlight some well-known, and some less well-known, comparability problems for categorical measures in longitudinal surveys. We focus on UK studies and their measures of occupational position, educational qualifications and ethnicity. These are widely measured and important categorical variables, whether being central analytical variables, or relevant controls in analyses with a different focus. We propose contributions in the form of (i) new online services that can assist in harmonizing measures, which we have generated through a recent ESRC-funded project on ‘data management’ (www.dames.org.uk), and (ii) our own suggestions on achieving effective harmonization of longitudinal variables, which focus upon documentation for replication, the recognition of age-cohort distributional differences, and a general advocacy of scaling categories as part of a strategy of ‘functional equivalence’. These cannot solve every comparability challenge in these areas, but, we argue, are steps in the right direction.

25

Data curation tool (for collecting metadata)

17/MAR/2010 DIR workshop: Handling Social Science Data

26

[Timing issues] Relation between time of survey sweep and exact timing of interview

NCDS(March)/BCS (April), & relative age effect in Engl. (Jul/A) & Scotland (Jan/Feb)

In some instances, categorical thresholds make timing matter more (e.g. b/a exam res)

…9/11 effects seem evident in the BHPS, but vanish after controlling for age

27

0 20 40 60

After 9/11

Just after 9/11

On 9/11

Before 9/11

Source: BHPS 2001 interviews. N =1132; 307; 4063; 13384 by time

Future looks good Age

28

WhB-a

WhO-b

Ind-c

WhB-c

PkB-a

WhO-c

WhI-c

WhB-b

WhO-a

Oth-b

BA-b

BC-b

Chi-bPkB-bBC-a

BC-c

Ind-b

Chi-c

Oth-a

PkB-c

BA-a

WhI-a

Ind-a

WhI-b

Oth-c

BA-cChi-a

I/II

IIIaIVabcV/VIVII/IIIb

UnemployedInactive

-2-1

01

2D

imen

sion

2 (

22.

1%)

-2.5 -2 -1.5 -1 -.5 0 .5Dimension 1 (58.4%)

a = Born in UK; b = Came to UK before 1970; c = came to UK 1970 or laterN=640295 (Data: Li and Heath, 2008)

LFS pooled data for men, 1991-2005Correspondence analysis dimension scores

Predicting poor subjective health, BHPS w15

29

legend: * p<0.05; ** p<0.01; *** p<0.001 bic 32364 32409 32468 32219 32363 32362 32485 ll -16154 -16139 -16131 -16072 -16148 -16143 -16134 r2 .071 .073 .074 .073 .072 .072 .074 N 12294 12294 12294 12239 12294 12294 12294 _cons 1.9*** 1.9*** 1.9*** 1.9*** 1.9*** 1.7*** 1.9*** _Iethim_29 .36* _Iethim_28 -.064 _Iethim_27 .96** _Iethim_26 .27 _Iethim_25 .32** _Iethim_24 .65 _Iethim_23 -.023 _Iethim_22 .66** _Iethim_21 -.057 _Iethim_19 .34 _Iethim_18 -.055 _Iethim_17 .44 _Iethim_16 -.066 _Iethim_15 .29 _Iethim_14 .044 _Iethim_13 .074 _Iethim_12 .0048 ethsa -.0011** ethscore -.018** .03 _Irace2_3 .26*** _Irace2_2 .19 _IracXoage_9 -5.2e-05 _IracXoage_8 -.0054 _IracXoage_7 .057** _IracXoage_6 .017 _IracXoage_5 .0057 _IracXoage_4 .022 _IracXoage_3 -.0029 _IracXoage_2 .016 _Irace_9 .35** .35 _Irace_8 -.06 .18 _Irace_7 .83** -1.1 _Irace_6 .14 -.54 _Irace_5 .31** .064 _Irace_4 .18 -.6 _Irace_3 .023 .14 _Irace_2 .29 -.42 educ4_4 .31*** .3*** .3*** .3*** .3*** .3*** .3*** educ4_2 -.1*** -.1*** -.1*** -.1*** -.1*** -.1*** -.1*** educ4_1 -.24*** -.25*** -.25*** -.25*** -.25*** -.25*** -.25*** oage .0062*** .0063*** .0062*** .0063*** .0062*** .011*** .0063*** fem .075*** .077*** .077*** .079*** .075*** .075*** .077*** Variable m1 m2 m2b m3 m5 m5b m4