comparability of categorical variables in longitudinal survey research paul lambert*, alison bowes*,...

of 29 /29
Comparability of categorical variables in longitudinal survey research Paul Lambert*, Alison Bowes*, Vernon Gayle*, Guy Warner*, and Tom Doherty** (*University of Stirling; ** University of Glasgow) Presented to the Society for Longitudinal & Life Course Studies Clare College, Cambridge 22-24 September 2010

Author: jenna-hurley

Post on 26-Mar-2015




0 download

Embed Size (px)


  • Slide 1

Comparability of categorical variables in longitudinal survey research Paul Lambert*, Alison Bowes*, Vernon Gayle*, Guy Warner*, and Tom Doherty** (*University of Stirling; ** University of Glasgow) Presented to the Society for Longitudinal & Life Course Studies Clare College, Cambridge 22-24 September 2010 Slide 2 Comparability of categorical variables in longitudinal survey research 1)Comparability strategies and selected problems 2)Some analytical prescriptions 3)The relevance of the GESDE services..Underlying motivation to study patterns/changes over time in distributions measured by categorical instruments....Comparisons Are the Essence.. (Treiman, 2009: 382) 2 Slide 3 1) Comparability strategies Harmonisation/standardisation is used to aid in the comparison of data across contexts Measurement vs meaning/functional equivalence [i.e. same absolute qualities of position, vs same relative meaning of position within the distribution] Ex post vs pre-harmonisation [e.g. Harkness et al. 2003; Hoffmeyer-Zlotnik & Harkness 2005] Comparability problem: when comparison (over time) is muddled by a lack of equivalence in absolute and/or relative position 3 Slide 4 4 Slide 5 Categorical variables: coding membership of groups Most social survey data is categorical! Importance of: fine levels of detail; boundaries; recoding strategies; cognitive issues 5 Interest in this talkHighMediumLow[n.a.]ordinal Country of birthUKIndiaPoland..nominal..Please describe your occupation... SecretaryLecturerShop assistant No jobnominal NS-SEC31.268(recode of 3) CAMSIS(F)62.494.242.8[n.a.](scaling of 3) Sports you enjoyFootballCricketDarts.nominal+MR Slide 6 Three important categorical variables StandardsKey correlates (r 2 )*Reference Educational qualifications ISCED; GHS; years of study year of birth (0.25) ; gender (0.04); Schneider (2010); Treiman (2007) OccupationsNS-SEC; RGSCgender (0.41) ; age (0.05) ; region (0.02) ; Rose and Harrison (2010) EthnicityONS; Years since immigr. year of birth (0.08) ; immigrant status (0.26) ; region (0.09) Bosveld et al. (2006) Longitudinal problems of comparison include: Changing structural contexts (distributions; correlations; sparsity) Changing measurement practices (e.g. decennial revisions; admin data) Changing international comparisons (e.g. ISCO88-SOC; ISCED-ONS) Existing measurement instruments are generally outside the domain of concept formation research (cf. Jowell, 2007) * 2008 BHPS 20+yrs, qfedhi, jbsoc, xeth using race{l} + 0.01 downwt for Wh Slide 7 Selected comparability problems for categorical measures in longitudinal analysis 7 Interpretation problems (same relative meaning over time?) Operationalisation problems Changing categories Changing distribution of categories Sparse categories Parsimony multivariate models and interactions Hidden change in administrative practices (e.g. recoding) Changing relevant correlates (e.g. with yob) Timing issues Documentation Slide 8 Common practice with longitudinal surveys Comparative model for categorical data is overwhelmingly one of measurement equivalence Nominal equivalence: offer, or locate, categories in the same scheme over time (often suppressing further detail in released data) Distribute explanatory metadata (e.g. Validity studies may be used to test correlation with expectations (e.g. Rose and Harrison 2010) This is commonly visible as coding to the lowest common denominator 8 Slide 9 9 A measurement equivalence example comparing ethnicity categories in context of demographic change Are the compatible categories equivalent? Who does the work (data distributors and/or analysts)? & who records it (e.g. Mohler et al., 2008) ? Slide 10 10 Here, measurement equivalence is compromised by administrative errors, & meaning equivalence is doubtful due to industrial restructuring (orig. occ. codes not available) Slide 11 [Changing correlates] Occupational measures change over time in total distribution and this interacts with occupational gender segregation 11 Slide 12 Meaning equivalence is preferable in longitudinal survey research!! Collect and preserve the data in its original detail Preserves information; easier for respondent; better documentation Allow analysts to harmonise/standardise using flexible/tailored equivalence strategies Same relative meaning may=diff. absolute properties (e.g. Educ-by-yob)? Ideally, trying and comparing more than one strategy Pressures for measurement equivalence from statistics agencies, analysts, consumers..goal of standardisation is to enhance comparability; inappropriate standardisation may do just the opposite (Harkness, 2008: 61)..when nominal equivalence is not enough.. (Schneider, 2010) 12 Slide 13 2) Some analytical prescriptions Measurement what we already do Appealing because we have existing standards/literature on most measures Improved practice: Use the published standards when they exist(!) cf. temptation to recode/collapse recommended measures Documentation/replication of derivations Comparison between a few different options Recognise parsimony/multivariate interactions 13 Slide 14 14 (Own analysis, for the NeISS project, Slide 15 15 Meaning equivalence For categorical data, equivalence for comparisons is often best approached in terms of meaning equivalence (because of non-linear relations between categories and shifting underlying distributions) (even if measurement equivalence seems possible) Arithmetic standardisation offers a convenient form of meaning equivalence by indicating relative position with the structure defined by the current context For categorical data, this can be achieved/approximated by scaling categories in one or more dimension of difference Slide 16 16 Slide 17 UK bornNot UK born Whi Iri 0 (base) Whi Other-5.4-3.4 Black Car.-9.6 Black Afr-12.7-5.9 Indian-16.1 Pak/Bang.-16.8-3.2 Chinese-15.8-6.5 Other-11.0-3.9 17 Slide 18 Is scaling useful?..sometimes.. Revealing exploratory exercise Parsimonious functional form If complex categorical measure is a control variable If interaction effects are considered If story told by (transformation of) a linear functional form is persuasive (e.g. exponential increase) Good for Multiple Response data (e.g. take arithmetic function of related cases) 18 Slide 19 3) The GESDE services DAMES ( node seeking to support data management in social research Data management = tasks involved in organising/ manipulating/enhancing data in preparation for research analysis Making tasks easier and more consistent Pre-analysis tasks conducted by researchers and data distributors Re-coding; harmonising; standardising; cleaning; linking Recording and distributing metadata for replication 19 Slide 20 20 Contributions: Preserving/facilitating data management Slide 21 DAMES provides online services for data coordination/organisation Tools for handing variables in social science data Recoding measures; standardisation / harmonisation; Linking; Curating 17/MAR/2010 DIR workshop: Handling Social Science Data 21 Slide 22 GESDE Search and browse supplementary data on occupations; educational qualifications; ethnicity 17/MAR/2010 DIR workshop: Handling Social Science Data 22 Slide 23 23 Conclusions: What we do and what we ought to do Research writers routinely select single simplifying sub- optimal collinear categorisations of concepts Due to coordinated instructions [e.g. Blossfeld et al. 2006] Due to perceived lack of available alternatives Due to perceived convenience To make longitudinal comparative analyses more scientific (cumulative & open to cross-examination) we should Acknowledge and discuss our equivalence strategies Operationalise and deploy various categorisations (measurement equivalence) and scalings/arithmetic measures (meaning equivalence), and explore their distributional properties and keep a replicable trail of all these activities.. ideally by using GESDE!! Slide 24 References Blossfeld, H. P., Mills, M., & Bernardi, F. (Eds.). (2006). Globalization, Uncertainty and Men's Careers: An International Comparison. Cheltenham: Edward Elgar. Bosveld, K., Connolly, H., Rendall, M. S., & (2006). A guide to comparing 1991 and 2001 Census ethnic group data. London: Office for National Statistics. Harkness, J. (2008). Comparative survey research: goals and challenges. In E. De Leeuw, J. Hox & D. A. Dillman (Eds.), International Handbook of Survey Methodology. London: Psychology Press. Harkness, J., van de Vijver, F. J. R., & Mohler, P. P. (Eds.). (2003). Cross-Cultural Survey Methods. New York: Wiley. Hoffmeyer-Zlotnik, J. H. P., & Harkness, J. (Eds.). (2005). Methodological Aspects in Cross-National Research. Mannheim: GESIS - ZUMA Zentrum fur Umfragen, Methoden und Analysen. Jowell, R., Roberts, C., Fitzgerald, R., & Eva, G. (2007). Measuring Attitudes Cross-Nationally. London: Sage. Lambert, P. S., Prandy, K., & Bottero, W. (2007). By Slow Degrees: Two Centuries of Social Reproduction and Mobility in Britain. Sociological Research Online, 12(1). Li, Y., & Heath, A. F. (2008). Socio-Economic Position and Political Support of Black and Ethnic Minority Groups in the United Kingdom, 1972-2005 [computer file]. 2nd Edition. Colchester: UK Data Archive [distributor], SN: 5666. Mohler, P. P., Pennell, B.-E., & Hubbard, F. (2008). Survey Documentation: Toward professional knowledge management in sample surveys. In E. De Leeuw, J. Hox & D. A. Dillman (Eds.), International Handbook of Survey Methodology (pp. 403-420). Hove: Psychology Press. Rose, D., & Harrison, E. (Eds.). (2010). Social Class in Europe: An Introduction to the European Socio-economic Classification London: Routledge. Schneider, S. L. (2010). Nominal comparability is not enough: (In-)Equivalence of construct validity of cross-national measures of educational attainment in the European Social Survey. Research in Social Stratification and Mobility. Simpson, L., & Akinwale, B. (2006). Quantifying Stablity and Change in Ethnic Group. Manchester: University of Manchester, CCSR Working Paper 2006-05. Treiman, D. J. (2007). The Legacy of Apartheid: Racial Inequalities in the New South Africa. In A. F. Heath & S. Y. Cheung (Eds.), Unequal Chances: Ethnic Minorities in Western Labour Markets. Oxford: Oxford University Press, for the British Academy. Treiman, D. J. (2009). Quantitative Data Analysis: Doing Social Research to Test Ideas. New York: Jossey Bass. Slide 25 Abstract In working with longitudinal social surveys, we sometimes ignore, or at the very least over-simplify, challenges of measurement comparability in variables over time. Often, our approaches to harmonization involve searching for categories with consistent names over time, frequently achieving such nominal equivalence by reducing the level of detail of measures to the so-called lowest common denominator. In this paper we highlight some well-known, and some less well-known, comparability problems for categorical measures in longitudinal surveys. We focus on UK studies and their measures of occupational position, educational qualifications and ethnicity. These are widely measured and important categorical variables, whether being central analytical variables, or relevant controls in analyses with a different focus. We propose contributions in the form of (i) new online services that can assist in harmonizing measures, which we have generated through a recent ESRC-funded project on data management (, and (ii) our own suggestions on achieving effective harmonization of longitudinal variables, which focus upon documentation for replication, the recognition of age-cohort distributional differences, and a general advocacy of scaling categories as part of a strategy of functional equivalence. These cannot solve every comparability challenge in these areas, but, we argue, are steps in the right direction. 25 Slide 26 Data curation tool (for collecting metadata) 17/MAR/2010 DIR workshop: Handling Social Science Data 26 Slide 27 [Timing issues] Relation between time of survey sweep and exact timing of interview NCDS(March)/BCS (April), & relative age effect in Engl. (Jul/A) & Scotland (Jan/Feb) In some instances, categorical thresholds make timing matter more (e.g. b/a exam res) 9/11 effects seem evident in the BHPS, but vanish after controlling for age 27 Slide 28 28 Slide 29 Predicting poor subjective health, BHPS w15 29