michael braun & dorothée behr gesis – leibniz institute for the social sciences, mannheim
DESCRIPTION
Using Probing Techniques to Assess Intercultural Validity The Case of Attitudes Towards Immigrants. Michael Braun & Dorothée Behr GESIS – Leibniz Institute for the Social Sciences, Mannheim. Overview. Establishing comparability across countries Conventional use of cognitive techniques - PowerPoint PPT PresentationTRANSCRIPT
Using Probing Techniques to Assess Intercultural Validity
The Case of Attitudes Towards Immigrants
Michael Braun & Dorothée Behr
GESIS – Leibniz Institute for the Social Sciences,
Mannheim
Overview
Establishing comparability across countries Conventional use of cognitive techniques Problems and alternatives Methods and data Results Conclusions
Establishing comparability across countries
• In intercultural research, measurement invariance has to be established across countries
• Application of data-analytic procedures does not get – or at least not always – at causes of incomparability (possible exception: ML-SEM)
• Possible solution: cognitive techniques (category-selection probing, comprehension probing, specific probes)
Conventional use of cognitive techniques
• as a pretesting device to detect bad items and improve a questionnaire
• mainly in a cognitive laboratory, i.e. not under regular field conditions
• respondents are briefed about the setting as being a pretest and their task being to help find (comprehension) problems
• often “professional” pretest respondents are sought who might develop lay theories
Conventional use of cognitive techniques
• normally conducted by interviewers proactively searching for hidden comprehension problems
• mostly applied to behavioral questions
• traditionally based on small quota samples (often not more than 20 interviews)
• low number of cases does not permit analysis of diverging argumentation patterns, shared only by a small proportion in the population, and quantification of results is impossible
•
Conventional use in intercultural studies
• mainly aiming at the improvement of translations
• often application to ethnic groups in only one country
• cognitive interviewing across countries very difficult to organize (availability of cognitive laboratories; standardization of procedures)
• different house styles in recruiting respondents and guidelines specifying interviewer behavior
Problems and an alternative
• personal interviews even counterproductive (standardization of interviewer qualification and behavior hard to achieve)
• time requirements of implementation by face-to-face interviews in the laboratory
• no compelling reason (other than costs under traditional implementation) for restrictions
• application of cognitive interviewing in the field as independent part of the research process
Advantages of using internet surveys
• allow for a higher degree of anonymity, improving data quality
• reduction of respondent burden
• online access panels as cost-efficient means to increase sample size and to investigate small subgroups with diverging response behavior
Our project
“Enhancing the Validity of Intercultural Comparative Surveys. The Use of Supplemental
Probing Techniques in Internet Surveys”
Funded by the German Science Foundation (DFG)applicants: Michael Braun, Wolfgang Bandilla &
Lars Kaczmirek
Project goal
Detection of interpretation differences in international surveys by cognitive methods (category-selection, comprehension and specific probing).
Demonstrating the feasibility to implement cognitive methods in non-probability online access panels.
Testing the approach with substantive survey topics, in the present case immigration items from the International Social Survey Program (ISSP) 2003 National Identity module.
There are different opinions about immigrants from other countries living in [country]. (By “immigrants” we mean people who come to settle in [country]) How much do you agree or disagree with each of the following statements?
a. (crime) Immigrants increase crime rates.
b. (economy) Immigrants are generally good for [country’s] economy.
c. (jobs) Immigrants take jobs away from people who were born in [country].
d. (culture) Immigrants improve society by bringing in new ideas and cultures.
[5-point Likert scale from agree strongly to disagree strongly; crime and jobs reverse coded, such that high values = negative attitudes.]
Study design and implementation
Non-probability online access panels in 7 countries/regions (ca. 500 cases each).
Canada, Denmark, United States, (western and eastern) Germany, Hungary, Spain.
Target population defined as country nationals aged btw. 18-65.
Quotas set according to gender, education, age. Data collection period January 2011.
Study design and implementation
Specific probing: “Which type of immigrants were you thinking of when you answered the question? The previous statement was: #.”
4 immigrant items were rotated using 4 splits, only the first item was probed
Challenges
Can we reproduce the answer patterns found in the ISSP data for the different countries?
Do answers to the specific probes reveal different patterns across countries?
Attitudes towards immigrants
Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM
Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6
Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7
United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7
W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6
E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7
Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8
Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7
Attitudes towards immigrants
Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM
Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6
Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7
United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7
W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6
E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7
Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8
Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7
Attitudes towards immigrants
Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM
Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6
Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7
United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7
W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6
E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7
Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8
Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7
Attitudes towards immigrants
Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM
Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6
Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7
United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7
W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6
E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7
Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8
Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7
Attitudes towards immigrants
Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM
Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6
Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7
United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7
W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6
E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7
Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8
Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7
CICOM compared to ISSP
ISSP data are neatly reproduced in the CICOM data in most cases.
As for deviations, it is not clear whether our sample or social change is the reason (ISSP data were collected in 2003).
It is particularly the Hungarians which seem to be less xenophobic in CICOM than ISSP
Answer patterns across countries
Probing question not answered Answers to probing not pertinent to question, not
categorizable answers (“other” category) Reference to immigrants in general, no specific group Reference to specific ethnicities, and to which Reference to immigrant groups defined by either the
positive or negative behavior referred to in the item Reference to the legal/illegal immigrant distinction
Results from specific probing: no answer and “other”
No answer Other
CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain
11.213.811.311.815.512.59.7
23.328.116.020.920.532.712.1
“other“: partly interpretation as category-selection probing
Results from specific probing:Percentages for major answer types
Immigrants in general
Specific ethnicities
Defined by pos/neg pole
Legal-illegal distinction
CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain
32.629.621.026.227.917.233.1
18.218.825.228.623.732.338.3
6.4 5.64.1 5.61.3 1.37.9 4.55.6 6.52.1 2.61.7 2.4
2.70.0
23.90.00.40.82.8
Results from specific probing:percentages of major specific ethnicities
Islamic countries
Eastern Europe
Asia Latin America
CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain
5.214.02.3
20.917.08.4
17.7
0.04.50.4
13.813.014.012.8
14.92.24.63.83.4
19.43.4
3.50.2
23.10.00.00.2
20.6
Probing and reality
This reflects nicely immigration reality in most of these countries:
Canada: really multi-ethnic, many Asians … Denmark: who knows this country??? United States: though also multi-ethnic, Latino migration
dominant West (and East) Germany: Turks first, Russians (but
perhaps partly ethnic Germans referred to as Russians), Latin Americans unfortunately missing
Probing and reality
Spain: Romanians as biggest single migrant group, followed by Marroccans; Latin Americans lumped together is biggest group (but in many cases not distinguishable from natives – or even originally Spanish remigrants from southern America, e.g. from Argentina)
But: What the hell have the Chinese to do with Hungary? Where are they?
Further results
Immigrants from EU-15 and from Subsaharan Africa are also mentioned, but not as frequently.
Sinti and Roma are rarely mentioned, and mostly restricted to Hungary.
Other groups are still less often mentioned and are put in the “other“ category in our coding schema.
Further results
Very small country differences in xenophobia. After inclusion of the probing codes no country
differences left. Unsurprisingly, thinking of immigrants defined by
the positive pole reduces and by the negative pole increases xenophobia.
Results from specific probing:general attitudes towards immigrants dependent on
thinking of the major specific ethnicities
Overall mean
No spec. group
Islamic countries
Eastern Europe
Asia Latin America
CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain
11.212.112.211.712.211.712.3
10.610.911.510.611.711.512.4
13.613.913.813.113.711.113.2
a13.2a
13.513.812.212.8
12.211.611.112.910.711.011.3
11.5a
13.0aaa
12.3
Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.
Results from specific probing:general attitudes towards immigrants dependent on
thinking of the major specific ethnicities
Overall mean
No spec. group
Islamic countries
Eastern Europe
Asia Latin America
CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain
11.212.112.211.712.211.712.3
10.610.911.510.611.711.512.4
13.613.913.813.113.711.113.2
a13.2a
13.513.812.212.8
12.211.611.112.910.711.011.3
11.5a
13.0aaa
12.3
Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.
Results from specific probing:general attitudes towards immigrants dependent on
thinking of the major specific ethnicities
Overall mean
No spec. group
Islamic countries
Eastern Europe
Asia Latin America
CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain
11.212.112.211.712.211.712.3
10.610.911.510.611.711.512.4
13.613.913.813.113.711.113.2
a13.2a
13.513.812.212.8
12.211.611.112.910.711.011.3
11.5a
13.0aaa
12.3
Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.
Results from specific probing:general attitudes towards immigrants dependent on
thinking of the major specific ethnicities
Overall mean
No spec. group
Islamic countries
Eastern Europe
Asia Latin America
CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain
11.212.112.211.712.211.712.3
10.610.911.510.611.711.512.4
13.613.913.813.113.711.113.2
a13.2a
13.513.812.212.8
12.211.611.112.910.711.011.3
11.5a
13.0aaa
12.3
Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.
Further results
In general: Thinking of immigrants in general is connected to less
xenophobia. Thinking of immigrants from Islamic and Eastern
European countries is connected to higher xenophobia. Thinking of immigrants from Asia is related to less
xenophobia. Thinking of Latin Americans does not show a clear
tendency (in the US: negative).
Further results
Differences between issues and specific immigrant groups mentioned are not really dramatic
Islamic countries are mentioned most frequently with crime item (15.4%) and least with economy item (9%)
The same for eastern Europeans: 14.3% in connection with crime, 4.8% in connection with the economy
There are no relevant issue-specific differences for Asians and Latin Americans
Some Conclusions
The reasoning in the seven countries/regions is highly similar on an abstract level, though the concrete groups of immigrants mentioned differ.
Respondents with positive beliefs tend to use inclusive statements (e.g. they refer to all immigrants), respondents with negative focus more on selective groups.
In the American context, the legal status of immigrants is mentioned more often than in the European context.
Latin Americans are clearly not equivalent in the Spanish and US-American contexts.