![Page 1: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/1.jpg)
1
Universitat de Girona
Measurement Quality in Social Networks
LLuís Coromina
Quantitative Methods, University of Girona Department of Economics,
Campus Montilivi, 17071 Girona, Spain E-mail: [email protected]
![Page 2: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/2.jpg)
2Universitat de Girona
Outline
• Studies on Social Network Data Quality
• Survey Design o Data collection o Questionnaire Design
• Types of hypothesis tested in other Quality studies
• Quality Measurement (Validity and Reliability)
• Measurement procedure o Multitrait-Multimethod Model (MTMM) o Multilevel MTMM o Interpretation
![Page 3: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/3.jpg)
3Universitat de Girona
Outline
• INSOC PROJECT: o Social Network Data collection Design o Multitrait-Multimethod Model results o Meta-analysis Design o Meta-analysis Results
• Results and comparison from others studies.
![Page 4: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/4.jpg)
Intro
4
Objective
Universitat de Girona
4
Main purpose of scientific research is the interpretation and/or prediction
of phenomena
Quality measurement instruments (empirical data) is of crucial importance
It is important to know how well we can measure with certain
measurement instruments and how much a researcher can rely on findings
based on data collected.
![Page 5: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/5.jpg)
Intro
5
Objective
Universitat de Girona
5
Egocentered network data are especially important.
Network characteristics (network size, structure and composition)
Characteristics of network members (feeling of closeness or importance;
frequency of advice, collaboration, social support) collected from the
respondent (ego).
![Page 6: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/6.jpg)
Intro
6Universitat de Girona
Quality in Social Network Survey Data
6
Studies on Social Network Data Quality
Killworth & Bernard, 1976, 1980; Bernard et al., 1979,1980, 1982;
Hammer, 1984; Sudman, 1985, 1988; Freeman and Romney, 1987;
Freeman, Romney & Freeman, 1987; Corman and Bradford, 1993;
Hlebec, 1993; Brewer and Webster, 1997,…
![Page 7: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/7.jpg)
Intro
7Universitat de Girona
Quality in Social Network Survey Data
7
People are generally very inaccurate in reporting on their past
interactions with other people.
People remember long-term or typical patterns of interaction with other
people rather well.
Accuracy of reporting about interactions is also influenced by the
frequency of interaction (more frequent contact with group members,
more accurate reports) and by the reliability of an individual
respondent on about actual interactions (Romney and Faust, 1982; Romney
and Weller, 1984).
![Page 8: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/8.jpg)
Intro
8Universitat de Girona
Quality in Social Network Survey Data
8
Quality of measurement, specially reliability, can be affected by the type of
social support (Ferligoj & Hlebec, 1999).
+ Emotional and informational support dimensions
Reliability
- Social companionship and material support dimensions
Reasons for ‘higher quality’
More “intimate” types of support � more important to the respondents.
� Relatively small number of very
important people.
Less effortful for the respondent. Network
members are important and close
![Page 9: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/9.jpg)
Intro
9Universitat de Girona
Data Collection Methods
9
DATA COLLECTION METHODS
In Social Sciences, the most frequent measurement instrument is a SURVEY
![Page 10: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/10.jpg)
Intro
10Universitat de Girona
Data Collection Methods
10
![Page 11: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/11.jpg)
Intro
11Universitat de Girona
Data Collection Methods
11
With interviewer Self-administration
Paper •••• Personal (Face-to-face)
•••• telephone
•••• “paper self-
administered”
Electronic •••• computer assisted personal
interview (CAPI)
•••• computer assisted telephone
interview (CATI)
•••• Web surveys
•••• Computer Assisted Self-
Interview (CASI)
•••• ACASI
•••• Mobile phone surveys
NO ONE COLLECTION METHOD IS BEST FOR ALL CIRCUMSTANCES.
THE CHOICE MUST BE MADE WITHIN THE CONTEXT OF THE PARTICULAR
OBJECTIVES OF THE SURVEY AND THE RESOURCES AVAILABLE.
![Page 12: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/12.jpg)
Intro
12Universitat de Girona
Data Collection Methods
12
Characteristics of the Population
+
Characteristics of The Sample
+
Types of Questions
+
Question Topic
+
Response Rate
+
€€ Cost $$
+
Time
![Page 13: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/13.jpg)
Intro
13Universitat de Girona
Data Collection Methods
13
PERSONAL INTERVIEWING
ADVANTAGES: DISADVANTAGES:
Generally: highest cooperation and lowest
refusal rates
Most costly mode of administration
Allows for longer, more complex interviews Longer data collection period
High response quality Interviewer concerns
Takes advantage of interviewer presence
Multi-method data collection
TELEPHONE INTERVIEWING
ADVANTAGES: DISADVANTAGES:
Less expensive than personal interviews Samples of general population Non-response
Shorter data collection period than personal Questionnaire constraints
Interviewer administration (vs. mail) Difficult to administer questionnaires on
sensitive or complex topics
Better control and supervision of interviewers
(vs. personal)
Biased against households without
telephones, unlisted numbers
Better response rate than mail for list samples
![Page 14: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/14.jpg)
Intro
14Universitat de Girona
Data Collection Methods
14
MAIL SURVEYS
ADVANTAGES: DISADVANTAGES:
Generally lowest cost Most difficult to obtain cooperation
Can be administered by smaller team of
people (no field staff)
No interviewer involved in collection of data
Access to otherwise difficult to locate, busy
populations
More likely to need an incentive for
respondents
Respondents can look up information or
consult with others
Need good sample
Slower data collection period than telephone
![Page 15: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/15.jpg)
Intro
15Universitat de Girona
Data Collection Methods
15
WEB SURVEYS
ADVANTAGES: DISADVANTAGES
Lower cost (no paper, postage, mailing,
data entry costs)
% of homes own a computer;
% have home e-mail.
Can reach international populations Representative samples difficult - cannot
generate random samples of general
population
Time required for implementation reduced Differences in capabilities of people's
computers and software for accessing
Web surveys
Complex skip patterns can be programmed Different ISPs/line speeds limits extent of
graphics that can be used
Sample size can be greater
![Page 16: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/16.jpg)
Intro
16Universitat de Girona
Data Collection Methods
16
MEASUREMENT QUALITY IMPLICATIONS OF MODE SELECTION
Social Desirability
� Under-represent (illegalities) or Over-represent (voting) different population groups.
� Sensitive information from respondents
� Self-administered methods produce fewer social desirability effects.
Response effects (wording, answer categories order, or questions order)
Response order (Primacy-visually- and recency –telephone)
Data completeness (with interviewers more questions answered)
Completion times
![Page 17: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/17.jpg)
Intro
17Universitat de Girona
Data Collection Methods
17
Mean completion times(in seconds)
by mode (web vs Telephone) and age group
![Page 18: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/18.jpg)
Intro
18Universitat de Girona
Data Collection Methods
18
Differences in data quality for different methods????
Data collection method for ego-centered network data.
Most used face-to-face data collection mode.
Face to face vs. Telephone (Kogovšek et al., 2002)
Validity and Reliability of web surveys
(Couper, 2000; Dillman, 2000; Couper et al., 2001; Vehovar et al., 2002)
Quality of Egocentered Network Data Collected via Web
Marin (2002) ; Lozar Manfreda et al. (2004).
![Page 19: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/19.jpg)
Intro
19Universitat de Girona
19
Questionnaire Design
QUESTIONNAIRE DESIGN
Problems in the response process Survey ERRORS
Krosnick (1999); Groves (2004),
Tourangeau, Rips & Rasinski (2000), Dillman, (2007)
![Page 20: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/20.jpg)
Intro
20Universitat de Girona
20
Questionnaire Design
1) Failure to encode the information required
Problem of asking people about what topics that they do not pay much
attention OR are not important.
Improvement: Pre-test; Filter questions (or multiple name generator)
2) Misinterpretation of the questions
Hospital study: During the past 12 months, since _______, how many times have you
seen or talked to doctor [NAME i ] about your health? Do not count any time you might
have seen a doctor while you were a patient in a hospital, but count all other times your
actually saw or talked to a medical doctor of any kind.
![Page 21: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/21.jpg)
Intro
21Universitat de Girona
21
Questionnaire Design
3) Forgetting and other memory problems
Example:
Low options High responses
Responses % Responses %
<1/2 hr 7.4 < 2 ½ hr 62.5%
½ to 1hr 17.7 2 ½ to 3 hr 23.4%
1 to 1 ½ hr 26.5 3 to 3 ½ hr 7.8%
1 ½ hr to 2 hr 14.7 3 ½ hr to 4 hr 4.7%
2 to 2 ½ hr 17.7 4 to 4 ½ hr 1.6%
>2 ½ hr 16.2 >4 ½ hr 0%
Question about television watching can go from 16% to 37% saying that they watched
more than 2.5 hours/day, depending on the set categories.
![Page 22: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/22.jpg)
Intro
22Universitat de Girona
22
Questionnaire Design
4) Sensitive (social desirability) questions (recommendations)
• More time to respondents to answer in order to not misreport information (Krosnick 1999)
• Question example for strategy for over-reported behaviors:
In talking about elections, we often find that a lot of people were not able to vote because they are not registered, they were sick, or they just didn’t have the time. How about you – did you vote in the elections this November?
• Use self-administration, computer-administered questionnaire or similar methods to improve reporting.
![Page 23: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/23.jpg)
Intro
23Universitat de Girona
23
Questionnaire Design
5) Failure to follow instructions (mainly for self-administered respondents)
If first page is half-page explaining instructions, respondent use to skip that part.
![Page 24: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/24.jpg)
Intro
24Universitat de Girona
24
Questionnaire Design
Instructions must be exactly where is needed and not in a separate section
![Page 25: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/25.jpg)
Intro
25Universitat de Girona
25
Questionnaire Design
Format of response scale (Network Questionnaires)
• Binary judgments (least difficult for respondents) about whether
respondents have a specified relationship with each actor on the roster.
• Ordinal ratings of tie strength
• Rankings: More demanding for respondents.
Majority of respondents preferred binary over ranking or rating…
![Page 26: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/26.jpg)
Intro
26Universitat de Girona
26
Questionnaire Design
…. BUT reliability for ratings > binary judgments (Ferligoj & Hlebec, 1999).
Recommendations (not only network questionnaires)
• Use closed questions for measuring attitudes
• Use 5 point response scale and labeled every scale point. With 9 or 11
point response scale the beginning and the ending.
• Use ranking (preferences order) only if the respondents can see all the
alternatives: otherwise use paired comparisons.
![Page 27: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/27.jpg)
Intro
27Universitat de Girona
27
Questionnaire Design
Name Generator Design
Studies suggest using minimal modules of name generators:
Van Der Poel (1993):
3-item: Discussing a major life change, aid with household tasks, and monthly
visiting.
5-items, add: Borrowing household money and going out socially.
Bernard et al. (1990):
5 items: Social activities, hobbies, personal problems, advice about important
decisions, closeness
Burt (1997):
3-item: GSS important matters issue, socializing, and discussion of a job
change
![Page 28: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/28.jpg)
Intro
28Universitat de Girona
28
Questionnaire Design
Clusters of persons named by relationships (families, workplaces,…) than by
similarity of individual features -gender, race or age- (Fiske ,1995)
Use Recognition rather than Recall (when possible) - Level of ‘forgetting’-
(Brewer, 2000)
Multiple name generators may limit forgetting because persons forgotten for one
generator are often name in response to others.
Single name generator maybe sufficient for ‘core networks’
![Page 29: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/29.jpg)
Intro
29Universitat de Girona
Quality in Social Network Data
29
Measurement Comparisons
Questionnaire components
Network ties
![Page 30: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/30.jpg)
Intro
30Universitat de Girona
Quality in Social Network Data
30
In summary,
Validity and Reliability of survey data, social networks, can be affected by
many characteristics of the measurement instrument ( Name Interpreters).
a) Response scale (binary; 5 ordinal ending; 5 ordinal all; line drawing scale; 11
ordinal)
b) Response category labels( all categories or end points).
c) Name Generator method (Recognition vs free recall).
c) Data collection method (face-to-face interview, telephone, web survey,…)
e) Question wording (after we obtain the list of alters with name generators,
we can ask name interpreter questions “by alters” or “by questions”).
f) Layout of the questionnaire: Plain vs Graphical
![Page 31: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/31.jpg)
Intro
31Universitat de Girona
Quality in Social Network Data
31
Types of networks analyzed:
Social support (Weiss, 1974; Hirsch, 1980; Wills, 1985:, vaux, 1988)
1) instrumental support (aid, material support, or tangible support) is the
provision of financial aid, material support, and necessary services),
2) informational support (advice support, appraisal support, or cognitive guidance)
involves help in defining, understanding, and coping with problematic events)
3) emotional support (or close support),
4) social companionship (diffuse support and belongingness) relates to time spent
with others in leisure activities).
![Page 32: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/32.jpg)
Intro
32Universitat de Girona
Quality in Social Network Data
32
I) Frequency of Scientific advice II) Frequency of Collaboration III)Asking for crucial Information IV) Social activities outside the work
Tie Characteristics: Demographic variables
a) Measures of tie strength Education Frequency of contact Gender Feeling of closeness Age
b) Feeling of importance
c) Frequency of the alter
upsetting the ego
![Page 33: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/33.jpg)
Intro
33Universitat de Girona
Hypothesis and Research Questions from Studies
33
Hypothesis and Research Questions from Studies
![Page 34: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/34.jpg)
Intro
34Universitat de Girona
Hypothesis and Research Questions from Studies
34
• To test whether the recognition data collection technique is more stable than that of free recall (Ferligoj & Hlebec,1999)
• The size of the social network and age of the respondent should not
affect the similarity of free-recall and recognition methods.
• Reliability and validity of the measurement of tie characteristics for emotional and informational support alters would be higher compared to that for social companionship and material support alters. Kogovsek & Ferligoj (2004)
• The stability of social support : Emotional support (close and important ties) more stable than Material support (provided by specialized sources).
Closeness is not required for providers of material support and informational support. Hlebec & Ferligoj (2002)
![Page 35: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/35.jpg)
Intro
35Universitat de Girona
Hypothesis and Research Questions from Studies
35
Hypothesis Predictors Kogovsek & Ferligoj (2005)
Measurement instrument Higher quality of measurement with by alters than by questions data collection
technique Higher quality of measurement of cognitively demanding questions with the
personal interview/by alters Higher quality of measurement of sensitive questions with the telephone
interview/by alters Network size Higher quality of measurement in small networks Personal characteristics Higher quality of measurement among female respondents Higher quality of measurement among younger respondents Higher quality of measurement among more educated respondents Personality characteristics Higher quality of measurement among extraverts
![Page 36: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/36.jpg)
Intro
36Universitat de Girona
Hypothesis and Research Questions from Studies
36
• We might expect that every tie or alter characteristic would be more reliably and more validly measured when the question is posed by alters than by questions ( Kogovšek , Ferligoj , Coenders , Saris, 2002)
We therefore expect that cognitively more demanding questions would be more prone to measurement errors in the telephone than in the face-to-face mode, therefore:
• It is expected that demanding questions would be more reliably and more validly measured in face-to-face interviews than by telephone
![Page 37: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/37.jpg)
Intro
37
Quality of measurement
Universitat de Girona
37
How to Measure
![Page 38: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/38.jpg)
Intro
38
Quality of measurement
Universitat de Girona
38
Reliability: The extent to which any questionnaire, test or measure
produces the same results on repeated experiments. Random error.
Validity: The extent to which any measure measures what is intended to
measure (Carmines & Zeller, 1979:12). Systematic error, bias.
Convergent validity refers to common trait variance and is inferred
from large and statistically significant correlations between measures of the
same trait using different methods.
Discriminant validity refers to the distinctiveness of the different traits;
it is inferred when correlations among different traits are less than one.
![Page 39: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/39.jpg)
Intro
39
Quality of measurement
Universitat de Girona
39
Test–retest stability of networks comparing data collection techniques of network data (Example: Free recall vs Recognition method) The test is performed twice. Giving a group of participants the same questionnaire on two different occasions, the test-retest reliability is the correlation between separate administrations of the test is high.
![Page 40: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/40.jpg)
Intro
40
Quality of measurement
Universitat de Girona
40
Q
U
A
L
I
T
Y
Multilevel
Multitrait
Multimethod
Data collection
Questionnaire Design
Reliability and validity estimates by country
Meta-analysis design (MCA)
Meta Analysis results
![Page 41: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/41.jpg)
Intro
41
Multitrait Multimethod model
Universitat de Girona
41
The MTMM model has rarely been used for measurement quality assessment in social network analysis. Hlebec (1999), Ferligoj and Hlebec (1999), Kogovšek et al. (2002), egocentered networks (TS model).
For model identification purposes the MTMM approach usually requires at least 3
repeated measurements of the same variable, using three different methods
(Kenny, 1976) � Burden on the respondent and increases the cost of data
collection.
To reduce these problems:
Split ballot MTMM experimental design (Saris, 1999; Saris & Coenders,
2000) combinations of just two repetitions (methods).
![Page 42: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/42.jpg)
Intro
42
Multitrait Multimethod model
Universitat de Girona
42
Reliability based on the classical test theory (Lord & Novick, 1968):
Yij = Sij + eij e Y S
• Yij is the response of variable i measured by method j.
• Sij is called the True Score (Saris & Andrews, 1991)
• eij is the random error, related to lack of reliability.
True Score is the result of Trait and Method:
Sij = mij Mj + tij Ti
![Page 43: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/43.jpg)
Intro
43
Multitrait Multimethod model
Universitat de Girona
43
T1= Trait / variable of interest M = reaction to S= True Score the method Y = observed response e = random error
• The strength of the relationship between T and S is called Validity
Validity = 1- var (M)
![Page 44: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/44.jpg)
Intro
44
Multitrait Multimethod model
Universitat de Girona
44
Confirmatory Factor Analysis (CFA) specification of the MTMM model:
Traits (Coromina, Coenders, Ferligoj, 2006): Yij = mij Mj + tij Ti + eij
Trait 1: Ask for scientific advice
Trait 2: Collaboration with your colleagues concerning research
Trait 3: Ask for information /data/software
Trait 4: Social activities outside the work
![Page 45: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/45.jpg)
Intro
45
Multitrait Multimethod model
Universitat de Girona
45
Assumptions (Andrews, 1984):
Cov(Ti,eij)=0 ∀ij
Cov(Mj,eij)=0 ∀ij
Cov(Ti ,Mj)=0 ∀ij
It makes possible to decompose the variance of Yij
Trait variance tij2Var(Ti)
method variance mij2Var(Mj)
random error variance Var(eij)
![Page 46: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/46.jpg)
Intro
46
Multitrait Multimethod model
Universitat de Girona
46
Y11 Y21 Y31 Y41 Y22 Y12 Y32 Y42
M1 M2
T1 T2 T3 T4
e11
e21 e31 e41 e12 e22 e32
e42
![Page 47: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/47.jpg)
Intro
47
Multitrait Multimethod model
Universitat de Girona
47
Reliability increases not only with true or trait variance, but also with method variance, which also belongs to the stable or repeatable part of the measurements.
)(
)(
ij
ij
YVar
SVar
= )(
)()( 22
ij
iijjij
YVar
TVartMVarm +
Validity, % of variance of the True Score explained by the Trait:
)()(
)(
)(
)(22
22
iijjij
iij
ij
iij
TVartMVarm
TVart
SVar
TVart
+=
QUALITY, strength of the relationship between Y and T
= RELIABILITY * VALIDITY
![Page 48: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/48.jpg)
Intro
48Universitat de Girona
Multilevel MTMM
48
Egocentered network data Two-level MTMM models.
The mean centred individual scores for group “g” and individual “k” can be decomposed into:
Ego (g) Between group component (g)
Alter1 Alter
2 … Alter
n Within group component (k)
YYY gkgkT −=
YYY ggB−=
ggkWgk YYY −=
![Page 49: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/49.jpg)
Intro
49Universitat de Girona
Multilevel MTMM
49
WITHIN BETWEEN
Between level: Units of measurement are egocentered networks as a whole,
average across all egocentric networks � Quality of averages studied.
Within level: Individual ego-alter
![Page 50: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/50.jpg)
Intro
50Universitat de Girona
Multilevel MTMM
50
Decomposition of the sample covariance matrix.
Total population covariance matrix ΣΣΣΣT
ΣΣΣΣT = ΣΣΣΣB + ΣΣΣΣW
Analysis of each component separately:
Yij = mij Mj + tij Ti + eij
Yij = mBij MBj + tBij TBi + eBij + mwij Mwj + twij Twi + ewij
YBij YWij
![Page 51: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/51.jpg)
Intro
51Universitat de Girona
Interpretation
51
We obtain two reliabilities and two validities for each trait-method combination:
Between and within groups
Groups are respondents Different interpretation
Individuals are stimuli evaluated by respondents from standard way
The between-group reliabilities and validities
Quality of the measurement of the egocentered network as a whole (average).
If the ego is the focus of interest and these averages are used as data instead of
the raw responses regarding individual alters � Between-group measurement
quality is the relevant one to look at.
![Page 52: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/52.jpg)
Intro
52Universitat de Girona
Interpretation
52
The within-group reliabilities and validities
Each subject is a separate unit of analysis and thus variance is defined across
stimuli presented to the same subject, not across subjects.
If the relationship is the focus of interest� within group measurement quality.
We know the between and within scores add to a total score:
Yij = mBij MBj + tBij TBi + eBij + mwij Mwj + twij Twi + ewij
YBij YWij
Then…
![Page 53: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/53.jpg)
Intro
53Universitat de Girona
Interpretation
53
It is possible to compute percentages of variance.
Decomposition of variance:
Var(Yij) = mwij2Var(MWj) + mBij
2Var (MBj) +
twij2Var(TWi) + tBij
2Var(TBi)+
Var(ewij) + Var(eBij)
From the decomposition one can:
• compute overall reliabilities and validities:
By aggregating all trait, method, and error components.
• compute overall percentages of within and between variance:
By aggregating all within components and all between components.
![Page 54: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/54.jpg)
Intro
54Universitat de Girona
Interpretation
54
• compute the percentage of between and within trait variance over the
total trait variance. A higher within percentage shows a higher alter variability
in the tie characteristics within a network and a lower variability of average
values of the tie characteristics between networks.
• compute a percentage of pure random error variance (i.e., within
error variance) over the total variance of the observed variables (grand total,
i.e., including all 6 components). The percentage of total variance explained by
any of the other 5 components can be computed in a similar way.
![Page 55: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/55.jpg)
Intro
55
INSOC Project
Universitat de Girona
55
INSOC project
International Network on Social Capital and Performance
![Page 56: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/56.jpg)
Intro
56
INSOC Project
Universitat de Girona
56
• Explain the academic PhD students success based on social capital.
• Develop and refine data collection methods.
• Develop social network analysis methods.
![Page 57: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/57.jpg)
Intro
57
INSOC Project
Universitat de Girona
57
Web Survey design
• Verbal (words and numbers) + rich visual.
• Special navigational features (progress indicators), animations…
• Lower costs
• Faster data collection and data analysis process
• Easy and fast questionnaire modification.
• Response when convenient
• Piping: Assigning questionnaire items based on earlier answers from the
respondent
![Page 58: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/58.jpg)
Intro
58
INSOC Project
Universitat de Girona
58
Coverage error !!!
Population with nearly universal internet access (PhD students)
![Page 59: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/59.jpg)
Intro
59
INSOC Project
Universitat de Girona
59
INSOC Questionnaire Design
Focus groups
Comparable versions (Catalan, Slovenian and Dutch)
Two independent translations
Pre-test
![Page 60: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/60.jpg)
Intro
60Universitat de Girona
Population, sample and data collection
60
PhD students at the Universities of Girona, Ljubljana and Ghent
April 2003: To know PhD students’ promoters
Personal interview to promoters � name generator questions
1. Name all the teaching assistants (or doctoral assistants) whose research is mainly under your supervision.
2. Name all the researchers of whom you are formally the mentor and who work on or participate in a
research project.
3. Name your colleague professors, senior researchers, junior researchers or people working in the private
sector with whom you substantially work together on those research projects in which PhD student X [name
PhD student] is involved.
![Page 61: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/61.jpg)
Intro
61Universitat de Girona
Population, sample and data collection
61
November 2003: e-mail invitations (86 in Girona, 191 in Ljubljana and 233
in Ghent).
Follow-up: after 1, 4 and 8, weeks
PhD responses Method 1
2weeks PhD responses Method 2
Girona 67 78% 61 91% Ljubljana 118 62% 81 69% Ghent 198 85% 55* 60%
Method 2: Different question order, different style of response category
labels and different graphical display and lay-out of the questions.
![Page 62: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/62.jpg)
Intro
62Universitat de Girona
Population, sample and data collection
62
0%10%20%30%40%50%60%70%80%
srv
Res
pons
es
Supervisors
PhD Students
1st reminder(2/12/03)
2nd reminder(23/12/03)
3rd reminder(23/01/04
to 26/01/04)
![Page 63: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/63.jpg)
Intro
63Universitat de Girona
Traits & Methods
63
Frequency Traits
• Trait 1: Ask for scientific advice
• Trait 2: Collaboration with your colleagues concerning research
• Trait 3: Ask for information/data/software
• Trait 4: Social activities outside the work
sample repetition Factor 1 Factor 2 Factor 3 Girona Only one Main questionnaire by questions all labels plain Girona Only one Follow-up by alters end labels plain
Kogovsek & Ferligoj (2004, 2005)
Group N First interview Second interview
1 320 Face-to-face/by alters Telephone/by alters
2 311 Face-to-face/by alters Telephone/by questions
3 402 Telephone/by alters Telephone/by questions
![Page 64: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/64.jpg)
Intro
64Universitat de Girona
Traits & Methods
64
![Page 65: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/65.jpg)
Intro
65Universitat de Girona
Traits & Methods
65
![Page 66: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/66.jpg)
Intro
66Universitat de Girona
Traits & Methods
66
Example of Method 2 in Girona
![Page 67: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/67.jpg)
Intro
67
MTMM Results -- University of Girona
Universitat de Girona
67
Advice Collaboration Info Social act
within trait variance M1 67.5% 65.3% 55.6% 67.0%M2 66.1% 59.3% 47.7% 42.4%within method variance M1 0.0% 0.0% 0.0% 0.0%M2 9.6% 9.2% 12.3% 13.1%within error variance M1 13.3% 17.2% 17.3% 14.5%M2 7.0% 12.3% 16.3% 25.2%between trait variance M1 12.2% 13.1% 18.5% 11.4%M2 12.0% 11.9% 15.9% 7.2%between method variance M1 0.4% 0.4% 0.6% 0.9%M2 5.2% 5.0% 6.7% 7.1%between error variance M1 6.6% 3.9% 8.0% 6.3%M2 0.0% 2.3% 1.2% 4.9%
Goodness of fit: Yuan-Bentler χ2= 103.1 with 33df. TLI: .945
![Page 68: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/68.jpg)
Intro
68
MTMM Results -- University of Girona
Universitat de Girona
68
% of within trait variance over all trait variance. The results show that most of the error free variance corresponds to the within level. This means that egos really discriminate between the different alters.
T1 T2 T3 T4 twij
2Var(TWi)/ [ twij2Var(Twi) + tBij
2Var(TBi)] 84.6% 83.3% 75.0% 85.4%
Within level Between level Overall level T1 T2 T3 T4 T1 T2 T3 T4 T1 T2 T3 T4 Reliability coef. M1 .91 .89 .87 .91 .81 .88 .84 .81 .90 .89 .86 .89 M2 .96 .92 .89 .83 1.00 .94 .98 .86 .96 .92 .91 .84 Validity coef. M1 1.00 1.00 1.00 1.00 .98 .98 .98 .96 1.001.001.00.99 M2 .93 .93 .89 .87 .83 .84 .84 .71 .92 .91 .88 .84 Quality coef. M1 .91 .89 .87 .91 .79 .86 .82 .78 .90 .89 .86 .88 M2 .89 .86 .79 .72 .83 .79 .82 .61 .88 .83 .80 .71
![Page 69: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/69.jpg)
Intro
69
Meta analysis Design
Universitat de Girona
69
Meta-Analysis Design and Results
![Page 70: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/70.jpg)
Intro
70
Meta analysis Design
Universitat de Girona
70
Summarizing the results of studies carried out by the universities of Girona, Ljubljana and Gent.
University sample repetition Factor 1 Factor 2 Factor 3 Girona (Spain) Only one Main questionnaire by questions all labels plain Girona (Spain) Only one Follow-up by alters end labels plain Ljubljana (Slovenia) 1 and 2 Main questionnaire by questions all labels plain Ljubljana (Slovenia) 1 Follow-up by questions end labels Graphical Ljubljana (Slovenia) 2 Follow-up by alters all labels Graphical Ghent (Belgium) Only one Main questionnaire by questions all labels plain Ghent (Belgium) Only one Follow-up by alters end labels plain
Categorical predictors (country, trait and factor 1 to factor 3)
Multiple classification analysis (MCA) was used
Estimation of the contribution of each factor on reliability and validity.
![Page 71: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/71.jpg)
Intro
71
Meta analysis Design
Universitat de Girona
71
Question order: by alters or by questions.
![Page 72: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/72.jpg)
Intro
72
Meta analysis Design
Universitat de Girona
72
Formulation by alters, with end labels and with a plain lay-out
Kogovšek et al. (2002) “by alters” seems to be more reliable for telephone
interviews.
![Page 73: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/73.jpg)
Intro
73
Meta analysis Design
Universitat de Girona
73
Response category labels: for all categories or for the end points of the
response scale.
Revilla & Saris (2011) find more quality in all labeled.
Költringer (1995) reports no effect of the way of labeling on either reliability or
validity. Personal and telephone interviews and mostly related to attitudinal (i.e.,
not network) variables using vague category labels of the type “rather satisfied”,
“completely agree”
Our study: Category labels are not vague quantifiers. Precise actual frequencies of
behavior and thus additional labels may help respondents give precise answers
about the frequency of contact with their social network.
![Page 74: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/74.jpg)
Intro
74
Meta analysis Design
Universitat de Girona
74
Lay-out of the questions and the web page: plain or graphical display.
Dillman et al. (1998b), who suggest that using a plain questionnaire provided
better results (response rate, less time…) than a graphical display version.
Deutskens et al. (2004) found that visual effects actually increase response quality.
![Page 75: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/75.jpg)
Intro
75Universitat de Girona
Meta-analysis results
75
Means of each level of all factors, corrected by the levels of other factors
country trait Girona Ljubl. Ghent trait 1 trait 2 trait 3 trait 4 % of between trait variance .133 .088 .117 .079 .125 .118 .111
% of within trait variance .610 .612 .555 .674 .654 .552 .508
% of between method variance .024 .025 .002 .017 .015 .022 .030
% of within method variance .049 .074 .077 .052 .047 .072 .114
% of between error variance .049 .064 .057 .053 .039 .063 .078
% of within error variance .152 .166 .189 .137 .132 .186 .219
Between reliability .874 .735 .839 .794 .875 .804 .698
Between validity .921 .907 .943 .916 .945 .919 .876
Within reliability .899 .895 .875 .916 .917 .877 .856
Within validity .974 .932 .931 .963 .965 .936 .887
Overall reliability .885 .882 .869 .892 .913 .867 .849
Overall validity .963 .923 .935 .955 .960 .932 .867
% of trait variance at within level .823 .864 .826 .895 .840 .823 .804
![Page 76: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/76.jpg)
Intro
76Universitat de Girona
Meta-analysis results
76
factor 1 factor 2 factor 3
by questions by alters all labels end labels plain lay-out
graph. lay-out
% of between trait variance .125 .082 .103 .116 .105 .117
% of within trait variance .650 .508 .595 .600 .593 .608
% of between method variance .013 .031 .006 .042 .026 .008
% of within method variance .035 .120 .070 .074 .082 .045
% of between error variance .065 .046 .061 .054 .059 .058
% of within error variance .136 .215 .181 .147 .163 .183
Between reliability .785 .809 .777 .825 .789 .806
Between validity .948 .880 .961 .856 .902 .958
Within reliability .912 .857 .884 .903 .894 .883
Within validity .975 .888 .942 .932 .927 .965
Overall reliability .895 .852 .873 .893 .884 .868
Overall validity .967 .893 .947 .914 .922 .962
% of trait variance at within level .835 .855 .850 .830 .844 .841
![Page 77: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/77.jpg)
Intro
77Universitat de Girona
Meta-analysis results
77
Overall reliabilities and validities are acceptable: around or above 0.85
By traits (pattern is consistent at different levels):
Collaboration (T2)
Advice (T1)
Asking for crucial information (T3)
Socializing (T4)
Overall reliability and validity is higher when:
Social network questions in a survey are organized by questions
All labeled categories
Graphical display questionnaire design (not within validities)
![Page 78: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/78.jpg)
Intro
78Universitat de Girona
Meta-analysis results
78
Country: No significant effect on reliability or validity.
% of trait variance at the within level:
Advice � Collaboration � Ask for information � Socializing
Scientific advice tends to be asked rather often to some group
members and not often to other group members by the same ego, but all
egos have a rather similar frequency average, while this occurs to a lesser
extent for socializing.
![Page 79: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/79.jpg)
Intro
79Universitat de Girona
Meta-analysis results from Other Studies
79
Meta-Analysis Results from other Studies
![Page 80: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/80.jpg)
Intro
80Universitat de Girona
Meta-analysis results from Other Studies
80
Kogovšek , Ferligoj , Coenders , Saris (2002) Coromina, Coenders, Kogovsek (2004) By type of networks: +: Frequency of contact is measured with the highest reliability Feeling of importance of the alter Feeling of closeness between ego and the alter - : Frequency of being upset.
Frequency of contact is less reliably measured over the telephone (speed of
telephone communication) compared with ‘the degree of closeness between the
ego and the alter’ or ‘importance of the alter’)
![Page 81: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/81.jpg)
Intro
81Universitat de Girona
Meta-analysis results from Other Studies
81
Answering the question about frequency of contact is probably more
cognitively demanding than answering about the degree of closeness for which the
answer may be more readily available
Cognitively more demanding name interpreter questions would be more reliably
measured by face-to-face than by telephone mode.
Kogovsek & Ferligoj (2005) Reliability Results : Method +reliable: Telephone/by alters (except contact) Personal/by alters -reliable: Telephone/by questions
![Page 82: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/82.jpg)
Intro
82Universitat de Girona
Meta-analysis results from Other Studies
82
When Telephone mode only � by alters (higher quality).
Telephone � speed of the mode � smaller elicited networks (would reduce
respondent burden and therefore lead to data of better quality)
BUT mean network size was actually smaller in the face-to-face (6.84) than in the
telephone mode (7.24).
Method: +reliable: face to face +valid: telephone (anonymity of phone) -reliable: telephone -valid: face to face Cognitively more demanding questions would be measured more reliably by face-to-face mode, and that questions that are potentially more sensitive would be measured more reliably in the telephone by alters condition.
![Page 83: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/83.jpg)
Intro
83Universitat de Girona
Meta-analysis results from Other Studies
83
It is quite possible that context effects are stronger when the data is collected by questions, since the respondent answers the same question for all alters first and not vice versa.
If we consider a question relating to feelings of closeness, it is more likely that the respondent would compare the alters while answering the question. In that case, it would not really be the actual feelings of closeness towards each individual alter that would be measured, but the feelings of closeness relative to previous alters on the list.
Coromina et al. (2004): Not so clear that telephone method is better than face-to
-face method at the between level.
Telephone produces better quality data at the within level, and an analysis of the
SB such as the one done by Kogovšek et al (2002) is inevitably contaminated by
the within level structure according to E(SB)= ΣW + cΣB
![Page 84: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/84.jpg)
Intro
84Universitat de Girona
Meta-analysis results from Other Studies
84
Social support (reliability) Ferligoj & Hlebec (1999)
+: Emotional support (discussion of important matters) +: Informational support (Illness ) Social Companionship (birthday party) -: Instrumental support (material) (importance of the topic to the respondents).
Response scale (the most important predictor of reliability estimates) Hlebec & Ferligoj (2002) +: 5 point category all labels. +: 11 point category scale ending
5 point category scale ending labels Line drawing scale
- : Binary scale
![Page 85: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/85.jpg)
Intro
85Universitat de Girona
Meta-analysis results from Other Studies
85
Response scale +reliable: all labelled -reliable: ending labels Name Generator: Recognition data collection enhances the reporting of both strong and weak ties, especially when the measurement scales employed can also measure the strength of ties.
Ferligoj & Hlebec (1999) ( Recognition / Free recall ) no differences Interview design (2nd most powerful predictor for reliability estimates) +reliable : Two measures (when two measures of the same trait are presented in the same interview) -reliable: One measure (a question is presented alone in a questionnaire)
![Page 86: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/86.jpg)
Intro
86Universitat de Girona
Meta-analysis results from Other Studies
86
Network size (not significant) 1–5 +valid: 1-5 6+ -valid: 6+ (larger proportion of weak ties and
measurement error more pronounced)
With smaller networks (important alters, consistent answers) the data collection technique (by alters/by questions) is more important than the data collection mode (telephone/face-to-face).
In larger networks: (seems that context effects are no longer so prominent). +valid: telephone -valid: face-to-face
![Page 87: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/87.jpg)
Intro
87Universitat de Girona
Meta-analysis results from Other Studies
87
Type of question +reliable: Behavior +valid: behaviour .975 -reliable Emotional, but not very different -valid: emotional .958
Education Up to compl. second. (no effect) College or more
Age +reliable: 40 years or less +valid: 40 years or less -reliable: 41+ -valid: +41 (memory or hearing problems;
more weak ties...)
![Page 88: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/88.jpg)
Intro
88Universitat de Girona
Meta-analysis results from Other Studies
88
Gender +reliable: female +valid: male -reliable: male -validity: female (maybe because women have
larger networks and more weak ties and might increase method effect)
Interaction Age / Gender -valid: older women +valid: other categories
Interaction Method / Gender Female/method Male/method +valid: face-to-face (.957) +valid: telephone (.999) - valid: telephone (.930) -valid: face-to-face (.928) Being interviewed about personal relationships by telephone suits both genders better, but face-to-face interviews suit women better than men.
![Page 89: Measurement Quality in Social Networks · Quality in Social Network Survey Data 8 Quality of measurement, specially reliability, can be affected by the type of social support (Ferligoj](https://reader033.vdocuments.mx/reader033/viewer/2022051815/603fb5e7da5f2a6a466047fd/html5/thumbnails/89.jpg)
Intro
89Universitat de Girona
Project Conclusions
89
Considering egocentered network data as hierarchical, Multilevel recommended.
Qualities at Within, Between and Overall level.
Combination of % from decomposed variance.
Egocentered Network data quality collected by Websurvey:
ordered by Questions
All categories of the scale labeled Overall and Within
Graphical display validity