content analysis: measurement and validity

Content Analysis:Measurement and

ValidityKimberly A. Neuendorf, Ph.D.

Cleveland State UniversityFall 2011

Measurement “Assignment of numerals to objects or

events according to rules” (Stevens, 1951)

Operationalization (Measurement)

Categories that are exhaustive and mutually exclusive

High and appropriate level of measurement Ratio Interval Ordinal Nominal

Evaluating Measures:Comparing Reliability, Accuracy (lack of Bias), and Precision

Targets…also see p. 114 of book. See also:

Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Newbury Park, CA: Sage Publications.

High Reliability

High Accuracy

High Precision

High Reliability

Low Accuracy

High Precision

Low Reliability

Moderate Accuracy

High Precision

Reliability?

High Accuracy

Low Precision

Validity

Internal Validity Conceptual definition =matches= Operationalization The standard of having a “good” measure—”Are we hitting

the target we want?”

External Validity Generalizability

Sample Ecological validity Replicability

Validity

“Face” Validity Criterion Validity

Relate to established standard or behavior Concurrent Predictive

e.g., Oliver & Kalyanaraman (2002) study of sexual and violent content in trailers and films. . . Criterion is the MPAA rating

Validity

Content Validity

Construct = ???

Validity

Content Validity

Validity

Content Validity

Validity

Content Validity

High internal consistency reliability. . .

Validity

Construct Validity Related to other measures (constructs) in ways

consistent with theory Convergent—relationships as expected Discriminant—lack of relationships, as expected

Construct Validity Smith volume (1992) on “thematic content

analysis”—psychographic applications—has examples Trust-mistrust coding system for TAT stories

construct validated against: Other thematic coding systems (e.g., Affiliation; Intimacy) Demographics (e.g., Mothers’ feeding habits during infancy

(more or less principled), moderated by gender & age) Self-reported behaviors (e.g., Reported positive relationships

with others; Divorce, family fights in last year; Drug use) Personality measures (e.g., Impulsivity; Locus of Control)

Validity Assessment & CATA

Short et al. (2010) provides a model for validity and CATA, using tests of a new dictionary for “entrepreneurial orientation” Content Validity

Deductive Inductive

External Validity Reliability Dimensionality (a type of Construct Validity--Discriminant) Predictive Validity (seems to really be Construct Validity—

Convergent???)

Validity Assessment & CATA

A Slight Adaptation of Short et al. (2010), converging with Carmines and Zeller (1979): Face Validity Content Validity

Deductive Inductive

External Validity Construct Validity--Dimensionality (Discriminant Validity) Construct Validity--Predictive (Convergent Validity) Criterion Validity (Concurrent or Predictive) [Reliability—separate, but without it, validity is also

compromised]

Apply to the iCarly Case? (Disrespectful Behaviors) A Slight Adaptation of Short et al. (2010), converging with

Carmines and Zeller (1979): Face Validity Content Validity

Deductive Inductive

External Validity Construct Validity--Dimensionality (Discriminant Validity) Construct Validity--Predictive (Convergent Validity) Criterion Validity (Concurrent or Predictive) [Reliability—separate, but without it, validity is also

compromised]

pause