Method & Implication for UX Research

University of Wisconsin-Madison Yang Liu, PhD in Communication

Estimating causal effect with survey data:

Using the research on “Internet effect in China” to showcase the application of PS methods

University of Wisconsin-Madison Yang Liu, PhD in Communication

Starting from a case of competitive analysis …The UX team of an Ecommerce company runs a survey through a banner ad on Yahoo!, hoping to compare its website design (D1) with its main competitor’s (D2) to see which performs better. They collect users’ social demographics, and two websites’ “performances” in terms of users’ attitudes & behaviors -- satisfaction of shopping experience, frequency of purchase, and expenditure. To convince management about further UX design & investment, they hope to get an answer for this QUESTION:

Exactly how much difference of user experience is created by D1 vs. D2?

University of Wisconsin-Madison Yang Liu, PhD in Communication

Starting from a case of competitive analysis …The UX team of an online shopping company runs a survey through a banner ad on Yahoo!, hoping to compare its website design (D1) with its main competitor’s (D2) and see which performs better. They collect users’ social demographic characteristics, and two websites’ “performances” in terms of users’ attitudes & behaviors -- satisfaction of shopping experience, frequency of purchase, and expenditure. To convince management about further UX investment, they hope to get an answer for this question:

Exactly how much difference of user experience is created by D1 vs. D2?

D1 vs. D2

Satisfaction Purchase Expenditure? ??

University of Wisconsin-Madison Yang Liu, PhD in Communication

Limitation of survey • Cross-sectional survey data cannot reveal causal effect.

Mere difference of UX between D1 vs. D2 is not its effect.

• Real-world users select into D1 vs. D2 by themselves.

Unlike in experiment, they are not randomly assigned.

University of Wisconsin-Madison Yang Liu, PhD in Communication

Limitation of survey • Cross-sectional survey data cannot reveal causal effect.

Mere difference of UX between D1 vs. D2 is not its effect.

• Real-world users select into D1 vs. D2 by themselves.

Unlike in experiment, they are not randomly assigned.

PS methods is used

for this limitation!

University of Wisconsin-Madison Yang Liu, PhD in Communication

Limitation of survey • Cross-sectional survey data cannot reveal causal effect.

Mere difference of UX between D1 vs. D2 is not its effect.

• Real-world users select into D1 vs. D2 by themselves.

Unlike in experiment, they are not randomly assigned.

PS methods is used

for this limitation!

Developed by statistician Paul R. Rosenbaum in 1983

Convenient analysis through using R package “MatchIt”

Increasingly applied in public health & sociology

Implemented by the Harris Poll in public opinion analysis

University of Wisconsin-Madison Yang Liu, PhD in Communication

Rationale

Of PS methods

Utilize social demographics to predict the probability of users selecting

into D1 vs. D2, creating propensity scores

Create a quasi-experimental condition of “random” assignment (P1=P2)

through matching or weighting their propensity scores

Therefore, estimate the causal effect of D1 vs. D2 on user experience,

which can be generalizable to a broad range of real-world users

Real-world users

D1

D2

UX 1

UX 2

P1

P2

University of Wisconsin-Madison Yang Liu, PhD in Communication

How to use

PS methods

Case: The effect of Internet use on citizen engagement in China (Yang Liu, 2015)

Although this study is not about UX of a specific website, it faces the same analytical

issue: How to use cross-sectional survey data to estimate the effect?

Solution: It used propensity score methods to realize the “random” assignment (P1=P2).

Chinese population

Internet user

Non-user

Engagement level 1

Engagement level 2

P1

P2

University of Wisconsin-Madison Yang Liu, PhD in Communication

Analysis

Step 1

Create PS

Predict the probability of Internet use (propensity score) with social demographics

Sample size N=37,279

Sampling strategy Multi-stage cluster sampling

Sampling frame Adults in all 31 provinces & municipalities of mainland China

Outcome variable Internet use vs. non-use

Predictors Age, gender, income, education, profession, city level, region

Model Logistic regression

Percentage predicted 87.11%

University of Wisconsin-Madison Yang Liu, PhD in Communication

Analysis

Step 2.1

Matching

Matching Internet users and non-users on propensity scores to generate matched data

Analytical tool R package “MatchIt”

Syntax Matched data = MatchIt (outcome variable ~ all predictors, original data)

Original dataN=37,279

Distribution of PS

Matched dataN=9,194

Distribution of PS

Internet users Non-users

University of Wisconsin-Madison Yang Liu, PhD in Communication

Analysis

Step 2.2

Matching

Use matched data to estimate the causal effect of Interne use

Matched data N=9194

Two ways of estimation Simple T-test (Or, use R package “Zelig” to run imputation)

The effects of Internet use on

citizen engagement

Results Highly consistent between the two ways of estimation

The effect of Internet use (vs. non-use)

Participation 0.004 0.005

Offline participation 0.325 *** 0.325 ***

Expression 1.237 *** 1.221 ***

Offline expression 0.664 *** 0.658 ***

T-Test Imputation

* p < .05, ** p < .01, *** p < .001

University of Wisconsin-Madison Yang Liu, PhD in Communication

Analysis

Step 3

Weighting

Use original data, but run regression while weighting the propensity score

Original data N=37,279

Create PS weight Weight = 1/(1-propensity score)

Results very consistent with those of PS matching

Estimation method Regression weighted by the PS weight

The effect of Internet use (vs. non-use)

Participation 0.168 0.005

Offline participation 0.439 *** 0.325 ***

Expression 1.140 *** 1.221 ***

Offline expression 0.659 *** 0.658 ***

Weighting Matching

* p < .05, ** p < .01, *** p < .001

University of Wisconsin-Madison Yang Liu, PhD in Communication

Conclusion • The consistency of results via different PS methods shows robustness of estimates.

• The power of Internet effect found is in line with prior longitudinal studies.

• These estimates are superior to experiment research in their generalizability.

PS methods are very effective for estimating causal effects with survey data.

University of Wisconsin-Madison Yang Liu, PhD in Communication

Suggestion:

UX research

Application 1 Estimation of causal effect of different designs in competitive analysis

Use social demographics to predict probability of using D1 vs. D2

Conduct PS matching or weighting to estimate the effect

Convince management of UX investment with robust & generalizable results

Application 2 Improve the generalizability of all web survey results on users

Use social demographics to predict probability of participating web survey

Conduct PS weighting in the following analysis

Correct the selection bias on the part of web survey respondents

University of Wisconsin-Madison Yang Liu, PhD in Communication

Resources Statistical package:MatchIt

Estimate causality with survey data

Correct selection bias in web survey results

Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software, 42(8), 1-27.

http://gking.harvard.edu/matchit Or, contact yliu224@wisc.edu for all R syntax needed for analysis

DuGoff, E. H., Schuler, M., & Stuart, E. A. (2014). Generalizing observational study results: applying propensity score methods to complex surveys. Health services research, 49(1), 284-303.

https://www.ncbi.nlm.nih.gov/pubmed/23855598

Schonlau, M., Van Soest, A., Kapteyn, A., & Couper, M. (2009). Selection bias in web surveys and the use of propensity scores. Sociological Methods & Research, 37(3), 291-318.

https://www.rand.org/content/dam/rand/pubs/working_papers/2006/RAND_WR279.pdf