understanding and measuring user engagement and attention in online news reading
TRANSCRIPT
Understanding and Measuring User Engagement and Attention
in Online News Reading Dmitry Lagun and Mounia Lalmas
1Thanks to Yahoo Faculty Research and Engagement Program for supporting this work.
User Engagement in Online News Reading
2
User engagement:“emotional, cognitive and behavioral connection that exists between a user and a resource” (Attfield et al., 2011)
Stickiness:concerned with users spending time on a news site.
User Attention in Online News Reading
Challenge II:identifying which aspects of the online interaction influence user engagement the most.
Challenge I:attract large shares of online attention by keeping users engaged.
Measuring user engagement with news content
3
Method PROS CONS
Dwell time (click duration)(Agichtein et al., 2006)
scalable; captures engagement at coarse level
cannot distinguish time spent on parts of the article
Eye tracking (Arapakis et al., 2014)
very detailed small scale; very expensive
Mouse cursor movement(Huang et al., 2011)
scalable; more fine grained than dwell time
cursor is often kept still during article reading, when no pointing action is required
coarse but more robust instrument to measure user attention at large scale during news reading
VIEWPORT TRACKING
Our Method: Viewport Tracking
4
viewport
time spent at i-th scroll position
i-th viewport defined by a rectangle (left, top, width, height)
viewport
Research questions
● Where do users spend their time during news article viewing?
● Does media image and video content affect time spent at a vertical position?
● What are typical patterns of news article reading?
● Can we accurately predict user engagement from textual content?
5
● 1,971 Yahoo news articles● 267,210 page views on desktopDATASET
6
Overall Pattern of Viewport Time (proxy for user attention)
Many users spend significantly smaller amount of time at lower scroll positions.
Some users find the article interesting enough to spend significant amount of time at the lower part of the article.
Some articles entice users to deeply engage with their content.
How do users browse through the article?
8
comment
header
top
middle
bottomartic
le b
odystart
top
middle
bottom
comment
leave
Markov States
beginning of a page view
top area occupies most of the viewport
middle area occupies most of the viewport
bottom area occupies most of the viewport
comment area occupies most of the viewport
user leaves the page
V1 V2 ... Vn
Mixture of Markov Chains Model
9
Single markov model:
Mixture of K markov models:
probability of starting at state v1
probability of transition from state Vi to V(i-1)
Markov States:{Start, Top, Middle, Bottom,
Comment, Leave}
weight of k-th mixture component
K=6 is optimal
Modeling of User Engagement from Article Content
16
?news article
%Bounce
%Shallow
%Deep
%Complete
user engagement profile
TUNE: Topics of User Engagement with News
17
TUNEnews article
%Bounce
%Shallow
%Deep
%Complete
user engagement profile
Unlike LDA, in TUNE topic is a combination of word co-occurrence and similarity of user
engagement profile.
Distribution of user engagement level
Experimental Setting
● Task○ Predict User Engagement Level Profile
● Model○ Linear regression
● Features○ Number of words in the article○ Presence of media content (e.g., image and video)○ Distribution of article topics with LDA○ Distribution of article topics with TUNE (our model)
● Evaluation Metric○ Pearson’s correlation between ground truth and predicted value○ Ten fold cross-validation
18
%Bounce
%Shallow
%Deep
%Complete
Results: Baselines
19
Feature Set %Bounce %Shallow %Deep %CompleteNumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0.071 0.571 0.410 0.185
Results: Baselines
20
Feature Set %Bounce %Shallow %Deep %CompleteNumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0.071 0.571 0.410 0.185NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328
NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405
Results: Baselines vs. TUNE
21
Feature Set %Bounce %Shallow %Deep %CompleteNumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0.071 0.571 0.410 0.185NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328
NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405NumWords + M + TUNE (T=5) 0.079 0.648 0.544 0.282
NumWords + M + TUNE (T=10) 0.311 0.713 0.660 0.400NumWords + M + TUNE (T=20) 0.349 0.724 0.682 0.409
NumWords + M + TUNE (T=50) 0.333(+132%)
0.742(+18%)
0.697(+29%)
0.428(+6%)
NumWords + M + LDA + TUNE 0.334 0.730 0.696 0.442Dwell 0.392 0.203 0.128 0.351
Conclusions● Unlike in search, user attention in news reading is not constantly decaying
with vertical position (e.g., can be bi-modal)
● Engagement with a news article can be categorized by depth of examination (Bounce, Shallow, Deep & Complete)
● The proposed engagement metrics go beyond “dwell time” as they capture user attention and engagement at sub-document level
● We can obtain accurate prediction of article engagement profile purely from its textual content
22
Summary: Viewport time attention as proxy of user engagement
23
Effect of position and content on viewport time at vertical position
V1 V2 ... VnArticle examination can be categorized by depth of examination
Four engagement classes: Bounce, Shallow, Deep and Complete
Joint model of article topics and user engagement classes improves prediction accuracy:
● Bounce (+140%)● Shallow (+18%) ● Deep (29%) ● Complete (+9%)
User Attention vs. Engagement Classes
25
MetricBounce
(N=26542)Shallow
(N=63982)Deep
(N=164197)Complete (N=12489)
dwell 6.17 (0.02) 63.75 (0.37) 99.02 (0.22) 228.35 (1.48)header time 2.99 (0.03) 15.39 (0.14) 18.48 (0.08) 17.41 (0.25)body time 5.06 (0.02) 35.13 (0.21) 86.24 (0.20) 85.00 (0.70)
comment time 0.56 (0.01) 17.27 (0.23) 9.72 (0.07) 110.90 (0.89)% header time 0.31 (0.00) 0.23 (0.00) 0.17 (0.00) 0.09 (0.00)% body time 0.62 (0.00) 0.58 (0.00) 0.76 (0.00) 0.40 (0.00)
% comment time 0.07 (0.00) 0.20 (0.00) 0.07 (0.00) 0.51 (0.00)% article read 0.12 (0.00) 0.23 (0.00) 0.83 (0.00) 0.84 (0.00)
# comment clicks 0.01 (0.00) 0.43 (0.01) 0.00 (0.00 3.14 (0.03)
User Engagement Classes and User Attention
26
Dwell time and viewport time on head, body and comment increase from Bounce to Complete.
Viewport time on head steadily decreases from Bounce to Complete: users spend an increasing amount of time reading content deeper in article.
Percentage of article read steadily increases from Bounce to Complete, as expected.
Deep and Complete correspond to the situations when the majority (83%) of the article was read.
Number of comment clicks is highest for Complete and then Shallow: users may engage with comments even if they do not read a large proportion of the article.
comment
headertop
middle
bottomartic
le b
ody
header