click models for web search - lecture...
Post on 12-Sep-2018
221 Views
Preview:
TRANSCRIPT
Evaluation Data and tools Results
Click Models for Web SearchLecture 3
Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§
a.chuklin@uva.nl i.markov@uva.nl derijke@uva.nl
§University of Amsterdam¶Google Research Europe
AC–IM–MdR Click Models for Web Search 1
Evaluation Data and tools Results
Course overview
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 2
Evaluation Data and tools Results
This lecture
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 3
Evaluation Data and tools Results
What do click models give us?
General:
Understanding of user behavior
Specific:
Conditional click probabilities
Full click probabilities
Attractiveness and satisfactoriness for query-document pairs
AC–IM–MdR Click Models for Web Search 4
Evaluation Data and tools Results
Lecture outline
1 EvaluationLikelihoodPerplexityRanking evaluation
2 Data and tools
3 Results
AC–IM–MdR Click Models for Web Search 5
Evaluation Data and tools Results
Evaluation summary
Click model’s output Evaluation
Conditional click probabilities Log-likelihoodFull click probabilities PerplexityParameter values Ranking evaluation
AC–IM–MdR Click Models for Web Search 6
Evaluation Data and tools Results
Lecture outline
1 EvaluationLikelihoodPerplexityRanking evaluation
AC–IM–MdR Click Models for Web Search 7
Evaluation Data and tools Results
Likelihood
Likelihood measures how well a click model estimatesconditional click probabilities given observed clicks.
LL(M) =1
|S|∑s∈S
logPM
(C1 = c
(s)1 , . . . ,Cn = c
(s)n
)Cr – binary random variable denoting a click at rank r
c(s)r – observed click at rank r in a search session s
P(Cr = c
(s)r
)– probability of observing c
(s)r in session s
P(C1 = c
(s)1 , . . . ,Cn = c
(s)n
)– probability of observing
sequence c(s)1 , . . . , c
(s)n in session s
AC–IM–MdR Click Models for Web Search 8
Evaluation Data and tools Results
Likelihood
PM
(C1 = c
(s)1 , . . . ,Cn = c
(s)n
)= PM
(C1 = c
(s)1
)· PM
(C2 = c
(s)2 , . . . ,Cn = c
(s)n | C1 = c
(s)1
)= PM
(C1 = c
(s)1
)· PM
(C2 = c
(s)2 | C1 = c
(s)1
)· PM
(C3 = c
(s)3 , . . . ,Cn = c
(s)n | C1 = c
(s)1 ,C2 = c
(s)2
)=
n∏r=1
PM
(Cr = c
(s)r | C<r = c
(s)<r
)
AC–IM–MdR Click Models for Web Search 9
Evaluation Data and tools Results
Likelihood: summary
LL(M) =1
|S|∑s∈S
logPM
(C1 = c
(s)1 , . . . ,Cn = c
(s)n
)LL(M) =
1
|S|∑s∈S
n∑r=1
logPM
(Cr = c
(s)r | C<r = c
(s)<r
)
Likelihood measures how well a click model estimatesconditional click probabilities given observed clicks.
LL(M) ∈ [−∞..0]
AC–IM–MdR Click Models for Web Search 10
Evaluation Data and tools Results
Lecture outline
1 EvaluationLikelihoodPerplexityRanking evaluation
AC–IM–MdR Click Models for Web Search 11
Evaluation Data and tools Results
Perplexity
Perplexity measures how well a click model estimatesfull click probabilities (i.e., when clicks are not observed).
pr (M) = 2− 1
|S|∑
s∈S
(log2 PM(C
(s)r =c
(s)r ))
pr (M) ∈ [1..2]
AC–IM–MdR Click Models for Web Search 12
Evaluation Data and tools Results
Lecture outline
1 EvaluationLikelihoodPerplexityRanking evaluation
AC–IM–MdR Click Models for Web Search 13
Evaluation Data and tools Results
Ranking evaluation
R̂el i Reli
αu1q 4
αu2q 2
αu3q 1
αu4q 4
αu5q 2
DCG =n∑
i=1
2Reli − 1
log2(i + 1)
AC–IM–MdR Click Models for Web Search 14
Evaluation Data and tools Results
Evaluation summary
Click model’s output Evaluation
Conditional click probabilities Log-likelihoodFull click probabilities PerplexityParameter values Ranking evaluation
AC–IM–MdR Click Models for Web Search 15
Evaluation Data and tools Results
Lecture outline
1 Evaluation
2 Data and tools
3 Results
AC–IM–MdR Click Models for Web Search 16
Evaluation Data and tools Results
Datasets
AOL2006: raw queries and clicked documents (no SERPs)
MSN2006: contains only clicked documents (no SERPs)
Workshop on Web Search Click Data (WSCD)
WSCD2012: predict document relevanceWSCD2013: detect search engine switchWSCD2014: search personalization
SogouQ
Tsinghua University: eye fixation
AC–IM–MdR Click Models for Web Search 17
Evaluation Data and tools Results
Dataset statistics
Dataset Queries URLs Users Sessions
AOL 2006 10,154,742 1,632,788 657,426 21,011,340MSN 2006 8,831,280 4,975,897 – 7,470,915SogouQ 2012 8,939,569 15,095,269 9,739,704 25,530,711WSCD 2012 30,717,251 117,093,258 – 146,278,823WSCD 2013 10,139,547 49,029,185 956,536 17,784,583WSCD 2014 21,073,569 70,348,426 5,736,333 65,172,853
AC–IM–MdR Click Models for Web Search 18
Evaluation Data and tools Results
Software
Click model packages
clickmodels project by Aleksandr ChuklinPyClick by Ilya Markov et al.
Infer.NET
General-purpose languages
OctaveMatlab
AC–IM–MdR Click Models for Web Search 19
Evaluation Data and tools Results
Lecture outline
1 Evaluation
2 Data and tools
3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation
AC–IM–MdR Click Models for Web Search 20
Evaluation Data and tools Results
Experimental setup
Data
first 1M query sessions from WSCD 2012 dataset75% for training, 25% for testingrepeat 15 times, each time with next 1M sessions
PyClick
50 iterations for EM
AC–IM–MdR Click Models for Web Search 21
Evaluation Data and tools Results
Studied click models
CTR models: counting clicks
Position-based model (PBM): examination and attractiveness
Cascade model (CM): previous examinations and clicks matter
Dynamic Bayesian network model (DBN): satisfactoriness
User browsing model (UBM): rank of previous click
AC–IM–MdR Click Models for Web Search 22
Evaluation Data and tools Results
Lecture outline
3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation
AC–IM–MdR Click Models for Web Search 23
Evaluation Data and tools Results
Log-likelihood
RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
Log-l
ikelih
ood
Cascade model: LL = −∞Complex models (DBN, UBM) win over simple onesMany examination parameters win over few: UBM > PBMSatisfaction parameters help: DBN > PBM
AC–IM–MdR Click Models for Web Search 24
Evaluation Data and tools Results
Lecture outline
3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation
AC–IM–MdR Click Models for Web Search 25
Evaluation Data and tools Results
Perplexity
RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN1.0
1.1
1.2
1.3
1.4
1.5
Perp
lexit
y
Complex models win over simple ones
Most complex models have similar perplexity
AC–IM–MdR Click Models for Web Search 26
Evaluation Data and tools Results
Perplexity by rank
1 2 3 4 5 6 7 8 9 100.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
GCTRRCTR
DCTRPBM
CMUBM
DCMCCM
DBNSDBN
Picture taken from A. Grotov, A. Chuklin, I. Markov, L. Stout, F. Xumara, and M. de Rijke. A comparative studyof click models for web search. In CLEF. Springer, September 2015.
AC–IM–MdR Click Models for Web Search 27
Evaluation Data and tools Results
Lecture outline
3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation
AC–IM–MdR Click Models for Web Search 28
Evaluation Data and tools Results
Training time
RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN0
500
1000
1500
2000
2500
3000
3500
4000
Tim
e (
sec)
MLE is much faster than EM
PBM and UBM are fast enough compared to DBN
AC–IM–MdR Click Models for Web Search 29
Evaluation Data and tools Results
Lecture outline
3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation
AC–IM–MdR Click Models for Web Search 30
Evaluation Data and tools Results
Experimental setup
Full WSCD 2012 dataset
146,278,823 query sessions30,717,251 unique queries117,093,258 unique URLs41,275 relevance labels (for 4,991 queries)
50% for training, 50% for testing
PyClick
AC–IM–MdR Click Models for Web Search 31
Evaluation Data and tools Results
Log-likelihood and perplexity
Click model Perplexity Log-likelihood
DBN 1.3510 −0.2824DCM 1.3627 −0.3613CCM 1.3692 −0.3560UBM 1.3431 −0.2646
UBM is the best in terms of predicting user click behavior
UBM has the largest number of examination parameters (55)
AC–IM–MdR Click Models for Web Search 32
Evaluation Data and tools Results
Ranking evaluation
NDCG
Click model @1 @3 @5 @10
DBN 0.717 0.725 0.764 0.833DCM 0.736 0.746 0.780 0.844CCM 0.741 0.752 0.785 0.846UBM 0.724 0.737 0.773 0.838
CCM is the best in terms of ranking
Not covered in this corse (but covered in the book)
AC–IM–MdR Click Models for Web Search 33
Evaluation Data and tools Results
Lecture 3 summary
Click model’s output Evaluation Best model
Conditional click probabilities Log-likelihood UBMFull click probabilities Perplexity UBMParameter values Ranking evaluation CCM
Training time MLE-based
AC–IM–MdR Click Models for Web Search 34
Evaluation Data and tools Results
Course overview
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Click Models for Web Search 35
Evaluation Data and tools Results
Up next
Practical Session 1
AC–IM–MdR Click Models for Web Search 36
top related