implementing an advanced click model · implementing an advanced click model practical session 2...
TRANSCRIPT
Implementing an Advanced Click ModelPractical Session 2
Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§
[email protected] [email protected] [email protected]
§University of Amsterdam¶Google Research Europe
AC–IM–MdR Implementing an Advanced Click Model 1
Challenge
1 Download code and slides fromclickmodels.weebly.com/russir-2016-course.html
No further internet access required
2 Work in teams up to 3 people
3 Submit your solution to [email protected] and [email protected]
4 Deadline is Thursday, August 25, 14:00
5 The winners will be announced at RuSSIR party
AC–IM–MdR Implementing an Advanced Click Model 2
Data and tools
Relevance prediction challenge dataset (WSCD 2012)
PyClick – an open-source Python library of click models forweb search
AC–IM–MdR Implementing an Advanced Click Model 3
Setup
Download files fromclickmodels.weebly.com/russir-2016-course.html
Install PyPy
Install enum34 package
Install PyClick:
git clone https://github.com/markovi/PyClick.gitcd PyClick && pypy setup.py install –user
http://clickmodels.weebly.com/russir-2016-setup.html
AC–IM–MdR Implementing an Advanced Click Model 4
Starting point
TCM class copied from PBM
Basic experiment
Load the train and test dataTrain TCMTest TCM
Additional classes explained during the session
AC–IM–MdR Implementing an Advanced Click Model 5
Preliminaries
AC–IM–MdR Implementing an Advanced Click Model 6
Preliminaries
1 Initial run(a) Load data and train the model(b) Get the same output as for PBM, save the output
2 Switch from single query sessions to search tasks(a) Extend SearchSession into TaskCentricSearchSession
(b) Create SearchTask
(c) Extend the data loading procedure, re-load data(d) Output the number of search tasks
3 Train on search tasks(a) Pass search tasks to TCM.train()
(b) Get an error4 Implement new inference
(a) Create TaskCentricInference as a subclass of EMInference(b) Pass a search task into Param.update()
(c) Use the new inference in TCM, re-run training(d) Get the same output as for PBM (compare with the previously
saved output)
AC–IM–MdR Implementing an Advanced Click Model 7
Implementing TCM
AC–IM–MdR Implementing an Advanced Click Model 8
Implementing TCM
1 Adding TCM-specific parameters
(a) Create TCMMatch, TCMNew, TCMFresh as subclasses of ParamEM(b) Add these parameters to TCM, re-run training(c) Still get the same output as in PBM
2 Update the EM formulas for attractiveness and examination
(a) Add matching and freshness to the EM formulas,re-run training
(b) Get the updated output
3 Implement EM formulas for other parameters
(a) Plug the EM updating formulas into Param.update(),re-run training
(b) Output the trained TCM parameters
AC–IM–MdR Implementing an Advanced Click Model 9
Session parameters
Rank Document
1 u12 u23 u34 u45 u5
Session Parameters
{αu1q, ε1, µ, ν, φ}{αu2q, ε2, µ, ν, φ}{αu3q, ε3, µ, ν, φ}{αu4q, ε4, µ, ν, φ}{αu5q, ε5, µ, ν, φ}
AC–IM–MdR Implementing an Advanced Click Model 10
EM update rules for TCM parameters
Attractiveness
α(t+1)uq =
1
|Suq|∑s∈Suq
(c(s)u +
(1− c
(s)u
) (1− εrµφ)αuq
1− εrαuqµφ
)
Examination
ε(t+1)r =
1
|Sr |∑s∈Sr
(c(s)u +
(1− c
(s)u
) (1− αuqµφ) εr1− εrαuqµφ
)Freshness
φ(t+1) =1
|S|∑s∈S
1
|s|∑
u∈s:H(s)u
(c(s)u +
(1− c
(s)u
) (1− εrαuqµ)φ
1− εrαuqµφ
)
AC–IM–MdR Implementing an Advanced Click Model 11
EM update rules for TCM parameters (cont’d)
Query matching the user’s information need
µ(t+1) =1
|S|∑s∈S
P(M(s) = 1 | C
)Submitting a new query after a matching query
ν(t+1) =
∑s∈S I ((s) 6= (l))P
(M(s) = 1 | C
)∑s∈S P
(M(s) = 1 | C
)Use TCMUtils to calculate P
(M(s) = 1 | C
)
AC–IM–MdR Implementing an Advanced Click Model 12
Summary
Implementing an advanced click model is relativelystraightforward if you have EM update formulae
Requires minimal changes in the infrastructure
TCM does not perform as well as PBM
Possible reasons
Simplification (previous occurrence instead of examination)Evaluation is not performed properly (discussed next)Problems with the model
AC–IM–MdR Implementing an Advanced Click Model 13
Next steps: Testing
Extend LogLikelihood to work with search tasks
Derive formulas for TCM click probabilities(full and conditional)
Plug the formulas into TCM.get full click probs()
and TCM.get conditional click probs()
AC–IM–MdR Implementing an Advanced Click Model 14
Next lecture
Basic Click Models
Parameter Estimation Evaluation
Data and ToolsResultsApplications
Advanced Models
Recent Studies
Future Research
AC–IM–MdR Implementing an Advanced Click Model 15
Acknowledgments
All content represents the opinion of the authors which is not necessarily shared orendorsed by their respective employers and/or sponsors.
AC–IM–MdR Implementing an Advanced Click Model 16