evaluation of cross-domain news article recommendations
DESCRIPTION
Presentation given at the UMAP 2013 Doctoral ConsortiumTRANSCRIPT
![Page 1: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/1.jpg)
Competence Center Information Retrieval & Machine Learning
10. April 2023
UMAP‘13 Doctoral Consortium
Evaluation of Cross-Domain News Article Recommendations
Benjamin Kille
![Page 2: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/2.jpg)
210. April 2023
Agenda
► Problem description► Challenges in News Article Recommendation
Sparsity Dynamic item collection Evaluation
► Research Questions► Data outline► Preliminary results► Conclusions► Next steps
![Page 3: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/3.jpg)
310. April 2023
Problem description
► Information overload amount of on-line accessible news articles increases limited user perception limited time capacity
► Solution: Recommender System filtering news articles with respect to relevance/utility
► Special challenges for news recommender systems Sparsity Dynamics
► General challenges for recommender systems Evaluation strategy
![Page 4: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/4.jpg)
410. April 2023
Problem formalization
► Set of domains each referring to a publisher► Each domain comprises a set of users , along with a set of items ► We observe preferences as users interact with items ► Let denote the set of items the target user interacted with, and
denote the set of users who interacted with a specific item► We define the recommendation task as
Rank the set of items according to estimated relevance/utility
![Page 5: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/5.jpg)
510. April 2023
Sparsity
► Cold-Start Problem ( and/or )Cacheda, F. et al., 2011. Comparison of collaborative filtering algorithms. ACM Transactions on the Web, 5(1), pp.1–33.► Providing preferences better recommendations (trade-off)Cremonesi, P., Milano, P. & Turrin, R., 2012. User Effort vs . Accuracy in Rating-based Elicitation. In 6th ACM Conferene on Recommender Systems. pp. 27–34.► News websites usually avoid log-in requirements to attract
larger user segments Incomplete user profiles Inconsistent user profiles
![Page 6: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/6.jpg)
610. April 2023
Dynamics
► News dynamic contentBillsus, D. & Pazzani, M.J., 2007. Adaptive News Access. In P. Brusilovsky, A. Kobsa, & W. Nejdl, eds. The Adaptive Web. Springer, pp. 550–570.► Unlike music or movies rarely re-consumed► For instance: Deutsche Presse Agentur (DPA)
750 messages 220k words 1,5k images
http://www.dpa.de/Zahlen-Fakten.152.0.html
![Page 7: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/7.jpg)
710. April 2023
Evaluation
► Strategy on-line: A/B testing (user-centric) off-line: data set (data-centric)
► Numerous facets utility relevance novelty serendipity …
► Dependending on the model formulation preference prediction (requires numerical preference data) item ranking
![Page 8: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/8.jpg)
810. April 2023
Evaluation (cont‘d)
► on-line
Dispatcher
● recommendation request click
●
●
●
●
●
●
CTR = # clicks# requests
![Page 9: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/9.jpg)
910. April 2023
Evaluation (cont‘d)
► off-line (replay)
Dispatcher
● recommendation request click
● ● ● ● ●●
CTR = # clicks# requests
[𝑖1 ,𝑖2 , 𝑖3 ] [𝑖1 ,𝑖2 , 𝑖3 ]
? ?
Li, L. et al., 2011. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM international conference on Web search and data mining - WSDM ’11. p. 297.
![Page 10: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/10.jpg)
1010. April 2023
Cross-domain setting
D1
D2
U
U
I
ID1
D2
U
U
I
I
D1
D2
U
U
I
I
D1
D2
U
U
I
I
no o
verla
p
Use
r ove
rlap
Item
ove
rlap
full
over
lap
Cremonesi, P., Tripodi, A. & Turrin, R., 2011. Cross-Domain Recommender Systems. In 2011 IEEE 11th International Conference on Data Mining Workshops. IEEE, pp. 496–503.
![Page 11: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/11.jpg)
1110. April 2023
Research Questions
► How can other publishers' user interactions contribute to
decrease sparsity for the target publisher?
► What characteristics must recommender algorithms exhibit to
successfully cope with dynamically changing item collections?
► How to evaluate cross-domain recommender systems with
dynamically changing item collections? How do standard
evaluation metrics compare to the observed clicks?
![Page 12: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/12.jpg)
1210. April 2023
Data outline
► > 1-2M impressions by 12 publishers (general news, local news, finance, information technology, sports, etc.) on a daily basis
► user features such as browser ISP OS device
► news article features such as title text URL Image
► http://www.dai-labor.de/en/irml/epen/
►Real interactions with actual users!
![Page 13: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/13.jpg)
1310. April 2023
Preliminary results
► Sparsity► Histogram of the relative frequency of user interactions
![Page 14: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/14.jpg)
1410. April 2023
Preliminary results (cont‘d)
► Dynamics
![Page 15: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/15.jpg)
1510. April 2023
Preliminary results (cont‘d)
► Popularity
![Page 16: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/16.jpg)
1610. April 2023
Conclusions
► News recommender systems must handle enormous sparsity► Dynamic item collections
Continuously, news items enter the system Continuously, items stop to be relevant/read
► We observe a serious popularity bias► We suppose that considering user preferences on additional
domains will decrease sparsity ► We suppose that identifying general preference pattern will
allow us to deal with item collection dynamics
![Page 17: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/17.jpg)
1710. April 2023
Next steps
► Implementation of existing cross-domain recommender algorithms
► Evaluating recommender algorithms with respect to CTR novelty diversity
► Investigate UI effects► Analyze applicability of context-sensitive recommendations► User/Item clustering to speed-up computation time
![Page 18: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/18.jpg)
1810. April 2023
Thank you for the attention!
Questions???
![Page 19: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/19.jpg)
1910. April 2023
Announcement: NRS 2013
► International News Recommender Systems Workshop and Challenge► In conjunction with ACM RecSys 2013IMPORTANT DATES
July 21, 2013 paper submission deadline July 1, 2013 data set release August 15, 2013 on-line challenge kick-off
HIGHLIGHTS Access to a real recommender system Real-time requirements Big Data Cross-domain Implicit feedback
Website: https://sites.google.com/site/newsrec2013/homeTwitter: @NRSws2013
![Page 20: Evaluation of Cross-Domain News Article Recommendations](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b73e5b4a7959404e8b47eb/html5/thumbnails/20.jpg)
Competence Center Information Retrieval &Machine Learning
20
www.dai-labor.de
FonFax
+49 (0) 30 / 314 – 74+49 (0) 30 / 314 – 74 003
DAI-Labor
Technische Universität BerlinFakultät IV – Elektrontechnik & Informatik
Sekretariat TEL 14Ernst-Reuter-Platz 710587 Berlin, Deutschland
10. April 2023
Benjamin KilleResearcher / PhD student