Effective Recommendation in Knowledge Portals

The SKYbrary case study

Roy HoeymansVrije Universiteit

Amsterdam, the Netherlandsroyhoeymans@gmail.com

ABSTRACTRecommender systems are used to recommend relevant itemsto a user, and have been applied to several types of applica-tions. In this paper we explore the effectiveness of a non-personalized recommender system in knowledge portals byconducting a case study on SKYbrary, a knowledge portalon aviation safety. We have developed several versions oftwo types of algorithms: a user-navigation based algorithm,which uses data on which articles users visit after viewinga current article, and a content based algorithm based on avector space model with a tf-idf weighting scheme that calcu-lates similarity between articles based on their textual con-tent. In our first experiment, the recommendations producedby these versions are compared to recommendations that aremanually selected by the content editors of SKYbrary. Thebest performing versions of each type of algorithm are thenused to create an additional hybrid algorithm. In our secondexperiment, we use a survey to ask the content editors ofSKYbrary to rate recommendations produced by these algo-rithms to determine which of the developed algorithms is themost effective. The results show that the hybrid algorithmproduces significantly more relevant recommendations thana user-navigation based algorithm, but there is not enoughevidence for the claim that a hybrid algorithm works betterthan a content based algorithm.

KeywordsRecommender system, algorithm, content based, knowledgemanagement, knowledge portals, MediaWiki, Google Ana-lytics, Python, tf-idf

1. INTRODUCTION

1.1 Recommender systemsA lot of websites that contain a significant amount of pages

have some sort of recommender system. A recommendersystem is a piece of software that provides suggestions forrelated items to a user. This way they help users to copewith the large amount of information that is available by rec-ommending pages, content, or products that they might beinterested in [9]. Examples of this are YouTube suggestingwhich videos to watch, Amazon suggesting which productsyou might want to buy, or a news website suggesting articlesthat you might like to read. The object that is recommendedto the user is commonly referred to as an ’item’ [9].

There are several different classes of recommender systemsbased on what type of data they use [4]. One of the most

widely implemented recommendation techniques is collabo-rative filtering, which predicts items that the current usermight like based on what other users with similar preferenceslike [17]. Another type of recommendation technique is con-tent based, which only looks at the current user’s historicactivity and features of items; it recommends items that aresimilar to the items that the user has liked in the past [11].A demographic type of recommender system recommendsitems based on the demographic profile of the user, for ex-ample based on the users language or age [20].

1.2 Knowledge portalsDifferent types of domains and situations require differ-

ent types of recommendation techniques. This paper fo-cuses on the application of a recommender system in thefield of knowledge management. Knowledge management isa broad term that generally consists of the collection, or-ganization and retrieval of information. An important partof this is developing and maintaining knowledge portals [5].A knowledge portal is a web-based single point of access toinformation that is available to the public or within an orga-nization. A well-organized knowledge portal makes it easyfor its users to find and retrieve the information they arelooking for, a quality that can be enhanced by the use ofa recommender system. A knowledge portal can consist ofweb pages with text, but also databases, file systems, and in-teractive components such as e-learning applications or dataanalysis tools [5]. In this paper we focus on recommendersystems in (the part of) knowledge portals that contain webpages with textual content.

1.3 Problem descriptionTo ensure the quality of the information, the content that

is available via knowledge portals is often created and main-tained by domain experts. To add recommendations to apage, expert knowledge is needed, and thus the content cre-ators often manually create and maintain a section with linksto related articles. A tedious task that would ideally besupported or replaced by a recommender system. However,where websites like Amazon or YouTube have demographicinformation about their users and explicit information aboutwhat they like (e.g. products they have bought, ratingsthey gave to movies) to personalize their recommendations,knowledge portals often do not. To design a recommendersystem for a knowledge portal, we have to look at the typesof data sources that are available. One of such data sourcescan be implicit user preferences [6], like how users navigatethrough the website and how many times a page is viewed.

Another data source that can be used is the textual con-tent of each page. Information retrieval techniques can beused to determine how similar each page is to other pages.Recommendations can then be generated based on this sim-ilarity.

Both types of data sources have their advantages and dis-advantages. The problem with the approach based on usernavigation patterns is that it suffers from a ’cold start’-problem: if a page is new or unpopular, there is not enoughinformation about how users navigate that a recommendersystem can use to generate useful recommendations [10]. Analgorithm based on similarity between texts does not havethis problem. However, such an algorithm does not takethe user’s preferences into account at all. In this paper, wealso propose a hybrid algorithm that combines these twoapproaches in an effort to diminish their distinct disadvan-tages. We refer to the algorithm based on user navigationdata as ’user-navigation based’, the algorithm based on sim-ilarity between textual content of articles as ’content based’,and the combination of these two algorithms as ’hybrid’.

2. RELATED WORK

2.1 Literature overviewIn a literature review by Xu (2014) , 403 articles that were

published between 2001 and 2013 were found that are rel-evant to the subject of recommender systems [21]. As wasmentioned in the introduction of this paper, several differenttypes of recommendation techniques exist. Collaborative fil-tering is by far the most popular one, mainly because of itssocial value and its usefulness in the entertainment field,which is the most common application field of recommendersystems [21, 13]. In applications were collaborative filteringis not useful due to the lack of user data, a content basedfiltering algorithm is used. Hybrid algorithms, with the pur-pose of combining the strengths and advantages of differenttypes of algorithms, are also common [21, 22].

However, these techniques provide personalized recom-mendations, because data about individual user preferencesand past behavior is available. The focus of this research isto find an effective way of generating recommendations forknowledge portals, which do not have access to this type ofdata. The recommendation techniques that we apply in thispaper are thus non-personalized.

2.2 Non-personalized recommender systemsNon-personalized recommender systems based on user pref-

erences can be divided into two main approaches. One ofthem is the aggregated opinion approach, which bases rec-ommendations on the average ratings of items given by users[2]. A prerequisite for this approach is the availability of ex-plicit ratings that users give to items, for example a scorebetween 1 and 5. Another type of non-personalized rec-ommender systems is a product association recommender.An example of this is Amazon’s ”Customers who boughtthis also bought”. The recommendations are not specificallybased on the user but on what the user is currently doing,which makes it a momentarily personalized technique [16].The versions of the user-navigation based algorithm that weuse in this paper use techniques from product associationrecommender systems.

In content based recommender systems, the content ofitems is used to predict how relevant it is to recommend

them to the user based on his or her profile. The con-tent data about an item can be structured or unstructured.Structured data consists of attributes or features. For exam-ple, a restaurant can have a ’cuisine’-attribute with possiblevalues like ’Italian’, ’French’, or ’Asian’ [15]. If these at-tributes are not available, which is often the case for webpages or documents, a content based recommender systemusually relies on unstructured text. In this case techniquesfrom the field of Information Retrieval can be utilized. Anexample of this is a Vector Space Model (VSM) with tf-idfweighting [9]. In a VSM, text documents and user profilesare spatially represented. The cosine similarity is then com-puted to predict a user’s interest in a document. We alsouse this approach in our content based algorithm but sinceknowledge portals often lack user profiles we only computethe similarity between text documents and use that to gen-erate recommendations.

While hybrid algorithms in recommender systems are com-mon, the way we have combined a user-navigation basedalgorithm and a content based algorithm into a hybrid algo-rithm (see section 6.2) is not described in the literature asof yet. The effectiveness of non-personalized recommenda-tion has been researched in the fields of e-commerce [14] andlibrary catalogs [19]. In both cases, non-personalized recom-mendations proved to be useful, be it by increasing sales orby improving search efficiency and perceived usability. Inthis paper the effectiveness of such a recommender systemis explored in knowledge portals.

2.3 Evaluation methodsThere are several evaluation methods available that can

be used to determine which recommendation algorithm per-forms best or is most suited for the application. The mostcommon evaluation methods are offline evaluations, onlineevaluations, and user studies [1]. In offline evaluations therecommendations produced by the algorithms are comparedto pre-compiled recommendations in an effort to analyzehow well they predict these pre-compiled recommendations.In some cases an offline evaluation is used to pre-select aset of algorithms for a more extensive evaluation [1]. Inonline evaluations, the behavior of users is observed wheninteracting with a recommender system that is already inproduction, for example by counting how often users clickon recommended links [9]. User studies are conducted by ob-serving test subjects while they perform tasks that requireinteraction with the recommender system. The results ofa user study can consists of quantitative measurements likethe percentage of tasks completed, the time it takes to com-plete a task, or qualitative answers to questions about thetasks [18].

In this paper we use an offline evaluation (section 5.4)to select two algorithms, one user navigation based and onecontent based algorithm, to combine into a hybrid algorithm.This hybrid algorithm is then compared to the separate user-navigation and content based algorithms by conducting asurvey to domain experts (section 6.3), in which they assigna relevancy score to the recommendations produced by thesealgorithms. This way we acquire empirical evidence that wecan use to determine which algorithm is the most effectivein recommending relevant articles in knowledge portals.

3. APPROACH

3.1 Research questionsTwo types of data that can be used for a recommender

system are available in a knowledge portal: user navigationpatterns, and the content of the articles. Respectively thesetypes of data can be used to build a user-navigation basedalgorithm and a content based algorithm. The aim of thispaper is to find out which of these two algorithms is more ef-fective in a knowledge portal, and if a hybrid algorithm thatcombines these two algorithms to compensate for their sep-arate disadvantages is more effective than these algorithmsseparately. This led us to define the main research questionof this paper as follows:

• Which type of recommendation algorithm is the mosteffective for a knowledge portal: a user-navigation basedalgorithm, a content based algorithm or a hybrid algo-rithm that combines these two?

In order to answer the main research question, the followingsub-questions need to be answered:

• Which type of user-navigation based algorithm worksbest for a recommender system in a knowledge portal?

• Which type of content based algorithm works best fora recommender system in a knowledge portal?

• Is a recommender system based on the combinationof a content based and a user navigation based algo-rithm more or less effective than a recommender sys-tem based on those separate algorithms?

3.2 ScopeTo answer the research question, we conduct a case study

on a specific knowledge portal called SKYbrary1, a Wiki sitethat contains articles about aviation and air traffic man-agement safety. SKYbrary is a good representation of aknowledge portal because it focuses on one subject, thuscontaining a small amount of articles compared to for ex-ample Wikipedia, and its content is created and maintainedby domain experts. To ensure that the algorithms that wehave developed in this paper do not only work for SKYbrarybut can be used in other knowledge portals as well, we baseour algorithm versions on existing techniques that are foundin the literature, with little to no tweaking for SKYbrary.

SKYbrary is built with MediaWiki and uses Google Ana-lytics to gather information about user navigation behavior.Conceptually, the algorithms that we have developed are de-signed to work for all knowledge portals, but the implemen-tation of our recommender system only works for knowledgeportals that are built with MediaWiki and use Google Ana-lytics. If a knowledge portal uses another content manage-ment system, the data that is needed for the recommendersystem needs to be acquired in a different way.

3.3 ImplementationMediaWiki is an open source software tool that was orig-

inally developed for Wikipedia and is also used for all theother projects of the Wikimedia Foundation and numerousother Wiki sites. MediaWiki uses a MySQL database tostore all the data that is relevant for a Wiki, such as data

1www.skybrary.aero

about pages, categories, links, and revisions. The raw text ofeach article, which is needed for a content based algorithm,can be retrieved from this database. Every knowledge portalthat is built with MediaWiki has the same data structure,which makes it easy to implement our recommender systemin other knowledge portals that are built with it.

Data about user navigation behavior for the user-navigationbased algorithm can be retrieved via the Google AnalyticsAPI. Another method of obtaining this data is by performinglog analysis on Apache server logs. However, we strongly fa-vor Google Analytics for this task as it is simpler and moreaccurate; Apache logs only show requests from certain IPaddresses to the server. If a user has caching enabled, mostof these requests are not even contained in the log file [7].

The source code2 for the recommendation algorithms wecreated is available online and can be used on knowledgeportals that are built with MediaWiki and use Google Ana-lytics with some modification.

3.4 Experiment setupThere are multiple ways to approach the development of

a user-navigation based or a content based algorithm, whichis why we have developed multiple versions of these algo-rithms. For pragmatic reasons, the quality of the producedrecommendations can not be rated by domain experts forevery version. For this reason, we conduct a first experi-ment in which we compare the produced recommendationsto the recommendations that are manually selected by thecontent management team of SKYbrary. This is not theactual evaluation that we want to do but just a way of de-termining which versions to select for the hybrid algorithmand the second experiment.

In the second experiment we survey the domain experts todetermine if the hybrid algorithm is better at recommendingrelevant articles than a separate user-navigation and contentbased algorithm. The survey shows an article and recom-mendations for that article, and asks the respondent to giveeach recommended article a score based on how relevant theythink it is for a user viewing the current article. A more de-tailed explanation of the experiments is given in section 5.4and 6.3.

4. SKYBRARYWe use SKYbrary for our case study on recommender sys-

tems in knowledge portals. SKYbrary is an internet plat-form containing articles about aviation and air traffic man-agement safety that was launched by EUROCONTROL3,the European Organization for the Safety of Air Navigation,in 2008. The objective of SKYbrary is to be a single pointof access to knowledge on these topics, and it is set up as aWiki site so that it is publicly available online. The mainaudience of SKYbrary consists of people that are profession-ally involved with aviation or air traffic managements, likepilots and air traffic controllers.

The articles on SKYbrary can only be added or edited bya small team of aviation safety experts4 to ensure a consis-tently high quality of the information. In addition to writingnew articles, the experts also regularly evaluate the already

2https://github.com/Royhoey/recommender-system-knowledge-portals3www.eurocontrol.int/4www.skybrary.aero/index.php/Skybrary Content Management

Article Y X → YTime of Useful Consciousness 449Explosive Depressurisation 282Emergency Depressurisation 230Aircraft Pressurisation Systems 175Cabin Altitude 60

Table 1: Top five recommendations for article X’Hypoxia’

existing articles to make sure that their content is up-to-date. This also means that the ’Related Links’-section ofeach article, which contains recommendations for other re-lated articles, is reviewed and edited if needed. The rec-ommendations that SKYbrary articles have at the bottomof each page are thus manually selected by domain experts.A recommender system that can automatically do this orsupport this process would be more ideal in this case. Themanually selected recommendations are used to determinewhich algorithm version works best in our first experiment,which is described in section 5.4.

5. USER-NAVIGATION ANDCONTENT BASED ALGORITHMS

In this chapter, the several versions of the user-navigationand content based recommendation algorithms that we havedeveloped are described in detail, and the first experimentis performed to determine which versions are used for thehybrid algorithm and the second experiment.

The recommendations are produced when selecting or vis-iting one SKYbrary article, which is referred to as article X.The recommended articles are referred to as articles Y. Eachpair of articles X and Y has a certain score attached to it,which is used to rank the recommended articles. How thisscore is calculated depends on which algorithm is used.

5.1 User-navigation basedTo create the user navigation based algorithm we have

used the Google Analytics API to retrieve data about usernavigation behavior on SKYbrary. Google Analytics onlystores aggregated data and no log files, so it is not possibleto retrieve information about one particular session. Whatcan be retrieved is the total amount of browsing sessions inwhich a certain page is visited, and how many users visiteda certain page after that. This metric is the basis for thethree versions of this algorithm that we developed.

Version 1This algorithm computes for each article pair XY in howmany sessions article X was followed by article Y. Thisamount can be retrieved via the Google Analytics API for acertain time period. In our experiment we have used a oneyear period, specifically from 01-04-2014 to 01-04-2015. Anexample of the results of this algorithm is shown in Table 1.For example, it shows that 449 users who have visited Hy-poxia, visited ’Time of Useful Consciousness’ later in theirbrowsing session. This algorithm produces accurate recom-mendations, mostly because the selected article X containslinks to the higher ranked recommendations. The first fourrecommendations in Table 1 are all in the ’Related Links’section of Hypoxia, and the ’Cabin Altitude’ article can bereached via a link on the page. While this algorithm is good

Air Temperature Dew Point Lapse RateAT 1 0.46 0.39DP 0.46 1 0.41LR 0.39 0.41 1

Table 2: Example of a small part of the similaritymatrix (AT = Air Temperature, DP = Dew Point,LR = Lapse Rate)

at predicting relatedness because of this, it relies heavily onwhich pages are linked in the article that it produces recom-mendations for. Consequently these recommendations areaccurate, but also quite obvious and therefore not immedi-ately useful. However, we have used this algorithm as a basefor the following more refined versions.

Version 2The second version of the user navigation based algorithm -in addition to the XY article pair metric of version 1 - alsoretrieves the number of sessions in which article X is notviewed but article Y is. This number is often used in rec-ommender system algorithms to mathematically correct forthe popularity of Y by applying the following formula [16]:

X→YX

!X→Y!X

By doing this the so-called ’banana problem’ is avoided [3].Most of the customers in a grocery store will have a bananain their basket, because it is the most frequently boughtitem. Unless a formula is applied takes the popularity ofthe banana into account, a recommender system will almostalways recommend a banana regardless of the other itemsthat the user has in his basket, simply because it is the mostpopular. This algorithm looks at whether viewing article Xmakes viewing article Y more likely than not viewing articleX, thus avoiding this problem.

Version 3Another solution to the ’banana problem’ can be obtainedby applying the following formula [16]:

P (X → Y )

P (X) ∗ P (Y )

This produces more or less the same results as version 2 ofthis algorithm. Which of these two algorithms are used isdetermined in our first experiment (section 5.4).

5.2 Content basedFor the content based algorithms we extracted the text of

each SKYbrary article from the MediaWiki database. Thewhole body of texts were used as input for the two algorithmversions that we describe below. The output of the contentbased algorithms is a similarity matrix. An example of asmall part of this matrix is shown in Table 2. To generaterecommendations for an article X we select the row corre-sponding to that article in the matrix and rank the articlesin the columns descendingly on similarity score. There aretwo versions of this algorithm, each with a different way ofcreating this similarity matrix.

Version 1This version of the content based algorithm uses the term

frequency in the articles to calculate their similarity. Anarray of all the SKYbrary article texts - the corpus - is con-verted into an index vocabulary that contains every termpresent in the whole corpus, along with its frequency. Theneach article in the corpus is converted to a vector of term fre-quencies - the measure of how many times the terms presentin the index vocabulary are present in the article - and re-turned as a term-document matrix. The similarity betweenthese term-frequency vectors is then calculated by comput-ing the cosine similarity, which is a number between 0 and1. An example of this similarity score is shown in Table 2.

Version 2The second version of this algorithm is a more sophisticatedversion of the previous one. It converts the whole corpus toa term-document matrix in the same way as version 1, butalso applies tf-idf (term-frequency times inverse document-frequency) normalization. Tf-idf is a term weighting schemethat is commonly used in the fields of text mining and infor-mation retrieval. It gives more importance to terms that arerare but more informative about the document than termsthat are frequently found among the whole corpus [15]. Forexample, a word like ’pilot’ gets a lower weight, becausethe corpus consists of aviation safety articles where ’pilot’ isnot a discriminative word, but a word like ’hypoxia’ gets ahigher weight because it is only mentioned in a few articles.If one article contains the word ’hypoxia’, it is more likelyto be related to an article that also contains this term, as’hypoxia’ is a term that does not occur in most articles.

5.3 ImplementationThe algorithm versions that we have described were de-

veloped in Python. For the user-navigation based algo-rithm versions, the google-analytics-python-client5 packagewas used to retrieve user navigation data via the Google An-alytics API. For the content based algorithm versions, scikit-learn6, a machine learning library in Python, was used.The first version of the content based algorithm uses theCountVectorizer of the scikit-learn library. This module al-lows us to convert an array of all the SKYbrary article textsinto a spatial representation of term frequencies. The sec-ond version of this algorithm does the same, but uses theTfIdfVectorizer instead, which applies the tf-idf weightingscheme. Both versions use some standard pre-processing ofthe texts. When converting text to a sequence of terms,scikit-learn uses the WordNGramAnalyzer, which filters outstop words, extracts tokens (e.g. ”pilots” and ”pilot’s” be-comes the token ”pilot”), removes accents and converts termsto lowercase. Additional Python packages that were usedare MySQLdb7 (to store recommendations in a database),numpy8 and scipy9.

5.4 Experiment 1 - offline evaluationIn this experiment, the recommendations made by each

algorithm version for each article X are compared to the ar-ticles in the ’Related Links’-section of that article, which aremanually selected by the SKYbrary content editors. This se-

5https://developers.google.com/api-client-library/python/apis/analytics/v36http://scikit-learn.org/stable/7https://pypi.python.org/pypi/MySQL-python8http://www.numpy.org/9http://www.scipy.org/

lection serves as a gold standard in this evaluation method;we compare the performance of the recommendation algo-rithms to the recommendations made by domain experts.

For each article the top X recommendations were selectedand the percentage of the recommended articles that are inthe ’Related Articles’-section was calculated. For example,if an article has five related articles and the recommendedarticles generated by the algorithm contains two of those,the score for that article is 0.4. The average score for allarticles for each algorithm version is referred to as ’recall’.

The recall is calculated for the top X recommendationsfor X is 2, 5, 10, and 30. Each number of X is deliberatelychosen: in our second experiment we use the top two rec-ommendations of each type of algorithm, most articles havea maximum of five recommendations in the ’Related Links’-section, we assume that a maximum of 10 articles can besomehow related, and 30 is the amount of recommendationsthat we generated for each algorithm version, on the as-sumption that a recommended article that is not in the top30 is unlikely to be relevant.

To calculate the amount of recommended articles thatmatch the articles in the ’Related Links’-section, we neededto know which articles are in the ’Related Links’-section.This information can not be found directly in the Medi-aWiki database. The text of each article was crawled untilthe string ”== Related Articles ==” was found. In the Me-diaWiki language, enclosing a string with two = symbolsmakes it appear as a header when viewed in a web browser.Then each line was analyzed until the next section startingwith two = symbols started. Each line contains a link (en-closed like ”[[ link text ]]” in the MediaWiki language), whichmatches the page title of the linked SKYbrary article. Thepage title was then matched to the page table in the Medi-aWiki database. However, some articles are redirects. Forexample the link ’Crew Incapacitation’ was not recognizedas a related article, because it is a redirect to the article ’Pi-lot Incapacitation’. Because of this, the recall score refers tothe amount of recommended articles generated by the algo-rithms that match the articles in the ’Related Links’-sectionthat we retrieved from the text.

Not every type of SKYbrary article was included in thisexperiment. The only articles we included were articles thatwere in at least one of the three main categories: ’Opera-tional Issues’, ’Human Performance’, and ’Glossary’. Arti-cles that were excluded are articles that describe one typeof aircraft, an airport, a plane crash, or safety regulationdocuments. The total amount of articles that are includedin at least one of the main categories is 882. Of those ar-ticles, we found at least one related article in 611 of them,so ultimately this experiment was done with 611 SKYbraryarticles.

5.5 ResultsThe recall for each algorithm version is shown in Table

3. Of all the user-navigation based algorithms, version 1performs best. Version 1 counts the amount of sessions inwhich article X was followed by article Y for each articlepair XY. This amount is naturally much higher for articlesY that are included in the ’Related Links-section of articleX, so the fact that this version scores higher does not meanmuch in this case. It does however show that the recall met-ric is properly calculated, because we expected this versionto predict the highest amount of related articles. Version 2

X=2 X=5 X=10 X=30un-v1 0.712 0.758 0.861 0.940un-v2 0.398 0.499 0.677 0.917un-v3 0.397 0.503 0.669 0.917cb-v1 0.545 0.590 0.720 0.838cb-v2 0.605 0.721 0.880 1

Table 3: Recall of ’Related Articles’ in top X rec-ommendations

and 3 of the user-navigation based algorithms are two dif-ferent approaches to solving the banana trap problem thatwe described earlier in section 5.2, with version 2 perform-ing slightly better, which is why it is selected for the hybridalgorithm and the second experiment.

Concerning the content based algorithms, the results inTable 3 clearly show that version 2 outperforms version 1for all sample sizes. Thus version 2 is selected for the hybridalgorithm and the second experiment as well.

6. HYBRID ALGORITHM

6.1 PurposeThe purpose of combining algorithms into a hybrid algo-

rithm is to make up for the disadvantages that they have sep-arately. The user-navigation based algorithm suffers fromthe ’cold start’-problem, which means that not enough datais present to generate recommendations for a new article be-cause it has not been viewed yet. This can be resolved bycombining it with a content based algorithm, which looksat the similarity between content and does not need dataabout user navigation behavior. However, a content basedalgorithm has several limitations as well. The content basedalgorithm in this paper calculates similarity between arti-cles based on term-frequency. A disadvantage of this is thatit is not possible to detect similarity between synonyms ormetaphorical language [8]. However, since the SKYbraryarticles contain a lot of jargon and technical terms, we thinkthe impact of this is negligible in this case. Another draw-back of content based algorithms that is often mentionedis over-specialization [9]. Because content based algorithmsare so accurate they have trouble recommending somethingunexpected or novel to the user. This is commonly referredto as the serendipity problem [12]. While this can be a prob-lem in fields like entertainment or e-commerce, where a rec-ommender system is used to help the user find new items, aknowledge portal typically favors accurate recommendationsover unexpected ones.

6.2 AlgorithmThe intuition behind the hybrid algorithm is as follows:

generate recommended articles using the content based al-gorithm and adjust these recommendations using its relativepopularity. For example, a recommended article can have ahigh similarity score, but if users hardly visit it, its rankingshould go down.

First we compute a set of recommended articles by usingthe content based algorithm. An example of a set of rec-ommended articles is shown in Table 4. The score is thesimilarity score between article X (in this case ’Hypoxia’)and article Y. For each recommended article, we look atthe score that is assigned by the user-navigation based algo-

Article Y ScoreEmergency Depressuriation 0.48Pressurisation Problems 0.38Aircraft Pressurisation Problems 0.34Cabin Altitude 0.34Explosive Depressurisation 0.26...Rapid Depressurisation 0.24Time of Useful Consciousness 0.19

Table 4: Content based recommendations for thearticle ’Hypoxia’

Article Y ScoreTime of Useful Consciousness 1Explosive Depressurisation 0.68Emergency Depressurisation 0.46Rapid Depressurisation 0.44Aircraft Pressurisation Systems 0.38...Cabin Altitude 0.23Pressurisation Problems 0.13

Table 5: User navigation based recommendationsfor the article ’Hypoxia’

Article Y ScoreTime of Useful Consciousness 0.6Emergency Depressurisation 0.47Explosive Depressurisation 0.47Aircraft Pressurisation Systems 0.36Rapid Depressurisation 0.34Cabin Altitude 0.27Pressurisation Problems 0.26

Table 6: Hybrid recommendations for the article’Hypoxia’

X=2 X=5 X=10 X=30un-v2 0.398 0.499 0.677 0.917cb-v2 0.605 0.721 0.880 1hybrid 0.710 0.791 0.917 1

Table 7: Comparison of hybrid algorithm touser-navigation based v2 and content based v2on the ’Recall of ’Related Articles’ in top Xrecommendations’-metric

rithm. This score is the outcome of the formula we used inversion 2 (see section 5.1), divided by the highest score inthe recommendation set of that article. An example of thisfor the article ’Hypoxia’ is shown in Table 5.

Table 4 and 5 show that the recommended article ’Time ofUseful Consciousness’ has a relatively low similarity score,but is by far the most popular article that is visited by usersafter reading ’Hypoxia’. We see that ’Pressurisation Prob-lems’ has a high similarity score, but it is hardly visited byusers. The hybrid algorithm simply takes the averages ofthese scores. The result set of recommendations is shownin Table 6. We can see here that ’Time of Useful Con-sciousness’ and ’Pressurisation Problems’ are now adjustedby their popularity. A comparison between the performanceof the hybrid algorithm and the user-navigation and contentbased algorithms based on the ’Recall of ’Related Articles’in top X recommendations’-metric that we applied in ourfirst experiment is displayed in Table 7. This shows thatthe hybrid algorithm can predict related articles with moreaccuracy than its underlying algorithms separately.

6.3 Experiment 2 - surveyTo determine if the hybrid algorithm is more effective than

the user-navigation and content based algorithm, we havecreated a survey that was sent to the domain experts ofthis knowledge portal: the SKYbrary content managementteam. The survey consists of 15 questions. Each questionshows a ’current’ SKYbrary article along with some recom-mended articles. The respondent is then asked the followingquestion: ”Imagine that a visitor of SKYbrary is interestedin the article displayed above, and the recommender sys-tem produces the recommendations for the articles below.Please indicate for each recommended article how relevantyou think it is for the visitor.” The respondent can rate eachrecommendation on a scale from 1 to 5. The meanings ofthe score are: 1 = completely irrelevant, 2 = irrelevant, 3 =neutral, 4 = relevant, 5 = very relevant. A screenshot of aquestion from the survey is shown in Figure 1.

The recommendations are produced by the different al-gorithms that we want to compare. Which algorithms pro-duces which recommendation is not shown to the respondentas not to influence their ratings. Each question consists ofseven recommended articles with the following underlyingalgorithms:

• 1 randomly selected article (baseline)

• 2 articles recommended by the user-navigation-basedalgorithm

• 2 articles recommended by the content based algorithm

• 2 articles recommended by the hybrid algorithm

Score 1 Score 2 Score 3 Score 4 Score 5BL 87.8% 4.4% 2.2% 4.4% 1.1%UN 11.7% 8.3% 28.9% 28.3% 22.8%CB 12.2% 6.7% 19.4% 32.3% 29.4%HY 9.4% 8.9% 17.2% 32.1% 32.3%

Table 8: Distribution of survey results per algorithm(UN = user-navigation based, C = content based, H= hybrid, BL = baseline)

Algorithm Mode MeanBaseline 1 1.27User-navigation based 4 3.42Content based 4 3.65Hybrid 5 3.69

Table 9: Descriptive statistics of the results of thesurvey

In practice, most questions have less than seven recommen-dations because some algorithms recommend the same arti-cles. If such an overlap takes place, the score assigned by therespondent is assigned to all the algorithms that recommendthat article.

7. RESULTS

7.1 Survey resultsThe survey was filled out by 6 domain experts of the SKY-

brary content management team. With each of them an-swering 15 questions that contain 2 recommendations to berated per algorithm, a total of 150 scores for each algorithmhave been gathered. The distribution of scores given for eachalgorithm type is displayed in Table 8.

Descriptive statistics for each algorithm are displayed inTable 9. Interestingly, the statistics show the same trend asthe results of the offline evaluation in our first experiment;the hybrid algorithm performs slightly better than the con-tent based algorithm, which in turn performs better thanthe user-navigation based algorithm. The algorithm scoredistribution shows that the baseline has a mode of 1 (veryirrelevant) and that the user-navigation and content basedalgorithm have a mode of 4, which stands for ’relevant’. Thehybrid algorithm has a mode of 5 as well as the highest meanscore.

While the results of the survey as displayed in Table 8and Table 9 show that the hybrid algorithm is better atgenerating relevant recommendations, we want to know ifthis result is not obtained by chance. To test if the differencein results is significant, we used a Mann Whitney-U testas it is well suited for two independent samples of ordinaldata with a non-normal distributed underlying population.For each pair of algorithms we define the null hypothesisas ’the performance of algorithm A is not different from theperformance of algorithm B’. We reject this hypothesis if thep-value is below 0.05. The results of the tests are displayedin Table 10.

The outcome of the statistical tests shows that every al-gorithm performs significantly better than the baseline. Italso shows that the hybrid algorithm is not significantly bet-ter at producing relevant recommendations than the contentbased algorithm. The difference between the content based

Figure 1: Screenshot of an example question of the survey

Algorithm pair P-value SignificantHY - CB 0.516 NoHY - UN 0.02 YesHY - BL 0.00 YesCB - UN 0.09 NoCB - BL 0.00 YesUN - BL 0.00 Yes

Table 10: Results of Mann Whitney-U tests for al-gorithm pairs (UN = user-navigation based,C = content based, H = hybrid, BL = baseline)

and user-navigation based algorithm is not significant either.However, the hybrid algorithm is significantly better at pro-ducing relevant recommendations than the user-navigationbased algorithm.

7.2 DiscussionThe results show that all the algorithms perform much

better than the baseline, which recommends randomly se-lected articles. Because knowledge portals are about oneparticular subject, it is not improbable that a randomly se-lected article is still somehow relevant to the current arti-cle. We included this baseline to see how the domain ex-perts rated randomly selected articles compared to articlesselected by our recommendation algorithms. The resultsshow that a random article is very likely to be completely

irrelevant and that the recommendation algorithms performsignificantly better by a large margin.

Another observation that can be made when looking atthe score distribution per algorithm in Table 8 is that theuser-navigation based algorithm has a large percentage ofrecommendations that were rated with a score of 3, whichcorresponds to neutral. An explanation for this could bethat users navigate to articles that the domain experts donot find relevant or not relevant enough. Therefore it wouldbe interesting to conduct the survey with users of SKYbrary.We did not do this in our research for practical reasons; be-cause visitors of SKYbrary are anonymous, it is hard to trackthem down. Also, we wanted to make sure that the recom-mendations were rated by experts with a similar amount ofexpertise on the subject of the knowledge portal, to ensurethat the knowledge of the respondent on the subject doesnot influence the ratings they give.

Due to time constraints we did not have the opportunityto discuss the results of the survey with the respondents;an explanation of why they rated some recommendations asirrelevant might allow for more insight and can be a startingpoint for improvement of the algorithms in further research.

8. RECOMMENDATION ENGINE DEMOAPPLICATION

There are two ways to implement a recommender systemin a knowledge portal: a system where the recommenda-

Figure 2: Screenshot of the recommendation engine demo application

tions are automatically generated in each article, or by usinga tool to show recommendations to the content managers,who can then manually select which recommendations willbe shown. This way, the content management team canstill filter out possibly irrelevant recommendations. Whichapproach is preferred depends on what the content manage-ment team of the specific knowledge portal finds an accept-able percentage of irrelevant recommendations, and if theyfavor a lower workload over the quality of recommendations.

To allow for further experimentation and to provide aninsight into how the different recommendation algorithmswork, we have developed such a recommendation enginedemo application. This tool allows the user to select anarticle to show recommendations for, and an algorithm touse for generating these recommendations. The tool is de-signed for the content management team of SKYbrary, so itis not publicly available. However, the source code10 can beadjusted to work with other knowledge portals. The tool isalso useful for researchers that want to see how the recom-

10https://github.com/Royhoey/Recommendation-engine-demo-application

mendation algorithms behave.Figure 2 shows a screenshot of this recommendation en-

gine demo application. In the screenshot, the article ’Hy-poxia’ is selected and recommendations generated by thecontent based algorithm are shown in the table below. Thetable consists of the title of the recommended article, alink to that article on SKYbrary, the relation of the recom-mended article to the current article (it can be linked in thearticle or already included in the ’Related Links’-section),and the score assigned by the algorithm.

A few extra filters were also added specifically for SKY-brary. The ’Hide Accidents & Incidents articles’ filter hidesrecommended articles that are in the Accidents & Incidentscategory. Articles that are from that category are never rec-ommended, because each article has a separate section con-taining links to related accidents and incidents. The othertwo filters can be used to hide recommended articles that arelinked in the article text or recommended articles that arealready included in the ’Related Links’-section. These fil-ters allow the content managers of SKYbrary to quickly seeif they have missed any relevant articles that are not linkedor included in the ’Related Links’-section in the article.

9. CONCLUSION

9.1 ConclusionThe main research question of this paper, which we have

stated in section 3.1, is: ’Which type of recommendationalgorithms is the most effective for a knowledge portal: auser-navigation based algorithm, a content based algorithmor a hybrid algorithm that combines these two?’. The firstexperiment showed us which version of each algorithm isbetter at predicting pre-selected related articles. This waywe could answer our sub questions and combine the bestuser-navigation based version and the best content basedversion into a hybrid algorithm. The purpose of the hy-brid algorithm is to compensate for the disadvantages of theseparate algorithms, and is thus expected to produce morerelevant recommendations. To test this we conducted a sur-vey to the content management team of SKYbrary, in whichthey were asked to rate the recommendations generated bythe different algorithms on relevancy.

The results of this survey show that there is enough ev-idence to say that a hybrid algorithm works better than auser-navigation based algorithm. However, we can not saywith absolute certainty that a hybrid algorithm works betterthan a content based algorithm, or that the content basedalgorithm works better than the user-navigation based al-gorithm. When implementing a recommender system in aknowledge base, a hybrid algorithm that also factors in con-tent similarity by using a tf-idf weighting scheme should beconsidered over an algorithm based on user navigation be-havior. The results of our survey show that a hybrid algo-rithm is likely to be better than a content based algorithm,but a statistically significant difference in performance couldnot be proven. To draw this conclusion, a different or moreelaborate evaluation method might be needed.

A recommendation engine demo application was designedto allow researchers and content managers of knowledge por-tals to experiment with recommendation algorithms. Thisway a decision can be made on which algorithm to imple-ment, or the tool can be used to support the process ofmanually selecting recommended articles.

9.2 Discussion and future workWe have used several existing algorithms to generate rec-

ommendations for a knowledge portal in this paper, whilepurposely not tweaking the algorithms for our case study toensure that they will work for other knowledge portals aswell. Studies on different knowledge portals with the samealgorithms can add support to this claim of generalizability.

A limitation of the user-navigation based algorithm in thiscase study is that the SKYbrary articles already contain a’Related Links’-section. An article that might be relevantbut is not linked in that section will never achieve a highrank in the set of recommendations. Therefore it would beinteresting to experiment with knowledge portals that donot have a ’Related Links’-section yet and see how a usernavigation based algorithm works there.

The results of the survey show that the difference in per-formance between the content based algorithm and the hy-brid algorithm is subtle, which made it difficult to find a sta-tistically significant difference. This can be caused by thesmall amount of respondents of the survey we conducted.This can be an inducement for further research on a knowl-edge portal where a higher amount of respondents is avail-

able to rate the algorithms. An evaluation technique thatcan be used is A/B testing, where half of the visitors of theknowledge portal see recommendations made by algorithm Aand the other half sees recommendations made by algorithmB. This will lead to a much larger amount of data points.Another cause of this subtle difference in performance couldbe the way the user-navigation and content based algorithmswere combined into a hybrid algorithm. Different ways ofcombining these algorithms into a hybrid algorithm can beexplored and evaluated in further research as well. Thesestarting points for future work might allow for a more evi-dent conclusion on which recommendation algorithm is themost suitable for knowledge portals.

10. REFERENCES[1] Joeran Beel and Stefan Langer. Research paper

recommender systems: A literature survey. 2014.

[2] Gleb Beliakov, Tomasa Calvo, and Simon James.Aggregation of preferences in recommender systems.In Recommender systems handbook, pages 705–734.Springer, 2011.

[3] Robin Burke. Integrating knowledge-based andcollaborative-filtering recommender systems. InProceedings of the Workshop on AI and ElectronicCommerce, pages 69–72, 1999.

[4] Robin Burke. Hybrid web recommender systems. InThe Adaptive Web, pages 377–408. Springer, 2007.

[5] Heidi Collins. Enterprise knowledge portals.AMACOM Div American Mgmt Assn, 2002.

[6] Michael Ekstrand. Coursera: Introduction torecommender systems - lecture 8. https://www.coursera.org/learn/recommender-systems/

lecture/LlcXz/preferences-and-ratings, 2015.(Last accessed 30-June-2015).

[7] Karl Groves. The limitations of server log files forusability analysis, 2007.http://boxesandarrows.com/the-limitations-of-server-log-files-for-usability-analysis/ (Last accessed30-June-2015).

[8] Marko Gulic, Ivan Magdalenic, and Boris Vrdoljak.Ontology Matching Using TF/IDF Measure withSynonym Recognition. Springer, 2013.

[9] Paul B Kantor, Lior Rokach, Francesco Ricci, andBracha Shapira. Recommender systems handbook.Springer, 2011.

[10] Blerina Lika, Kostas Kolomvatsos, and StathesHadjiefthymiades. Facing the cold start problem inrecommender systems. Expert Systems withApplications, 41(4):2065–2073, 2014.

[11] Pasquale Lops, Marco De Gemmis, and GiovanniSemeraro. Content-based recommender systems: Stateof the art and trends. In Recommender systemshandbook, pages 73–105. Springer, 2011.

[12] Valentina Maccatrozzo. Burst the filter bubble: Usingsemantic web to enable serendipity. In The SemanticWeb -ISWC 2012, volume 7650 of Lecture Notes inComputer Science, pages 391–398. Springer BerlinHeidelberg, 2012.

[13] Deuk Hee Park, Il Young Choi, Hyea Kyeong Kim,and Jae Kyeong Kim. A review and classification ofrecommender systems research. School of

Management, KyungHee University, Seoul, Korea,IPEDR vol, 5, 2011.

[14] Bhavik Pathak, Robert Garfinkel, Ram D Gopal,Rajkumar Venkatesan, and Fang Yin. Empiricalanalysis of the impact of recommender systems onsales. Journal of Management Information Systems,27(2):159–188, 2010.

[15] Michael J Pazzani and Daniel Billsus. Content-basedrecommendation systems. In The adaptive web, pages325–341. Springer, 2007.

[16] Anil Poriya, Tanvi Bhagat, Neev Patel, and RekhaSharma. Non-personalized recommender systems anduser-based collaborative recommender systems. Int. J.Appl. Inf. Syst, 6(9):22–27, 2014.

[17] J Ben Schafer, Dan Frankowski, Jon Herlocker, andShilad Sen. Collaborative filtering recommendersystems. In The Adaptive Web, pages 291–324.Springer, 2007.

[18] Guy Shani and Asela Gunawardana. Evaluatingrecommendation systems. In Recommender systemshandbook, pages 257–297. Springer, 2011.

[19] Simon Wakeling, Paul Clough, and Barbara Sen.Investigating the potential impact of non-personalizedrecommendations in the opac: Amazon vs. worldcat.org. In Proceedings of the 5th Information Interactionin Context Symposium, pages 96–105. ACM, 2014.

[20] Yuanyuan Wang, Stephen Chi-fai Chan, and GraceNgai. Applicability of demographic recommendersystem to tourist attractions: a case study on tripadvisor. In Proceedings of the The 2012IEEE/WIC/ACM International Joint Conferences onWeb Intelligence and Intelligent AgentTechnology-Volume 03, pages 97–101. IEEE ComputerSociety, 2012.

[21] Jin Xu, Karaleise Johnson-Wahrmann, and ShuliangLi. The development, status and trends ofrecommender systems: a comprehensive and criticalliterature review. 2014.

[22] Zui Zhang, Hua Lin, Kun Liu, Dianshuang Wu,Guangquan Zhang, and Jie Lu. A hybrid fuzzy-basedpersonalized recommender system for telecomproducts/services. Information Sciences, 235:117–129,2013.