J Data Semant (2016) 5:99–113DOI 10.1007/s13740-016-0060-9

ORIGINAL ARTICLE

Content-Based Video Recommendation System Based on StylisticVisual Features

Yashar Deldjoo1 · Mehdi Elahi1 · Paolo Cremonesi1 · Franca Garzotto1 ·Pietro Piazzolla1 · Massimo Quadrana1

Received: 10 November 2015 / Accepted: 20 January 2016 / Published online: 11 February 2016© Springer-Verlag Berlin Heidelberg 2016

Abstract This paper investigates the use of automaticallyextracted visual features of videos in the context of rec-ommender systems and brings some novel contributions inthe domain of video recommendations. We propose a newcontent-based recommender system that encompasses a tech-nique to automatically analyze video contents and to extracta set of representative stylistic features (lighting, color, andmotion) grounded on existing approaches of Applied MediaTheory. The evaluation of the proposed recommendations,assessed w.r.t. relevance metrics (e.g., recall) and com-paredwith existing content-based recommender systems thatexploit explicit features such as movie genre, shows that ourtechnique leads tomore accurate recommendations. Our pro-posed technique achieves better results not only when visualfeatures are extracted from full-length videos, but also whenthe feature extraction technique operates on movie trailers,pinpointing that our approach is effective also when full-length videos are not available orwhen there are performancerequirements. Our recommender can be used in combinationwith more traditional content-based recommendation tech-niques that exploit explicit content features associated to

B Yashar Deldjooyashar.deldjoo@polimi.it

Mehdi Elahimehdi.elahi@polimi.it

Paolo Cremonesipaolo.cremonesi@polimi.it

Franca Garzottofranca.garzotto@polimi.it

Pietro Piazzollapietro.piazzolla@polimi.it

Massimo Quadranamassimo.quadrana@polimi.it

1 Politecnico di Milano, Milan, Italy

videofiles, to improve the accuracy of recommendations.Ourrecommender can also be used alone, to address the problemoriginated from video files that have no meta-data, a typicalsituation of popular movie-sharing websites (e.g., YouTube)where every day hundred millions of hours of videos areuploaded by users and may contain no associated informa-tion. As they lack explicit content, these items cannot beconsidered for recommendation purposes by conventionalcontent-based techniques even when they could be relevantfor the user.

1 Introduction

Recommender Systems (RSs) are characterized by the capa-bility of filtering large information spaces and selecting theitems that are likely to be more interesting and attractive toa user [1]. Recommendation methods are usually classifiedinto collaborative filtering methods, content-based methodsand hybrid methods [1–4]. Content-based methods, that areamong popular ones [5–7], suggest items which have contentcharacteristics similar to the ones of items a user liked in thepast. For example, news recommendations consider wordsor terms in articles to find similarities.

A prerequisite for content-based filtering is the availabilityof information about relevant content features of the items.In most existing systems, such features are associated tothe items as structured or un-structured meta-information.Many RSs in the movie domain, for instance, consider moviegenre, director, cast, (structured information), or plot, tagsand textual reviews (un-structured information). In contrast,our work exploits “implicit” content characteristics of items,i.e., features that are “encapsulated” in the items and must becomputationally “extracted” from them.

123

100 Y. Deldjoo et al.

We focus on the domain of video recommendations andpropose a novel content-based technique that filters itemsaccording to visual features extracted automatically fromvideo files, either full-length videos or trailers. Such featuresinclude lighting, color, and motion; they have a “stylistic”nature and, according to Applied Media Aesthetics [8], canbe used to convey communication effects and to stimulatedifferent feelings in the viewers.

The proposed recommendation technique has been evalu-ated w.r.t. relevance metrics (e.g., recall), using conventionaltechniques for the off-line evaluation of recommender sys-tems that exploit machine learning methods [9,10]. Theresults have been then compared with existing content-basedtechniques that exploit explicit features such as movie genre.We consider three different experimental conditions—(a)visual features extracted from movie trailers; (b) visual fea-tures extracted by full-length videos; and (c) traditionalexplicit features based on genre—to test two hypotheses:

1. Our recommendation algorithm based on visual featuresleads to a higher recommendation accuracy in com-parison with conventional genre-based recommendersystems.

2. Accuracy is higher when stylistic features are extractedfrom either full-length movies or when they originatefrom movie trailers only. In other words, for our recom-mender movie trailers are good representatives of theircorresponding full-length movies.

The evaluation study has confirmed both hypotheses and hasshown that our technique leads to more accurate recommen-dations than the baselines techniques in both experimentalconditions.

Our work provides a number of contributions to the RSfield in the video domain. It improves our understanding onthe role of implicit visual features in the recommendationprocess, a subject which has been addressed by a limitednumber of researches. The proposed technique can be usedin two ways:

• “In combination with” other content-based techniquesthat exploit explicit content, to improve their accuracy.This mixed approach has been investigated and evaluatedby other works [11,12]. Still, prior off-line evaluationshave involved a limited number of users (few dozens )against the thousands employed in our study.

• “Autonomously”, to replace traditional content-basedapproaches when (some) video items (typically the newones) are not equipped with the explicit content fea-tures that a conventional recommender would employto generate relevant recommendations. This situation,which hereinafter we refer to as “extreme new itemproblem” [13] typically occurs for example in popular

movie-sharing websites (e.g., YouTube) where every dayhundredmillions of hours of videos are uploaded by usersand may contain no meta-data. Conventional content-based techniques would neglect to consider these newitems even if they may be relevant for recommendationpurposes, as the recommender has no content to analyzebut video files. To our knowledge, the generation of rec-ommendations that exploit automatically extracted visualfeatures “only” has not been explored nor evaluated inprior works.

As an additional contribution, our study pinpoints that ourtechnique is accurate when visual feature extraction oper-ates both on full-length movies (which is a computationallydemanding process) and on movie trailers. Hence, ourmethod can be used effectively also when full-length videosare not available or when it is important to improve perfor-mance.

The rest of the paper is organized as follows. Section 2reviews the relevant state of the art, related to content-basedrecommender systems andvideo recommender systems.ThisSection also introduces some theoretical background onMedia Esthetics that helps us to motivate our approach andinterpret the results of our study. Section 3 describes the pos-sible relation between the visual features adopted in our workand the esthetic variables that are well known for artists in thedomain of movie making. In Sect. 4, we describe our methodfor extracting and representing implicit visual features of thevideo and provide the details of our recommendation algo-rithm. Section 5 introduces the evaluation method. Section 6presents the results of the study and Sect. 7 discusses them.Section 8 draws the conclusions and identifies open issuesand directions for future work.

2 Related Work

2.1 Content-Based Recommender Systems

Content-based RSs create a profile of a user’s preferences,interests and tastes by considering the feedback provided bythe user to some items together with the content associated tothem. Feedback can be gathered either explicitly from users,by explicitly asking them to rate an item [14], or implicitlyby analyzing her activity [15]. Recommendations are thengenerated by matching the user profile against the featuresof all items. Content can be represented using keyword-basedmodels, in which the recommender creates a Vector SpaceModel (VSM) representation of item features, where an itemis representedby avector in amulti-dimensional space.Thesedimensions represent the features used to describe the items.By means of this representation, the system measures a rel-evance score that represents the user’s degree of interest

123

Content-Based Video Recommendation System Based on Stylistic Visual Features 101

toward any of these items [6]. For instance, in the moviedomain, the features that describe an item can be genre,actors, or director [16]. This model may allow content-basedrecommender systems to naturally tackle the new item prob-lem [13]. Other families of content-based RSs use semanticanalysis (lexicons and ontologies) to create more accurateitem representations [17–19].

In the literature, a variety of content-based recommenda-tion algorithms have been proposed. A traditional exampleis the “k-nearest neighbor” approach (KNN) that computesthe preference of a user for an unknown item by comparingit against all the items known by the user in the cata-log. Every known item contributes to predict the preferencescore according to its similarity with the unknown item. Thesimilarity can bemeasured by typically using Cosine similar-ity [6,7]. There are also works that model the probability forthe user to be interested to an item using a Bayesian approach[20], or use other techniques adopted from IR (InformationRetrieval) such as the Relevance Feedback method [21].

2.2 Recommender Systems in the Multimedia Domain

In the multimedia domain, recommender systems typicallyexploit two types of item features, hereinafter referred to asHigh-Level features (HL) or Low-Level features (LL). High-Level features express properties of media content that areobtained from structured sources of meta-information suchas databases, lexicons and ontologies, or from less struc-tured data such as reviews, news articles, item descriptionsand social tags [16,20–24]. Low-Level features are extracteddirectly from media files themselves [25,26]. In the musicrecommendation domain, for example, Low-Level featuresare acoustic properties such as rhythm or timbre, which areexploited to find music tracks that are similar to those likedby a user [27–30].

In the domain of video recommendation, a limited numberof works have investigated the use of Low-Level features,extracted from pure visual contents, which typically rep-resent stylistic aspect of the videos [11,12,31,32]. Still,existing approaches consider only scenarios where Low-Level features are exploited in addition to another type ofinformation with the purpose of improving the quality ofrecommendations. Yang et al. [11] proposes a video recom-mender system, calledVideoReach,which use a combinationof High-Level and Low-Level video features of differentnature—textual, visual and aural—to improve the click-through rate. Zhao et al. [12] proposes a multi-task learningalgorithm to integrate multiple ranking lists, generated usingdifferent sources of data, including visual content. As none ofthese works use Low-Level visual features only, they cannotbe applied when the extreme new item problem [33] occurs,i.e., when only video files are available and high-level infor-mation is missing.

2.3 Video Retrieval

A Content-Based Recommender System (CBRS) for videosis similar to aContent-BasedVideoRetrieval system (CBVR)in the sense that both systems analyze video content to searchfor digital videos in large video databases. However, thereare major differences between these systems. For example,people use popular video sharing websites such as YouTubefor three main purposes [34]: (1) Direct navigation: to watchvideos that they found at specific websites, (2) Search: towatch videos around a specific topic expressed by a set ofkeywords, (3) Personal entertainment: to be entertained bythe content that matches their taste. A CBVR system is com-posed of a set of techniques that typically address the firstand the second goal, while a CBRS focuses on the third goal.Accordingly, themain differences betweenCBRSandCBVRcan be listed as follows [11]:

1. Different objectives: The goal of a CBVR system is tosearch for videos that match “a given query” provideddirectly by a user as a textual or video query etc. The goalof a video CBRS is, however, to search for videos thatarematchedwith “user taste” (also known as user profile)and can be obtained by analyzing his past behavior andopinions on different videos.

2. Different inputs: The input to a CBVR system typicallyconsists of a set of keywords or a video query where theinputs could be entirely un-structured and do not have anyproperty per se. The input to a video CBRS on top of thevideo content includes some or many features obtainedfrom user modeling (user profile, tasks, activities), thecontext (location, time, group) and other sources of infor-mation.

3. Different features: In general, video content features canbe classified into three rough hierarchical levels [35]:

• Level 1: Stylistic low-level that deals with modelingthe visual styles in a video.

• Level 2: Syntactic level that dealswith finding objectsand their interaction in a video.

• Level 3: Semantic level that deals with conceptualmodeling of a video.

People most often rely on content features derived fromlevel 2 and 3 to search for videos as they reside closer tohuman understanding. Even for recommender systems,most CBRSs use video meta-data (genre, actor etc.) thatreside in higher syntactic and semantical levels to pro-vide recommendations. One of the novelties of this workis to explore the importance of stylistic low-level fea-tures in human’s perception of movies. Movie directorsdrastically use the human’s perception in stages of moviecreation, to convey emotions and feeling to the users. Wethus conclude that CBVR system and CBRS deal with

123

102 Y. Deldjoo et al.

video content modeling at different levels depending onsuitability of the feature for a particular application.

While low-level features have been marginally exploredin the community of recommender systems, they have beenextensively studied in other fields such as Computer Visionand Content-Based Video Retrieval [36–38]. For differentobjectives, these communities share with the community ofrecommender systems, the research problems of definingthe “best” representation of video content and of classify-ing videos according to features of different nature. Hence,they offer results and insights that are of interest also inthe movie recommender systems context.Hu et al. [36] andBrezeale and Cook [39] provide comprehensive surveys onthe relevant state of the art related to video content analysisand classification, and discuss a large body of low-level fea-tures (visual, auditory or textual) that can be considered forthese purposes. Rasheed et al. [38] propose a practical moviegenre classification scheme based on computable visual cues.Rasheed and Shah [40] discuss a similar approach by consid-ering also the audio features. Finally, Zhou et al. [41] proposea framework for automatic classification, using a temporallystructured features, based on the intermediate level of scenerepresentation.

2.4 Video Features from a Semiotic Perspective

The stylistic visual features of videos that we exploit in ourrecommendation algorithm have been studied not only inComputer Science but also from a semiotic and expressivepoint of view, in the theory and practice of movie making(see Sect. 3). Lighting, color, and camera motion are impor-tant elements that movie makers consider in their work toconvey meanings, or achieve intended emotional, esthetic, orinformative effects. AppliedMedia Aesthetic [8] is explicitlyconcerned with the relation of media features (e.g., lights,shadows, colors, space representation, camera motion, orsound) with perceptual, cognitive, and emotional reactionsthey are able to evoke in media consumers, and tries to iden-tify patterns in how such features operate to produce thedesired effect [42]. Some aspects concerning these patterns([38,43]) can be generated from video data streams as sta-tistical values and can be used to computationally identifycorrelations with the user profile, in terms of perceptual andemotional effects that users like.

3 Artistic Background

In this section, we provide the artistic background to the ideaof using stylistic visual features for movie recommendation.We describe the stylistic visual features from an artistic pointof view and explain the possible relation between these low-

level features and the esthetic variables that are well knownfor artists in the domain of movie making.

As noted briefly in Sect. 2, the study on how variousesthetic variables and their combination contribute to estab-lish the meaning conveyed by an artistic work is the domainof different disciplines, e.g., semiotics, traditional estheticstudies, etc. The shared belief is that humans respond to cer-tain stimuli (being them called signs, symbols, f eaturesdepending on the discipline) in ways that are predictable, upto a given extent. One of the consequences about this is thatsimilar stimuli are expected to provoke similar reactions, andthis in turn may allow to group similar works of art togetherby the reaction they are expected to provoke.

Among these disciplines, of particular interest for thispaper, Applied Media Aesthetic [8] is concerned with therelation of a number of media elements, such as light, cam-era movements, colors, with the perceptual reactions they areable to evoke in consumers of media communication, mainlyvideos and films. Such media elements, that together buildthe visual images composing the media, are investigated fol-lowing a rather formalistic approach that suits the purposes ofthis paper. By an analysis of cameras, lenses, lighting, etc.,as production tools as well as their esthetic characteristicsand uses, Applied Media Aesthetic tries to identify patternsin how such elements operate to produce the desired effectin communicating emotions and meanings.

The image elements that are usually addressed as fun-damental in the literature, e.g. in [42] , even if with slightdifferences due to the specific context, are lights and shad-ows, colors, space representation, motion and sound. It hasbeen proved, e.g. in [38,43], that some aspects concerningthese elements can be computed from the video data streamas statistical values. We call these computable aspects as fea-tures.

We will now look into closer details of the features, inves-tigated for content-based video recommendation in this paperto provide a solid overviewon how they are used to producingperceptual reaction in the audience. Sound will not be fur-ther discussed, since it is out of scope of this work, as wellas the space representation, that concerns, e.g., the differentshooting angles that can be used to represent dramatically anevent.

3.1 Lighting

There are at least two different purposes for lighting in videoandmovies. Themost obvious is to allow and define viewers’perception of the environment, to make visible the objectsand places they look at. But light can also manipulate how,emotionally, an event is to be perceived, acting in a way thatbypass rational screens. The two main lighting alternativesare usually addressed to as chiaroscuro and f lat lighting,but there are many intermediate solutions between them. The

123

Content-Based Video Recommendation System Based on Stylistic Visual Features 103

Fig. 1 a Out of the past (1947)an example of highly contrastedlighting. b The wizard of OZ(1939) flat lighting example

Fig. 2 a An image fromDjango Unchained (2012). Thered hue is used to increase thescene sense of violence. b Animage from Lincoln (2012).Blue tone is used to produce thesense of coldness and fatigueexperienced by the characters(color figure online)

first is a lighting technique characterized by high contrastbetween light and shadow areas that put the emphasis onan unnatural effect: the borders of objects are altered by thelights. The latter instead is an almost neutral, realistic, way ofilluminating, whose purpose is to enable recognition of stageobjects. Figure 1a, b exemplifies these two alternatives.

3.2 Colors

The expressive quality of colors is closely related to that oflighting, sharing the same ability to set ormagnify the feelingderived by a given situation. The problem with colors is thedifficulty to isolate their contribution to the overall ‘mood’ ofa scene from that of other esthetic variables operating in thesame context. Usually their effectiveness is higher when thecontext as a whole is predisposed toward a specific emotionalobjective.

Even if an exact correlation between colors and the feelingthey may evoke is not currently supported by enough scien-tific data, colors nonetheless have an expressive impact thathas been investigated thoroughly, e.g. in [44]. An interestingmetric to quantify this impact has been proposed in [45] asperceived color energy, a quantity that depends on a color’ssaturation, brightness and the size of the area the color coversin an image. Also the hue plays a role as if it tends towardreds, the quantity of energy is more, while if it tends more on

blues, it is less. These tendencies are shown in the examplesof Fig. 2a, b.

3.3 Motion

The illusion of movement given by screening a sequence ofstill frames in rapid succession is the very reason of cinemaexistence. In a video or movie, there are different types ofmotions to consider:

• Profilmic movements Every movement that concerns ele-ments, shot by the camera, falls in this category, e.g.performers motion, vehicles, etc.. The movement can bereal or perceived. By deciding the type and quantity ofmotion an ‘actor’ has, considering as actor any possibleprotagonist of a scene, the director defines, among others,the level of attention to, or expectations from, the scene.As an example, the hero walking slowly in a dark alley,or a fast car chasing.

• Camera movements Are the movements that alter thepoint of view on the narrated events. Cameramovements,such as the pan, truck, pedestal, dolly, etc. can be usedfor different purposes. Someuses are descriptive, to intro-duce landscapes or actors, to follow performers actions,and others concern the narration, to relate two or more

123

104 Y. Deldjoo et al.

different elements, e.g., anticipating a car’s route to showan unseen obstacle, to move toward or away from events.

• Sequences movements As shots changes, using cutsor other transitions, the rhythm of the movie changesaccordingly. Generally, a faster rhythm is associated withexcitement, and a slower rhythm suggests a more relaxedpace [46].

In this paper, we followed the approach in [38], con-sidering the motion content of a scene as a feature thataggregate and generalize both profilmic and camera move-ments. Sequence movements are instead considered in theAverage shot length feature, both being described withdetail in the next section.

4 Method Description

The first step to build a video CBRS based on stylistic low-level features is to search for features that complywith humanvisual norms of perception and abide by the grammar of thefilm—the rules that movie producers and directors use tomake movies. In general, a movieM can be represented as acombination of threemainmodalities:MV the visual,MA theaudio and MT textual modalities, respectively. The focus ofthis work is only on visual features, thereforeM = M(MV).The visual modality itself can be represented as

MV = MV(fv) (1)

where fv = ( f1, f2, . . . , fn) is a set of features that describethe visual content of a video. Generally speaking, a videocan be considered as contiguous sequence of many frames.Consecutive video frames contain a lot of frames that arehighly similar and correlated. Considering all these framesfor feature extraction not only does not provide new infor-mation to the system but also is computationally inefficient.Therefore, the first step prior to feature extraction is struc-tural analysis of the video, i.e. to detect shot boundaries andto extract a key frame within each shot. A shot boundary is aframe where frames around it have significant difference intheir visual content. Frames within a shot are highly similaron the other hand, therefore it makes sense to take one rep-resentative frame in each shot and use that frame for featureextraction. This frame is called the Key Frame.

Figure 3 illustrates the hierarchical representation of avideo. Two types of features are extracted from videos: (1)temporal features (2) spatial features. The temporal featuresreflect the dynamic perspectives in a video such as the aver-age shot duration (or shot length) and object motion, whereasthe spatial features illustrate static properties such as color,light, etc. In the following we describe in more detail these

Fig. 3 Hierarchical video representation and feature extraction in ourframework

features and the rationale behind choosing them in additionto how they can be measured in a video.

4.1 Visual Features

To demonstrate the effectiveness of the proposed videoCBRS, after carefully studying the literature in computervision, we selected and extracted the five most informativeand distinctive features to be extracted from each video

fv = ( f1, f2, f3, f4, f5)

= (Lsh, μcv, μm, μσ 2m, μlk) (2)

where Lsh is the average shot length, μcv is the mean colorvariance over key frames, μm and μσ 2

mare the mean motion

average and standard deviation across all frames, respec-tively, and μσ 2

mis the mean lightening key over key frames.

As can be noted, some of the features are calculated acrosskey frames and the others across all video frames (see Fig. 3).Each of these features carry a meaning and are used in thehands of able directors to convey emotions when shootingmovies. Assuming that there exists n f frames in the video, tbeing the index of each single frame and nsh key frames (orshots), q being the index of a numbered list of key frames,the proposed visual features and how they are calculated ispresented in the following [25,26,38]

• Average shot length (Lsh) A shot is a single camera actionand the number of shots in a video can provide usefulinformation about the pace at which a movie is beingcreated. The average shot length is defined as

Lsh = n f

nsh(3)

where n f is the number of frames and nsh the num-ber of shots in a movie. For example, action moviesusually contain rapid movements of the camera (there-fore, they contain higher number of shots or shorter shot

123

Content-Based Video Recommendation System Based on Stylistic Visual Features 105

lengths) compared to dramaswhich often contain conver-sations between people (thus longer average shot length).Because movies can be made a different frame rates, Lsh

is further normalized by the frame rate of the movie.• Color variance (μcv) The variance of color has a strong

correlation with the genre. For instance, directors tendto use a large variety of bright colors for comedies anddarker hues for horror films. For each key frame repre-sented in Luv color space we compute the covariancematrix:

ρ =⎛⎜⎝

σ 2L σ 2

Lu σ 2Lv

σ 2Lu σ 2

u σ 2uv

σ 2Lv σ 2

uv σ 2v

⎞⎟⎠ (4)

The generalized variance can be used as the represen-tative of the color variance in each key frame given by

�q = det(ρ) (5)

in which a key frame is a representative frame within ashot (e.g. the middle shot). The average color variance isthen calculated by:

μcv =∑nsh

q=1 �q

nsh(6)

where nsh is the number of shots equal to number of keyframes.

• Motion within a video can be caused mainly by the cam-eramovement (i.e. cameramotion) or movements on partof the object being filmed (i.e. object motion). Whilethe average shot length captures the former characteris-tic of a movie, it is desired for the motion feature to alsocapture the latter characteristic.Amotion feature descrip-tor based on optical flow [47,48] is used to measure arobust estimate of themotion in sequence of images basedon velocities of images being filmed. Because motionfeatures are based upon sequence of images, they arecalculated across all video frames.At frame t , if the average motion of pixels is representedby mt and the standard deviation of pixel motions is(σ 2

m)t :

μm =∑n f

t=1mt

n f(7)

and

μσ 2m

=∑n f

t=1(σ2m)t

n f(8)

whereμm andμσ 2mrepresent the average of motion mean

and motion standard deviation aggregated over entire n f

frames.• Lighting Key is another distinguishing factor betweenmovie genres in such a way that the director uses it asa factor to control the type of emotion they want to beinduced to a viewer. For example, comedy movies oftenadopt lighting keywhich has abundance of light (i.e. highgray-scale mean) with less contrast between the brightestand dimmest light (i.e. high gray-scale standard devia-tion). This trend is often known as high-key lighting. Onthe other hand, horror movies or noir films often pickgray-scale distributions which is low in both gray-scalemean and gray-scale standard deviation, known by low-key lighting. To capture both of these parameters, aftertransforming all key-frames to HSV color space [49],we compute the mean μ and standard deviation σ of thevalue component which corresponds to the brightness.The scene lighting key ξ defined by multiplication of μ

and σ is used to measure the lighting of key frames

ξq = μ · σ (9)

For instance, comedies often contain key-frames whichhave a well distributed gray-scale distribution whichresults in both the mean and standard deviation of gray-scale values to be high, therefore, for comedy genre onecan state ξ > τc, whereas for horror movies the light-ing key with poorly distributed lighting the situation isreverse and wewill have ξ < τh , where τc and τh are pre-defined thresholds. In the situation where τh < ξ < τcother movie genres (e.g. Drama) exists where it is hardto use the above distinguish factor for them. The averagelighting calculated over key frames is given by (10)

μlk =∑nsh

q=1 ξq

nsh(10)

It isworth noting that the stylistic visual features have beenextracted by using our own implementation. The code andthe dataset of extracted features will be publicly accessiblethrough the webpage of the group.1

4.2 Recommendation Algorithm

To generate recommendations using our Low-Level stylisticvisual features, we adopted a classical “k-nearest neighbor”content-based algorithm. Given a set of users u ∈ U and acatalog of items i ∈ I , a set of preference scores rui givenby user u to item i has been collected. Moreover, each itemi ∈ I is associated to its feature vector fi . For each couple

1 http://recsys.deib.polimi.it/.

123

106 Y. Deldjoo et al.

of items i and j , the similarity score si j is computed usingcosine similarity:

si j = fi T f j‖ fi‖

∥∥ f j∥∥ (11)

For each item i the set of its nearest neighbors NNi isbuilt, |NNi | < K . Then, for each user u ∈ U , the predictedpreference score rui for an unseen item i is computed asfollows

rui =∑

j∈NNi ,ru j>0 ru j si j∑j∈NNi ,ru j>0 si j

(12)

5 Evaluation Methodology

We have formulated the following two hypotheses:

1. the content-based recommender system, that exploits aset of representative visual features of the video contents,may have led to a higher recommendation accuracy incomparison to the genre-based recommender system.

2. the trailers of the movies can be representative of theirfull-lengthmovies, with respect to the stylistic visual fea-tures, and indicate high correlation with them.

Hence, we speculate that a set of stylistic visual fea-tures, extracted automatically, may be more informative ofthe video content than a set of high-level expert annotatedfeatures.

To test these hypotheses, we have evaluated the Top-N rec-ommendation quality of each content-based recommendersystems by running a fivefold cross-validation on a subset ofthe MovieLens-20M dataset [50]. The details of the subsetare described later in the paper. The details on the evaluationprocedure follow.First, we generated five disjoint random splits of the ratingsin the dataset.Within each iteration, the evaluation procedureclosely resembles the one described in [9]. For each iteration,one split was used as the probe set Pi , while the remainingoneswere used to generate the training setMi andwas used totrain the recommendation algorithm. The test set Ti containsonly 4-star and 5-star ratings from Pi , which we assume tobe relevant.For each relevant item i rated by user u in Ti , we form alist containing the item i and all the items not rated by theuser u, which we assume to be irrelevant to her. Then, weformed a top-N recommendation list by picking the top Nranked items from the list. Being r the rank of i , we have ahit if r < N , otherwise we have a miss. Since we have onesingle relevant item per test case, recall can assume value 0

Table 1 General informationabout our dataset

# Items 167

# Users 139, 190

# Ratings 570, 816

Table 2 Distribution of movies in our catalog

Action Comedy Drama Horror Mixed Total

# 29 27 25 24 62 167

% 17 16 15 14 38 100

or 1 (respectively in case of a miss or a hit). Therefore, therecall(N) on the test set Ti can be easily computed as:

recall(N ) = #hits

|Ti | (13)

We could have also evaluated the precision(N) on the rec-ommendation list, but since it is related to the value of therecall(N) by a simple scaling factor 1/N [9], we decided toomit it to avoid redundancy. The values reported throughoutthe paper are the averages over the fivefolds.

We have used a set of full-length movies and their trailersthat were sampled randomly from all the main genres, i.e.,Action, Comedy, Drama and Horror. The summary of thedataset is given in Table 1.

As noted before, the movie titles were selected randomlyfrom MovieLens dataset, and the files were obtained fromYouTube [51]. The dataset contained over all 167 movies,105 of which belonging to a single genre and 62 moviesbelonging to multiple genres (see Table 2).

The proposed video feature extraction algorithm wasimplemented in MATLAB R2015b2 on a workstation withan Intel Xeon(R) eight-core 3.50 GHz processor and 32 GBRAM. The Image Processing Toolbox (IPT) and ComputerVision Toolbox (CVT) in MATLAB provide the basic ele-ments for feature extraction and were used in our work forvideo content analysis. In addition, we used the R statisti-cal computing language3 together with MATLAB for dataanalysis. For video classification, we took advantage of allthe infrastructure in Weka4 that provides an easy-to-use andstandard framework for testing different classification algo-rithms.

2 http://www.mathworks.com/products/matlab.3 https://www.r-project.org.4 http://www.cs.waikato.ac.nz/ml/weka.

123

Content-Based Video Recommendation System Based on Stylistic Visual Features 107

6 Results

6.1 Classification Accuracy

We have conducted a preliminary experiment to understandif the genre of the movies can be explained in terms of thefive low-level visual features, described in Sect. 4. The goalof the experiment is to classify the movies into genres byexploiting their visual features.

6.1.1 Experiment A

To simplify the experiment, we have considered 105 moviestagged with one genre only (Table 2). We have experimentedwith many classification algorithms and obtained the bestresults under decision tables [52]. Decision tables can beconsidered as tabular knowledge representations [53]. Usingthis technique, the classification for a new instance is doneby searching for the exact matches in the decision table cells,and then the instance is assigned to the most frequent classamong all instances matching that table cell [54].

We have used tenfold cross-validation and obtained anaccuracy of 76.2 % for trailers and 70.5 % for full-lengthmovies. The best classification was done for comedy genre:23 out of 27 movie trailers were successfully classified intheir corresponding comedy genre. On the other hand, themost erroneous classification happened for the horror genrewheremore number ofmovie trailers have beenmisclassifiedinto the other genres. For example, 4 out of 24 horror movietrailers have been mistakenly classified as action genre. Thisis a phenomenon that was expected, since typically there aremany action scenes occurred in horror movies, and this maymake the classification very hard. Similar results have beenobserved for full-length movies.

From this first experiment we can conclude that the fivelow-level stylistic visual features used in our experimentcan be informative of the movie content and can be usedto accurately classify the movies into their correspondinggenres.

6.2 Correlation between Full-Length Movies andTrailers

Oneof the research hypotheseswehave formulated addressesthe possible correlation between the full-length movies andtheir corresponding trailers. Indeed, we are interested toinvestigate whether or not the trailers are representative oftheir full-length movies, with respect to the stylistic visualfeatures. To investigate this issue, we have performed twoexperiments.

6.2.1 Experiment B

We have first extracted the low-level visual features fromeach of the 167 movies and their corresponding trailers inour dataset. We have then computed the cosine similaritybetween the visual features extracted from the full-lengthmovies and the trailers. Cosine is the same metric used togenerate recommendations in our experiments (as explainedin Sect. 5), and hence, it is a reliable indicator to evaluate ifrecommendations based on trailers are similar to recommen-dations based on the full-length movies.

Figure 4 plots the histogram of the cosine similarity. Aver-age is 0.78, median is 0.80. More than 75 % of the movieshave a cosine similarity greater than 0.7 between the full-length movie and trailer. Moreover, less than 3 % of themovies have a similarity below 0.5.

Overall, the cosine similarity shows a substantial correla-tion between the full-length movies and trailers. This is aninteresting outcome that basically indicates that the trailersof the movies can be considered as good representatives ofthe corresponding full-length movies.

6.2.2 Experiment C

In the second experiment, we have used the low-level fea-tures extracted from both trailers and the full-length moviesto feed the content-based recommender system described inSect. 4.2. We have used features f3–f5 (i.e., camera motion,object motion, and light) as they proved to be the best choiceof stylistic visual features (as described in Sect. 6.3). Qual-ity of recommendations has been evaluated according to themethodology described in Sect. 5.

Cosine Similarity0.4 0.5 0.6 0.7 0.8 0.9 1

Pro

babi

lity

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35Histogram of Similarities

Fig. 4 Histogram distribution of the cosine similarity between full-length movies and trailers

123

108 Y. Deldjoo et al.

1 1.5 2 2.5 3 3.5 4 4.5 5

N

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

reca

ll

LL (stylistic)HL (genre) random

(a)

1 1.5 2 2.5 3 3.5 4 4.5 5

N

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

reca

ll

LL (stylistic)HL (genre) random

(b)

Fig. 5 Performance comparison of different CB methods under best feature combination for full-length movies (a), and trailers (b). K = 2 for LLfeatures and K = 10 for HL features

Figure 5 plots the recall@N for full-length movies (a) andtrailers (b), with values of N ranging from 1 to 5. We notethat the K values have been determinedwith cross-validation(see Sect. 6.3.1). By comparing the two figures, it is clear thatthe recall values of the content-based recommendation usingthe features extracted from the full-lengthmovies and trailersare almost identical.

The results of this second experiment confirm that low-level features extracted from trailers are representative of thecorresponding full-lengthmovies and can be effectively usedto provide recommendations.

6.3 Recommendation Quality

In this section,we investigate ourmain researchhypothesis: iflow-level visual features can be used to provide good-qualityrecommendations. We compare the quality of content-basedrecommendations based on three different types of features:

Low Level (LL): stylistic visual features.High Level (HL): semantic features based on genres.Hybrid (LL+HL): both stylistic and semantic features.

6.3.1 Experiment D

To identify the visual features that aremore useful in terms ofrecommendation quality, we have performed an exhaustiveset of experiments by feeding a content-based recommendersystemwith all the 31 combinations of the five visual featuresf1–f5. Features have been extracted from the trailers.Wehavealso combined the low-level stylistic visual features with thegenre, resulting in 31 additional combinations. When usingtwo or more low-level features, each feature has been nor-malized with respect to its maximum value (infinite norm).

Table 3 reports Recall@5 for all the different experimentalconditions. The first column of the table describes whichcombination of low-level features has been used (1 =featureused, 0 =feature not used). The last column of the tablereports, as a reference, the recall when using genre only. Thisvalue does not depend on the low-level features. The optimalvalue of K for the KNN similarity has been determined withcross-validation: K = 2 for low level and hybrid, K = 10for genre only.

Recommendations based on low-level stylistic visual fea-tures extracted from trailers are clearly better, in termsof recall, than recommendations based on genre for anycombination of visual features. However, no considerabledifference has been observed between genre-based andhybrid-based recommendations.

It is worth noting that, when using low-level feature f2(color variance), recommendations have a lower accuracywith respect to the other low-level features, although alwaysbetter with respect to genre-based recommendation. More-over, when using two or more low-level features together,accuracy does not increase. These results will be furtherinvestigated in the next section.

6.4 Feature Analysis

In this section, we wish to investigate why some low-levelvisual features provide better recommendations than the oth-ers, as highlighted in the previous section. Moreover, weinvestigate why combinations of low-level features do notimprove accuracy.

6.4.1 Experiment E

In a first experiment, we analyze if some of the low-levelfeatures extracted from trailers are better correlated than the

123

Content-Based Video Recommendation System Based on Stylistic Visual Features 109

Table 3 Performance comparison of different CBmethods, in terms ofRecall metric, for different combination of the Stylistic visual features

Features Recall@5

LL stylistic LL+HL Hybrid HL genre

f1 f2 f3 f4 f5 (K = 2) (K = 2) (K = 10)

0 0 0 0 1 0.31 0.29 0.21

0 0 0 1 0 0 32 0.29

0 0 1 0 0 0.31 0.22

0 1 0 0 0 0.27 0.23

1 0 0 0 0 0.32 0.25

0 0 0 1 1 0.32 0.21

0 0 1 0 1 0.31 0.22

0 0 1 1 0 0.32 0.22

0 0 1 1 1 0.32 0.23

0 1 0 0 1 0.24 0.20

0 1 0 1 0 0.25 0.20

0 1 0 1 1 0.25 0.22

0 1 1 0 0 0.24 0.20

0 1 1 0 1 0.23 0.20

0 1 1 1 0 0.25 0.18

0 1 1 1 1 0.25 0.22

1 0 0 0 1 0.31 0.26

1 0 0 1 0 0.31 0.29

1 0 0 1 1 0.31 0.18

1 0 1 0 0 0.30 0.23

1 0 1 0 1 0.30 0.22

1 0 1 1 0 0.31 0.24

1 0 1 1 1 0.31 0.23

1 1 0 0 0 0.25 0.20

1 1 0 0 1 0.23 0.20

1 1 0 1 0 0.25 0.22

1 1 0 1 1 0.25 0.21

1 1 1 0 0 0.22 0.20

1 1 1 0 1 0.21 0.20

1 1 1 1 0 0.25 0.21

1 1 1 1 1 0.25 0.21

others with respect to the corresponding features extractedfrom the full-length movie. This analysis is similar to the onereported in Sect. 6.2, but results are reported as a function ofthe features.

Figure 7 plots the cosine similarity values between visualfeatures extracted from the full-length movies and visualfeatures extracted from trailers. Features f2 and f4 (colorvariance and object motion) are the less similar features, sug-gesting that their adoption, if extracted from trailers, shouldprovide less accurate recommendations.

We also performed Wilcoxon test comparing featuresextracted from the full-length movies and trailers. Theresults, summarized in Table 4, prove that no significant dif-

Table 4 Significance test with respect to features in 2 set of datasets(movie trailers and full movies)

f1(Lsh) f2(μcv) f3(μm) f4(μσ 2m) f5(μlk)

Wilcox.test 1.3e−9 5.2e−5 0.154 2.2e−16 0.218

H0/H1 w -> H1 w -> H1 w -> H0 w -> H1 w -> H0

The numbers in italic represent the features that do not reject the nullhypothesis

ference exists between features f3 (camera motion) and f5(light), which clearly shows that the full-length movies andtrailers are highly correlation with respect to these two fea-tures. For the other features, significant differences have beenobtained. This basically states that some of the extracted fea-tures may be either not correlated or not very informative.

6.4.2 Experiment F

In Fig. 6, scatter plots of all combinations of the five styl-istic visual features (intra-set similarity) are plotted. Havingvisually inspected, it can be seen that overall the featuresare weakly correlated. However, there are still features thatmutually present high degree of linear correlation. For exam-ple, features 3 and 4 seem to be highly correlated (see row3, column 4 in Fig. 6). Moreover, we have observed similar-ity by comparing the scatter plots of full-length movies andtrailers. Indeed, any mutual dependency between differentfeatures extracted either from full-length movies or trailerswas similar. This is another indication that trailers can beconsidered as representative short version of the full-lengthmovies, in terms of stylistic visual features.

6.4.3 Informativeness of the Features

Entropy is an information theoretic measure [55] that is anindication of the informativeness of the data. Figure 8 illus-trates the entropy scores computed for the stylistic visualfeatures. As it can be seen, the entropy scores of almost allvisual stylistic features are high. The most informative fea-ture, in terms of entropy score, is the fifth one, i.e. LightingKey, and the least informative feature is the second one, i.e.,Color Variance (see Sect. 4 for detailed description). Thisobservation is in the full consistency with the other findings,that we have obtained from, e.g. Wilcoxon test (see Table 4),and correlation analysis (see Fig. 7).

Having considered all the results, we remark that our con-sidered hypotheses have been successfully validated, and wehave shown that a proper extraction of the visual stylisticfeatures of videos may have led to higher accuracy of videorecommendation, than the typical expert annotation method,either when the features are extracted from full-length videosorwhen the features originate frommovie trailers only. Theseare promising results, as they overall illustrate the possibility

123

110 Y. Deldjoo et al.

Fig. 6 Scatter plot for differentcombination of the stylisticvisual features extracted fromthe movie trailers

Scatter plot of the Features (Trailers)

0 0.5 10 0.5 10 0.5 10 0.5 10 0.5 1

0

0.5

1

0

0.5

1

0

0.5

1

0

0.5

1

0

0.5

1

Features1 2 3 4 5

Cos

ine

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Similarities between Features of Full Movies and Trailers

Fig. 7 Cosine similarity between stylistic visual features extractedfrom the full-length movies and their corresponding trailers

to achieve higher accuracy with an automatic method than amanual method (i.e., expert annotation of videos) since themanual method can be very costly and in some cases evenimpossible (e.g., in huge datasets).

7 Discussion

The results presented in the previous section confirm bothour research hypothesis:

1. recommendations based on low-level stylistic visualfeatures are better than recommendations based on high-level semantic features, such as genre;

Features1 2 3 4 5

Ent

ropy

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Entropy of the Features (10 classes)

Fig. 8 Entropy of the stylistic visual features

2. low-level features extracted from trailers are, in gen-eral, a good approximation of the corresponding featuresextracted from the original full-length movies.

7.1 Quality of Recommendations

According to Table 3, all the low-level visual features providebetter recommendations than the high-level features (gen-res). The improvement is particularly evident when usingeither scene duration, light, camera movement, or objectmovement, with an improvement of almost 50 % in termsof recall with respect to genre-based recommendations. Theimprovement is less strong when using color variance, sug-gesting that user opinions are not strongly affected by howcolors are used in movies. This is partially explained by the

123

Content-Based Video Recommendation System Based on Stylistic Visual Features 111

limited informative content of the color variance feature, asshow in Fig. 8, where color variance is the feature with thelowest entropy.

The validity of this finding is restricted to the actual exper-imental conditions considered and may be affected by thelimited size of the dataset. In spite of these limitations, ourresults provide empirical evidence that

the tested low-level visual features may provide predic-tive power, comparable to the genre of the movies, inpredicting the relevance of movies for users.

Surprisingly, mixing low-level and high-level features doesnot improve the quality of recommendations and, in mostcases, the quality is reduced with respect to use of low-level only, as shown in Table 3. This can be explained byobserving that genres can be easily predicted by low-levelfeatures. For instance, action movies have shorter scenesand shot lengths, than other movies. Therefore, a correlationexists between low-level and high-level features that leadsto collinearities and reduced prediction capabilities of themixed approach.

When using a combination of two or more low-level fea-tures, the quality of recommendations does not increasesignificantly and, in same cases, decreases, although it isalways better than the quality obtained with high-level fea-tures. This behavior is not surprising, considering that thelow-level features are weakly correlated, as shown in Fig. 6.

7.2 Trailers vs. Movies

One of the potential drawbacks in using low-level visual fea-tures is the computational load required for the extraction offeatures from full-length movies.

Our research shows that low-level features extracted frommovie trailers are strongly correlated with the correspondingfeatures extracted from full-length movies (average cosinesimilarity 0.78). Scene duration, camera motion and lightare the most similar features when comparing trailers withfull-length movies. The result for the scene duration is some-how surprising, as we would expect scenes in trailers tobe, on average, shorter than scenes in the correspondingfull movies. However, the strong correlation suggests thattrailers have consistently shorter shots than full movies.For instance, if an action movie has, on average, shorterscenes than a dramatic movie, the same applies to their trail-ers.

Our results provide empirical evidence that

low-level visual features extracted from trailers can beused as an alternative to features extracted from full-length movies in building content-based recommendersystems.

8 Conclusion and Future Work

In this paper, we have presented a novel content-basedmethod for the video recommendation task. The methodextracts and uses the low-level visual features from videocontent to provide userswith personalized recommendations,without relying on any high-level semantic features—suchas, genre, cast, or reviews—that are more costly to collect,because they require an “editorial” effort, and are not avail-able in many new item scenarios.

We have developed a main research hypothesis, i.e., aproper extraction of low-level visual features from videosmay led to higher accuracy of video recommendations thanthe typical expert annotation method. Based on a largenumber of experiments, we have successfully verified thehypothesis showing that the recommendation accuracy ishigher when using the considered low-level visual featuresthan when high-level genre data are employed.

The findings of our study do not diminish the impor-tance of explicit semantic features (such as genre, cast,director, tags) in content-based recommender systems. Still,our results provide a powerful argument for exploring moresystematically the role of low-level features automaticallyextracted from video content and for exploring them.

Our future work can be extended in a number of challeng-ing directions:

• Wewill widen our analysis by adopting bigger and differ-ent datasets, to provide a more robust statistical supportto our finding.

• Wewill investigate the impact of using different content-based recommendation algorithms, such as those basedon Latent-Semantic-Analysis, when adopting low-levelfeatures.

• We will extend the range of visual features extracted andwe will also include audio features.

• We will analyze recommender systems based on low-level features not only in terms of accuracy, but also interms of perceived novelty and diversity, with a set ofonline user studies.

Acknowledgments This work is supported by Telecom Italia S.p.A.,Open Innovation Department, Joint Open Lab S-Cube, Milan.

References

1. Ricci F, Rokach L, Shapira B (2011) Introduction to recommendersystems handbook. In: Ricci F, Rokach L, Shapira B, Kantor PB(eds) Recommender Systems Handbook. Springer, pp 1–35

2. Adomavicius G, Tuzhilin A (2005) Toward the next generation ofrecommender systems: a survey of the state-of-the-art and possibleextensions. IEEE Trans Knowl Data Eng 17(6):734–749

123

112 Y. Deldjoo et al.

3. Burke R (2002) Hybrid recommender systems: Survey andexperiments. User Model User Adapt Interact 12(4):331–370./papers/burke-umuai-ip-2002.pdf

4. Su X, Khoshgoftaar TM (2009) A survey of collaborative filteringtechniques. Adv Artif Intell 2009:4

5. Balabanovic M, Shoham Y (1997) Fab: Content-based, collabora-tive recommendation. Commun ACM 40(3):66–72

6. Lops P, De Gemmis M, Semeraro G (2011) Content-based rec-ommender systems: State of the art and trends. In: Recommendersystems handbook. Springer, pp 73–105

7. Pazzani MJ, Billsus D (2007) The adaptive web. chap. Content-based Recommendation Systems, pp 325–341. Springer-Verlag,Berlin, Heidelberg. http://dl.acm.org/citation.cfm?id=1768197.1768209

8. Zettl H (2002) Essentials of applied media aesthetics. In: Dorai C,Venkatesh S (eds) Media Computing, The Springer InternationalSeries in Video Computing, vol 4. Springer, New York, pp 11–38

9. Cremonesi P, Koren Y, Turrin R (2010) Performance of recom-mender algorithms on top-n recommendation tasks. In: Proceed-ings of the 2010 ACM Conference on Recommender Systems,RecSys 2010, Barcelona, Spain, September 26–30, 2010, pp 39–46

10. Deshpande M, Karypis G (2004) Item-based top-n recommenda-tion algorithms. ACM Trans Inf Syst (TOIS) 22(1):143–177

11. Yang B, Mei T, Hua XS, Yang L, Yang SQ, Li M (2007) Onlinevideo recommendation based on multimodal fusion and relevancefeedback. In: Proceedings of the 6th ACM international conferenceon Image and video retrieval ACM, pp 73–80

12. Zhao X, Li G,WangM, Yuan J, Zha ZJ, Li Z, Chua TS (2011) Inte-grating rich information for video recommendation withmulti-taskrank aggregation. In: Proceedings of the 19th ACM internationalconference on Multimedia ACM, pp. 1521–1524

13. Elahi M, Ricci F, Rubens N (2013) Active learning strategies forrating elicitation in collaborative filtering: a system-wide perspec-tive. ACM Trans Intell Syst Technol (TIST) 5(1):13

14. Billsus D, Pazzani MJ (1999) A hybrid user model for news storyclassification. Springer, New York

15. Kelly D, Teevan J (2003) Implicit feedback for inferring user pref-erence: a bibliography. In: ACM SIGIR Forum ACM, vol 37, pp18–28

16. Musto C, Narducci F, Lops P, Semeraro G, de GemmisM, BarbieriM, Korst J, Pronk V, Clout R (2012) Enhanced semantic tv-showrepresentation for personalized electronic program guides. In: UserModeling, Adaptation, and Personalization. Springer, pp 188–199

17. Degemmis M, Lops P, Semeraro G (2007) A content-collaborativerecommender that exploits wordnet-based user profiles for neigh-borhood formation. User Model User Adapt Interact 17(3):217–255

18. Eirinaki M, Vazirgiannis M, Varlamis I (2003) Sewep: using sitesemantics and a taxonomy to enhance the web personalizationprocess. In: Proceedings of the ninth ACM SIGKDD internationalconference on Knowledge discovery and data mining ACM, pp99–108

19. Magnini B, Strapparava C (2001) Improving user modelling withcontent-based techniques. In: User Modeling 2001. Springer, pp74–83

20. Mooney RJ, Roy L (2000) Content-based book recommendingusing learning for text categorization. In: Proceedings of the fifthACM conference on Digital libraries ACM, pp 195–204

21. Ahn JW, Brusilovsky P, Grady J, He D, Syn SY (2007) Open userprofiles for adaptive news systems: help or harm? In: Proceedingsof the 16th international conference onWorld Wide Web ACM, pp11–20

22. Billsus D, Pazzani MJ (2000) User modeling for adaptive newsaccess. User Model User Adapt Interact 10(2–3):147–180

23. Cantador I, Szomszor M, Alani H, Fernández M, Castells P (2008)Enriching ontological user profiles with tagging history for multi-domain recommendations

24. Middleton SE, Shadbolt NR,DeRoureDC (2004)Ontological userprofiling in recommender systems. ACM Trans Inf Syst (TOIS)22(1):54–88

25. Deldjoo Y, Elahi M, Quadrana M, Cremonesi P (2015) Towardbuilding a content-based video recommendation system basedon low-level features. In: E-Commerce and Web Technologies.Springer

26. Deldjoo Y, Elahi M, Quadrana M, Cremonesi P, Garzotto F (2015)Toward effective movie recommendations based on mise-en-scènefilm styles. In: Proceedings of the 11th Biannual Conference onItalian SIGCHI Chapter ACM, pp 162–165

27. Bogdanov D, Herrera P (2011) How much metadata do we needin music recommendation? A subjective evaluation using prefer-ence sets. In: International Society forMusic Information RetrievalConference (ISMIR), Miami, Florida, USA

28. Bogdanov D, Serrà J, Wack N, Herrera P, Serra X (2011) Unifyinglow-level and high-level music similarity measures. IEEE TransMultimed 13(4):687–701

29. Knees P, Pohle T, Schedl M, Widmer G (2007) A music searchengine built upon audio-based and web-based similarity measures.In: Proceedings of the 30th annual international ACM SIGIR con-ference on Research and development in information retrievalACM, pp 447–454

30. Seyerlehner K, Schedl M, Pohle T, Knees P (2010) Using block-level features for genre classification, tag classification and musicsimilarity estimation. Submission to Audio Music Similarity andRetrieval Task of MIREX 2010

31. Canini L, Benini S, Leonardi R (2013) Affective recommendationof movies based on selected connotative features. IEEE Trans Cir-cuits Syst Video Technol 23(4):636–647

32. Lehinevych T, Kokkinis-Ntrenis N, Siantikos G, Dogruöz AS,Giannakopoulos T, Konstantopoulos S (2014) Discovering simi-larities for content-based recommendation and browsing in multi-media collections. In: IEEE 2014 Tenth International Conferenceon Signal-Image Technology and Internet-Based Systems (SITIS),pp 237–243

33. Rubens N, ElahiM, SugiyamaM,KaplanD (2015) Active learningin recommender systems. In: Recommender Systems Handbook.Springer, pp 809–846

34. Davidson J, Liebald B, Liu J, Nandy P, Van Vleet T, Gargi U,Gupta S, He Y, Lambert M, Livingston B et al (2010) The youtubevideo recommendation system. In: Proceedings of the fourth ACMconference on Recommender systems ACM, pp 293–296

35. Wang Y, Xing C, Zhou L (2006) Video semantic models: surveyand evaluation. Int J Comput Sci Netw Secur 6:10–20

36. HuW, Xie N, Li L, Zeng X, Maybank S (2011) A survey on visualcontent-based video indexing and retrieval. IEEE Trans Syst ManCybern Part C Appl Rev 41(6):797–819

37. Lew MS, Sebe N, Djeraba C, Jain R (2006) Content-based multi-media information retrieval: State of the art and challenges. ACMTrans Multimed Comput Commun Appl (TOMM) 2(1):1–19

38. Rasheed Z, Sheikh Y, Shah M (2005) On the use of computablefeatures for film classification. IEEE Trans Circuits Syst VideoTechnol 15(1):52–64

39. Brezeale D, Cook DJ (2008) Automatic video classification: a sur-vey of the literature. IEEE Trans Syst Man Cybern Part C ApplRev 38(3):416–430

40. Rasheed Z, Shah M (2003) Video categorization using semanticsand semiotics. In: Video mining. Springer, pp 185–217

41. Zhou H, Hermans T, Karandikar AV, Rehg JM (2010) Movie genreclassification via scene categorization. In: Proceedings of the inter-national conference on Multimedia ACM, pp 747–750

123

Content-Based Video Recommendation System Based on Stylistic Visual Features 113

42. Dorai C, Venkatesh S (2001) Computational media aesthetics:Finding meaning beautiful. IEEE Multimed 8(4):10–12

43. Buckland W (2008) What does the statistical style analysis of filminvolve? A review of moving into pictures. more on film history,style, and analysis. Lit Linguist Comput 23(2):219–230

44. Valdez P,MehrabianA (1994) Effects of color on emotions. Journalof Experimental Psychology: General 123(4):394

45. WangHL,CheongLF (2006)Affective understanding infilm. IEEETrans Circuits Syst Video Technol 16(6):689–704

46. Choros K (2009) Video shot selection and content-based scenedetection for automatic classification of tv sports news. In: Tkacz E,Kapczynski A (eds) Internet Technical Development and Applica-tions,Advances in Intelligent andSoftComputing, vol 64. Springer,Berlin Heidelberg, pp 73–80

47. Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of opticalflow techniques. Int J Comput Vis 12(1):43–77

48. Horn BK, Schunck BG (1981) Determining optical flow. In: 1981Technical Symposium East. International Society for Optics andPhotonics, pp 319–331

49. Tkalcic M, Tasic JF (2003) Colour spaces: perceptual, historicaland applicational background. In: EUROCON 2003. Computer asa Tool. The IEEE Region 8, vol 1, pp 304–308

50. Datasets–grouplens. http://grouplens.org/datasets/. Accessed:2015-05-01

51. Youtube. http://www.youtube.com. Accessed: 2015-04-0152. Kohavi R (1995) The power of decision tables. In: 8th European

Conference on Machine Learning. Springer, pp 174–18953. Kohavi R, Sommerfield D (1998) Targeting business users with

decision table classifiers. In: KDD, pp 249–25354. Freitas AA (2014) Comprehensible classification models: a posi-

tion paper. ACM SIGKDD Explor Newsl 15(1):1–1055. Guyon I, Matic N, Vapnik V et al (1996) Discovering informative

patterns and data cleaning

123