IEEE COMMUNICATIONS LETTERS, VOL. 18, NO. 5, MAY 2014 781

Chasing the Most Popular Video: An Evolutionary Video Clip SelectionJianting Yue, Bo Yang, Cailian Chen, and Xinping Guan

Abstract—Video is experiencing an unprecedented develop-ment in the recent decade. Based on content reuse, multiplevideo users could share same video clips. Also, as a commonscene, video users are always chasing popular video clips. Inthis paper, we apply video popularity as a spontaneous label todistinguish different video clips. Moreover, we associate videopopularity with video quality, thus facilitating video users inpursuing video clips with both satisfying video popularity andvideo quality. We first apply evolutionary game to model thevideo clip selection process, and then propose a Q-learning basedalgorithm.

Index Terms—Video, popularity, evolutionary game, Q-learning.

I. INTRODUCTION

V IDEO applications are rapidly developing these years.Thanks to video content providers like YouTube, users

are able to get access to miscellaneous kinds of video clips.In order to satisfy such dramatic demands, a range of newtechnologies including cloud streaming [1] and femtocaching[2] have emerged, which in turn foster the video services.

Compared with other mobile content types, video contenthas its own distinctive characteristics. Firstly, based on contentreuse, lots of people are able to share a same video content,which differs from voice content and message content thatmerely serve the individual. Secondly, mobile users constantlyprefer video clips with high quality. To be more explicit, if avideo clip experiences too much distortion or buffering time,there would be a great possibility that the disappointed userswitches to other video clips that could play smoothly. Thirdly,different video contents have various popularity which couldbe deemed as a spontaneous label to distinguish video clips.For instance, a hot movie would definitely attract a greaternumber of clicks than an old documentary, and in this sensethe notion of video popularity reflects such a trend. Lastly,video contents, particularly High Definition videos which need

Manuscript received January 15, 2014. The associate editor coordinatingthe review of this letter and approving it for publication was Y.-D. Lin.

J. Yue is with the Department of Electrical and Computer Engineering, Mc-Master University, Hamilton, Ontario, Canada (e-mail: yuej3@mcmaster.ca).This work was mainly done when he was with the Department of ElectronicEngineering, Shanghai Jiao Tong University, Shanghai, China.

B. Yang, C. Chen, and X. Guan are with the Department of Automation,and the Key Laboratory of System Control and Information Processing,Ministry of Education, Shanghai Jiao Tong University, Shanghai, China (e-mail: {bo.yang, cailianchen, xpguan}@sjtu.edu.cn). B. Yang and X. Guan arealso with the Cyber Joint Innovation Center, Hangzhou, China.

This work was supported partially by the National Basic Research Projectof China under Grant 2010CB731803, the NSFC under Grants 61221003,61174127, 61104033, 61273181, 61203104, 61374107 and 61301223, theResearch Found for the Doctoral Program of Higher Education underGrants 20110073130005, 20110073120025, and 20121333120012; the Sci-ence and Technology Commission of Shanghai Municipal, China under grants13QA1401900 and 13ZR1421800 and the Cyber Joint Innovation Center,Hangzhou, China. This work was also sponsored by Huawei InnovationResearch Program with no. YB2014030021.

Digital Object Identifier 10.1109/LCOMM.2014.031414.140097

much higher bit rates, require more wireless resources forstreaming.

In literature, [3] is among those early papers referringto video popularity models. Different from its considerationabout pre-acquired data and hierarchical servers, we investi-gate how mobile users select video clips based on real-timeinformation in a dynamic manner. Scores of recent worksare investigating video contents as well. For example, [4]explicitly presents the popularity factor, and [5] proposes anenergy-spectrum-aware scheduling scheme for video stream-ing. However, most of previous works only consider “popularvideo” that the wireless network propagates, while the problemof video’s service and business model has been addressed infew papers such as [6] and [7].

As a matter of fact, because mobile video users (MVUs)are able to switch among different video clips and rationallyselect the video clip with a satisfying utility, their behaviorneeds to be taken into consideration as well. Meanwhile, aswe would elaborate in Section II that a more popular video clipindicates a lower video quality, it is of great significance tocarefully balance the MVUs’ requirements of video popularityand video quality. In this sense, we investigate a situationwhere MVUs switch among multiple video clips.

The main contributions of our work are as follows:• Based on the content reuse of video, we discuss the

behavior of MVUs, which mimics the video clip selectionprocess in practice.

• We apply popularity to label the video and then seam-lessly connect it with video quality. Thus the utility ofevery MVU is depicted in terms of video popularity.

• We use replicator dynamics in evolutionary game tomodel the interaction of MVUs, therefore every MVUs issatisfied with the utility obtained from the accessed videoclip. Also, Q-learning is used to propose an evolutionalgorithm without too much information exchange.

The rest of this paper is organized as follows. In Section II,we give our system model and formulate the problem. Thenin Section III we present detailed solutions to the problem.Some simulation results are discussed in Section IV, and wefinally conclude our work in Section V.

II. SYSTEM MODEL AND PROBLEM FORMULATION

A. System Model

We consider a downlink model consisting of a base station(BS) and N MVUs within a certain range of area. MVUswould request video clips from the BS. Meanwhile, we assumethat there are overall M video clips available. Obviously, eventhough there are a number of video clips for MVUs to choose,each MVU could watch only a single one video clip at onetime. Let Xm denote the amount of MVUs selecting videoclip m ∈ {1, 2, . . . ,M}, and we have

∑Mm=1 Xm = N . Then

1089-7798/14$31.00 c© 2014 IEEE

782 IEEE COMMUNICATIONS LETTERS, VOL. 18, NO. 5, MAY 2014

define xm � Xm

N , ∀m representing the proportion of MVUsrequesting video clip m. Also note that xm ∈ [0, 1].

At this moment, it is worthwhile to point out here that{xm, ∀m} pinpoints the popularity of video clips. Apparently,a video clip becomes increasingly popular when a greaternumber of MVUs are willing to get access to it (i.e., xm

rises), and vice versa. In this sense, {xm} exactly reflects theextent of popularity as a spontaneous label.

B. Utility of MVU

As discussed in the previous section, MVUs care for bothvideo popularity and video quality. If either element frustratesthe MVU, the MVU would then switch to another video clip.In this sense, we denote the utility function of MVU watchingvideo clip m as:

um = P (xm)[1− 1

1+eαm(qm−βm)

]− μm, ∀m, (1)

where P (xm) is the unit utility gain with respect to the popu-larity xm, αm describes the sensitivity toward the increaseof video quality, βm represents the reserved video qualityrequirement of MVUs, qm stands for the quality of video clipm, and μm denotes the price that each MVU pays for itsaccess to video clip m. In this paper we simply presume that

P (xm) = pmxρmm , ∀m (2)

where pm and ρm are positive parameters, indicating thataccessing a popular video clip is of great help in raising utility.

Notice that Sigmoid function in (1) performs the role ofchurn rate depicting the satisfaction of users with regard toservice quality [8]. Moreover, although different video clipsmight have different prices depending on factors such ascopyright fee and producing cost, we assume that the accessingprice μm is fixed and open to all MVUs.

On the other hand, considering that multiple MVUs requestvideo clips from the BS, it is reasonable to denote the qualityof each video clip based on the per-MVU serving bandwidthas well as the proportion of MVUs experiencing smooth play[9]. According to [10], there should have:

qm = γm(

Bm

xmNrm

)dm (3)

where γm and dm are adjustable positive parameters indicatingthe improvement of video quality, Bm is the bandwidth thatthe BS allocates for video clip m [11], and rm denotes thevideo streaming rate. Without loss of generality, consideringthat there is rm = Bm log(1+νm) for video clip m where νmis the Signal to Interference plus Noise Ratio, we assume thatthere is a positive correlation between rm and Bm providedwith a fixed νm, such that Bm

rmequals to some constant value.

Combining (1), (2) and (3), we would have:

um = pmxρmm

[1− 1

1 + eαm(γm( BmxmNrm

)dm−βm)

]− μm. (4)

As a consequence, it is seen in (4) that xm, representingboth the proportion of MVUs and the video popularity simul-taneously, influences the utility um dynamically. Thus, MVUsthat choose the same video clip share the same utility. Inaddition, we make um = 0 when xm = 0 as the specialcase.

Besides, all information is allowed to be updated dynami-cally such that variable-population cases [12] could be dealtwith. To be more explicit, if new video clips enter the system,or old ones leave the system, or transmission conditionschange, the BS could reformulate the problem and cope withnew scenarios by updating parameters and variables in (4).Then MVUs are able to make decisions accordingly. Othersolutions include formulating the problem in a repeated gamemodel, but it is out of our main scope and we do not discussit in detail.

Till now, we have derived the utility function in terms ofthe popularity and the quality of video clip. MVUs could thenswitch among multiple video clips in order to pursue a higherutility according to (4). Towards this end, we would solvethe problem by analyzing (4) mainly, which covers lots ofvariables in a complicated manner.

III. EVOLUTION OF MVUS

A. Evolution Algorithm

For MVUs, there are totally M choices of video clip, andthey could change their decisions in order to pursue a higherutility. In this sense, such a selection game could be modeledby replicator dynamics in evolutionary game.

In an evolutionary game, all of the players are groupedinto a population. Each player would independently choosea certain strategy, and tend to switch to the strategy providinga higher utility. The solution of evolutionary game is the evolu-tionary equilibrium where there is no change in proportion ofplayers choosing different strategies [13]. Typically, replicatordynamics are used to model the process of strategy replication:

xm = σxm(um − u), ∀m (5)

where σ is a constant step size representing the evolutionaryspeed, um denotes the utility of the MVUs getting access tovideo clip m, and u represents the average utility of all MVUs.Obviously, (5) corresponds to the intuition that MVUs wouldselect video clip m if getting access to it triggers a greaterutility than the average. In addition, x = {x1, x2, . . . , xM}denotes the population state.

Through the evolutionary process, MVUs’ choices wouldultimately reach a stable state defined as evolutionary stablestrategy (ESS).

Definition 1: A strategy s∗ is an ESS, iff, for ∀s �= s∗,• Equilibrium condition: f(s, s∗) ≤ f(s∗, s∗), and

Algorithm 1 Evolution Algorithm1: Each MVU selects to get access to a video clip in random.2: The utility of MVUs accessing video clip m is calculated

according to (4).3: The average utility is derived by u =

∑Mm=1 xmum.

4: For a MVU accessing video clip m, if its utility fails toreach the average utility, i.e., um < u, it would randomlyswitch to another video clip with the probability u−um

u .If it determines to change its selection, it would choosein random among the video clips satisfying um′ > um

where m′ �= m.5: Repeat from Step 2 to Step 4.

YUE et al.: CHASING THE MOST POPULAR VIDEO: AN EVOLUTIONARY VIDEO CLIP SELECTION 783

• Stability condition: if f(s, s∗) = f(s∗, s∗), f(s, s) <f(s∗, s).

When reaching a stable state after evolving over time, thedifferential equation should equal to 0. Then none of theMVUs would like to change its choice since its utility equalsto the average. In this way, we could acquire ESS by solving:

xm = 0, ∀m. (6)

Since MVUs could select all of the video clips, every videoclip’s utility as well as the proportion of accessed MVU isavailable to all MVUs. Hence, we could derive the evolutionalgorithm as described in Algorithm 1. Also, the existence anduniqueness of the solution are clarified in Theorem 1.

Theorem 1: For dynamical system xm = σxm(um −u), ∀m with initial state condition x(0) = x0, there exists aunique solution x if each element is a measurable function on[0,∞).

Proof: To start with, substitute um and u in (5) and thus

g(xm) � xm = σ(xmGm(xm)−xm

∑Mn=1 xnGn(xn)), (7)

where Gm(xm) = Gm(xm) − μm, and Gm(xm) =pmxρm

m

(1− 1

1+eαm(γm(

BmxmNrm

)dm−βm)

).

For every fixed case, the partial derivative of g(xm) withrespect to xm is continuous, and if each element is measurableon [0,∞), then g(xm) is measurable.

Let y1, y2 ∈ (0, 1] and y1 > y2. Notice here that we excludethe cases where y1 = 0 or y2 = 0 which are deemed as specialcases as claimed in Section II-B, in the sense that both of themlead to zero utility and leave no influence on the system.

Hence, for the first term in (7), we could derive (8)shown at the top of next page, which further indicatesthat |y1Gm(y1)− y2Gm(y2)| ≤ V0|y1 − y2|, where ymax =max{ym �= 0, ∀m}, ymin = min{ym �= 0, ∀m}, zmax =

eαm(γm( Bm

yminNrm)dm−βm), zmin = eαm(γm( Bm

ymaxNrm)dm−βm),

and V0 = pm(ρm+1)(ymaxzmax−yminzmin)ymax−ymin

+ μm.Meanwhile, for the second term in (7), it is obvious that,

for ∀n, |y1xnGn(xn) − y2xnGn(xn)| ≤ Vn|y1 − y2| whereVn = max{xnGn(xn) �= 0}. In this sense, we have

|g(y1)− g(y2)| ≤∑M

n=0 σVn|y1 − y2|,which implies that g(xm) satisfies the Lipschitz condition [14]and further proves the theorem.

Algorithm 2 Q-Learning Based Evolution Algorithm

1: Initialization: Set Qm(t) = 0, ∀m.2: Repeat

for MVUs accessing to different video clipsif rand() < ε

choose video clip m in randomelse

choose video clip m = argmaxmQm(t)end ifcalculate the utility um, and update

Qm(t+ 1) = (1− δ)Qm(t) + δ[um + ηmaxm Qm(t)]

end for

5 10 15 20 25 300.01

0.02

0.03

0.04

Iteration(a) Proportion of MVU choosing certain video clip

Pro

port

ion

of M

VU

5 10 15 20 25 3038

40

42

44

46

48

Iteration(b) Utility of MVU choosing certain video clip

Util

ity o

f MV

U

Fig. 1. Proportion and utility of MVU with Algorithm 1.

5 10 15 20 25 300

0.05

0.1

0.15

Iteration(a) Proportion of MVU choosing certain video clip

Pro

port

ion

of M

VU

5 10 15 20 25 3030

35

40

45

50

Iteration(b) Utility of MVU choosing certain video clip

Util

ity o

f MV

U

Fig. 2. Proportion and utility of MVU with Algorithm 2.

Besides, the evolutionary equilibrium is stable when eigen-values of Jacobian matrix of xm have negative real parts [13].

B. Q-Learning Based Evolution Algorithm

As specified in Algorithm 1, MVUs need to calculate theaverage utility, which means that the proportion of MVUsgetting access to each video clip needs to be known. In order toavoid too much information exchange, we apply reinforcementlearning here, which facilitates each MVU the ability to workwithout knowing the strategies of other MVUs.

Specifically, we introduce Q-learning approach [15] asthe useful tool. On this line, each MVU maintains Q-valuerepresenting the knowledge of different video clips and thenmakes decision according to Q-value. Let Qm(t) denote the Q-value (i.e., the corresponding utility) of MVUs accessing videoclip m at time t. Hence we could derive a Q-learning basedevolution algorithm in Algorithm 2, wherein ε determines theprobability of exploiting the video clip and δ ∈ (0, 1) repre-sents the learning rate which controls the speed of adjusting Q-value [16]. It is apparent that the expected Q-value Qm(t+1)

784 IEEE COMMUNICATIONS LETTERS, VOL. 18, NO. 5, MAY 2014

|y1Gm(y1)− y2Gm(y2)| = pm[yρm+1

1 (1 + eαm(γm(

Bmy2Nrm

)dm−βm))− yρm+1

2 (1 + eαm(γm(

Bmy1Nrm

)dm−βm))

(1 + eαm(γm( Bm

y1Nrm)dm−βm)

)(1 + eαm(γm( Bm

y2Nrm)dm−βm)

)− (yρm+1

1 − yρm+12 )

]

≤ pm[yρm+11 e

αm(γm(Bm

y2Nrm)dm−βm) − yρm+1

2 eαm(γm(

Bmy1Nrm

)dm−βm)] ≤ pmymaxzmax − yminzmin

ymax − ymin(yρm+1

1 − yρm+12 ) (8)

5 10 15 20 25 3030

35

40

45

Ave

rage

util

ity o

f MV

U

Iteration

Algorithm 1Algorithm 2

Fig. 3. Average utility of MVU.

is updated based on the previous value as well as the bestaction maxm Qm(t).

IV. SIMULATION RESULTS

We consider the BS provides 50 video clips to 400 MVUs,and MVUs switch among these video clips. For better illus-tration, we randomly choose 3 out of 50 groups and presentcorresponding curves in Fig. 1 and Fig. 2. Although thelegends are not explicitly given out over there, we claim herethat the curve with the same symbol indicates the same groupin all 4 subfigures.

Fig. 1 and Fig. 2 show the proportion and the achievedutility of MVUs accessing different video clips during theevolution process, using Algorithm 1 in Fig. 1 and Algorithm2 in Fig. 2, respectively. To be more explicit, all curvesabout proportion start at the initial point 0.02. However, afterseveral iterations, they change in different directions, andultimately reach some stable values. At the same time, thosecurves about utility finally converge to the average. It is seenthat two algorithms trigger various outcomes as a result ofthe difference in information exchange. At this point, it isworthwhile to point out that to take the advantage of bothpopularity and utility simultaneously is never an easy task, asshown in both figures that the utilities achieved from all videoclips are identical in the end regardless of popularity.

The average utility of MVU is illustrated in Fig. 3. Wecould find that both algorithms achieve better utility at last.However, there are also some notable differences. Apparentlythe curve of Algorithm 1 gradually increases until reaching thefinal value, in that MVUs are always pursuing video clips thatcould bring them greater utility. In contrast, there are a coupleof fluctuations in the curve of Algorithm 2, since MVUs makedecisions independently with little information exchange andeven randomly sometimes. Due to the same reason, Algorithm2 fails to perform as well as Algorithm 1 in achieving anaverage utility at the same level. In addition, all three figuresindicate that both evolution algorithms are able to reach stablestates quickly within a small number of iterations.

V. CONCLUDING REMARKS

In this paper, we mainly discuss the behavior of MVUs.Considering the general scene in practice where MVUs switch

among multiple video clips, we model their behaviors usingreplicator dynamics in evolutionary game, and subsequentlyderive the evolution algorithm. In particular, we have videopopularity as a spontaneous label to distinguish different videoclips, and then associate it with video quality. Moreover,we also take the advantage of Q-learning based evolutionalgorithm in order to avoid too much information exchange.In this way, we deal with the video clip selection process byconsidering a number of relevant influential factors.

REFERENCES

[1] G. Lawton, “Cloud streaming brings video to mobile devices,” Com-puter, vol. 15, no. 2, pp. 14–16, Feb. 2012.

[2] N. Golrezaei, K. Shanmugam, A. G. Dimakis, A. F. Molisch, andG. Caire, “FemtoCaching: wireless video content delivery throughdistributed caching helpers,” in Proc. 2012 IEEE INFOCOM, pp. 1107–1115.

[3] C. Griwodz, M. Bar, and L. C. Wolf, “Long-term movie popularitymodels in video-on-demand systems: or the life of an on-demandmovie,” in Proc. 1997 ACM International Conference on Multimedia,pp. 349–357.

[4] A. Fiandrotti, J. Chakareski, and P. Frossard, “Popularity-aware rateallocation in multi-view video,” in Proc. 2010 Conference on VisualCommunications and Image Processing.

[5] L. Zhou, R. Q. Hu, Y. Qian, and H.-H. Chen, “Energy-spectrumefficiency tradeoff for video streaming over mobile ad hoc networks,”IEEE J. Sel. Areas Commun., vol. 31, no. 5, pp. 981–991, May 2013.

[6] J. Huang, Z. Li, M. Chiang, and A. K. Katsaggelos, “Joint source adap-tation and resource allocation for multi-user wireless video streaming,”IEEE Trans. Circuits and Systems for Video Technol., vol. 18, no. 5, pp.582–595, May 2008.

[7] W. S. Lin and K. J. R. Liu, “Game-theoretic pricing for video streamingin mobile networks,” IEEE Trans. Image Process., vol. 21, no. 5, pp.2667–2680, May 2012.

[8] Y. Chen, J. Zhang, and Q. Zhang, “Utility-aware refunding frameworkfor hybrid access femtocell network,” IEEE Trans. Wireless Commun.,vol. 11, no. 5, pp. 1688–1697, May 2012.

[9] J. Dai, F. Liu, B. Li, B. Li, and J. Liu, “Collaborative caching inwireless video streaming through resource auctions,” IEEE J. Sel. AreasCommun., vol. 30, no. 2, pp. 458–466, Feb. 2012.

[10] C. Wu, B. Li, and S. Zhao, “Diagnosing network-wide P2P livestreaming inefficiencies,” in Proc. 2009 IEEE INFOCOM, pp. 2731–2735.

[11] R. Haddad, M. McGarry, and P. Seeling, “Video bandwidth forecasting,”IEEE Commun. Surveys & Tutorials, vol. 15, no. 4, pp. 1803–1818, Apr.2013.

[12] Z. Zhang and H. Zhang, “A variable-population evolutionary gamemodel for resource allocation in cooperative cognitive relay networks,”IEEE Commun. Lett., vol. 17, no. 2, pp. 361–364, Jan. 2013.

[13] D. Niyato, E. Hossain, and Z. Han, “Dynamics of multiple-seller andmultiple-buyer spectrum trading in cognitive radio networks: a game-theoretic modeling approach,” IEEE Trans. Mobile Computing, vol. 8,no. 8, pp. 1009–1022, Aug. 2009.

[14] J. Engwerda, LQ Dynamic Optimization and Differential Games. Wiley,2005.

[15] X. Chen, H. Zhang, T. Chen, and M. Lasanen, “Improving energyefficiency in green femtocell networks: a hierarchical reinforcementlearning framework,” in Proc. 2013 IEEE International Conference onCommunications, pp. 2241–2245.

[16] D. Niyato and E. Hossain, “Dynamics of network selection in heteroge-neous wireless networks: an evolutionary game approach,” IEEE Trans.Veh. Technol., vol. 58, no. 4, pp. 2008–2017, May 2009.