notube: ad insertion [compatibility mode]

WP 4Automatic Ad insertionAutomatic Ad insertion

Anne‐Lore MEVEL & Raoul MONNIER (TVN)

ChallengeChallenge

A t ti ll i t Ad ti i i id• Automatically insert Advertising in a video (PIP) at the right time and in the right corner

Low interest in this corner

B

…

Best moment

26‐27 March 2012 NoTube 3rd review 2

Where were we last year ?Where were we last year ?

• Preliminary tests showed a mismatch between the automatic ranking andPreliminary tests showed a mismatch between the automatic ranking and the opinion of field testers

• Some improvements were identified:• Improve the algorithm based on corner saliency

di• Use audio• Use scene cuts analysis• Use global saliency maps analysis (not only corners)• Adjust the Ad visibilityj y

• Open questions: • How letterbox/pillarbox can be used for Ad insertion when present ?

A di i d i h Ad S l i• Audio processing during the Ad: Several options• Mix film sound track with Ad audio • Replace film sound track by Ad Audio • Keep film sound track


Where were we last year ?Where were we last year ?• Suggestions from the last review

F th t ' i t f i it ld b f l t b bl t– From the operator's point of view, it would be useful to be able to insert a specific number of multiple ads in an entire program (e.g. a movie)

• The algorithm allows to find a given number of sequences where the Ad can b i t d Thi b i t f th l ith th t th tbe inserted. This number is a parameter of the algorithm that the operator can use

– Sound is not currently used in this placement, and it definitely should• Sound processing was taken into account for the improvement of the

l ith d d i thalgorithm and was assessed in the survey– Ad choice relating to content is also very important, e.g. to avoid

alcohol advertisements on a driving scene• The metadata describing scenes of the film and the Ad (EgtaMETA) could be

d b l d ( f h f h )used but was not implemented (out of the scope of the project)– Furthermore additional user testing mocking‐up a ‘real’ movie

consumption situation needs to look into general acceptance of the ad insertion concept

• General acceptance was evaluated in the survey


Methodology used to improve the i d i i h lAutomatic Ad insertion technology

M difi ti f th l ith t h• Modification of the algorithm to have a ranking with different criteria

• Survey to assess the automatic Ad insertion technology

• Feedback from this survey was used to try and find a way to avoid proposing sequences where the Ad disturbs the viewer

526‐27 March 2012 NoTube 3rd review

Improvements in the algorithmsImprovements in the algorithms

Corners picture saliency analysis: This is the basis of– Corners picture saliency analysis: This is the basis of the algorithm to isolate N sequences per film

• On each picture, the saliency of each corner is calculated• The algorithm looks after sequences minimizing the integration of corner saliency throughout the duration of the Ad

– 3 other values are calculated for the sequences found by the previous algorithm

• Global picture saliencyGlobal picture saliency• Number of scene cuts• Sound level analysis


Ad insertion technologyAd insertion technology

• The Ad insertion algorithms are based on the succession of• The Ad insertion algorithms are based on the succession of two workflows The first workflow analyses the movie in order to extract ythe metadata describing the n “best sequences” available to insert the Ad and writes these metadata in an XML file.

The second workflow inserts the Ad in the video thanks toThe second workflow inserts the Ad in the video thanks to the metadata produced by the first workflow.


First workflow : AnalysisFirst workflow : Analysis

Input modules to get the movie Video decoding to Video Analysis TS input file

and extract the compressed data

Video decoding to uncompress the data to produce the

metadata

Result of this workflowResult of this workflowXML file describing the n « best sequences »


Demo

Second workflow : Insertion of the Ad

Video decoding to

Input modules to get the movie TS

The initial movie PiP Insertion

Generation of H264 decoding to uncompress the movie

data Video Processing to insert Video Output

mod les to

movie TS input file and extract the compressed

data

video output file

Input modules to get the movie TS

input file and

Video Processing

Offset to delay

to insert Picture in Picture the

AdVideo decoding to uncompress

H264 encoding

modules to create a TS output file

input file and extract the compressed

data

to resize the Ad

the Ad insertion

uncompress the movie

data

The ad resized and delayed


DemoThe ad resized and delayed

Survey to evaluate the algorithmsSurvey to evaluate the algorithms

30 d f l ti• 30 sequences were prepared for evaluation– 6 films– 5 sequences per film

• They were uploaded on YouTube • Questionnaires (Google docs) were prepared and sent to partnersp


Results of the survey (1/5)Results of the survey (1/5)

F 20 t 26• From 20 to 26 answers• Few answers, but results are consistent

– Standard deviation of sequence ranking is between 0.7 and 1.3 (ranking is between 1 and 5)

Stand Dev Seq1 Seq2 Seq3 Seq4 Seq5300 1,1 1,3 1,0 1,1 1,0Doc 0,7 0,7 0,7 0,7 1,0Drama 0,9 0,8 1,1 1,1 1,1

Sequences ranking

, , , , ,Sherlock 0,7 0,7 0,7 0,9 0,9StayIn Alive 1,1 0,9 0,9 1,0 1,07years 1,1 0,7 0,9 1,0 1,2



• Good results• Good results – Average mark: 3,6 (3 = acceptable, 4 = good)– Only 3 sequences among the 30 sequences (10%) were considered as

“unacceptable” (mark ≤ 2.5)– 83% of the sequences were, at least, “acceptable” (mark ≥ 3)– 50 % of the sequences were judged “good” to “very good”

(mark ≥ 3.9)

Average Seq1 Seq2 Seq3 Seq4 Seq5300 2,4 2,4 3,5 3,7 3,3

Sequences ranking

300 2,4 2,4 3,5 3,7 3,3Doc 4,6 4,4 4,2 4,5 3,8Drama 3,1 4,0 3,1 3,1 3,9Sherlock 4,3 3,9 4,1 2,9 3,5StayIn Alive 2,8 3,9 4,0 4,2 3,7


7years 4,0 4,0 3,9 3,8 2,5


• Very dependant on the film• Very dependant on the film

80%90%100%

Does the way the Ad is inserted disturb you to understand what is happening ?

80%90%100%

126

Does the way the Ad is inserted disturb you to understand what is happening ?

0%10%20%30%40%50%60%70%80%

"Doc", "Doc", "Doc", "Doc", "Doc",

0 0 0 1 1

20 20 20 19 19 No

Yes

0%10%20%30%40%50%60%70%80%

"300", "300", "300", "300", "300",

1420

4 6 4

12

22 20 22No

Yes

• Most people are disturbed because, sometimes, the Ad hides part of the face of people (hairs)

Seq 1 Seq 2 Seq 3 Seq 4 Seq 5 Seq 1 Seq 2 Seq 3 Seq 4 Seq 5

300 Doc Drama Sherlock Staying 7 years All filmsBottom left 23% 3% 8% 19% 2% 4% 39%Bottom Right 15% 6% 10% 1% 5% 8% 29%Top left 6% 1% 2% 2% 0% 3% 10%Top Right 5% 0% 1% 2% 22% 6% 22%



A l d t t thi t h l ?• Are people ready to accept this technology ?Yes 52%No 33%Don't know 14%



R ki i b th i i l l ith• Ranking given by the original algorithm (corner saliency only) is not always enough to t “ d”get “good” sequences

Average Seq1 Seq2 Seq3 Seq4 Seq5300 2 4 2 4 3 5 3 7 3 3

Sequences ranking

300 2,4 2,4 3,5 3,7 3,3Doc 4,6 4,4 4,2 4,5 3,8Drama 3,1 4,0 3,1 3,1 3,9Sherlock 4,3 3,9 4,1 2,9 3,5StayIn Alive 2 8 3 9 4 0 4 2 3 7

• Other criteria were studied to improve the l

StayIn Alive 2,8 3,9 4,0 4,2 3,77years 4,0 4,0 3,9 3,8 2,5

results26‐27 March 2012 NoTube 3rd review 18

Use of Global SaliencyUse of Global Saliency

• The saliency is calculated on the full picture and• The saliency is calculated on the full picture and integrated throughout the Ad duration

rGlobal sail.300 39 21 17 13 11 -0,7Doc 40 49 46 27 29 0,3Drama 31 32 49 31 21 -0,5Sherlock 14 10 11 11 17 0,0

• The correlation coefficient (Global saliency versus

StayIn Alive 21 20 21 39 39 0,47years 27 24 28 30 30 -0,6

• The correlation coefficient (Global saliency versus survey ranking) varies too much (+/‐) to use the Global Saliency to improve the resultsy p


Use of Scene CutsUse of Scene Cuts

• The number of scene cuts are calculated• The number of scene cuts are calculated throughout the Ad duration

300 5 11 2 1 6 -0 8Scene cuts

300 5 11 2 1 6 -0,8Doc 6 2 13 7 6 -0,1Drama 6 5 12 5 2 -0,6Sherlock 2 6 3 12 5 -0,9StayIn Alive 5 4 5 2 2 0 5

• The correlation coefficient (Nb of scene cuts ki ) i l ti d th

StayIn Alive 5 4 5 2 2 -0,57years 1 9 5 3 7 -0,3

versus survey ranking) is always negative and the number of scene cuts could be used to improve the results


Use of Sound levelUse of Sound level

• The LUFS is calculated throughout the Ad• The LUFS is calculated throughout the Ad duration

LUFS300 7,79E-04 1,26E-04 5,10E-04 1,44E-04 1,14E-04 -0,3Doc 4,94E-03 4,76E-03 5,08E-03 5,67E-03 4,71E-03 0,5Drama 9,48E-03 2,15E-03 4,50E-02 8,91E-02 3,56E-03 -0,7Sherlock 1,05E-02 7,72E-03 8,23E-03 3,21E-03 1,20E-03 0,87years 2 33E 04 2 56E 03 5 32E 04 4 75E 04 9 42E 04 0 1

• The correlation coefficient (LUFS versus survey

7years 2,33E-04 2,56E-03 5,32E-04 4,75E-04 9,42E-04 0,1

• The correlation coefficient (LUFS versus survey ranking) varies too much (+/‐) to use the LUFS to improve the resultsp


ConclusionsConclusions• Conclusions

Th f li i d lt The use of corner saliency gives good results Global saliency and sound level analysis doesn’t give results which

could improve the algorithm The number of scene cuts may be used to get slightly better results e u be o sce e cu s ay be used o ge s g y be e esu s

but we are lacking enough experimental data to tune the algorithm The main remaining problem is that the algorithm isn’t able to detect

facesP l f f t t di• Proposals for future studies– Add a face detection algorithm to discard sequences where the Ad

would hide faces– Take into account the number of scene cutsTake into account the number of scene cuts– Carry additional field tests to fine tune the algorithm

• More data (more sequences/films) • Higher dynamic (good and bad sequences)

M l• More people
