fiat-ifta 2013 - television linked to the web: the case for audiovisual archives

www.linkedtv.eu

Johan Oomen and Lotte Belice Baltussen@johanoomen and @lottebeliceNetherlands Institute for Sound and Vision

Television linked to the web: the case for audiovisual archives

FIAT/IFTA | Dubai, United Arab Emirates – 27 October 2013

www.linkedtv.eu

2

multimediacontentanalysis

new distributionplatforms

lots of archive content

www.linkedtv.eu

2

multimediacontentanalysis

new distributionplatforms

lots of archive content

where the magic happens!

www.linkedtv.eu

Where the magic happens

3

multimediacontentanalysis

new distributionplatforms

lots of archive content

Audiovisual broadcasts

Audiovisual broadcasts

Audiovisual broadcasts

Professional annotation

Audiovisual broadcasts

Professional annotation

Search engine

Audiovisual broadcasts

Professional annotation

Search engine

Television makers

Audiovisual broadcasts

Professional annotation

Search engine

Television makers

Academics

Audiovisual broadcasts

Professional annotation

Search engine

Television makers

General public

Academics

General public

Academics

Audiovisual broadcasts

Professional annotation

Search engine

Television makers

Machine analysis Data gathering

WikisTwitter

Subtitles

Importer

NewsCrowdsourcing

multimedia content analysis

Data gathering

Data Gathering:Pros and Cons

Data Gathering:Pros and Cons

• Large-scale data for some programs

• Access to opinions about television

• Puts programs in larger national context

• Affordable

• Noisy data (sometimes)

• Not created for the archive

• Unclear how the data can be used (sometimes)

multimedia content analysis

Machine Analysis

Low-level features

Multimediacontent analysis

multimedia content analysis

Machine Analysis

Low-level features

Multimediacontent analysis

today sniper fire disrupted the funeral of an eleven year old ethnic albanian boy he was killed yesterday while while chopping wood his family blames serb police before his death louisiana state police now say six workers were killed after a natural gas well exploded and caught fire about forty five miles east of shreveport four others were injured in yesterday's blast a police spokesman says the derek started to melt in the intense heat the

Audio transcripts Concept detectors

Speaker recognition Face recognition

Machine Analysis: Pros and Cons

Machine Analysis: Pros and Cons

• Extremely detailed, millisecond-level descriptions

• Directly describe video content

• Affordable

• Large scale!

• Noisy

• Limited to what the machine “knows”

• No summarisation of what is important

• Confusing for users

www.linkedtv.eu

10

Media fragments

linkedtv.eu

www.linkedtv.eu

Where the magic happens

11

multimediacontentanalysis

new distributionplatforms

lots of archive content

www.linkedtv.eu

12

Context guestimates

linkedtv.eu

thousands of hrs of new television content produced byDutch public broadcasters every year

Hours of archival content of Sound and Vision used in this new content

www.linkedtv.eu

How do broadcast professionals use archive content?§ Query log analysis.* (50k user

sessions, 264k queries)

13

§ 33% programs§ 17% stories, and fragments§ 50% audiovisual fragments with a start and end time

specified by the users themselves.

* Paper: Search Behavior of Media Professionals at an Audiovisual Archive: ATransaction Log Analysis. By Bouke Huurnink

0 500

1000 1500 2000 2500 3000 3500 4000

0-11-22-33-44-55-66-77-88-99-1010-1111-1212-1313-1414-1515-1616-1717-1818-1919-2020-2121-2222-2323-2424-2525-2626-2727-2828-2929-30>30

Num

ber o

f ord

ers

Order duration in minutes

broadcast orderstory order

fragment order

www.linkedtv.eu

Preference for new material

§ 46 % of the orders are for footage broadcast more than 1 year before the order date

§ 12% of orders are for material that is over 10 years old.

§ => a lot of high quality content is not used.

14

www.linkedtv.eu

15

Context guestimates

linkedtv.eu

thousands of hrs of new television content produced byDutch public broadcasters every year

Hours of archival content of Sound and Vision used in this new content

www.linkedtv.eu

16

Context – the potential

linkedtv.eu

Total av-collection Sound and Vision750k of hours (& growing daily)

www.linkedtv.eu

17

Context – the potential: on-demand personalised services

linkedtv.eu

Total av-collection Sound and Vision750k hours (& growing daily)

www.linkedtv.eu

Where the magic happens

18

multimediacontentanalysis

new distributionplatforms

lots of archive content

www.linkedtv.eu

19

§ 86% of TV viewers surf the Web at the same time (Yahoo!, USA)

§ 88% of USA consumers use mobile as second screen (Business Insider, May 2013)

§ Manual efforts to link TV program to additional content (Shazam, IntoNow)

Web and TV exist in parallel

linkedtv.eu

www.linkedtv.eu

Samsung advert

20 http://www.youtube.com/watch?v=UwmoBmlkaGQ

www.linkedtv.eu

21

Examples – BBC Interactive Video Player

linkedtv.eu

www.linkedtv.eu

22

Examples – Hulu Face Match

linkedtv.eu

www.linkedtv.eu

23

LinkedTV: interweaving Web and TVinto a single experience

The LinkedTV vision

linkedtv.eu

§ Mixing lean back and lean forward§ how to maintain immersion while

removing fourth wall illusion?§ will this work across genres and

audiences?§ Does second screen help or hinder?

www.linkedtv.eu

24 linkedtv.eu

Research Organisations

SME Technology Providers

User Interface and Scenario Partners

Technical coordinator Market Studies

www.linkedtv.eu

25

LinkedTV – Use cases

Interactive News

§ Professional news content produced by RBB for general audience

§ local news show rbb Aktuell§ People, locations and events

can be linked to previous events.

Hyperlinked Documentary: Antiques Interactive

§ Program watched predominantly by 50+ers

§ AVRO:Tussen Kunst en Kitsch (TKK)

§ Objects, locations and experts provide anchors to related materials (targets).

www.linkedtv.eu

26

Use case – Antiques InteractiveBert and Anne

“I like to relax and sit back when I watch Antiques Road Show. It’s

always exciting when something’s worth a lot. In some cases, I like to

look up extra info about an object or art style after the show on the PC,

but if something doesn’t interest me, I’ll just read a magazine.”

“As an antiques shop owner, by work is also my hobby. I’m always looking up

information about art objects on my PC or tablet. I like watching my colleagues

talking about are on Antiques Road Show and I use my tablet to bookmark specific objects or other information that I like. I

also do this for Anne, and send her overviews.”

www.linkedtv.eu

27

How LinkedTV helps Bert and Anne

linkedtv.eu

Video Object and word detection

Presentation engine

Selecton of relatedconcepts

Selection of relatedcontent

Connection to concepts

www.linkedtv.eu

27

How LinkedTV helps Bert and Anne

linkedtv.eu

“...painting of Jan Sluijters....”

Video Object and word detection

Presentation engine

Selecton of relatedconcepts

Selection of relatedcontent

Connection to concepts

www.linkedtv.eu

27

dbpedia.org/resource/Jan_Sluyters

How LinkedTV helps Bert and Anne

linkedtv.eu

“...painting of Jan Sluijters....”

Video Object and word detection

Presentation engine

Selecton of relatedconcepts

Selection of relatedcontent

Connection to concepts

www.linkedtv.eu

27

dbpedia.org/resource/Jan_Sluyters

How LinkedTV helps Bert and Anne

linkedtv.eu

Paintings by Jan Sluijters

“...painting of Jan Sluijters....”

Video Object and word detection

Presentation engine

Selecton of relatedconcepts

Selection of relatedcontent

Connection to concepts

www.linkedtv.eu

27

dbpedia.org/resource/Jan_Sluyters

How LinkedTV helps Bert and Anne

linkedtv.eu

Paintings by Jan Sluijters

“...painting of Jan Sluijters....”

Video Object and word detection

Presentation engine

Selecton of relatedconcepts

Selection of relatedcontent

Connection to concepts

www.linkedtv.eu

27

dbpedia.org/resource/Jan_Sluyters

How LinkedTV helps Bert and Anne

linkedtv.eu

Paintings by Jan Sluijters

“...painting of Jan Sluijters....”

Video Object and word detection

Presentation engine

Selecton of relatedconcepts

Selection of relatedcontent

Connection to concepts

www.linkedtv.eu

28

From one screen...

http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2

www.linkedtv.eu

28

From one screen...

http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2

www.linkedtv.eu

28

From one screen...

http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2

www.linkedtv.eu

§

29

www.linkedtv.eu

30

...to paper prototype...

linkedtv.eu

www.linkedtv.eu

31

www.linkedtv.eu

32

...to second screen demo

linkedtv.eu

www.linkedtv.eu

33

dbpedia.org/resource/Jan_Sluyters

How do we give broadcasters some control?

linkedtv.eu

Paintings by Jan Sluijters

“...painting of Jan Sluijters....”

Hypervideo analysis

Presentation engine

Named Entity Recognition

Enriching NEs with web content

Generation of media fragments

Editor tool

www.linkedtv.eu

34

[Useful tool example] editor tool

, Roeland Ordelman1

§ Broadcasters want to have some control over noisy data.§ Editors have the expertise to determine relevance +§ The user interface can only show a limited amount of links, and there are many

§ Research: sample 805 videos, average 55 NEs per video. 55 NEs = many, many possible links depending on the size of the white list [Li et al., 2013]

NE - person: Jan Sluijters

NE – art syle: Fauvism

NE - person: Emiel Aardewerk

Link white list 1

Link white list 2

Link white list 3

Link white list 6

Link white list 7

Link white list etc etc

Link white list 4

Link white list 5

www.linkedtv.eu

35

Editor tool overview – selecting entities and enrichments

www.linkedtv.eu

36

Editor tool overview – selecting entities and enrichments

www.linkedtv.eu

37

Editor tool overview – editor‘s choices

www.linkedtv.eu

38

The project ends in May 2015. The coming year, we’ll focus on:

•Improving noisy data from automatic enrichments• eg. through aggregating multiple entities and visual cues

•Multiple user studies and iterative development of front-end functionalities and interface based on feedback

•Making enrichments more relevant for users by further refining personalisation technologies

•Further Market studies and business model development

Future work in LinkedTV

www.linkedtv.eu

Online resources!

39

www.linkedtv.eu

40

Do stuff now! Media Fragment Enricher

, Roeland Ordelman1

Eurecom’s tool: http://mfe.synote.org/mfe/

www.linkedtv.eu

Lets explore!

41

multimediacontentanalysis

new distributionplatforms

lots of archive content

www.linkedtv.eu

Lets explore!

41

multimediacontentanalysis

new distributionplatforms

lots of archive content

where the magic happens!

www.linkedtv.eu

42

Thanks!

, Roeland Ordelman1

Find me here:

@johanoomen| joomen@beeldengeluid.nl @lottebelice lbbaltussen@beeldengeluid.nl

Credits:§All LinkedTV partners§Slide hat-tip to: Lynda Hardman (CWI), Lyndon Nixon (MODUL), Raphaël Troncy (Eurecom), Bouke Huurnink (Sound and Vision)

§ LinkedTV overview: From SmartTV to LinkedTV. slideshare.net/linkedtv/linked-tv-introduction§ The importance of Linked Media to the Future Web. slideshare.net/linkedtv/www-linked-media-

keynote§ Describing media assets: media fragment specification and description. slideshare.net/troncy/

describing-media-assets-media-fragment-specification-and-description §Heaps of thanks to AVRO for so generously letting LinkedTV use their content.

Links:§www.linkedtv.eu. See demo’s, tools, research, scenario’s and more!§twitter.com/BenGlabs §www.avro.nl