www.linkedtv.eu

Lotte Belice Baltussen, Roeland Ordelman, Jaap Blom

Netherlands Institute for Sound and Vision

VideoHypE: An Editor Tool for Supervised Automatic Video Hyperlinking

Intetain | Mons, Belgium – 03 July 2013

2

www.linkedtv.eu Video hyperlinking

§  Topic that is receiving increasing attention §  Relate video (fragments) to other multimedia sources

§  Various perspectives and approaches: §  Semantic/Event Web §  Recommendation: detail-on-demand §  Storytelling/narratives: connected video fragments tell a story §  Summarization: links between video fragments to create summary

§  Link up communities by means of special sessions, workshops and benchmarks §  Multimedia Modeling (MMM 2014) – Dublin. Special session on Multimedia Hyperlinking

and Retrieval – CfP due 01 August 2013 http://mmm2014.org/special-sessions/ §  MediaEval Benchmark Evaluation on Search & Hyperlinking – www.multimediaeval.org

7/2/13

3

www.linkedtv.eu Experiencing the web on TV

§  Making the web available on TV

§  Typically this could take form in a Second Screen App

§  “Reference Device”, “Companion Device”

§  Second Screen dimensions §  Social §  Channel §  Program

§  Provide dynamic, contextually relevant links §  Give background information §  Link to e-shops

4

www.linkedtv.eu Dynamic contextually relevant links

§  Current approaches: editorial (manual)

§  What could automation bring in? Automatic link suggestion?

§  End-user perspective §  Automation to enable really dynamic/user driven links

§  Broadcaster’s perspective §  What are potentially interesting links for users?

§  Automatic link suggestion could help §  However, some potential links should be avoided!

§  We need control over the links §  Which links “add value” to the program?

§  We need to take program context into account §  How to maximize link potential & reduce manual effort?

§  …

5

www.linkedtv.eu VIDEO HYPerlink Editing tool

§  Automatic link suggestion based on video analysis §  Segmentation, visual concept/face detection, speech &

speaker recognition, entity extraction, …

§  Editing tool for: §  Selecting/deselecting/adding (suggested) link anchors §  Selecting/adding link targets §  (optionally) pre-view the results

§  Development in use scenarios based on specific programming:

§  News

§  Antiques Roadshow

6

www.linkedtv.eu Scenarios and content

 Enriched News

§  Professional news content

produced by RBB for general audience

§  local news show rbb Aktuell §  People, locations and

events can be linked to previous events.

Antiques Interactive

§  Program watched predominantly by 50+ers

§  AVRO: Tussen Kunst en Kitsch (TKK)

§  Objects, locations and experts provide anchors to related materials (targets).

linkedtv.eu

7

www.linkedtv.eu

dbpedia.org/resource/Jan_Sluyters

Link suggestion workflow

linkedtv.eu

Paintings by Jan Sluijters

“...painting of Jan Sluijters....”

Video analysis

Presentation engine

Named Entity Recognition

Linking NEs to web content

Generation of media fragments

Editing tool

8

www.linkedtv.eu VideoHypE & Automatic Link Suggestion

, Roeland Ordelman1

§  Supervision on suggested anchors

§  Deselect undesired links

§  Favour ‘authored’ websites (e.g., using white-list)

§  Determine relevance of links (e.g., on the basis of knowledge about audience)

§  Narrow down amount of links: potentially there are many §  Research: sample 805 videos, average 55 NEs per video. 55 NEs = many,

many possible links depending on the size of the white list [Li et al., 2013]

NE - person: Jan Sluijters

NE – art syle: Fauvism

NE - person: Emiel Aardewerk

Link white list 1

Link white list 2

Link white list 3

Link white list 6

Link white list 7

Link white list etc etc

Link white list 4

Link white list 5

9

www.linkedtv.eu Additional VideoHypE requirements

, Roeland Ordelman1

§  Requirements elicitation session with 5 editors 1.  Create and edit chapters

2.  Timeline with chapters

3.  See annotations per chapters, distinction between entity types

4.  Select annotations + related hyperlinks

5.  Add selected, approved hyperlinks

6.  Timeline with enities and hyperlinks

7.  Add new hyperlinks to media fragments (e.g. shots/chapters/spatial)

8.  Remove hyperlinks and/or entities (per media fragment or episode)

9.  Adding to white list

10.  Sorting on entity types

10

www.linkedtv.eu Demo overview – choose your video

, Roeland Ordelman1

11

www.linkedtv.eu Demo overview – shot overview

, Roeland Ordelman1

12

www.linkedtv.eu Demo overview - chapters

, Roeland Ordelman1

13

www.linkedtv.eu Demo overview – selecting entities

, Roeland Ordelman1

14

www.linkedtv.eu Demo overview – selecting links per entity

, Roeland Ordelman1

15

www.linkedtv.eu Demo overview – editor‘s choices overview

, Roeland Ordelman1

16

www.linkedtv.eu Demo overview – adding links

, Roeland Ordelman1

17

www.linkedtv.eu

•  Status: •  Implementing remaining requirements •  Next milestone: user testing with real editors

•  How do editors appreciate the tool? •  Overload versus added value of aut. suggestions •  Is the technology mature enough?

•  Additional questions: •  How do end-users appreciate the output of the tool? •  Can we make the tool adaptive (`learn’ on the basis of editors input) •  How to make the editor tool scalable? Different programmes, different

requirements •  Visualisation of linked content (in tool+application) may also influence

selection process •  LinkedTV Demo session @ ~16:00

VideoHypE – discussion & future work

18

www.linkedtv.eu Thanks!

, Roeland Ordelman1

Roeland Ordelman Audiovisual Content Access University of Twente, The Netherlands Netherlands Institute for Sound and Vision Cross-Media Interaction Check out: §  www.linkedtv.eu (project website) §  www.multimediaeval.org (benchmark evaluation on Search & Hyperlinking in

conjunction with ACM Multimedia 2013, october 18-19, Barcelona)