gareth jones 1209

Multimedia Information Seeking ThroughSearch and Hyperlinking

Gareth J. F. Jones

Centre for Next Generation LocalisationSchool of Computing, Dublin City University, Dublin, Ireland

Overview

Background and Motivation

MediaEval Multimedia Evaluation Benchmark

Search and HyperlinkingSearch and Hyperlinking at MediaEval 2012Search and Hyperlinking at MediaEval 2013

Conclusions and Research Prospects

Introduction

I Search and hyperlink navigation are well establishedapproaches to information seeking on the textual web.

I Increasing amounts of online digital multimedia content iscreating new challenges and opportunities for informationaccess.

I Realizing the potential of this content requires users to beprovided with engaging ways to interact with it, helpingthem to discover, browse, navigate and search.

I Different classes of users, such as media professionals,students, researchers and home users will potentially havedifferent needs and preferred means of access.

Introduction

I Existing research on multimedia information retrievalmainly focused on:

I search for individual relevant items - generally focusing onvisual information needs and features; or

I linking of content based on sharing visual features withoutattention to user information needs, e.g. linking videoscontaining the same individuals.

I Linking of multimedia content to text resources, e.g.wikipedia.

I Social remmendation based on user behaviour.

I Our work in search and hyperlinking aims to exploremultimodal access over multimedia content, and linking ofvideo content to support potential user interests or needs.

Introduction

Current Video Recommendation

Search for “Things to do in London”

Vision of Hyperlinked Video

Search and Hyperlinking

I These studies take place within the context of theMediaEval Multimedia Evaluation Benchmark campaign(www.multimediaeval.org).

I An evaluation benchmarking initiative focusing oninformation access and indexing tasks for multimediacontent.

I Established in 2010, following on from earlier VideoCLEFtasks at CLEF 2008 and CLEF 2009.

I Evaluates new algorithms primarily for novel and emergingtasks linked to a use case.

I Emphasis on the “multi” in multimedia: audio, speech,visual content, tags, users, context.

I TREC style annual evaluation cycle:I call for task proposals,I call for participation,I data release,I participants submit results,I organisers release evaluation of results,I participants gather at a workshop to discuss the results of

the task.

I Search and Hyperlinking task at MediaEval 2012 andMediaEval 2013.

I TREC style annual evaluation cycle:I call for task proposals,I call for participation,I data release,I participants submit results,I organisers release evaluation of results,I participants gather at a workshop to discuss the results of

the task.

I Search and Hyperlinking task at MediaEval 2012 andMediaEval 2013.

MediaEval Workshops

I MediaEval 2012 workshop in Pisa, Italy - co-located withECCV 2012.

I MediaEval 2013 workshop in Barcelona, Spain -co-located with ACM Multimedia 2013.

MediaEval Workshops

I MediaEval 2012 workshop in Pisa, Italy - co-located withECCV 2012.

I MediaEval 2013 workshop in Barcelona, Spain -co-located with ACM Multimedia 2013.

I Audio-visual archive exploration scenario.I Video-to-video linking.I Start with archive search assuming an “information need”.I Assume search becomes serendipitous or exploratory by

following hyperlinks.

I Video, e.g. 2 hoursI Interesting segment,

e.g. 10 minutesI Anchor: segment for

which a user would liketo link from (e.g. 1minute), e.g. “I want toknow more about ...”

I HyperlinkI Target: relevant

segment for a givenanchor, e.g. 5 minutes

I MediaEval 2012 Search and Hyperlinking task (datasetblip.tv):

I Search: retrieve known-item video segment given a naturallanguage description

I Hyperlinking: link known-item video segment to similarsegments in the collection

I MediaEval 2013 Search and Hyperlinking task (datasetBBC archive):

I Search: retrieve known-item video segment given a naturallanguage description

I Hyperlinking: link user-defined anchor within theknown-item to relevant target video segments in thecollection

Search and Hyperlinking at MediaEval 2012

I Multimodal search where relevance can be related to bothvisual and/or audio information streams.

I Models scenario where user seeks single relevantknown-item fragment of multimedia content.

I Once the relevant fragment has been located the user canengage in browsing or exploratory search by followingautomatically created hyperlinks to related multimediafragments within the same collection.

Blip10000 dataset

The Blip10000 dataset was created by the PetaMedia EUNetwork of Excellence:

I Contains 14,838 Creative Commons videos crawled fromblip.tv, and corresponding user provided metadata.

I Total of ca. 3,260 hours of data: Divided into 5,288development videos, and 9,550 test videos.

I Automatically created visual features:I shot boundaries - ave. shot length 30 secsI single keyframe for each shotI visual concepts from a set of 589

I Two automatic speech recognition transcripts created:LIMSI/Vocapia and LIUM.

I Comparative experiments using only English languagedataset: 4,890 test videos.

Test Query Set

I 30 textual queries consisting of natural language typestatements and search engine style queries.

I Textual information collected via crowdsourcing using theAmazon Mechanical Turk (AMT) platform; multimodalfeatures added manually afterwards.

I Query set for video selected at random from the top 10genre categories.

I Turkers required to:I locate an interesting region of the videoI mark the beginning and end points of the interesting regionI transcribe the words spoken exactlyI create natural language and search engine type queries

which they would use to refind this region of the video

Evaluation

Average Precision

AP =1n.

N∑r=1

Generalized Average Precision:

GAP =1n.

N∑r=1

P[r ] ·(

1 − DistanceGranularity

· 0.1)

Evaluation

Average Precision

AP =1n.

N∑r=1

Generalized Average Precision:

GAP =1n.

N∑r=1

P[r ] ·(

1 − DistanceGranularity

· 0.1)

Evaluation

Segment Precision (SP[r ]) at rank r :

Average Segment Precision:

ASP =1n.

N∑r=1

SP[r ] · rel(sr )

SP[r ] = Segment Precision [r ], rel(sr ) = 1, if relevant content ispresent, otherwise rel(sr ) = 0

Evaluation

Segment Precision (SP[r ]) at rank r :

Average Segment Precision:

ASP =1n.

N∑r=1

SP[r ] · rel(sr )

SP[r ] = Segment Precision [r ], rel(sr ) = 1, if relevant content ispresent, otherwise rel(sr ) = 0

Hyperlinking

I Linking effectiveness evaluated using MAP.I Relevance of each of the top 10 proposed links for each

submitted run evaluated using AMT.I Separate Qrel file created for each run; pooled unified Qrel

created by merging run qrels.I Turkers given the following options:

I Video segments totally unrealtedI Video segments related, same topic or focus, but different

informationI Video segments related, different perspective or view on

the same informationI Video segments are basically the same.

I Working with 270 hours of BBC archival data:I one complete weekI material not owned by the BBC excluded

I Task designed to explore the information needs and taskactivities of home users.

I Expected to favour exploratory search options.

I Search queries created by 30 public users with computerand online experience.

I Recruited using external agency.I Profile:

I Nationality: UKI Age Group: 16-30I Computer familiarity: highI Online activity: high

I Users given a general introduction on video search usingthe AXES Pro system.

I Scope and type of the collection explained, and usersallowed to explore the collection.

I User creates text query expressing information need whichshould be satisfied by the available dataset.

I User searches for a relevant video, reformulating theirquery if necessary.

I Ask the user to identify anchors within their chosen video:specify region, spoken words or whole segmennts.

I Explain reasons why these anchors were selected.

Specification of Anchors

Classification of Selected Clips

Classification of Anchors

I Set of 50 known-item experimental queries selectedI Corresponding set of 100 anchors selected

I Run submissions 10th September!I Relevance assessment of links ongoing using returning

test volunteers at BBC and using AMT (to increase amountof relevance assessment in time available)

I MediaEval 2013 workshop 18th-19th October

I Set of 50 known-item experimental queries selectedI Corresponding set of 100 anchors selectedI Run submissions 10th September!

I Relevance assessment of links ongoing using returningtest volunteers at BBC and using AMT (to increase amountof relevance assessment in time available)

I Set of 50 known-item experimental queries selectedI Corresponding set of 100 anchors selectedI Run submissions 10th September!I Relevance assessment of links ongoing using returning

I Search is a more established task, but little research so faron identifying reliable jump-in points.

I Promising results from Search and Hyperlinking in 2012encouraged us to move to user studies with BBC data.

I Work so far with volunteer subjects suggests that users willmake use of video hyperlinks.

I Results and other findings from Search and Hyperlinking in2013 will be used to help develop focused researchagenda for 2014 and beyond.

References

I M.Larson, M.Soleymani, M.Eskevich, P.Serdyukov, R.Ordelman and G.J.F.Jones,The Community and the Crowd: Developing large-scale data collections formultimedia benchmarking. IEEE Multimedia. Vol.19, No.3, pp15-23, 2012.

I M.Eskevich, W.Magdy and G.J.F.Jones. New Metrics for Meaningful Evaluationof Informally Structured Speech Retrieval, In Proceedings of the 34th EuropeanConference on Information Retrieval (ECIR 2012), Barcelona, April 2012.

I M.Eskevich, G.J.F.Jones, S.Chen, R.Aly, R.Ordelman and M.Larson, Search andHyperlinking Task at MediaEval 2012, Proceedings of the MediaEval 2012Workshop, Pisa, Italy, October 2012.

I M.Eskevich, G.J.F.Jones., R.Aly, R.J.F. Ordelman, S.Chen, D.Nadeem,C.Guinaudeau, G.Gravier, P.Sebillot, T.de Nies, P.Debevere, R.van de Walle,P.Galuscakova, P.Pecina and M.Larson, Information Seeking through Search andHyperlinking, In Proceedings of ACM International Conference on MultimediaRetrieval (ICMR 2013), Dallas, Texas, USA, April 2013.

I R.Aly, R.Ordelman, M.Eskevitch, G.J.F.Jones and S.Chen, Linking inside a videocollection – what and how to measure?, Proceedings of the First Worldwide WebWorkshop on Linked Media (LiME-2013) at WWW 2013, Rio de Janeiro, Brazil,May 2013.

Acknowledgements

Coordinators of the MediaEval Search and Hyperlinking task:Robin Aly, Maria Eskevich, Gareth Jones, Roeland Ordelman

Search and Hyperlinking activities supported by:

gareth jones 1209

Technology