icme 2016 - tutorial on interactive search in video & lifelog repositories

InteractiveSearchinVideo&Lifelog Repositories

KlausSchoeffmann,PhDKlagenfurtUniversityInstituteof InformationTechnologyKlagenfurt,Austria

FrankHopfgartner,PhDUniversityof GlasgowSchoolof HumanitiesGlasgow,UK

Interactive Search in Video & Lifelog Repositories

• Part1:InteractiveVideoSearchØ Searchinvideocontent:motivationandchallengesØ Automaticvideoretrievalvs.interactivevideosearchØ Toolsforinteractivesearch

§ Browsing,Navigation,Visualization,Similarity&Sketch-basedSearchØ EvaluationofIVSTools

§ TRECVID,VideoBrowserShowdown(VBS)

Shortbreak

• Part2:LifeloggingØ QuantifiedSelfØ LifelogrepositoriesØ LifeloggingtechniquesØ Interactivevisualization

KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 2

Motivation

3KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016

Video Everywhere

• UbiquitoususeofvideosnowadaysØEntertainment andcommercialsØSocialgaming(screencasts)ØPersonalvideos(family,kids,…)ØSportsdocumentationandanalysis(e.g.,GoPro)ØProductusageinstructions(e.g.,furniture)ØSurveillance(buildings,places,street,…)ØHealthcareandmedicalscience(endoscopicprocedures)ØLifelogging

• Enormousamountofdata,challengingtosearch!


Video – The Ultimate Media?


[MaryMeeker,LiangWu,InternetTrends,D11Conference,May,2013]

Asof 2014,everyminute 300hours ofvideo are uploaded

to YouTube!

Video Cameras

• IncreasinglypowerfulØThesedaysyoucanrecord4Kcontentwithyourmobile!ØVideosensorsuseauto-focus,objecttracking,colorcorrection,and imagestabilization

ØStoragespacenotabigproblem§ Currentsmartphoneshave128GBofmemory§ NASdevicescheaplyavailable

ØNetworkbandwidthalsodramaticallyincreasedoveryears§ Videostreamingonthegoissimpleandcommon§ LTEconnectionsprovide30Mbit/sandevenmuchmore!



[MaryMeeker,LiangWu,InternetTrends,D11Conference,May,2013]

Challenge: Finding Content

• EvenwithretrievaltoolsstillchallengingtofindcontentlaterØEspeciallyifnotpubliclyavailable(andpopular+annotated)ØManyproblemswithquerying,inparticularfornon-experts

• Ultimategoal:makesearchaseffectiveasfortextØQuicklyfindrelevantcontentØComparetointeractivityofatextbook

§ Index,ToC,listoffigures/tables,etc.§ Change,extend,copy,bookmark,highlight,etc.


SearchforVideoContent


Example Scenario

10

Why? (e.g.,showtosomeone,includeineditedvideo,findsomeinformation,extractimage,etc.)

Youwanttofindthisvideoclipinyourcollection:

KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016

Large Video Collection

11

IACCdataset,asusedforTRECVID:146,788shots

(~9,000videos)

Page123….383940


How a Novice Would Solve ThisNovice users typically employ a file browser and a simple video player!

VCRinthe1970sprovidedasimilarfunctionality!

12

?KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016

Fileexplorerandvideoplayer

13

Factor>1Mio!

[en.wikipedia.org]


How a Novice Would Solve ThisNovice users typically employ a file browser and a simple video player!

VCRinthe1970sprovidedasimilarfunctionality!


Fileexplorerandvideoplayer

• Videoretrievaltoolwithcontentanalysisandsearch• Queryby

ØText,Concept,Example

• AutomaticsearchØContent-baseddatasuchas:

§ Text (e.g.,metadata,ASR,OCR,transcripts,…)

§ Globalfeatures(e.g.,color,texture,motion)

§ Localfeaturesandconcepts(e.g.,VLAD,BoVW,…)

ØRankedresultlist

15

IBMTRECVID2007VideoRetrieval System[1]

How a Retrieval Expert Would Solve This


16

Content-basedFeature

ExampleImage

Text

Rankedlistofshots

InIACCabout5800pages.L

TemporalContext

[Heesch,D.,Howarth,P.,Magalhaes,J.,May,A.,Pickering,M.,Yavlinsky,A.,&Rüger,S.(2004,November).Videoretrieval using search and browsing.InTRECVideoRetrieval EvaluationOnlineProceedings.]

How a Retrieval Expert Would Solve This


17

Thiswas10yearsago,whataboutstate-of-the-art?


A More Recent Video Retrieval Tool


[A.Moumtzidou etal.,“VERGE:AMultimodalInteractiveVideoSearchEngine”,Proc.of the 21stInternationalConferenceonMultiMedia Modeling(MMM2015),Sydney,2015]

kNN Similarity searchbased onVLADvectors

Concept detection with SVMandfive local descriptors (SIFT,SURF,

ORB,...)and PCAor CNNs

Hierarchicalkeyframe clustering

19URL: http://mklab-services.iti.grKlausSchoeffmann

21Similarity SearchResultsKlausSchoeffmann

22

Concept-basedsearchstillfarfromoptimal(evenwithCNNs)!Evenwithperfectresults,whowouldbrowseafew1000shots?

ShortcomingsoftheQuery-and-Browse Approach


Common Video Retrieval Approach

WorkswellifØuserscanproperlyexpresstheirneeds.Øcontentfeaturescansufficientlydescribevisualcontent.Øcomputervisioncanaccuratelydetectsemantics.

24

Content-basedSearch

Ranked Results

Unfortunately,inpracticetheseassumptionsdonothold.


ØContent-basedfeatures§ Howtounderstandsemanticsfrompixels? SemanticGap

Bothimagesshowbearsinfrontofalandscape.


Mind the Gap!

ØDatabaseaffinityofconceptclassifiersØLowperformanceinbroaddomain

P(k) Precisionatlevelk(afterkresults)rel(k) definesifkth retrieveddocumentisrelevant

PerformanceGap

26

TRECVID2015SemanticIndexing(60concepts):median“inferredaverageprecision”(infAP)=0.24

Inotherwords:morethan75%

ofresultsarewrong!


Mind the Gap!

Ø Query-by-concept§ Whichconcepttouse?Choosefromalonglistofresults…

Ø Query-by-example§ Typicallynoperfectexampleavailable.

Ø Query-by-sketch§ UsersarenoartistsJ (seealsonextslide)

Ø Query-by-text§ Howtodescribeadesiredimagebytext?

UsabilityGap

27

Apicturetellsa1000words.

bymarfis75

Howtodescribeadesiredvideoclipbytext???KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016

Mind the Gap!

Needs More Focus on the User (Interface)!

Ø Insomesituationsuserscannotformulateaquery§ à provideexploratorysearchfeatures!§ Forexample:browsing,filtering,similaritysearch

ØUsersexpectgoodresults(onfirstpage!)§ à Userelevancefeedback/activelearning insteadoflonglists!

ØVideosaredynamic§ Staticthumbnailsarenotinformative§ Esp.trueforlongshotsandself-similarcontent§ à skimsandvisualsummaries(“smartplayback”)§ à sophisticatednavigation&contentstructurevisualization

ØShotshaveatemporalcontextØGridinterfacesarenotalwaysthebestchoice


UsabilityGap

InteractiveVideoSearch


Interactive Video Search

30

• HCIcommunity• Methodsforinteractivesearch• Humancomputation• Nocontentunderstandingbutsimple

• Multimediacommunity• Mostlyautomaticsearch• Retrievalengine• Complicatedtouse

Mismatch

Novices Experts

à CombineHCIwithCVandMIRforbettersearchtools


User-Centric Exploratory Search

• Stronglyintegrateuser intosearchprocessØ AssumeasmartuserØ Givehim/hermorecontroloversearchprocess

§ Inspectsandinteracts§ Selectsmostmeaningfultoolforcurrentneeds,e.g.

• ContentBrowsing/Navigation• ContentVisualizationandSummarization• Ad-hocQuerying(e.g.,bysketch, filtering,ad-hocexample)• Aspect-basedexploration,parallelsearchpaths

Ø Iterative:Search– Inspect– Think– Repeat§ Exploratorysearch(“willknowitwhenIseeit”)§ Insteadof„query-and-browse-results“


Aspects of Interactive Video Search (IVS)

IVS

Navigation &Browsing

DifferentQueryTypes

Dynamics&Convenience

ContentVisualization


UnderlyingStructure

Abstracts/summaries

Overview(TOC)

Skims

SmartPlayback

Bookmarks

History

TextorConcept

ExampleImage

ExampleClip

(SimilaritySearch)

Sketch

Filter(Spatial&Temporal)

CoarseNavigationFineNavigation

BrowsingSequences/Scenes/Shots

Similarity-BasedArrangements(e.g.,byColor)

Outline

InteractiveVideoSearch(IVS)Tools:ØVideoNavigationØVideoBrowsingØContentVisualizationØSketch-basedSearch


VideoNavigation


Improving Navigation

35

e.g.,onYouTubedefaultwindow:

640pixels=frames(25seconds)

Commonseeker-barlimitsnavigationgranularity

[Huerst etal.,ICME2007]

ZoomSlider

Improvements(selected):


Improving Seeker-Bar Navigation

36

WolfgangHürst,GeorgGötz,andMartinaWelte,“Interactivevideobrowsingonmobiledevices”,inProceedingsofthe15thInternationalConferenceonMultimedia (MULTIMEDIA'07).ACM,pp.247-256,2007


ZoomSlider[Huerst etal.,ICME2007]

Improving Navigation

38

e.g.,onYouTubedefaultwindow:

640pixels=frames(25seconds)

Commonseeker-barlimitsnavigationgranularity

[Dragicevic etal.,CHI2008]

DirectManipulation

[Huerst etal.,ICME2007]

ZoomSlider

Improvements(selected):


Relative Flow DraggingBackground Stabilization

39

PierreDragicevic,GonzaloRamos,Jacobo Bibliowitcz,DerekNowrouzezahrai,Ravin Balakrishnan,andKaranSingh.“Videobrowsingbydirectmanipulation”,inProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems(CHI'08).ACM,pp.237-246,2008

Videobrowsingbydirectmanipulation/relativeflowdragging


Relative Flow Dragging

• EvaluationwithauserstudyØ 16participants(18-44yearsold)Ø Directcomparisontoseeker-barnavigationØ Navigationtasks,2videos(ladybug,cars)

§ “FindthepositionwheretheladybugpassesovermarkerX”§ “FindthemomentwhencarXstartsmoving”

Ø Flowdraggingsignificantlyfaster(RM-ANOVA)byatleast250%(alsosignificantlylesserrors)

40

PierreDragicevic,GonzaloRamos,Jacobo Bibliowitcz,DerekNowrouzezahrai,Ravin Balakrishnan,andKaranSingh.“Videobrowsingbydirectmanipulation”,inProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems(CHI'08).ACM,pp.237-246,2008


Scrubbing Wheel

• RequirementsØSimpleandeffectivenavigationontouchscreens

ØEfficientnavigationthatallowsforcontentsearchinbothshortandlongvideos

• IdeaØ improvenavigationbyusingacircularnavigationarea

Ø inspiredbyAppleiPod(c)device


KlausSchoeffmann andLukasBurgstaller,“ScrubbingWheel:AnInteractionConcepttoImproveVideoContentNavigationonDeviceswithTouchscreens“,inProceedingsoftheIEEEInternationalSymposiumonMultimedia2015(ISM2015),Miami,FL,USA,2015,pp.351-356

Scrubbing Wheel Implementation (iOS)


IEEEInternationalConferenceonMultimedia&Expo(ICME)2016

DemoVideo

KlausSchoeffmann 44

VideoBrowsing


46

Video Browsing

[F.Arman,R.Depommier,A.Hsu,andM-Y.Chiu,Content-basedBrowsingofVideoSequences,inProc.ofACMInternationalConferenceonMultimedia,1994,pp.97-103]


Video Browser for the Digital Native

47

[Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on. IEEE, 2012.]

“TemporalSemanticCompression”basedontempofunctionandshotpopularity(insight)


Video Browser for the Digital Native

• Userstudywith8participantsØ Testconfigurationelementsbytwotasks(afterpresentation+5minutestraining)§ (i)Browseafamiliarmovietofindscenesyouremember§ (ii)Browseanunfamiliarmovietogetafeelforitsstoryorstructure

Ø QuestionnairewithLikert-scaleratings

48

[Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on. IEEE, 2012.]


The Video Explorer

49

[Schoeffmann,K.,Taschwer,M.,&Boeszoermenyi,L.(2010,February).Thevideo explorer:atool for navigation and searching within asingle video based onfastcontent analysis.InProceedings of the first annualACMSIGMMconference onMultimediasystems (pp.247-258).ACM.]


Interactive Navigation Summaries

Allowsausertoquicklyidentifysimilar/repeatingscenes

50

[Schoeffmann,K.,&Boeszoermenyi,L.(2009,June).Videobrowsing using interactive navigation summaries.InContent-Based MultimediaIndexing,2009.CBMI'09.Seventh Int.Workshop on (pp.243-248).IEEE.]


Motion Layout: Direction + Intensity

MotionVector (µ)classification intoMotionhistogram with K=12

equidistant motion directions (bins)Mappingto Hue channel

51

[Schoeffmann,K.,Lux,M.,Taschwer,M.,&Boeszoermenyi,L.(2009,June).Visualization of video motion incontext of video browsing.InMultimediaand Expo,2009.ICME2009.IEEEInt.Conf.on (pp.658-661).IEEE.]


52

[Schoeffmann,K.,Lux,M.,Taschwer,M.,&Boeszoermenyi,L.(2009,June).Visualization of video motion incontext of video browsing.InMultimediaand Expo,2009.ICME2009.IEEEInt.Conf.on (pp.658-661).IEEE.]

Similarity Search (SOI) with Motion Layout


• SOISearchØ Motion-basedsearchbyexamplesequence

§ UsingMotionDirection histogramDb

§ User-selectedsequence

Ø Findmostsimilarsequences§ Computedistance toanypossibleseq. ofsamelength§ Matchifbelowspec.threshold


MotionLayout(Db)

Match1 Match2 Match3

frame1 framen

Similarity Search (SOI) with Motion Layout

Region-of-Interest(ROI)SearchØ Userselectsspatialregion-of-interestØ Onsearch

§ ComputeEuclidiandistance offrameFtoeveryotherframe f (acc.toselectedregion)

§ Basedoncolorlayout descriptor

…

frameF

frame1 framek framen

User-selectedregion(I)

…

d(F,1)=350 d(F,k)=8 d(F,n)=400

54


Similarity Search (ROI) with Color Layout


55


Similarity Search (ROI) with Color Layout


The ForkBrowser

• Thread:linkedsequenceofshotsinaspecifiedorderØ Queryresults,visualsimilarity,semanticsimilarity,textualsimilaritytime,…


[DeRooij,Ork,CeesGMSnoek,andMarcelWorring."Balancingthreadbasednavigationfortargetedvideosearch."Proceedingsofthe2008internationalconferenceonContent-basedimageandvideoretrieval(CIVR).ACM,2008.]

IEEEInternationalConferenceonMultimedia&Expo(ICME)2016

DemoVideo

KlausSchoeffmann 58

Goal:improvetwo-handeduse

The ThumbBrowser


[MarcoHudelist,KlausSchoeffmann,LaszloBöszörmenyi.“MobileVideoBrowsingwiththeThumbBrowser”,Proc.oftheInternationalConferenceonMultimedia,2013,pp.405-406]

ContentVisualization


Grid Interfaces Aren‘t Enough!

• ManyvideoretrievalsystemsuseaGridinterface!?

Moreover,agridinterfacedoesnotallowforfasthumanvisualsearch(seelater)!

61

Arankedlistofresultsdoesnotconveythetemporalcontentstructure!• Towhichvideodoesashotbelongto?• Whatisthesequenceofshots?• Howlongisashot/scene?


TableofVideoContent(TOVC)

[Goeau etal.,ICME2007]

62

Squeeze/FisheyeRapidVisualSerialPresentation(RSVP)

Improving Visualizationaka “Video Surrogates”

[Wildemuth etal.,2003]

[Wittenburg etal.,2005]


63

VideoTree[Jansenetal.,CBMI2008]

However,outperformedbysimple“gridofkeyframes”intermsofsearchtime.

Similarconceptproposedlater[Girgensohn etal.,ICMR2011]

• Split-basedclustering algorithmwithcolorcorrelograms.

• Treenotdirectlyshown totheuser(onlyonelevel).

Improving Visualizationaka “Video Surrogates”


Hierarchical Video BrowsingAnother Tree-based Approach

FrontalView TopView

From:[Schoeffmann andDelFabro,2011]

64

• Goal:improvecontentoverview• Nocontentanalysis(justuniformsamplingofframes)


3D Ring Instead of Grid!

• UtilizationofscreenrealestateØ LargesetofimagesØ Minorocclusion,slightdistortion

• IntuitiveinteractionØ Rotateandzoom

• Content-based sorting• “Pop-outimages”(intheback)• Furtheradvantages

Ø Immediatelycontinueonmiss,scaling

66

Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“, in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951.


3D Ring Interface - Perspectives

PreferredDesignacc.touserstudy

25%Vertical66%Horizontal 8.3%Frontal

67



3Dinterface significantly faster than grid by 12.7%

User Study: Grid vs. Ring (both sorted)150 images, 12 participants, 1440 trials

68



Extension: Multiple Rings with Vertical Scrolling

69

KlausSchoeffmann.2014.TheStack-of-RingsInterfacefor Large-Scale ImageBrowsing onMobileTouchDevices.InProc.of the ACMInt.ConferenceonMultimedia(MM'14).ACM,NewYork,NY,USA,1097-1100.

Significantly faster search (by about 48%)than common image browser oniPad!


Sketch-BasedSearch


• Colorsketchesmappedtofeaturesignatures

• Matchedtothoseofkeyframes

71

1. Samplingkeypoints2. Descriptionthroughlocation(x,y),

CIELab,contrastandentropyofsurroundingpixels

3. k-meansclustering

Feature Signatures

[Kruliš,M.,Lokoč,J.and Skopal,T.(2013).Efficient Extraction of FeatureSignatures Using Multi-GPUArchitecture.SpringerBerlinHeidelberg,LNCS7733,pp.446-456.]


Feature Signature-Based Video Browser

72

ColorSketch(Signature)

Player

WinnerofVideoBrowserShowdown2014+2015Downloaddemoat:http://siret.ms.mff.cuni.cz/lokoc/vbs.zip

2nd ColorSketch(optional)

[Lokoč,J.,Blažek,A.,&Skopal,T.(2014,January).Signature-Based VideoBrowser.InMultiMedia Modeling (pp.415-418).SpringerInternationalPublishing.]


Compactvisualization

Simplecolor-positionsketch

Negativeexample

Matchedkey-frames

Timeto2nd sketch

2nd optionalsketch

Interactive-navigationsummaryOndemandneighborhoodexpansion

[Slide:AdamBlazeketal.(siret researchgroup,CzechRepublic)]


Compact Visualization to Save Space


[CourtesyofJakubLokoc etal.]

Another Example of a Sketch-Based Browser


[KaiUweBarthel,Nico Hezel,Radek Mackowiak.Navigatingagraphofscenesforexploringlargevideocollections,inProc.of22ndInternationalConferenceonMultiMedia Modeling(MMM2016),LectureNotesinComputerScience(LNCS),Vol.tbd,SpringerInternationalPublishing,2016,pp.1-7]

WinnerofVideoBrowserShowdown2016

Break


EvaluationofIVSTools


User Studies with Significance Tests!

• Manyinterfacesproposedwithoutproperevaluation• InterfaceAbetterthaninterfaceB?à comparativeuserstudyneeded!

Ø Performsearchtasksinexactlythesamesetting(data,environment,etc.)

Ø Loggingofinteractionbehaviorandtasksolvetime

Ø QuestionnaireaboutsubjectiveworkloadsØ Statisticalanalysiswithpropertests(e.g.,t-test,ANOVA,Wilcoxonsigned-rank,etc.)

• Usersimulations?• Evaluationcompetitions

Ø SamedatasetØ ComparativeevaluationØ TRECVID,MediaEval,VideoBrowserShowdown


Video Browser Showdown (VBS)

• AnnualperformanceevaluationcompetitionØ LiveevaluationofsearchperformanceØ SpecialsessionatInt.ConferenceonMultiMedia Modeling(MMM)Ø Demonstratesandevaluatesstate-of-the-artinteractivevideosearchtoolsØ IdeainfluencedbyVideOlympics (Snoek etal.,IEEEMultimedia2008)

• FocusØ Known-itemSearchtasks

§ Targetclipsarepresentedonsite§ Teamssearchinshareddataset

Ø Highlyinteractivesearch§ Shouldpushresearchoninterfaces

andinteraction/navigationØ Experts andNovices

§ Easy-to-usetoolsandmethodsØ Ad-HocVideoSearch(TRECVIDAVS)tasksstartingfrom2017

79

http://videobrowsershowdown.org/


Video Browser Showdown (VBS)

• Liveevaluation/scoringthroughVBSServer• Score(s)[0-100]fortaski andteamk isbasedon

ØSolvetime(t)ØPenalty(p)basedonnumberofsubmissions(m)

80

Maximumsolvetime(Tmax)typically5minutes

[Schoeffmann,K.,Ahlström,D.,Bailer,W.,Cobârzan,C.,Hopfgartner,F.,McGuinness,K.,...&Weiss,W.(2013).TheVideoBrowserShowdown:aliveevaluation of interactive video search tools.InternationalJournalof MultimediaInformationRetrieval,1-15.]


Correctbutsubmittedlaterthanfirstteam Penaltyduetotoomany

wrongsubmissions


Video Browser Showdown 2016

• Searchinmid-sizedvideocollectionsØ Originallyonlysinglevideosearch

• TwodifferentkindofKIStasks:Ø Visual:visualpresentationofa30stargetclipØ Textual:textualdescriptionofa30stargetclip

• SharedvideodatafromBBCØ 2016:441videofiles,about320.000shots(250hours)

[Schoeffmann,Klaus."Auser-centricmediaretrievalcompetition:Thevideobrowsershowdown2012-2014."MultiMedia,IEEE 21.4(2014):8-13.]


Visual Task Example (2016)


Textual Task Example (2016)

“Stevecuttingadrawingintohisblockofwood.Youcanseehishandandacutterandflowersymbols.”



2012:Klagenfurt11 teams

2013:Huangshan6teams

2014:Dublin7teams

2015:Sydney9teams

2016:Miami9teams

VBS2017:January4,2017,Reykjavik,Iceland(MMM2017)http://www.videobrowsershowdown.org/

Winner 2014 and 2015(2014: single video and collection search, 2015: collection only)

86

ColorSketch(Signature)

Player

2nd ColorSketch(optional)

[Lokoč,J.,Blažek,A.,&Skopal,T.(2014,January).Signature-Based VideoBrowser.InMultiMedia Modeling (pp.415-418).SpringerInternationalPublishing.]


Video Browser Showdown 2015Two other examples of the 9tools (collection search only)

87

Moumtzidou,A.,Avgerinakis,K.,Apostolidis,E.,Markatopoulou,F.,Apostolidis,K.,Mironidis,T.,...&Patras,I.(2015,January).VERGE:AMultimodalInteractiveVideoSearchEngine.InMultiMedia Modeling(pp.249-254).SpringerInternationalPublishing.

• Shotandscenedetection• HLF(Concepts)with

SIFT/SURFandVLAD• Similaritysearch

• Uniformsampledframes• Humancomputation

Hürst,W.,vandeWerken,R.,&Hoet,M.(2015,January).AStoryboard-BasedInterfacefor MobileVideoBrowsing.InMultiMedia Modeling (pp.261-265).SpringerInternationalPublishing.

3rd place


Human vs. Machine

• UtrechtUniversity@VBS2015Ø WolfgangHuerst etal.,TheNetherlandsØ Strongexperience inHCI

• FeaturesØ Uniformly sampled thumbs(1second distance)

Ø Huge storyboard ontabletØ Vertical scrolling,paging

88

625thumbnails inone screen

[Hürst,W.,vandeWerken,R.,&Hoet,M.(2015,January).AStoryboard-Based Interfacefor MobileVideoBrowsing.InMultiMedia Modeling (pp.261-265).SpringerInternationalPublishing.]


Winner 2016


FrankHopfgartnerSchoolofHumanities

Universityof Glasgow,UK

Tutorial:InteractiveSearchinVideo&LifelogRepositoriesPart2:TheQuantifiedSelfandLifelogging

IEEEInternationalConferenceonMultimediaandExpo(ICME)2016

Afewwordsaboutme

Research on Multimedia Analysis, Quantified Self, Lifelogging

Lecturer(AssistantProfessor)inInformationStudies(UGlasgow)

PhDinComputingScience(UniversityofGlasgow)

Past:VariouspositionsinBerlin(TUB),Dublin(DCU),Berkeley(ICSI),andLondon(QMUL)

What is The Quantified Self?

TheQuantifiedSelfisaboutobtainingself-knowledgethroughself-tracking.

What is The Quantified Self?

Self-trackingisalsoreferredtoaslifelogging,self-analysis,orself-hacking.

Memex

Bush,Vannevar."AsWeMayThink."TheAtlanticMonthly.July1945.

ImagesofM

emex:http://trevor.sm

ith.nam

e/mem

ex/

MyLifeBits

• GordonBell(Microsoft)digitizedhislife:ØBookswrittenØPersonaldocumentsØPhotosØPosters,paintings,photoofthings

ØHomemoviesandvideosØCDcollectionØPCfilesØ…

Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009

http://research.microsoft.com/en-us/projects/mylifebits/

MyLifeBits

Slidefrom:G.Bell.ChallengesinUsingLifetimePersonalInformationStoresbasedonMyLifeBits.PresentationatAlpbach Forumon26August2004.

Self-tracking devices

Self-tracking apps

Creating Personal Lifelog Repositories

Alifelogrepositoryconsistsofheterogeneousdatarecordedusingmanydifferentsensors.

In this tutorial, we will…

• getanintroductionintothecreationoflifelogrepositories

• understandthemajorchallengesofcreatinglifelogrepositories

• discusshowtoevaluatelifeloggingtechniques.

So what are the challenges?

Thechallengesarehowtosensetheperson,capturetheiractions,theirlifeandmakeitaccessibleusing

appropriategraphicaluserinterfaces,search/recommendationenginesandvisual/auralfeedback.Further,exploitingthelifelog toidentify

contextforadaptiveinformationservices.

Research communities

Multimedia

ACMMultimedia

IEEEICME

MultimediaModeling

HCI

ACMCHI

AugmentedHuman

ACMUbiComp

MachineLearning

ICML

KDD

ECML

The Key Challenges

Capturing

SemanticAnalysis Access

EvaluationLifelogrepository

Challenge 1: Capture

Automaticallyandunobtrusivelycapturelifelogger’s lifeexperiences.

Image:@morberg,flickr.com

Communication

Interests

Health

Travel

Socialnetworks

Recording my media consumption

Brusilovsky,P.andKobsa,AlfredandNejdl,Wolfgang.“TheAdaptiveWeb:MethodsandStrategiesofWebPersonalization."LectureNotesinComputerScience,SpringerVerlag,2007.

Recording my communicationIm

age:http://www.wire

d.co.uk/news/archive/2013-

06/10/sim

ple-guide-to-prism/viewgallery/304880

Recording my online behaviour

Recording how I feel

https://exist.io/

Recording how I feel

http://measuredme.com/

Recording what I hear

http://lifeboxapp.com/

Record where I go

Recording where I travel

http://flightdiary.net/

Recording my activities

Source:https://jawbone.com/blog/jawbone-up-data-by-city/

Recording who I meet

http://linkedin.com/

(Automatically) recording who I meet

• Inferred,weightedfriendshipnetworkvs.reported,discretefriendshipnetwork.

Eagle,NathanandPentland,Alex(Sandy)andLazer,David.“Inferringfriendshipnetworkstructurebyusingmobilephonedata."ProceedingsoftheNationalAcademyofSciencesoftheUnitedStatesofAmerica,106(36):15274-15278,2009.

Recording what I eat

Aizawa,Kiyoharu,Maruyama,Yutu,Li,He,andMorikawa,Chamin.“FoodBalanceEstimationbyUsingPersonalDietrary TendenciesinaMultimediaFoodLog."IEEETransactionsonMultimedia,15(8):2176-2185,2013.

SemanticGap

http://foodlog.jp/

http://mealsnap.com/

Recording what I eat

Source:http://edition.cnn.com/2014/01/29/world/asia/korea-eating-room/

Recording what I see

"LifeGlogging cameras1998200420062013labeled"byGlogger - Ownwork.LicensedunderCCBY-SA3.0viaCommons-https://commons.wikimedia.org/wiki/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg#/media/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg

Visual Lifelogging

Example: Visual Lifelog of a day

2,000picturesaday

Slide:C.Gurrin

Big Data

CathalGurrin,AlanF.SmeatonandAidenR.Doherty(2014),"LifeLogging:PersonalBigData",FoundationsandTrends®inInformationRetrieval:Vol.8:No.1,pp1-125.

Vision: Recording what I see(Black Mirror, S01E03)

The Key Challenges

Capturing



Challenge 2: Semantic Analysis

How not to do it…

A day

Thisdoesnotworkwell…Let’saddeventsegmentation.

Event Segmentation & Annotation

• Segment5,500photosperdayintoasetofeventsØ SimilartoSBDindigitalvideoprocessingØ Weemployvisualfeaturesandoutputofon-devicesensors

MultipleEvents

Finishingworkinthelab

Atthebusstop ChattingatSkylon Hotellobby Movingtoaroom

Teatime Onthewaybackhome

EventSegmentation

Summarization

Slide:CathalGurrin

Context is key

• Contextcueshelpustoremember(Naaman etal.)

• Contextinlifeloggingdata:Ø Location,bluetooth,time,date,…

Ø DerivedKnowledge(e.g.activities)

• Approaches:Ø Combinecuesfromdifferentsources

Ø Performcontentanalysistoidentifyobjects,people,events…

Ø Annotatelifelogsinformofnarrativetext

Mor Naaman,SusumuHarada,QianYing Wang,HectorGarcia-Molina,AndreasPaepcke:Contextdataingeo-referenceddigitalphotocollections.ACMMultimedia2004:196-203

Visual Feature Extraction

Ø Steeringwheel(72%)Ø Shopping(75%)Ø Insideofvehiclewhennotdriving(airplane,taxi,car,

bus)(60%)Ø Toilet/Bathroom(58%)Ø GivingPresentation/Teaching(29%)Ø ViewofHorizon(23%)Ø Door(62%)Ø Staircase(48%)Ø Hands(68%)Ø Holdingacup/glass(35%)Ø Holdingamobilephone(39%)Ø Eatingfood(41%)Ø Screen(computer/laptop/tv)(78%)Ø Readingpaper/book(58%)Ø Meeting(34%)Ø Road(47%)Ø Vegetation(64%)Ø OfficeScene(72%)Ø Faces(61%)Ø People(45%)Ø Grass(61%)Ø Sky(79%)Ø Tree(63%)

Byrne,Daragh,Doherty,AidenR.,Snoek,CeesG.M.,Jones,GarethJ.F.,Smeaton,AlanF.“Everydayconceptdetectioninvisuallifelogs: validation,relationshipsandtrends."MultimediaToolsandApplications,49(1):119-144,2010.

Non-supervised Event Segmentation

2. Arriving in the office

6. Walking inthe building 12. Leaving

the office

NaLietal.“RandomMatrixEnsemblesofTimeCorrelationMatricestoAnalyze VisualLifelogs."InProc.MultimediaModeling Conference,Dublin,Ireland,pp.400-411,2014.

EventSegmentationbasedontheextractionoflowlevelfeaturesandcomputationofsemanticconceptsrequiresknowledgeaboutdataset.

Alternative:Highlight“significantevents”byperformingtimeseriesanalysis

The Key Challenges

Capturing



People access memory for five reasons

Sellen,AbigailandWhittaker,Steve.“BeyondTotalCapture:AConstructiveCritiqueofLifelogging."CommunicationsoftheACM,53(5):70-77,2010.

•Relivingpastexperiencesforvariousreasons

Recollecting

•Story-tellingorsharinglifeexperienceswithothers

Reminiscing

•Findspecificinformationsuchasanaddress,oradocument

Retrieving

•Gaininginsights(QuantifiedSelf)

Reflecting

•Planningfutureactivities.

Remembering

Quantified Self

P. Kostopoulos. Stress Detection using Smartphone Data. In Proc. HealthWear’16, Budapest, Hungary, 2016

Quantified Self

http://quantifiedself.com/data-visualization/

Reflecting

• Reflecting isaformofquantifiedself-analysisoverthelifearchivedatatodiscoverknowledgeandinsightsthatmaynotbeimmediatelyobvious.

• Example: NickFeltronAnnualReports

Image:©NickFeltron.

MyLifeBits

Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009

Interactive visualization

Hwang,Keum-SungandCho,Sung-Bae.“ALifelogbrowserforvisualizationandsearchofmobileeveryday-life."MobileInformationSystems,10(2013):243-258.Jeon,JaeHo andYeon,Jongheum andLee,Sang-gooandSeo,Jinwook.“ExploratoryVisualizationofSmartphone-basedLifeloggingDatausingSmartRealityTestbed.”InProc.BigDataandSmartComputing,pp.29-33,2014

Virtual reality

“BadTripisanimmersivevirtualrealityinstallation[…]thatenablespeopletonavigatethecreator'smindusingagamecontroller.SinceNovember2011,everymomentsofhislifehasbeendocumentedbyavideocameramountedonglasses,producinganexpandingdatabaseofdigitalizedvisualmemories.Using customvirtualrealitysoftware,hecreatedavirtualmindscapewherepeoplecouldnavigate,andexperiencehismemoriesanddreams.”

Souce:http://www.kwanalan.com

Virtual reality

Souce:http://www.kwanalan.com

Art installations

Kelly,PhilipandDoherty,AidenR.andSmeaton,AlanF.andGurrin,CathalandO’Connor,NoelE.“TheColourofLife:NovelVisualisationsofPopulationLifestyles."InProc.ACMMultimedia,pp.1063-1066,2010.

Image:Cou

rtesyofC.G

urrin

Displaying photo stream

Image:http://thenextweb.com/gadgets/2013/07/29/autographer-review-we-put-this-615-wearable-life-logging-camera-to-the-test/

Video Summary

Browsing in the Living Room

• Controlwithasuiteofgestures:ØNext/previouseventØNext/previousimageØNext/previousday,week,…

• Possibilityofpivotviewacrossmultipleaxes,e.g.,people,locations,…

Gurrin,CathalandLee,Hyowon andCaprani,NiamhandZheng,Zhenxing andO’Connor,NoelandCarthy,Denise.“BrowsingLargePersonalMultimediaArchivesinaLean-backEnvironment."InProc.MultimediaModeling Conference,pp.98-109,2010.

SenseCam Viewer

Doherty,AidenR.,Moulin,ChrisJ.A.,andSmeaton,AlanF.(2011)AutomaticallyAssistingHumanMemory:ASenseCam Browser.,Memory:SpecialIssueonSenseCam:TheFutureofEverydayResearch?TaylorandFrancis,19(7),785-795

Browsing Interface

Lee,Hyowon,Smeaton,AlanF.,O’Connor,NoelE.,Jones,GarethJ.F.,Blighe,Michael,Byrne,Daragh,Doherty,AidenR.,Gurrin,Cathal.“ConstructingaSenseCam visualdiaryasamediaprocess."MultimediaSystems,14(6):341-349,2008.

Lifelog Insight Tool

AaronDuane,RashmiGupta,LitingZhou,andCathalGurrin.“VisualInsightsfromPersonalLifelogs."InProc.NTCIR12,2016.

Highlighting Key Moments

Hopfgartner,F.andYang,YangandZhou,Lijuan andGurrin,Cathal.“UserInteractionTemplatesfortheDesignofLifeloggingSystems."InSemanticModelsforAdaptiveInteractiveSystems.Chapter10,pp.187-204,2013.

Lifelog Moment Retrieval“FindthemomentswhenI’mdrinkingcoffeeinfrontofmylaptop”

G.DeOliveiraBarra,A.CartasAyala,M.Bolanos,M.Dimiccoli,X.Giro-i-Nieto,P.Radeva.“LEMoRe:ALifelogEngineforMomentsRetrievalattheNTCIR-LifelogLSATTask."InProc.NTCIR12,2016.

Reminiscing

• Reminiscing isaboutstory-tellingorsharinglifeexperienceswithothers.

Image:CourtesyofC.Gurrin

With Events and Narrative

The Key Challenges

Capturing



Open Research Questions

• Multimediasummarisation• Handlingheterogeneousdatastreams• Visualisation oflifelogs• RetrievalandRecommendation• …

NTCIR

• WorkshopseriesfocusingonresearchonInformationAccess technologies(informationretrieval,questionanswering,textsummarisation,etc)

• InitiallysponsoredbyJapanSocietyforPromotionofScience (JSPS)

• Organisedsince1997inan18-monthscycle• NTCIR-12:January2015– June2016

NIITestCollectionforIRSystems

NTCIR-12 Tasks

NTCIR-12

§ Secondround:§ Search-IntentMining§ MobileClick§ TemporalInformationAccess§ SpokenQuery&SpokenDocumentRetrieval§ QALabforEntranceExam

§ Firstround:§ MedicalNLPforClinicalDocuments§ PersonalLifelog Access&Retrieval§ ShortTextConversation

Encourageresearchadvancesinorganisingandretrievingfromlifelogdata.

LifeLog @ NTCIR-12

C.Gurrin,H.Joho,F.Hopfgartner,L.Zhou,R.Albatal.OverviewofNTCIR-12LifelogTask.InProc.NTCIR-12,Tokyo,Japan,2016

Multimodal dataset with information needs

Createdbythreeindividualsover

10+days

TESTCOLLECTION

§ 18.18GB§ 88,124images§ Accompanyingoutputof

1,000concepts (825MB)§ Dataprocessedpre-release

(removalofpersonalcontent;faceblurring,translationofconcepts)

§ Detaileduserqueriesandjudgmentsgeneratedbythelifeloggingdatagatherers

C.Gurrin,H.Joho,F.Hopfgartner,L.Zhou,R.Albatal.NTCIRLifelog:TheFirstTestCollectionforLifelogResearch.InProc. SIGIR’16,toappear.

Tasks

Evaluatedifferentmethodsofretrievalandaccess.

T1:LIFELOGSEMAN

TICAC

CESS(LSAT) § Modelstheretrievalneed

fromlifelogs(Known-itemSearch)

§ RetrieveNsegmentsthatmatchinformationneed

§ InteractiveorAutomaticparticipation

§ Interactive:Timelimitforfairandcomparativeevaluationinaninteractivesystemwithusers

§ Automatic:Fully-automaticretrievalsystem.Automatedqueryprocessing

T2:LIFELOGINSIGH

T

§ Modelstheneedforreflectionoverlifelogdata

§ Exploratorytask,theaimisto:§ Encouragebroad

participation§ Novelmethodsto

visualizeandexplorelifelogs

§ SamedataasLSATtask§ Presentedviademo/poster

Tasks

Evaluatedifferentmethodsofretrievalandaccess.

T1:LIFELOGSEMAN

TICAC

CESS(LSAT) § Aknownitemsearchtaskto

findmoments§ Automaticandinteractive

(4&1participants)§ 48queries§ Unitofretrievalwasthe

moment§ Anyimagewithina

momentcanbesubmitted

T2:LIFELOGINSIGH

T

§ Modelstheneedforreflectionoverlifelogdata

§ Exploratorytask,theaimisto:§ Encouragebroad

participation§ Novelmethodsto

visualizeandexplorelifelogs

§ SamedataasLSATtask§ Threeparticipants

Example LSAT Topic

Title: TowerBridge

Description: Findthemoment(s)whenIwaslookingatTowerBridgeinLondon

Narrative: Tobeconsideredrelevant,thefullspanofTowerBridgemustbevisible.MomentsofcrossingtheTowerBridgeorshowingsomesubsetofTowerBridge

arenotconsideredrelevant

Evaluation

topvtypicalautomaticruns Interactivevautomatic(best)runs

Example LIT Topics

Title:Whohasamorehealthylifestyle?

Description: Comparethelifestyleofallthreeuserswithinthedimensionofpersonalhealthandwellness

Narrative: Therearemanyaspectstoahealthylifestyle,suchastheamountofexercise,thefoodanddrinkconsumed,environmentalfactors,thelevelofsocialinteractionsandsleeptime.Thistopicisseekingtounderstandwhichoftheuserswouldbeconsideredtobethemosthealthy.Any

dimension(orcombinationofdimensions)ofhealthylifestyleisconsideredacceptableasapointofcomparison.

AaronDuane,RashmiGupta,LitingZhou,andCathalGurrin.“VisualInsightsfromPersonalLifelogs."InProc.NTCIR12,2016.

Task 1: Lifelog Semantic Access

Findthemoment(s)

whereIusemycoffeemachine.

Findthemoment(s)

whereIaminthekitchen

Findthemoment(s)whereIam

playingwithmyphone.

Findthemoment(s)whereIampreparingbreakfast.

http://ntcir-lifelog.computing.dcu.ie/

Task 2: Lifelog Insight Task

ProvideinsightsonthetimeIspendtakingbreakfast.

ProvideinsightsonthetimeI

spenddrivingtowork.

ProvideinsightsonthetimeI

spendreadingapaper.

ProvideinsightsonthetimeIspendworking

onthecomputer.


Evaluation (Task 1)

• Automaticrunsassumethattherewasnouserinvolvementinthesearchprocessbeyondspecifyingthequery.Thesearchsystemgeneratesarankedlistofupto100momentsforeachtopicandnotime.

• Interactiverunsassumethatthereisauserinvolvedinthesearchprocessthatgeneratesaqueryandselectswhichmomentsareconsideredcorrectforeachtopic.

Ø 1.Ininteractiveruns,themaximumtimeallowedforanytopicis5minutes

Ø 2.WeusedthetimeelapsedtocalculaterunperformanceatdifferenttimeCut-offs.TheCut-offswereselectedas10s,30s,60s,120s,300s.

• EvaluationMetricsØ MeanAveragePrecision(MAP)Ø NormalisedDiscountedCumulativeGain(NDCG)


Example results (Interactive Runs)


Shameless advertisement

ConsiderparticipatinginNTCIRLifelog2andpresentyourworkinEuropeor

Japan


NTCIR-12: Lifelog Glasgow-Tokyo session

Thankyouforyourattention


[email protected]

@OkapiBM25www.hopfgartner.co.uk

icme 2016 - tutorial on interactive search in video & lifelog repositories

Presentations & Public Speaking