TRECVID2016WorkshopNa6onalIns6tuteofStandardsandTechnology

Mul6mediaEventDetec6onTaskNov.15,2016

DavidJoy,JonathanFiscus,AndrewDelgado

Pleasecontactmed_poc@nist.govforques6ons/comments

MEDSessionSchedule9:00–11:20 Tuesday,Nov.15

9:00–9:20 MEDTaskOverview

9:20–9:40 VIREO(CityUniversityofHongKong)

9:40–10:00 INF(CarnegieMellonU.;BeijingU.ofPostsandTelecommunica6on;U.AutonomadeMadrid;ShandongU.;XianJiatongU.;SingaporeManagementU.)

10:00–10:20 MediaMill(UniversityofAmsterdam)

10:20–10:40 Break

10:40–11:00 BUPT-MCPRL(BeijingUniversityofPostsandTelecommunica6ons)

11:00–11:20 MEDDiscussion

Quicklyfindinstancesofeventsinalargecollec6onofsearchvideos

Multimedia Event Detection (MED)

Execu6onHardwareRepor6ng•  3ClassesofCompu6ngHardware•  Small: 100CPUcores,1,000GPUcores•  Medium: 1,000CPUcores,10,000GPUcores•  Large: 3,000CPUcores,30,000GPUcores

Evaluation Conditions

EventModel

No6onalSystemDiagram

Evalua6onData

MetadataMED

ModelGenera6on

QueryExecu6on

RankedVideosid036id839id983id312id033id239id783id912…

EventKit

QueryTrainingCondi6ons

NumberofExemplars

Pre-SpecifiedEvents 0 10 100

Ad-HocEvents 10

Interac6veAd-HocEvents

10

SearchCollec6on•  MED16Eval-Full->198Kvideos,4,738hours•  MED16Eval-Sub->32Kvideosubset,783hours

Mul6mediaEventDetec6onTask

AMEDeventisacomplexac6vityoccurringataspecificplaceand6meinvolvingpeopleinterac6ngwithotherpeopleand/orobjects

MED‘16Overview

•  MEDevalua6onsfrom2010-2016– SupportedbytheIARPAAladdinProgramandLDCcollecteddata

– Severalsimplifica6onsin2015,whichwerecon6nuedin2016

•  What’snewinMED2016–  Introduc6onofnewtestdataset,asubsetofthe*Yahoo!FlickrCrea6veCommons100Million(YFCC100M)videos

– 10newAd-Hocevents*-Disclaimer:Certaincommercialequipment,instruments,ormaterialsareiden6fiedinthispaperinordertospecifytheexperimentalprocedureadequately.Suchiden6fica6onisnotintendedtoimplyrecommenda6onorendorsementbytheNa6onalIns6tuteofStandardsandTechnology,norisitintendedtoimplythatthematerialsorequipmentiden6fiedarenecessarilythebestavailableforthepurpose.

TheTRECVIDMED2016EventsPre-SpecifiedEvents Ad-HocEvents

MED‘14PSEvents MED‘14AHEvents NewEvents

Aqemp6ngabiketrick Beekeeping Camping

Cleaninganappliance Weddingshower CrossingaBarrier

Dogshow Non-motorizedveh.repair OpeningaPackage

Givingdirec6onstoaloca6on Fixingmusicalinstrument MakingaSandSculpture

Marriageproposal Horseridingcompe66on MissingaShotonaNet

Renova6ngahome Fellingatree Opera6ngaRemoteControlledVehicle

Rockclimbing Parkingavehicle PlayingaBoardGame

Townhallmee6ng Playingfetch MakingaSnowSculpture

WinningaracewithoutavehicleTailga6ng MakingaBeverage

Workingonametalcrassproject

Tuningmusicalinstrument Cheerleading

OperaTngaRemoteControlledVehicle

Illustrative Examples •  Positive instances of the event •  Non-Positive “miss” clips that do not contain the event

EvidenTalDescripTon:•  scene:indoorsoroutdoors•  objects/people:remotecontrolvehicles(cars,trucks,planes,

helicopters,trains,etc.),remotes,antennas,racetrack,plas6ctakeofframp

•  ac6vi6es:direc6ngremotecontrolvehicles,turningonvehicles,crashingvehicles

•  audio:enginesrevving,motorwhirring,explana6onoftypeofvehicle,discussionofwherethevehicleisgoingorcouldgo,discussionofwhatisseenonavideofeedfromthevehicle

Definition: An individual operates a vehicle remotely with a controller Explication: Remote controlled vehicles are self-propelled machines that are powered by a motor or engine of some kind and whose movement is controlled from a distance by human inputs to a remote control device …

ExampleEventKit

Miss!

TheTestDataDatacollecTon

Testset #ofvideos

DuraTon(Hrs)

Avg.duraTon(Secs)

HAVICProgress

MED16EvalFull 98,003 3,713 136

MED16EvalSub 16,000 620 139

YFCC100MSubset

MED16EvalFull 100,000 1,025 37

MED16EvalSub 16,000 163 37

Total MED16EvalFull 198,003 4,738 86

MED16EvalSub 32,000 783 88

•  HAVICProgress–  Engineeredtarget

richness–  Controlledsampling

ofInternetvideodomain

•  YFCC100MSubset–  Randomselec6on*–  Shorterdura6on

videos

*-ExcludingYLI-MEDcorpusvideos(~50kvideos);Excludingvideosnotavailablebymmcommons.org’sAWSS3datastore(~5kvideos)

12MED2016FinishersByCondi6onYears

Team

AH PS

Organiza6on10Ex 0Ex 10Ex 100Ex

SML MED SML MED SML MED SML MED

6

INF Sub Sub Sub CarnegieMellonUniversityetal.MediaMill Full Full MediaMill-UniversityofAmsterdamNIIHitachiUIT Full Sub Na6onalIns6tuteofInforma6csTokyoTech Full Full Full TokyoIns6tuteofTechnology

5 VIREO Full Full Full Full CityUniversityofHongKong&TNO

3

ITICERTH Sub Sub Sub Informa6csandTelema6csInst.KU-ISPL Sub Sub KoreaUniversity

nwudan Full Full FullNTTMediaIntelligenceLaboratoriesandFudanUniversity

MCIS Sub Sub BeijingIns6tuteofTechnologyMcislab

2 BUPTMCPRL Sub SubMul6mediaCommunica6onandPaqernRecogni6onLabsBUPT

Eqer Sub Sub EqerSolu6ons1 PKUMI Full PekingUniversity

3 1 4 1 10 1 8 1

0

10

20

2010(Pilot)2011 2012 2013 2014 2015 2016

NumberofMEDFinishers

AH–Ad-Hoceventcondi6onPS–Pre-Specifedeventcondi6on0Ex–0exemplarcondi6on10Ex–10exemplarcondi6on100Ex–100exemplarcondi6onSML–Small-sizedhardwareMED–Medium-sizedhardwareFull–processedMED16EvalFulltestsetSub–processedMED16EvalSubtestset-Redoutlineindicatesarequiredcondi6on

Pre-SpecifiedEventMAPPrimarySystems–HAVICProgressSubset

MAP(EvalSub-ProgressSubset)=1.02*MAP(EvalFull-ProgressSubset)+5.96R^2=0.996

EvalFull EvalSub

Pre-SpecifiedAPbySystemandEventPrimarySystems–10Ex–MED16EvalSub–MixedSystemSize

ProgressSubset

MAPàMeanInferredAveragePrecision(MInfAP)

•  FollowsAslametal.procedure,Sta6s6calMethodforSystemEvalua6onUsingIncompleteJudgmentsProceedingsofthe29thACMSIGIRConference,Seaqle,2006.

–  Stra6fied,variabledensity,pooledassessmentproceduretoapproximateMAP

•  MInfAPinthe2016evalua6on–  Progress–MAPandMInfAP200

(simulated)onPSandAH–  Progress+YFCC100M–MInfAP200onPS

andAH•  ForMED‘15,NISTranexperimentswith

2014datatoop6mizethestratasizesandsamplingrate.ThissamesamplingratewasusedforMED‘16

–  Define2strata•  1-60->100%•  61-200->20%

PS-EvalSub-ProgressSubset--MInfAP200(Simulated)=1.14*MAP+0.421R^2=0.99

Pre-SpecifiedEvalSubSimulatedMInfAP200ProgressSubset

Pre-SpecifiedEventMInfAP200Progress+YFCC100M

EvalFull EvalSub

PerformanceonHAVICvs.Yahoo!OurAqempttoScorePrecision@10

•  UnabletoscorePrecision@10forbothHAVICandYahoo!–  Stra6fiedsamplingdidnotyieldsufficientjudgements

•  Forexample:–  E022–Cleaninganappliance

–  Propor6onofsubsetoftop200clips,binnedbyrank

–  Teamsshownarerepresenta6vebasedonMInfAP200scores

PerformanceonHAVICvs.Yahoo!Good,Bad,UglyEvents

Event AP(top5) Descrip6onE022 8.42 CleaninganapplianceE028 53.78 Townhallmee6ngE039 61.4 Tailga6ng

•  Stra6fiedrandomsamplenotsufficientforheterogeneousdata

•  Wewillcon6nuetoworkonscoringYahoo!separately

Ad-HocEventResults

•  10newevents–  10Exemplartrainingonly

•  MED16EvalFulltherequiredcondi6on

•  ReferenceGenera6on–  Pooledassessmentwithusingall

submissions–  Stratadefini6on

•  1:60:100%•  61:200:20%

Ad-HocMInfAP200onEvalFull

Ad-HocInfAPbySystemandEventPrimarySystems–10Ex–MED16EvalFull–MixedSystemSize

Ad-HocPooledAssessmentEventRichnessvs.InfAP

E051 CampingE052 CrossingaBarrierE053 OpeningaPackageE054 MakingaSandSculptureE055 MissingaShotonaNetE056 Opera6ngaRemoteControlledVehicleE057 PlayingaBoardGameE058 MakingaSnowSculptureE059 MakingaBeverageE060 Cheerleading

Ad-Hoc10Ex–MInfAP200BoxPlotsEventRichnessinAnnota6onPools

MED‘16Summary

•  Pre-SpecifiedResults–  Onlyoneteambuilta“Medium”hardwaresystem– Mostteamsprocessedthesubset(783hr.)testset(MED16EvalSub)

–  No6ceableimprovementoverlastyear’sPre-SpecifiedresultsonProgress

–  Stra6fiedrandomsamplingonheterogeneousdatanotpowerfulenoughtodeterminedifferencesinperformance

•  Ad-HocResults–  Only4of12teamspar6cipated–  Noteamspar6cipatedinInterac6veEventQuerytest

MED’17Plans

•  NISTintendstocon6nueMEDinastreamlinedfashion

•  NISTtoreleaseProgressannota6ons•  MakeupofdatasetsTBD,HAVIC+YFCC100M•  Discon6nuingsupportforInterac6veAd-Hoccondi6on

•  Counter-proposals?

Thankyou!

Ques6ons?