![Page 1: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/1.jpg)
Video retrieval using inference network
A.Graves, M. Lalmas
In Sig IR 02
![Page 2: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/2.jpg)
Outline
Background Mpeg 7 Inference network model Experiment Conclusion
![Page 3: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/3.jpg)
Background
CBVR: find the video upon demand The semantic gap: high level concepts and low level
features Traditional method: retrieve by example Relevance feedback: tedious for video retrieval Retrieval by semantics?
![Page 4: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/4.jpg)
Mpeg 7
Multimedia Content description interface Attach metadata to multimedia content
Semantics: event, actor, place……Structure: shot, scene, group……
Video as a structured document The information contained in the Mpeg 7 annotation
can be exploited to perform semantic based video retrieval
Mpeg 7 does not provide the solution to extract the annotation
![Page 5: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/5.jpg)
Mpeg 7
Description definition language Descriptors Description schemes
![Page 6: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/6.jpg)
Mpeg 7
Structured document and description <TextAnnotation>
<FreeTextAnnotation> Basil attempts to mend the car without success
</FreeTextAnnotation> <StructuredAnnotation>
<Who>Basil</Who> <WhatObject>Car</WhatObject> <WhatAction>Mend</WhatAction> <Where>Carpark</Where>
</StructuredAnnotation> </TextAnnotation> <Video structures>……
<Video shot 13>… …</>
![Page 7: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/7.jpg)
Inference network
Perform a ranking given many sources of evidence Document network (DN)
Constructed from the document data, represents all the retrievable units
Query network (QN)Constructed from the query, represents the information needs
![Page 8: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/8.jpg)
Inference network
The document networkDocument layer (retrievable units collection)Contextual layer (represents the contextual information about
the document-concept links)Concept layer (represent all the concepts in the network)
![Page 9: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/9.jpg)
Inference network
Document network
![Page 10: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/10.jpg)
Inference network
Link weight calculationStructural: duration ratio (Between document nodes)Contextual (Between contextual nodes) sibling number,
context size, frequencyContext – concept (tf, idf)
![Page 11: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/11.jpg)
Inference network
Query networkA framework of nodes that represents the information needConcept nodes Constraint operators (and, or, sum, not…), context-conceptual
constraints
![Page 12: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/12.jpg)
Inference network
![Page 13: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/13.jpg)
Inference network
Attachment and EvaluationAttachment: match the DN and QN, get a set of links between
their nodesAll the constraint satisfiedLink inheritance: in the DN, document node can share the
context nodes of its parent node Evaluation: calculated quantized similarity for each
document node in the DN
![Page 14: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/14.jpg)
Inference network
AttachmentLink the QN and DN such that:
The concept nodes that contains same concepts (need a synonym dictionary) with weight 1 (firm)
For constraint queries, adjust the weight
![Page 15: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/15.jpg)
Inference network
Attachment
![Page 16: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/16.jpg)
Inference network
EvaluationThe evaluation process can be done toward all
document nodes, it is calculated according to the query network
Can be used in different applications:Retrieve scene, shot from one videoRetrieve video from video collections…
![Page 17: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/17.jpg)
Inference network
EvaluationBack-propagate the QN-DN link weight back to each
document nodesAll the node will have a valueCan derive a rank at different granularity level (Video, scene,
shot)
![Page 18: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/18.jpg)
Experiment
3 manually annotated video are employed as test data, totally 329 shots
Annotations:<abstract> <Structure Annotation> <Freetext Annotation> for video shots and
scenes Avg. precision: 69.25 Similarity order are as expected
![Page 19: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/19.jpg)
Experiment
The performance are quite dependent on the quality of the annotation data
Efficient annotation methods will be quite helpful
![Page 20: Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02](https://reader038.vdocuments.mx/reader038/viewer/2022110322/56649d3b5503460f94a15e04/html5/thumbnails/20.jpg)
Conclusion
Propose a semantic video retrieval model based on Inference Network model, fully exploits the structural, conceptual, and contextual aspects of Mpeg 7.
Parry the semantic gap problem