livre: a video extension to the lire content-based image retrieval system
TRANSCRIPT
![Page 1: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/1.jpg)
LIVRE: A VIDEO EXTENSION TO THE LIRE CONTENT-BASED IMAGE RETRIEVAL SYSTEM
Degree’s Final Project DissertationTelecommunications Engineering
Gabriel de Oliveira
Supervisors:
Assoc. Prof. Mathias LuxAssoc. Prof. Xavier Giró
![Page 2: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/2.jpg)
Outline of the Thesis
1. Introductioni. Motivationii. Overview and previous work
2. Proposed solution: The LIvRE system i. Parsingii. Indexingiii.Retrieval
3. Validationi. Dataset
- Stanford I2V Newscasts datasetii. Experiments
- Quantitative evaluation- Qualitative evaluation - The thinking-aloud test.
4. Conclusions and Further Work
March – October 2015
Slide 2
![Page 3: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/3.jpg)
MotivationGoal: To develop an all-in-one open source system for CBVR.
• Server side requeriments:• Fast• Scalable• Flexible• Automated
• User interface requeriments: • Fast• OS and device independent• Mobile
Slide 3
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 4: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/4.jpg)
Overview and previous work
Slide 6
• Open source CBIR Library in Java• Apache Lucene core• Solr plugin
• Supports parsing, indexing and retrieval• Global and local descriptors• Web-based interface
[1] Mathias Lux. LIRE: Open source image retrieval in java. In Proceedings of the 21st ACM international conference on Multimedia, pages 843{846.ACM, 2013.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 5: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/5.jpg)
Developed solution: The LIvRE CBVR system
Slide 7
CBVR system - concept and requirements.
database
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 6: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/6.jpg)
Developed solution: The LIvRE CBVR system
Slide 7
User sideServer side
LIvRE CBVR system architecture.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 7: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/7.jpg)
Block 1: Parsing
System Architecture - Parsing
Slide 8
1. Find videos in any given folder structure.
2. Extract keyframes from those videos.
3. Parse extracted keyframes with selected image descriptors.- Color Layout, Edge Histogram, JCD and PHOG
4. Generate XML Documents with the Feature Vectors.
Tools are provided to:
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 8: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/8.jpg)
Block 2: Indexing
Fig. System Architecture
Slide 8
1. Find XML Documents containing the Feature Vectors
(generated from Parsing Block).
2. Upload XML documents to Solr.
3. Commit changes in Solr core.
Tools are provided to:
Fig. System Architecture - Indexing
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 9: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/9.jpg)
Block 3: Retrieval
System Architecture - Retrieval
Slide 8
1. Image search field.
2. Settings.
User web-based interface input:
Web-based user interface input as displayed on small screen devices.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 10: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/10.jpg)
Block 3: Retrieval
System Architecture - Retrieval
Slide 8
1. Image search field.
2. Settings.
User web-based interface input:
Web-based user interface input as displayed on small screen devices.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 11: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/11.jpg)
Block 3: Retrieval
Slide 8
1. Candidate videos displayed using HTML5.
2. Thumbnails with other similar frames.
3. Time refinement.
4. Video information.
User web-based results presentation:
System Architecture - Retrieval Retrieval results presentation for small screen devices
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 12: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/12.jpg)
Block 3: Retrieval
Slide 8
1. Candidate videos displayed using HTML5.
2. Thumbnails with other similar frames.
3. Time refinement and ranking.
4. Video information.
User web-based results presentation:
Fig. System Architecture - Retrieval Fig. Retrieval results presentation for small screen devices
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 13: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/13.jpg)
LIvRE CBVR system demo
Introduction · Overview · LIvRE CBVR system · Dataset · Experiments · Conclusions
Slide 9
![Page 14: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/14.jpg)
ValidationStanford I2V Dataset
Freely available data set. Large (~1TB Video)
• 23,443 video clips• Average video duration: 2,65min.• Keyframes @1fps: 3,808,760• Video hours: 1,035h
Ground-truth• 78 queries
Some query images and video frames from the Stanford I2V dataset.
Slide 14
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 15: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/15.jpg)
ValidationExperiments
LIvRE CBVR system tested with 2 different evaluation methods:
Slide 16
12
Quantitative evaluation
Qualitative evaluation(Thinking-aloud Test)
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 16: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/16.jpg)
Quantitative study:
• Use ground-truth provided with the dataset for:
• Scene Retrieval evaluation (finding the right video).
• Time Refinement evaluation (finding the right moment of time at the right video).
Qualitative study:
• Web-based user interface.
• Thinking-aloud Test (offline).
• Participants are expert and non expert users.
• 4 Non-expert users.
• 2 Expert users. Slide 17
1
2
ValidationExperiments
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 17: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/17.jpg)
Quantitative study
Slide 18
LIvRE
Quantitative study evaluation process from provided ground-truth
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 18: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/18.jpg)
Quantitative study
Slide 18
1st Stage: Scene Retrieval
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 19: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/19.jpg)
Quantitative study
Slide 18
2nd Stage: Temporal Refinement
Temporal Refinement results for 100k candidates
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 20: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/20.jpg)
Qualitative study
Thinking-aloud Test
• Volunteer participants perform specific tasks with the web-based user interface.
• LIvRE CBVR system is running locally (offline) on the machine.
• Participants show their thoughts in loud-voice.
• Sessions are recorded and evaluated.
Slide 18
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 21: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/21.jpg)
Qualitative studyThinking-aloud Test
Slide 19
Sample input query frames Screenshots from Thinking-aloud test 1
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
Timing results for 50K candidates (in miliseconds)
![Page 22: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/22.jpg)
Conclusions and Future Work
- New CBVR System, LIvRE, was developed as an extension of LIRE. - LIvRE is now a branch of the LIRE Solr Project.
Slide 28
Future work: • Local image descriptors.• Integration of sound descriptors.• Simplified set-up and deployment.• Demo paper at ICMR 2016.• Add Video annotation tool.• Integration with computer vision / deep learning projects.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
![Page 23: LIvRE: A Video Extension to the LIRE Content-Based Image Retrieval System](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a5b2841a28ab1a628b66f5/html5/thumbnails/23.jpg)
Thank you for your attentionDo you have any question?
8 October 2015
LIvRE: A Video Extension to the LIREContent-Based Image Retrieval System.
Gabriel de Oliveira