a critical view of context
DESCRIPTION
A Critical View of context. Biologically Inspired Models of Vision course. Alexandru Rusu , Guillaume Lemaître , Isabel Rodes and Oscar Ramos. Introduction. Extraction Low Level Image Features Extraction Semantic Image Features Building the Context Features - PowerPoint PPT PresentationTRANSCRIPT
A CRITICAL VIEW OF CONTEXTBiologically Inspired Models of Vision course
Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos
INTRODUCTION
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 2
• Extraction Low Level Image Features
• Extraction Semantic Image Features
• Building the Context Features
• Experiments and Results
• Improvements
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
INTRODUCTION
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 3
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
LOW LEVEL IMAGE FEATURES EXTRACTIONOVERVIEW
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 4
• Downsize the images to 60 × 80 pixels
• Extract color information
• Extract texture information
• Extract global position information
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 5
LOW LEVEL IMAGE FEATURES EXTRACTIONCOLOR FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
L* Component
a* Component
b* Component
RGB to CIE L*a*b*
RGB image
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 6
LOW LEVEL IMAGE FEATURES EXTRACTIONCOLOR FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
L* Component
a* Component
b* Component
L* Component
a* Component
b* Component
RGB image
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 7
LOW LEVEL IMAGE FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
Color Features
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 8
LOW LEVEL IMAGE FEATURES EXTRACTIONTEXTURE FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
Polarity
Anisotropy
Contrast
RGB image
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 9
LOW LEVEL IMAGE FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
Color Features Texture Features
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 10
LOW LEVEL IMAGE FEATURES EXTRACTIONPOSITION FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
RGB image
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 11
LOW LEVEL IMAGE FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.[2] - Carson, C., Belongie, S., Greenspan, H., and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV.
Color Features Texture Features Position Features
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 12
SEMANTICS FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
• Semantic Layers used:
• Example (for building): 1=building, 0=no building
- Building - Tree
- Road - Sky
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 13
SEMANTICS FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
• In Test images: no ground truth → Use 4 SVM binary classifiers (input: low-level feature image)
• Training set: 10 000 samples per category
True Semantic
Label
LearnedSemantic
Label
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 14
SEMANTICS FEATURES EXTRACTION
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
• ROC curve for the SVM classifiers
Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 15
BUILDING THE CONTEXT FEATURES
21/11/2010
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
• Image has been converted to 20 layers:- 4 binary semantic features - 3 color features- 3 texture features - 10 global position features
• Data sampled at 8 orientations and radii of 3,5,10,15,20 pixels
• 40 samples:• (40)(20) = 800 dimensional
context feature per pixel
Green: size of carRed: size of pedestrian
Experiments and results
Fidelity of semantic information
Empirical semantic features:• Four SVMs• Four classes: building, tree, road, sky
• Three features: colour, texture, position
• Training and testing• Cross-validation and ROC
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 16
Performance of the context detector• Comparison with ROC curves of true high-level context detector and
appearance detector• Appearance detector outperforms the context-based
Experiments and results
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 17
Relative importance of context features• Comparison of four context classifiers• Low-level feature-based detection only marginally improved by addition
of semantic features
Experiments and results
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 18
Relative importance of context features• Testing of possible overlap of context with target object• Low-level and high-level classifiers at d ϵ{3,5,10,15,20}• Semantic features only important at farther distances
Experiments and results
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 19
IMPROVING OBJECT DETECTION WITH CONTEXTDataflow of the a rejection cascade
• Tune the thresholds THC and THA.
• Different ROCs
• Validation set of 200 images[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 20
IMPROVING OBJECT DETECTION WITH CONTEXT• Tuning the context threshold
• The ROCs of full system performance
• Three different objects• Horizontal lines indicate the performance of the system with no context• The marks the selected parameters for the system
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 21
CONCLUSIONS
• An effective context detection system
• Rejection cascade architecture
• Importance of contextual cues
• Good performance when the appearance information is weak (critically low resolution and very noisy images)
• Ways of extracting context information
[1] - Wolf, L., and Bileschi, S. 2006. A critical view of context, IJCV.
21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 22