classifying visual objects regardless of depictive style qi wu, peter hall department of computer...
TRANSCRIPT
Classifying Visual Objects Regardless of Depictive Style
Qi Wu, Peter HallDepartment of Computer Science
University of Bath
Summary
• Conventional Comp.Vis. classifiers do not generalise well across depictive styles.
• We propose new visual class model,one invariant to depictive style.
• Experiments validate our model.
People can see objects in a wide variety of depictive styles.
Photos Artwork
Literature Gap: BoW does not generalise across depictive styles
Photos Artwork47% (Dense SIFT)
Photos Artwork51% (Dense SIFT)
Our solution: A new Visual Class Model that does generalise across styles.
Photos Artwork47% (Dense SIFT)
Photos Artwork51% (Dense SIFT)
64%
67%
A New Visual Class Model• We assume an object class is characterised by:
– the qualitative shape of object parts,– the structural arrangement of those parts.
• A hierarchical graph model per image:– coarse-to-fine representation (layered),– nodes labelled by primitive shapes,
• abstracting region shape brings greater robustness.
– arcs labelled with displacement vectors
• Median graph models:– aggregates models from several instances,– single class model.
Making a VCM
(a): An input collection. (b): Probability maps for each input image, and graph models for each map. (c): The median graph model for the whole class. (d): The refined median graph as the final class model
A schematic VCM• A hierarchical description
• Berkeley segmentation• Filtering process using cLge
• A graph• Arcs at same level denote
touching neighbours.• Arcs between layers link
parent – children.
• Nodes label• A 6-elements
probability vector.• The probability that a
region belongs to a given prime shape class.
Prime Shapes,BMVC 2012
Prime Shapes• Does a set of elementary planar shapes
appear commonly in the world ?• Art provides strong anecdotal evidence “yes”– 20th century Western Art --- Cubism
Determine Prime Shapes• A fully unsupervised framework
Determine Prime Shapes
Back to our model…
Build graphs, one for each image
Left: graph model. Right: Object broken in primitive shapes
Compute an initial Visual Class Model• Median Graph
• First compute the graph distance between each pair.
• Using the distance matrix to embed graph into a vector space
• Compute the Euclidean Median of all the data points.
• Transfer the median vector back to graph using a state-of-art method proposed in [Ferrer and Valveny, 2008]
Refine the Visual Class ModelThe initial model contains nodes and arcs that derive from visual clutter in back ground of images in the training set
• Refine the model• Match the median back
into each training image.
• Count the number of times a given node in the model appears in the training data.
• Delete all nodes below a frequency threshold., which is computed via maximising matching score.
Some Examples
Experiments
• Compare with other two methods• PHOW features (Dense SIFT) [Bosch and Zisserman, ICCV
2007]• Local PAS features [Ferrari and Jurie, IJCV 2010]• Structure Only [Bai and Song, CVIU 2011]
Results
Conclusions
• It’s possible to learn models of objects classes that generalise across depictive styles.
• Many applications are promised.
• Just a first step– Simplify the model, still too much nodes and arcs.– Time consuming.– Additional labelling– Move to object localisation.
Questions?
One application of Prime shapes