Content based Image Content based Image Retrieval using SVD Retrieval using SVD

and SVMand SVM

Need for image data Need for image data managementmanagement

► The process of digitisation does not in The process of digitisation does not in itself make image collections easier to itself make image collections easier to manage. Some form of cataloguing and manage. Some form of cataloguing and indexing is still necessary – the only indexing is still necessary – the only difference being that much of the difference being that much of the required information can now potentially required information can now potentially be derived automatically from the images be derived automatically from the images themselves.themselves.

► While it is perfectly feasible to identify a While it is perfectly feasible to identify a desired image from a small collection desired image from a small collection simply by browsing, more effective simply by browsing, more effective techniques are needed with collections techniques are needed with collections containing thousands of items which need containing thousands of items which need some form of access by image content. some form of access by image content.

What is CBIR?What is CBIR? Process of retrieving desired images Process of retrieving desired images

from a large collection on the basis from a large collection on the basis of features (such as colour, texture of features (such as colour, texture and shape) that can be automatically and shape) that can be automatically extracted from the images extracted from the images themselves. themselves.

Also known as query by image Also known as query by image content (content (QBICQBIC) and content-based ) and content-based visual information retrieval (visual information retrieval (CBVIRCBVIR))

► ““Content-based” means that the search Content-based” means that the search will analyze the actual contents of the will analyze the actual contents of the image. The term 'content' in this image. The term 'content' in this context might refer to colors, shapes, context might refer to colors, shapes, textures, or any other information that textures, or any other information that can be derived from the image itself. can be derived from the image itself.

► Indexing is often used as identifying Indexing is often used as identifying features within an image; with indexing features within an image; with indexing data structures we here mean data structures we here mean structures to speed up the retrieval of structures to speed up the retrieval of features within image collections. features within image collections.

Practical applications of Practical applications of CBIR CBIR

► Crime preventionCrime prevention► The military The military ► Intellectual propertyIntellectual property► Architectural and engineering design Architectural and engineering design ► Fashion and interior designFashion and interior design► Journalism and advertisingJournalism and advertising► Medical diagnosis Medical diagnosis ► Geographical information and remote sensing Geographical information and remote sensing

systems systems ► Cultural heritageCultural heritage► Education and trainingEducation and training► Home entertainmentHome entertainment► Web searchingWeb searching

Query techniquesQuery techniques a)Query by examplea)Query by example Query by example is a query Query by example is a query

technique that involves providing the CBIR technique that involves providing the CBIR system with an system with an example imageexample image that it will that it will then base its search upon. The underlying then base its search upon. The underlying search algorithms may vary depending on search algorithms may vary depending on the application, but result images should the application, but result images should all share common elements with the all share common elements with the provided example.provided example.

b) Semantic retrievalb) Semantic retrieval The ideal CBIR system from a The ideal CBIR system from a user user

perspectiveperspective would involve what is referred would involve what is referred to as to as semanticsemantic retrieval.This type of open- retrieval.This type of open-ended task is very difficult for computers ended task is very difficult for computers to perform.to perform.

Current CBIR systems therefore generally Current CBIR systems therefore generally make use of lower-level features like make use of lower-level features like texture, color, and shape, although some texture, color, and shape, although some systems take advantage of very common systems take advantage of very common higher-level features like faces. higher-level features like faces.

Other query methods include Other query methods include browsing for example images, browsing for example images, navigating customized/hierarchical navigating customized/hierarchical categories, querying by image region categories, querying by image region (rather than the entire image), (rather than the entire image), querying by multiple example images, querying by multiple example images, querying by visual sketch, querying by querying by visual sketch, querying by direct specification of image features, direct specification of image features, and multimodal queries (e.g. and multimodal queries (e.g. combining touch, voice, etc.). combining touch, voice, etc.).

c)Other query methodsc)Other query methods

Semantic representation for Semantic representation for image retrieval image retrieval

It is very difficult to manually annotate images It is very difficult to manually annotate images and videos as well as retrieve raw data. The and videos as well as retrieve raw data. The main problem in semantic labeling is a wide main problem in semantic labeling is a wide variation in visual appearance within objects variation in visual appearance within objects of the same class. of the same class.

An alternative approach is to build a statistical An alternative approach is to build a statistical learning system that starts with a minimal learning system that starts with a minimal knowledge and simple descriptors. Users then knowledge and simple descriptors. Users then train the systemtrain the system via formative feedback via formative feedback before it can work with large databases. before it can work with large databases.

Feature extraction using SVDFeature extraction using SVDIn linear algebra, the In linear algebra, the singular value singular value decompositiondecomposition ( (SVDSVD) is an important ) is an important factorization of a rectangular real or factorization of a rectangular real or complex matrix, with several applications complex matrix, with several applications in signal processing and statistics. in signal processing and statistics. Statement Statement

Suppose Suppose MM is an is an mm-by--by-nn matrix whose matrix whose entries come from the field entries come from the field KK, which is , which is either the field of real numbers or the field either the field of real numbers or the field of complex numbers. Then there exists a of complex numbers. Then there exists a factorization of the formfactorization of the form

where where UU is an is an mm-by--by-mm unitary matrix over unitary matrix over KK, , Σ is Σ is mm-by--by-nn diagonal matrix with diagonal matrix with nonnegative numbers on the nonnegative numbers on the

diagonaldiagonal V*V* denotes the conjugate transpose of denotes the conjugate transpose of

V,V, an an nn-by--by-nn unitary matrix over unitary matrix over KK.. Such a factorization is called a Such a factorization is called a singular-singular-

value decompositionvalue decomposition of of MM..

Matrix Matrix VV contains a set of orthonormal contains a set of orthonormal "input" or "analysing" basis vector "input" or "analysing" basis vector directions for directions for MM.. Matrix U contains a set of orthonormal "output" basis vector directions for M.

Matrix Σ contains the singular values, which can be thought of as scalar "gain controls" by which each corresponding input is multiplied to give a corresponding output.

The diagonal matrix Σ is uniquely determined by M.

Classification of data classes Classification of data classes using support vector machine using support vector machine (SVM)(SVM) SVMsSVMs are a set of related are a set of related

supervised learning methods used for supervised learning methods used for classification .classification .

Viewing input data as two sets of Viewing input data as two sets of vectors in an vectors in an nn-dimensional space, an -dimensional space, an SVM will construct a separating SVM will construct a separating hyperplane in that space, one which hyperplane in that space, one which maximizes the maximizes the marginmargin between the between the two data sets. two data sets.

To calculate the margin, two To calculate the margin, two parallel hyperplanes are parallel hyperplanes are constructed, one on each side of constructed, one on each side of the separating hyperplane, which the separating hyperplane, which are "pushed up against" the two are "pushed up against" the two data sets.data sets.

In general the larger the margin In general the larger the margin the better the generalization error the better the generalization error of the classifier.of the classifier.

In the case of SVM, a data point is In the case of SVM, a data point is viewed as a viewed as a pp-dimensional vector and -dimensional vector and we can separate such points with a we can separate such points with a pp − 1-dimensional hyperplane. This is − 1-dimensional hyperplane. This is called a called a linear classifier.linear classifier.

There are many hyperplanes that There are many hyperplanes that might classify the data. We pick the might classify the data. We pick the hyperplane so that the distance from hyperplane so that the distance from the hyperplane to the nearest data the hyperplane to the nearest data point is maximized.point is maximized. such a linear such a linear classifier is known as a classifier is known as a maximum-maximum-margin classifier. margin classifier.

A set of features that describes one case is A set of features that describes one case is called a called a vector. vector. The vectors near the hyper The vectors near the hyper plane are the plane are the support vectorssupport vectors. . The distance between the dashed lines shown in the figure is called the margin.

Formalization of SVMFormalization of SVMWe are given some training data, a set of points We are given some training data, a set of points of the formof the form

where the where the cici is either 1 or −1, indicating the class is either 1 or −1, indicating the class to which the point belongs. Each is a to which the point belongs. Each is a pp--dimensional real vector. We want to give the dimensional real vector. We want to give the maximum-margin hyperplane which divides the maximum-margin hyperplane which divides the points having points having cici = 1 from those having = 1 from those having cici = − 1. = − 1. Any hyperplane can be written as the set of Any hyperplane can be written as the set of points satisfying points satisfying

The vector is a normal vector: it is The vector is a normal vector: it is perpendicular to the hyperplane.perpendicular to the hyperplane.

The parameter determines the offset of the The parameter determines the offset of the hyperplane from the origin along the normal hyperplane from the origin along the normal vector .vector .

We want to choose the and We want to choose the and bb to maximize to maximize the margin.the margin.

The hyperplanes can be described by the The hyperplanes can be described by the equations equations and and If the training data are linearly separable, we If the training data are linearly separable, we can select the two hyperplanes of the margin can select the two hyperplanes of the margin in a way that there are no points between in a way that there are no points between them and then try to maximize their distance. them and then try to maximize their distance. By using geometry, we find the distance By using geometry, we find the distance between these two hyperplanes is , so we between these two hyperplanes is , so we want to minimize . As we also have to want to minimize . As we also have to prevent data points falling into the margin, we prevent data points falling into the margin, we add the following constraint: add the following constraint:

Contd…….Contd……. for each for each ii either either

for of the first class for of the first class oror

for of the second for of the second class.class.

We can put this together to get the We can put this together to get the optimization problem: choose to minimize optimization problem: choose to minimize subject tosubject to

Contd….Contd….► The optimization problem presented is The optimization problem presented is

difficult to solve because it depends on ||difficult to solve because it depends on ||ww||, the norm of ||, the norm of w'w', which involves a , which involves a square root. square root.

► It is possible to alter the equation by It is possible to alter the equation by substituting ||substituting ||ww|| with without || with without changing the solution .This is a quadratic changing the solution .This is a quadratic programming (QP) optimization problem. programming (QP) optimization problem. More clearly, minimize , subject to More clearly, minimize , subject to

Experimental work and Experimental work and resultsresults

► We have taken a database consisting We have taken a database consisting of 20 different classes of images each of 20 different classes of images each class consisting of 72 images.class consisting of 72 images.

► The different classes of images that The different classes of images that were taken in the database are as were taken in the database are as shown below:shown below:

Contd…Contd…

Consider that the image of class that is shown Consider that the image of class that is shown below is given as querybelow is given as query

And the images that are retrieved for And the images that are retrieved for the the query if the first ten relevant images are retrieved query if the first ten relevant images are retrieved are as follows:are as follows:

The top eight images among the retrieved The top eight images among the retrieved images are relevant to the given query and images are relevant to the given query and the last two are not.the last two are not.

Thus each image of the database is given Thus each image of the database is given as query and the number of relevant as query and the number of relevant images retrieved amongst the total images images retrieved amongst the total images that fall into the zone of query is that fall into the zone of query is calculated.calculated.

ConclusionConclusion• The classification is done based on the The classification is done based on the

ratio of number of images that fall into ratio of number of images that fall into the relevant category.the relevant category.

• The final result is 64.985%.The final result is 64.985%.• The number of relevant images The number of relevant images

retrieved and the total number of retrieved and the total number of images that fall into a particular zone images that fall into a particular zone concerned to the given query images concerned to the given query images from different classes of database are from different classes of database are tabulated.tabulated.