semi-supervised biased maximum margin analysis...

47
Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback Image Retrieval Institute for Media Innovation and School of Electrical & Electronic Engineering Supervisor : Assoc Prof. Wang Lipo School of Electrical &Electronic Engineering Co-supervisor: Assoc Prof. Lin Weisi School of Computer Engineering PhD Student: Zhang Lining

Upload: others

Post on 12-Jul-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Semi-Supervised Biased Maximum

Margin Analysis for SVM Relevance

Feedback Image Retrieval

Institute for Media Innovation

and School of Electrical amp Electronic Engineering

Supervisor Assoc Prof Wang Lipo School of Electrical ampElectronic

Engineering

Co-supervisor Assoc Prof Lin Weisi School of Computer Engineering

PhD Student Zhang Lining

Outline

bull Background

bull Problem and motivation

bull Proposed solutions

bull Experimental Results

bull Conclusions

Background

bull Digital cameras and mobile phone cameras are prevalent rapidly

bull More and more digital images

bull Retrieving an image from a large image collection becomes a more and

more important research topic

Generate

Background

bull Early Image Retrieval System(Text-based image retrieval)

tree

green

grass

sky

yellowAnnotate Retrieve

grass

Background

bull Drawbacks of the early text-based image retrieval system

ndash Hard labor

ndash Beyond the technology

80 million images collection

Annotate

Bill GatesWho is he

80 000 000 3600 24 926 days

Background

bull Dynamic interpretation of one image

bull An image says thousands of words

SpidermanSkyscraper

Background

bull Some Representative Content-Based Image Retrieval systems

ndash Query By Image Content system IBM corporation

ndash Virage system Viarage corporation

ndash Photobook system MIT media lab

ndash Multimedia Analysis and Retrieval System UIUC

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 2: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Outline

bull Background

bull Problem and motivation

bull Proposed solutions

bull Experimental Results

bull Conclusions

Background

bull Digital cameras and mobile phone cameras are prevalent rapidly

bull More and more digital images

bull Retrieving an image from a large image collection becomes a more and

more important research topic

Generate

Background

bull Early Image Retrieval System(Text-based image retrieval)

tree

green

grass

sky

yellowAnnotate Retrieve

grass

Background

bull Drawbacks of the early text-based image retrieval system

ndash Hard labor

ndash Beyond the technology

80 million images collection

Annotate

Bill GatesWho is he

80 000 000 3600 24 926 days

Background

bull Dynamic interpretation of one image

bull An image says thousands of words

SpidermanSkyscraper

Background

bull Some Representative Content-Based Image Retrieval systems

ndash Query By Image Content system IBM corporation

ndash Virage system Viarage corporation

ndash Photobook system MIT media lab

ndash Multimedia Analysis and Retrieval System UIUC

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 3: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Background

bull Digital cameras and mobile phone cameras are prevalent rapidly

bull More and more digital images

bull Retrieving an image from a large image collection becomes a more and

more important research topic

Generate

Background

bull Early Image Retrieval System(Text-based image retrieval)

tree

green

grass

sky

yellowAnnotate Retrieve

grass

Background

bull Drawbacks of the early text-based image retrieval system

ndash Hard labor

ndash Beyond the technology

80 million images collection

Annotate

Bill GatesWho is he

80 000 000 3600 24 926 days

Background

bull Dynamic interpretation of one image

bull An image says thousands of words

SpidermanSkyscraper

Background

bull Some Representative Content-Based Image Retrieval systems

ndash Query By Image Content system IBM corporation

ndash Virage system Viarage corporation

ndash Photobook system MIT media lab

ndash Multimedia Analysis and Retrieval System UIUC

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 4: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Background

bull Early Image Retrieval System(Text-based image retrieval)

tree

green

grass

sky

yellowAnnotate Retrieve

grass

Background

bull Drawbacks of the early text-based image retrieval system

ndash Hard labor

ndash Beyond the technology

80 million images collection

Annotate

Bill GatesWho is he

80 000 000 3600 24 926 days

Background

bull Dynamic interpretation of one image

bull An image says thousands of words

SpidermanSkyscraper

Background

bull Some Representative Content-Based Image Retrieval systems

ndash Query By Image Content system IBM corporation

ndash Virage system Viarage corporation

ndash Photobook system MIT media lab

ndash Multimedia Analysis and Retrieval System UIUC

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 5: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Background

bull Drawbacks of the early text-based image retrieval system

ndash Hard labor

ndash Beyond the technology

80 million images collection

Annotate

Bill GatesWho is he

80 000 000 3600 24 926 days

Background

bull Dynamic interpretation of one image

bull An image says thousands of words

SpidermanSkyscraper

Background

bull Some Representative Content-Based Image Retrieval systems

ndash Query By Image Content system IBM corporation

ndash Virage system Viarage corporation

ndash Photobook system MIT media lab

ndash Multimedia Analysis and Retrieval System UIUC

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 6: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Background

bull Dynamic interpretation of one image

bull An image says thousands of words

SpidermanSkyscraper

Background

bull Some Representative Content-Based Image Retrieval systems

ndash Query By Image Content system IBM corporation

ndash Virage system Viarage corporation

ndash Photobook system MIT media lab

ndash Multimedia Analysis and Retrieval System UIUC

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 7: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Background

bull Some Representative Content-Based Image Retrieval systems

ndash Query By Image Content system IBM corporation

ndash Virage system Viarage corporation

ndash Photobook system MIT media lab

ndash Multimedia Analysis and Retrieval System UIUC

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 8: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The framework of the CBIR system

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 9: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Background

bull Popular Relevance Feedbacks (Rui et al 1998 Zhou et al 2003)

ndash Query Movement and Reweighting algorithms

ndash Density Estimation RF algorithm

ndash Subspace learning algorithms

ndash Statistical sampling algorithms

ndash Classification-based algorithms

ndash Etchellip

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 10: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Problem and motivation

bull The most famous classification method

Vladimir Vapnik

Cite Burr Settles

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 11: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

ndash SVM RF was first proposed in (Huang et al ICIP 2000)

P Hong Q Tian and T S Huang Incorporate support vector machines to

content-based image retrieval with relevant feedbackICIP00 VancouverBC

Canada 2000

ndash Improvement for SVM RF image retrieval

Guo et al TNN 2001 Tong et al ACM MM 2001 Wang et al CVPR 2003

Wang et al ICCV 2005 Tao et al TAMPI 2007 Li et al TIP 2006 Hoi et al

ACM TOIS 2009 Tao et al TCSVT 2008 etchellip

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 12: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Review of the SVM RF for CBIR

bull Support Vector Machine Relevance Feedback for Image Retrieval

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 13: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

A typical set of feedback samples in a RF iteration

ldquoAll positive examples are alike and each negative example is negative

in its own wayrdquo ----------X Zhou T Huang 2001

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 14: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Two drawbacks when using the SVM RF

bull The SVM treats positive and negative feedbacks equally However it is

not proper

bull The SVM ignores the unlabelled samples However they are very helpful

in constructing a good classifier

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 15: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

How to explore solutions to the first drawback

bull Assumption ldquoDifferent semantic concepts live in different subspaces and

each image lives in many different subspaces----X Zhou T Huang 2001rdquo

Similar

Color

Texture

Shape

Different

Concept

Girl

Dog

bull Solution Therefore we can solve the first problem by finding a semantic

subspace associated with the human perception

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 16: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

How to explore solutions to the second drawback

bull Assumption Unlabelled samples lie on or around a low dimensional

manifold in a high dimensional Euclidean space

bull Solution Therefore we can use the manifold structure of the unlabelled

samples to refine the semantic subspace

One Example

From 3-D data to 2-D data

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 17: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

How to explore solutions to the two drawbacks

bull Subspace Learning Techniques

ndash Principle Component Analysis(PCA)

ndash Linear Discriminant Analysis(LDA)

ndash Biased Discriminant Analysis (BDA)

ndash Etchellip

bull Graph Embedding (GE) framework (Yan et al TPAMI 2007)

ndash GE can unify all the representative subspace learning method within

one framework

ndash The manifold learning algorithms can be listed as follows

bull ISOMAP Locally Linear Embedding Locality Preserving Projection Margin

Fisher Analysis Etchellip

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 18: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Graph Embedding Framework

bull Two graphs in graph embedding framework

ndash The intrinsic graph

bull Characterize the similarity relationship

between vertex pairs

ndash The penalty graph

bull Characterize the dissimilarity relationship

between vertex pairs

bull The formula of Graph Embedding techniques

2

( ) ( )

arg min || || arg min ( )T T

T

i j ij

tr YBY c tr YBY ci j

y y y W tr YLY

( )arg min

( )

T

TY

tr YLYY

tr YBY

or

Similar objective

Dissimilar objective

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 19: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Proposed solutions to solve the two drawbacks

bull Biased Maximum Margin Analysis (BMMA) for SVM RF

ndash Assumption ldquoAll positive examples are alike and each negative

example is negative in its own way----X Zhou T Huang et alrdquo

ndash Construct two graphs to describe the similarity and dissimilarity

bull Intrinsic graph we use the intrinsic graph to describe the similarity

between positive feedbacks

bull Penalty graph we use the penalty graph to describe the

dissimilarity between positive feedbacks and negative feedbacks

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 20: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The intrinsic graph in BMMA

bull All positive feedbacks are alike (Decrease the pairwise distance)

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 21: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The intrinsic graph in BMMA

bull The objective function of the intrinsic graph

2

|| ||

2 [ ( ) ]

s s

i j

T TI i j ij

i j j ori

T T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

s s s

j i

ij

if l i and l j i or jW

else

Similarity

objective

Weighting

matrix

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 22: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The penalty graph in BMMA

bull Each negative sample is negative in its own way (Increase the pairwise distance)

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 23: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The penalty graph in BMMA

bull The objective function of the penalty graph

Dissimilarity

objective

Weighting

matrix

2

|| ||

2 [ ( ) ]

p p

i j

T T pp i j ij

i j j ori

T p p T

S x x W

tr X D W X

1 | | ( ) 1 ( ) 1

0

p p p

j ip

ij

if l i and l j i or jW

else

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 24: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The proposed BMMA for SVM Relevance Feedback

bull Biased Maximum Margin Analysis for SVM RF

ndash Maximize the objective function of the penalty graph

ndash Minimize the objective function of the intrinsic graph

arg m ax 2 [ ( ) ] 2 [ ( ) ]

arg m ax ( ) ( )

arg m ax [ ( ) ]

T p p T T T

T T T T

T T

tr X D W X tr X D W X

tr XBX tr XLX

tr X B L X

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 25: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The proposed BMMA for SVM Relevance Feedback

bull Solutions

ndash The solution to the above problem is trivial Therefore we should

remove an arbitrary scaling factor in the projection

ndash Note that we can use other constraints for example However it will encounter the

ldquoSmall Sample Sizerdquo problem

T

1

m ax ( ( ) )

= ( )

1 0 1 2

T

l

T T

k k

k

T

k k

tr X B L X

X B L X

s t k l

( ) 1T T

tr XBX

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 26: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

The proposed BMMA for SVM Relevance Feedback

bull The dimensionality of the embedding subspace

ndash In general we have

where are the associated eigenvalues and we have

Therefore we should preserve all directions associated with the

nonnegative eigenvalues

1

m ax ( ( ) )

l

T T

i

i

tr X B L X

is

1 2 10

d d l

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 27: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Semi-Supervised Biased Maximum Margin Analysis

bull Another intrinsic graph for unlabeled samples in SemiBMMA

Unlabelled Images

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 28: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Semi-Supervised Biased Maximum Margin Analysis

bull The objective function of the intrinsic graph for unlabeled samples

2

1|| ||

2

[ ( ) ]

[ ]

s s

i j

T T uU i j ij

i j j ori

T u u T

T T

S x x W

tr X D W X

tr XUX

2 21

| |exp( || || ) ( ) ( ) 0

0

n

u u

i j j iu

ij

x x if l i l j i or jW

else

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 29: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Semi-Supervised Biased Maximum Margin Analysis

bull Introduce the intrinsic graph of unlabelled samples into BMMA we can

have Semi-Supervised BMMA (SemiBMMA)

bull Comparison between the BMMA SVM and the SemiBMMA SVM

arg max [ ( ) ]

T Ttr X B L U X

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 30: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments

bull Corel Image Database (10763 images with 80 concepts)

bull Content-Based Image Retrieval System framework

User

Relevance Feedback Model

Visual

Features

Query

Image

Similarity Metric

Image

Database

Final ResultsRetrieval Results

NOYES

Positive

Negative

Unlabelled

User

Labels

System

Labels

System

Selects

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 31: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiment Platform

bull Image Representations

ndash Color hue saturation value color histograms and color moments to

form 256+9 dimensional features

ndash Local Image Descriptor Weber Local Descriptor(WLD)-(Jie Chen

etal TPAMI 2009) 240 dimensional features

ndash Shape Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 32: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments

bull Subsection I Visualization of problems in the SVM RF

ndash A feature subset selection problem for the SVM RF

ndash An unstable problem for the SVM RF

bull Subsection II Subspace learning based on different methods

ndash Six group of comparison results for toy problems

bull Subsection III Statistical experimental results

ndash Experiments on a small size image database

ndash Experiments on a large scale image database

bull Subsection IV Visualization of the image retrieval results

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 33: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection I

ndash A feature subset selection problem for the SVM RF

Random select 2-d features the resultant SVM classifiers

Positive

Negative

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 34: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection I

bull An unstable problem for the SVM RF in the first few interations

ndash (a) 4 positive feedbacks and 6 negative feedbacks

ndash (b) 5 positive feedbacks and 6 negative feedbacks

ndash (c) 6 positive feedbacks and 6 negative feedbacks

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 35: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection II

bull Subspace learning based on different methods

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

-4 -3 -2 -1 0 1 2 3 4-5

-4

-3

-2

-1

0

1

2

3

4

5

Positive

Negative

LDA

BDA

MFA

BMMA

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 36: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III

bull Part I On a small scale Corel image database

ndash Objective To validate the effectiveness of BMMA combined with the SVM RF

ndash Experimental settings

bull Database 3899 corel images with 30 different categories

bull Feedback sample number first 5 relevant images and first 5 irrelevant images in

top 20 results are labeled as positive and negative feedbacks

bull Feedback iteration number as queries to do 2 rounds of feedback iterations

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 37: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III

bull The average precision values for all 30 categories after 2nd feedback

0 5 10 15 20 25 300

01

02

03

04

05

06

07

08

09

1

Category No

Ave

rage

Pre

cisi

on a

t T

op 2

0

BaseLine

SVM

OCCASVM

BMMASVM

Observation BMMA combined with the SVM RF can significantly improve the

performance of the SVM RF for most of the 30 categories

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 38: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III

bull Average Precision after 1st feedback and 2nd feedback iteration

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Average P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

20 40 60 80 100 120 14001

02

03

04

05

06

07

08

09

Scope

Avera

ge P

recis

ion

Baseline

SVM

OCCASVM

BMMASVM

1st fedback iteration 2nd fedback iteration

Observation BMMA combined with the SVM RF can significantly

improve the performance of the SVM RF after 1st feedback However

over fitting problem will occur after more feedback iterations

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 39: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III bull Part II On a large scale Corel image database

ndash Objective To evaluate the effective of the BMMA and SemiBMMA combined

with the SVM RF

ndash Experimental settings

bull Database 10763 corel images with 80 different categories five cross

validation dataset

bull Feedback schemes first 3 relevant and all irrelevant images in top 20

results are labeled as positive and negative feedbacks respectively 400

independent queries 9 feedback iterations

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 40: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III

bull Experimental results on large scale image database (Average Precision)

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

03

04

05

06

07

08

09

1

Number of IterationsA

vera

ge P

recis

ion

Average Precision in Top 20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 901

02

03

04

05

06

07

08

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

045

05

055

06

065

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Observations SemiBMMA combined with the SVM RF shows much better performance

BMMA combined with the SVM RF also show good performance comparing with the

SVM RF

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 41: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III

bull Experimental results on large scale image database (Standard Deviation)

0 1 2 3 4 5 6 7 8 9

02

025

03

035

04

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

034

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top20 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9023

024

025

026

027

028

029

03

031

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top30 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9021

022

023

024

025

026

027

028

029

03

031

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 40 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 902

022

024

026

028

03

032

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 50 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

0 1 2 3 4 5 6 7 8 9018

02

022

024

026

028

03

Number of Iterations

Sta

ndard

Devia

tion

Standard Deviation in Top 60 Results

SemiBMMASVM

BMMASVM

OCCASVM

BDASVM

SVM

OneSVM

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 42: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III

bull The average precision results after 9th feedback

bull Observation SemiBMMA SVM RF and BMMA SVM RF can improve the performance

of SVM RF after 9-th feedback significantly in all top 10-top 90 results OCCA ignores the

negative feedbacks and thus will cause the over-fitting problems as shown in top 50-top 90

results

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 43: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection III

bull Conclusions on the experimental results on a large scale image database

ndash a) BMMA combined with the SVM RF can improve the performance

of the SVM RF therefore BMMA SVM can integrate the distinct

properties of feedbacks in to the SVM RF

ndash b) SemiBMMA combined with the SVM RF show much better

performance comparing with the SVM RF and BMMA SVM RF

therefore SemiBMMA can integrate the information of the unlabeled

samples into the SVM RF

ndash c) It should be noted that the different performance between the small

and large image database are because of the different feedback schemes

in two experiments

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 44: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Subsection IV

bull Visualization of the retrieval results

ndash Query images Randomly select 4 images as the queries

(eg bobsled cloud cat and car)

ndash Feedback Scheme around 4 positive and 4 negative

feedbacks in top 20 images

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 45: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Experiments Visualization of the results

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 46: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback

Conclusions

bull Support Vector Machine based RFs have been widely used to

enhance the image retrieval system

bull BMMA can combine the properties of the feedbacks with

SVM RF and enhance the performance of the CBIR system

bull Semi-Supervised BMMA can integrate the information of

unlabelled samples into SVM RF and thus further enhance the

performance of the CBIR system

Page 47: Semi-Supervised Biased Maximum Margin Analysis …imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/...Semi-Supervised Biased Maximum Margin Analysis for SVM Relevance Feedback