visual typo correction by collocative optimization

1

Visual Typo Correction by Collocative OptimizationA Case Study on Merchandize Images

Xiao-Yong Wei, Member, IEEE, Zhen-Qun Yang∗, Chong-Wah Ngo, Member, IEEE, Wei Zhang

Abstract—Near-duplicate retrieval (NDR) in merchandize im-ages is of great importance to a lot of online applications on e-Commerce websites. In those applications where the requirementof response time is critical, however, the conventional techniquesdeveloped for a general purpose NDR are limited, becauseexpensive post-processing like spatial verification or hashing isusually employed to compromise the quantization errors amongthe visual words used for the images. In this paper, we arguethat most of the errors are introduced because of the quantizationprocess where the visual words are considered individually, whichhas ignored the contextual relations among words. We proposea “spelling or phrase correction” like process for NDR, whichextends the concept of collocations to visual domain for modelingthe contextual relations. Binary quadratic programming is usedto enforce the contextual consistency of words selected for animage, so that the errors (typos) are eliminated and the qualityof the quantization process is improved. The experimental resultsshow that the proposed method can improve the efficiency ofNDR by reducing vocabulary size by 1,000% times, and under thescenario of merchandize image NDR, the expensive local interestpoint feature used in conventional approaches can be replacedby color-moment feature which reduces the time cost by 9,202%while maintaining comparable performance to the state-of-the-art methods.

Index Terms — near-duplicate retrieval;visual word quantiza-tion; binary quadratic programming.

I. INTRODUCTION

Given the overwhelming number of online business web-sites nowadays, merchandize imagery, as a practical meansof product promotion, is forming one of the most rapidlyincreasing set among web images. Many merchandize images,however, are indeed exact duplicates or near-duplicates (i.e.,the images that share the same objects/scenes but appeardifferently due to the variations in capturing conditions, ac-quisition times, rendering conditions, or editing operations).For example, according to our study on a dataset includingover one million merchandize images crawled from eBay.comand Taobao.com1, more than 12.9% of those images are near-duplicates, and together with exact duplicates, the proportion

Copyright (c) 2013 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposes must beobtained from the IEEE by sending an email to [email protected].

The work described in this paper was supported in part by the grantsfrom the National Natural Science Foundation of China (No. 61001148 andNo. 61272256) and the Research Grants Council of the Hong Kong SpecialAdministrative Region, China (CityU 119610).

Xiao-Yong Wei and Zhen-Qun Yang are with the College of Com-puter Science, Sichuan University, Chengdu 610065, China (e-mail: [email protected], [email protected]). Chong-Wah Ngo and WeiZhang are with the Department of Computer Science, City Universi-ty of Hong Kong, Kowloon, Hong Kong (e-mail: [email protected],[email protected]).

∗ Corresponding author.1eBay.com and Taobao.com are the most popular e-Commerce websites in

US (or worldwide) and China respectively.

of the duplicate images is up to 29.7%. Therefore, searchingnear-duplicates in merchandize images is of great commercialvalue for the E-Commerce industry. It can help to identifypirated products, especially when they are sold at websites ofdifferent languages where the text-based retrieval may workinadequately; similarly, it can be used to detect unlicensedsellers, to monitor how a product is distributed among websitesor countries through Internet, or even to find potential clientswho are interested in a particular product. As shown inFigure 1, near-duplicate images, once detected, can provideadditional valuable clues such as price/comment comparisonacross websites or countries, and evaluation of the marketpotentials for products.

To conduct near-duplicate retrieval (NDR) in merchandizeimages, one can first think of employing general purpose NDRtechniques, which have reached a level of sophistication. Aftera decade of intensive study, it is generally recognized thatthe use of the local features based on local interest points(LIPs) within a bag-of-word (BoW) framework [1–5] performsthe best for NDR. LIPs are salient points of images whosefeatures are invariant to the changes of scale, illumination, andviewpoint [6, 7]. With LIPs detected for each image, the ideaof BoW is borrowed from the text retrieval, in the way thatan image is considered a document with its LIPs as “words”.On this basis, images can thus be indexed by LIPs so as tobe retrieved with the advanced techniques as developed in textretrieval.

The LIPs+BoW framework is simple and straightforward,and due to its solid foundation in information retrieval, ithas demonstrated promising performance in a wide rangeof applications such as copyright violation detection [8–10], object retrieval [4], scene recognition [11], and videosummarization [12]. However, when facing large-scale datasetand online applications, the time efficiency of the frameworkis seriously below the expectations. First, the extraction ofLIPs is inherently time-consuming, because to select candidatepoints, the process has to conduct a great amount of tests forevery pixel of an image. Second, a point-to-point matchinghas usually been employed when determining the similaritybetween two images. The situation could be worse whensome methods further integrate the checking of geometricconsistency among LIPs into the process to improve theaccuracy [13–16]. Therefore, the framework has seldom beenstudied in scenarios of Internet or mobile applications wherethe requirement for response time is critical and the targetdatasets are usually in web-scale.

In this paper, we conduct a domain-specific study of N-DR on merchandize images, with the hope of serving theaforementioned online applications. We argue that the con-

2

Images

Prices

Ratings

Reviews

Fig. 1: Valuable clues provided by cross-website (countries) near-duplicate detection.

ventional LIPs+BoW framework does not exploit the potentialrepresentativeness of the LIPs, because LIPs are traditionallyprocessed individually when mapping them into “words”,disregarding the contextual information among LIPs and caneasily lead to an inconsistent mapping. Therefore, we proposeto integrate a “spelling or phrase correction” like process intothe framework, which we name “visual typo correction”. Thebasic idea is to extend the concept of collocations to the visualdomain for modeling the contextual relations among visualwords, on the basis of which we can enforce the mapped wordsfor an image to be contextually consistent, so that the visual“typos” (i.e., the mis-mapped words) can be eliminated. Thecollocative optimization of visual words leads to the followingadvantages that are yet to be explored in the literature:

• Robust Mapping: Due to the variations in capturingconditions, acquisition times, rendering conditions, orediting operations, the appearances of a LIP can varysignificantly in different near-duplicate images, whichmay introduce “typos/errors” into the mapped “word-s” when LIPs are considered individually. We proposea co-occurrence-based model to learn the visual wordcollocations and use it as a contextual information tocorrect unreasonable mapping. For example, a LIP fromthe surface of a car may be mapped to a “word” usuallyrepresenting glass surfaces with reflections. This mappingcan be corrected when the majority of other LIPs inthe same image are mapped to car-related “words” (e.g.,those representing wheels, lights or license plates);

• Complexity Reduction: To improve the representativenessof the LIPs within the LIPs+BoW framework, one hasto increase the size of the vocabulary so as to coveras many “words” as possible. Several researches haveshown that, to achieve a state-of-the-art performance, onehas to build a vocabulary up to a scale of hundreds ofthousands (e.g., 100k in [13] and 1M in [4]), whichdramatically increases the time for feature quantizationand thus is infeasible to online applications. Therefore,by introducing the contextual constraint in the proposedmethod, LIPs can work collaboratively to form patternsfor feature representation, which relaxes the requirementfor the large scale vocabulary. Our study will show thatthe proposed method can reduce the vocabulary size by1,000% times while maintaining comparable performance(cf. Section V-C);

• Feature Simplicity: Existing works on NDR are mainlyfocusing on improving the accuracy (e.g., those by op-timizing the detectors or descriptors for LIPs [17–19],by optimizing the vocabulary size [13], by introducinggeometric constraints [13, 14]), while the efficiency hasbeen simply addressed by employing standard techniqueslike inverted files [1] or hashing functions [20]. To thebest of our knowledge, this is the first work studying theefficiency optimization for the domain-specific NDR onmerchandize images. We will show that under certain sce-narios, the expensive LIPs-based features can be replacedby simple features (e.g., color-moments) with comparable

3

... ... ...

... ... ...

= + + + + + ...

= + + + + + ...

... ... ...

...

...

... ... ...

Training Image Set

Target Image Set LIPs Detection &

Description

LIPs Detection

LIPs Description

Clustering Vocabulary

Vocabulary Learning

Indexing

Quantization Inverted File

Fig. 2: LIPs+BoW framework for NDR.

or even better performance, while reducing the time costby 9,202% (cf. Section V-C).

The remainder of this paper is organized as follows. Sec-tion II reviews the LIPs+BoW framework and NDR techniquesin the literature. Section III introduces the contextual consis-tency in a NDR sense, and Section IV describes our modelof visual typo correction by collocative optimization. Theexperimental results will be presented in Section V. Finally,Section VI concludes this paper.

II. RELATED WORK

A. A general framework of NDR

A description of the LIPs+BoW based NDR framework isgiven in Figure 2. The procedure starts by extracting LIPs foreach image, where every pixel’s stability to transformations(for example, the change of scale, illumination) is evaluated byinvestigating the patch(es) around the pixel, resulting in a set ofpixels with good invariance properties as LIPs. A patch aroundeach LIP is then extracted and described by the local featureswhich are usually invariant to the geometric and photometrictransformations [6, 7, 21]. LIPs of all images in the datasetare pooled and clustered into groups. A model feature vectorrepresenting the “word” of each cluster is calculated usingthe mean or medoid of member vectors within the cluster.By composing all “words” into a vocabulary, every LIP canbe “quantized”2 by assigning it to the closest word, so thatone image is represented as a set of “words”, resembling atraditional text document [1–5]. Finally, images are indexedinto an inverted file and the rest of the process is almost thesame as the conventional text retrieval.

2Note that the quantization process sometimes is called “encoding”. Weuse the term “quantization” throughout this paper, because it is more oftenused in the literature of near-duplicate retrieval.

The LIPs+BoW framework is simple and straightforward,and has demonstrated promising performance in many appli-cations. However, when facing web-scale data, its performancehas been challenged in both accuracy and time efficiency. Onone hand, we should notice that decomposing an image intoLIPs decreases the geometrical relations among pixels. Con-sequently, the precision of retrieval can be seriously affected,especially when facing a large amount of web images wherethe chance of encountering non-near-duplicate images havingsimilar sets of visual words becomes non-negligible. On theother hand, as discussed in Section I, the searching of LIPswith a pixel-by-pixel manner is inherently inefficient, and thematching between the LIP sets of two images (usually over1,000 LIPs for each) is even more costly.

B. Recent trends of NDR

Efforts to improve the accuracy and efficiency are themain drive behind the recent trends in NDR. To improvethe accuracy, the most intuitive idea is to add the geomet-rical constraint when comparing visual words (LIPs afterquantization) between two images. Words in one image areusually related to their neighbors to construct (explicitly orimplicitly) graph(s), hence image comparison using graphsresulting from LIPs of respective images yield more accuratesimilarity measures [13–16]. Alternatively, to mitigate someof the frequently occurring visual words from dominatingthe others, “burstiness” of visual words is often adopted toadjust the similarities between images by further consideringthe inter-image or intra-image frequency of the words [22].Meanwhile, to improve the efficiency, a majority of researchesis focusing on developing better hashing functions to map LIPsinto a low-dimensional space so as to speed up the match-ing process (e.g., locality sensitive hashing (LSH) [17, 23],hamming embedding (HE) [13, 24], and min-Hash [25, 26]).State-of-the-art methods often integrate these two aspects by

4

TABLE I: Techniques used in conventional Text Retrieval andtheir analogues in Near-Duplicate Retrieval.

Conventional Text Retrieval Near-Duplicate RetrievalTokenization LIPs Extraction [6] [7] [21]Stop-List Visual Word Stop-List [1]Stemming Vector Quantization [1-20]Spelling/Phrase Correction (No Analogues)Inverted File Visual Word Inverted File [1, 15]Query Expansion Contextual Synonyms [30]Reranking Spatial Reranking [13–16]

using hashing to retrieve the whole dataset and then employingthe geometrical constraint to re-rank the top-ranked images.Nevertheless, the quest about which geometrical constraintshould be exploited continues, since one has to set a balancebetween recall and precision. A similar dilemma also occursin the hashing or encoding stage where one has to determineparameters of the hashing algorithms, the hashing functionitself, the dimension of the target space, or the number/size ofthe bins.

Despite being accuracy or efficiency oriented, most ofrecent methods put their focus on post-quantization process.As discussed in Section I, however, due to the variations inthe appearances of LIPs, the quantization usually introducesa certain amount of wrongly mapped words. Those “typos”will be propagated and further degrade performance of theprocesses afterwards. Therefore, in this paper, we propose apre-quantization method which utilizes the contextual relationsamong visual words to correct the “typos” and thus improvethe quality of LIPs-to-Words mapping. Specifically, instead ofdirectly mapping each LIP to a single word, we select severalcandidates for every LIP and finally find an optimal set ofwords (one for each LIP) where the contextual consistencyamong words has been maximized. In this regard, the proposedmethod is also related to soft-weighting [27] or visual wordambiguity [28] which map each LIP to several words witheach word assigned a weight proportional to the significanceof the LIP. While soft assignment in soft-weighting or visualword ambiguity has been proven to be a practical way tocompensate for the shortcomings of the single-word-mapping,though it will cause an increase in the number of wordsneeded to represent the image as well as the complexity ofthe feature and thus affect the efficiency. Furthermore, it bringsadditional computational cost or may even cause ambiguity forthe post-quantization processes. Therefore, soft assignment isonly applied to the query side in most of the applications.

C. NDR in a text retrieval point of view

While NDR has been studied over one decade resultingin a large number of methods, a significant amount of ideasemployed in the LIPs+BoW framework have been obviouslyborrowed from text retrieval (TR) domain [29]. Therefore,when reviewing those ideas, it is better to refer to theiranalogues in TR that most readers are more familiar with,as summarized in Table I.

As introduced, the first step in TR is to parse documents intowords (tokenization), while its analogue in NDR is to extractLIPs from images. Therefore, early efforts have been put into

seeking effective types of “words” (i.e., LIPs). Representativeones include SIFT [6], SURF [7] and MSER [21]. Second,the quantization in NDR maps LIPs to visual words, whichis analogous to stemming in TR. For each LIP, the mappedvisual word is usually the one(s) with the largest similarity. Alist of stop words is sometimes employed to filter out the non-informative visual words, which is directly adopted from TR[1]. In indexing, both NDR and TR use inverted file to indeximages (documents) [1, 15]. Alternatively, one can use “termfrequency-inverse document frequency” (known as tf-idf) torepresent images (documents) to vectors for similarity-basedretrieval (i.e., Vector Space Model) [4]. In addition, queryexpansion, which expands query terms (i.e., the visual wordsof the query image) with their contextual synonyms (i.e., visualwords that frequently co-occur within the same context), hasalso been practiced in NDR [30]. Finally, spatial consistencyis measured between query images and the top-ranked onesin the initial retrieved list for re-ranking. There might be alot of correspondences in TR for the idea of re-ranking. Butgiven the fact that every graph edge is actually representingthe spatial relation of a word-pair, the re-ranking scheme usingedge-wise comparison in NDR is more like a bi-gram basedmethod [13–16].

In Table I, one can observe that the spelling/phrase correc-tion widely used in TR is missing its analogue in NDR. In TR,it is intuitive to remove typos/errors by using spelling/phrasecorrection to infer that a user should mean “a drop of water”even she has typed “a drap of wafer”, indicated by the contextof the sentence. One question may be raised immediately isthat “Do visual words in NDR also include typos/errors?” Infollowing sections, we will show that the answer is YES andthe performance of NDR can be further improved when con-textual consistency is considered in the LIPs+BoW framework.

III. CONTEXTUAL CONSISTENCY IN NDR

Compared with that of text retrieval, contextual consistencyin NDR might be a less intuitive concept to the reader. Inthis section, we identify the “typos/errors” in visual words,on the basis which we then define the concept of contextualconsistency for correcting visual spelling in NDR.

A. Typos/Errors in visual words

There are several chance events where “typos/errors” canoccur in visual words. The inevitable variation in appearancesis the first one to consider. Various conditions of capturing orvarious types of editing can make the same objects or scenesappear significantly different in near-duplicate images. Underthese conditions, visual feature of an individual LIP may varydramatically from the original. Serious deformation is thesecond important factor. This can happen during intentionalediting (e.g., resizing, captioning, rotating), deformations of aproduct consisting of a soft material, and recapturing photosin an environment different than the query with significantchanges in illumination, view-points and so on. Specular re-flection is also responsible, given the overwhelming popularityof shiny surfaces in commercial products. The last one goesto the quantization step in NDR (cf. Figure 2). Representing

5

LIPs in the same cluster with its mean or medoid vector hasreduced the computational cost, but inevitably will introduceambiguity, which increases the probability of mapping theLIPs to the wrong visual words. When all those factors cometogether in NDR, the set of visual words for representing acertain image could be poor, and this can seriously degradethe subsequent steps.

B. What’s the contextual consistency in NDR?Nearly all the spelling/phrase correctors in TR are utilizing

the contextual consistency (e.g., collocation of words) betweenwords for eliminating typos/errors. In NDR, the concept ofcontextual consistency can be conveniently adopted sincevisual words work together to compose an image in almostthe same way as text words do in a document. For example,in the conventional NDR approaches, LIPs from the surfaceof a car can easily be quantized as visual words representingsurface of an airplane or other metal-like materials. However,if a certain amount of LIPs representing road, tires or otherconcept related to ground transportation are detected in thesame image, one can be more confident that the mis-quantizedLIPs are from the surface of a car. This is similar to thelogic we use in text retrieval that “a drop” most likely comeswith “water” instead of “wafer”. Using contextual consistencywhich considers visual words in an image collaboratively,therefore, has the advantage over conventional quantizationprocess where visual words are mapped individually.

One may argue that in reranking methods based on geomet-ric consistency, the contextual consistency has already beenconsidered. Although this is partly true, these two conceptshave fundamental differences: a) geometric consistency isemployed to compare the relative positions among words,while contextual consistency is used to verify co-occurrence.In other words, geometric consistency prevents “drap of wafer”to be matched to phrases like “wafer of drap” or “wafer drapof”, but it does not take into consideration if “drap of wafer”is meaningful to the context. On the contrary, contextualconsistency makes sure all visual words have semantic mean-ings3 consistent to the context, forcing “drap of wafer” to bemapped to “drop of water” during quantization; b) geometricconsistency is used in the post-processing of the retrieval toeliminate the irrelevant results, while contextual consistencyis used in the pre-processing to improve the quality of theinputs. These two are thus not mutually exclusive. It is worthmentioning that in contextual consistency checking discussedin this article, the order of the words is ignored, which meansthat “drap of wafer” can be mapped to “water of drop” or“water drop of”. Given the fact that how to define a linearorder for visual words in a 2D image is still an open question,this relaxation is reasonable. Moreover, when both geometricconsistency and contextual consistency are adopted in the samesystem, they can be complementary to each other.

3The wording “semantic” means that a visual word shall carry some visualcharacteristics that is unique to the type of objects where it is extracted from.A visual word can thus be viewed as a symbol semantically meaningful fordepicting its host objects, analog to using a text word (e.g., drop) to describean object (e.g., water). Note that the visual words are neither associated toany semantic tags nor its meanings are learnt through any supervised learningmethods.

IV. UTILIZING CONTEXTUAL CONSISTENCY FOR NDR

To utilize contextual consistency for NDR, we introducetwo additional components into the vocabulary learning andquantization processes respectively. During vocabulary learn-ing, we will first identify the visual word collocations forrepresenting the contextual information. During quantization,instead of mapping each LIP to the visual word with the largestsimilarity (as in conventional approaches), we select for eachLIP the top-k words as its candidates, and finally choose thebest set of words that has the largest contextual consistencywith the collocations learned in vocabulary learning. In thissection, we will introduce these two components in detail.

A. Modeling Visual Word Collocations

In text domain, a collocation is defined as a sequence ofwords that co-occur more often than would be expected bychance [31]. A popular method for finding collocation candi-dates is to slide an n-word window along sentences to calculatethe in-window co-occurrence of words so as to identify wordsequences with high frequency. We extend this idea to imagedomain by defining the window as a circle of radius d pixels.Therefore, for any two words under investigation, it is easy tocalculate the in-window statistics about their co-occurrence4.Finally we employ X 2 test to select the word pairs whichco-occur frequently but not by chance.

To model the contextual consistency in NDR, the mostconvenient way is to borrow existing techniques developed innatural language processing. We choose (point-wise) mutualinformation in this case, which is popularly employed tomeasure the collocation of words. More specifically, given twovisual words w1 and w2, the mutual information is defined as

I(w1, w2) = logp(w1, w2)

p(w1)p(w2)(1)

where p(w1, w2) , p(w1) and p(w2) are the frequencies of theco-occurrence of w1 and w2, the occurrence of w1, and theoccurrence of w2, respectively. Note that not like text wordpairs which appear in sentence in order, visual words do notappear sequentially in an image. Therefore, in Eq. (1), werelax the order constraint so that I(w1, w2) = I(w2, w1).

It is worth mentioning that the collocation of visual wordshas also been studied by other authors (e.g., [33] and [34])but mostly with the focus on how to identify different typesof visual collocations, which is different from our purpose ofenforcing contextual consistency for NDR. Therefore, in thispaper, we employ X 2 test and mutual information to modelthe collocations whose effectiveness has long been recognizedin TR. While which technique for collocation extraction is thebest remains an interesting topic for future work, it is beyondthe topic of this paper.

4Similar ideas have been employed in [30] and [32] for learning the co-occurrence among visual words, but for different purposes (i.e., for detectingvisual synonyms in [30] and for modeling visual phrases in [32]). Morediscussions can be found in the Sections IV-B and V-C.

6

B. Contextual Consistency Enforcement

During quantization, assuming that top-k visual words (withthe largest similarities) have been selected as candidates foreach LIP, the task of finding the best set of visual words (i.e.,one word for each LIP) for representing an image then fallsinto a typical combinational optimization problem – to find anoptimal combination (set) of visual words that maximizes thecontextual consistency. To this end, we first define an energyfunction to measure the contextual consistency among wordsin a given set W as

E(W) =

|W|∑i=1

|W|∑j=i+1

I(wi, wj), (2)

which is straightforwardly the accumulated mutual informationof words in W . In the rest of this section, we will see thatsearching an optimal set W∗ which maximizes E(·) can beformulated as a binary quadratic programming (BQP) problem.

Assuming there are n LIPs in the current image and for eachLIP, the top-k visual words (with the largest similarities) havebeen selected as candidates, the candidate pool then includesn× k words as

w1, w2, · · · , wk︸︷︷︸For the 1st LIP

, wk+1, · · · , w2k︸︷︷︸For the 2nd LIP

, · · · , w(n−1)k+1, · · · , wnk︸︷︷︸For the nth LIP

.

To each candidate word wi, let us assign a binary variablexi ∈ {0, 1} to indicate its selection status, on the basis ofwhich we can modify Eq. (2) to a BQP object function as

f(x1, x2, · · · , xnk) =nk∑i=1

nk∑j=i+1

xixj · I(wi, wj) ·N(wi, wj)

(3)where N(wi, wj) is a function which returns 1 if the corre-sponding LIPs of wi and wj fall into the same d-radius win-dow, 0 otherwise. Note that the term xixj ensures I(wi, wj)has its impact to the object function only if both wi and wj

are selected (i.e., xi = xj = 1), while N(wi, wj) enforces theselection of visual word for a LIP only need to be contextuallyconsistent with those of its neighbors, which will significantlyreduce the number of nonzero terms in Eq. (3) and thussimplify the problem. In addition, to make sure the selectionsof the visual words among the candidates of the same LIP areexclusive, we add n constraints for the BQP model as

k∑j=1

x(i−1)k+j = 1, i = 1, 2, · · · , n. (4)

Further composing all xi’s into a vector x and the pair-wise neighboring and selection statuses (i.e., N(wi, wj) andI(wi, wj)) into a matrix C (a nk-by-nk matrix) where

Cij = I(wi, wj) ·N(wi, wj), i, j = 1, 2, · · · , nk (5)

the optimization problem can be summarized into a standardBQP form as

maximize f(x) = x′Cx (6)subject to Ax = 1 (7)

xi ∈ {0, 1}, i = 1, 2, · · · , nk (8)

where A is a n-by-nk binary matrix for encapsulating theconstraints in Eq. (4). Therefore, for each row (denoted as theith row) of A, the (i−1)k+1 to (i−1)k+k entities are all oneswhile the rest are all zeros. The BQP, even being a NP-hardproblem, has been intensively studied, and can be efficientlysolved with well-know discrete optimization techniques suchas the branch and bound algorithm [35]. Moreover, both C andA are highly sparse, which will inherently reduce the time forsolving of Eq. (6). Furthermore, to meet the need of onlineapplications, we implement a coarse-to-fine branch and boundalgorithm. We first perform clustering on all the LIPs andselect only the medoids from the resulting clusters to conducta coarse BQP. Then we fix the selections for those medoids soas to filter out a large number of branches in the search treewith which we conduct a fine BQP involving all the LIPs.This is based on the observation that similar LIPs will usuallyselect similar words so that the selections of member LIPsin a cluster will generally follow that of the medoid. In ourexperiment, by simplifying the search tree, the computationalcost has been reduced from 19 ∼ 116 to 12 ∼ 27 milliseconds.

One may argue that the BQP can be replaced by classi-cal dynamic programming by arranging LIPs sequentially asstages and their candidate words as states. However, this isnot exactly true since selections of visual words are highlyinterdependent on LIPs, it is difficult to enforce contextualconsistency on an one-by-one basis. In terms of dynamicprogramming, it makes the optimization of the i-th statenot only dependent on those of the previous states but alsothose of the succeeding ones. Therefore, arranging the statessequentially is tantamount to ignoring some of the inter-dependencies.

Another fact worth mentioning is that the BQP process de-pends on the learning of contextual consistency among visualwords which were themselves obtained through learning. Inprinciple, one could be concerned about the effect of wordquantization errors to contextual consistency (i.e., Eq. (1)) asthe quantization errors (i.e., some LIPs are mapped to wrongwords) on the training dataset are inevitable. We argue thisis not a critical issue for the proposed method, because thecontextual consistency is learnt from the visual collocationswhich must be the words that co-occur “frequently”. Sincethe quantization errors are basically “random” (in a statisticalsense) and the mis-mapped words rarely have the chance toform collocations with others, these noisy cases can practicallybe ignored given a large enough training dataset.

Note that the use of spatial context in BQP is differentfrom [30] in which the authors mine frequently appearingwords as contextual synonyms for query expansion. Whileboth approaches share similarity in learning co-occurrencesstatistics, BQP enforces the spatial consistency in visual wordquantization and thus can be viewed as a pre-processingstep for indexing. In contrast, [30] leverages the statisticsduring query time by expanding query words with contextualsynonyms for boosting the recall of relevant images. Sincethe consistency is only considered in query time, mismatchedwords will still be indexed for online retrieval. In brief, BQPfundamentally corrects the mismatched words with spatialcontext, while [30] compensates the mismatches, rather than

7

Fig. 3: Examples of reconstructing images with their visual words. The first row (from left to right): the original image, imagereconstructions with and without contextual consistency. The second row: the enlarged details of images on the first rows.

making correction, with contextually consistent words.

C. Visualization of Contextual Consistency

Up to this point, we have introduced and established themodel of contextual consistency in NDR. However for thesake of clarity, we have borrowed examples from the textdomain during description. In this section, we verify whetherit is applicable to NDR in an “image” sense.

To visualize the effect of utilizing the contextual consistencyon images, we modify the processing of quantization as fol-lows: instead of using LIP detectors, images (with normalizedresolutions) are decomposed into patches with a fixed size(20 × 20 pixels in our case, as shown in Figure 3); patchesare described using color-moment features (the first threemoments, namely the mean, standard deviation, and skewness,are calculated per RGB channel [36]); feature vectors of allpatches are clustered using K-Means so that the medoids areselected as visual words to construct the vocabulary; everytime a new image comes, we decompose it into patches in thesame way and assign the patches to the corresponding visualwords using the BQP quantization approach. By this means,the resulting visual words can be attached back to the imageso that the quantization process is visualized as a process ofsymbolic image reconstruction.

Figure 3 shows an example of reconstructing image withits visual words (more examples can be found at our demowebpage5). It is easy to see that, by enforcing contextualconsistency, the visual words after the BQP quantization areable to recover the original image more faithfully, by usingmore metal-like patches for the fish, more plant-like patchesfor the flower and leaves, and more wood-like patches forthe drawers at background. From the enlarged details, we cansee that the appearances of different parts of the metal fishvary largely due to the illumination and reflection. Therefore,simply selecting the visual words with the largest similaritiesto the corresponding patches will result in a diverse andrandom set of patches for reconstruction. For example, someof the highlights are represented by patches with white text andgray background, which are highly inconsistent to the patchesof the other parts. By contrast, this disadvantage is eliminatedwhen contextual consistency is enforced during quantization.

V. EXPERIMENTS

A. Dataset Construction and Evaluation Metric

To evaluate the performance, we have crawled a datasetincluding one million (i.e., 1,106,280) merchandize images

5http://vireo.cs.cityu.edu.hk/wxy/product.images/

8

from eBay.com and Taobao.com, and divided them into threesubsets: 1) Training Set which includes 1% (i.e., 10,000images) of the entire dataset; 2) Basic Set which includesanother 10% (i.e., 112,092 images) of the dataset; 3) Dis-tracter Set which includes the rest part of the dataset (i.e.,984,188 images). There are no exact duplicates in the datasetand no overlaps among the three subsets. The ground-truthis generated by fully annotating the Basic Set, where 12,901images are found to form 4,141 near duplicate groups. We useeach of those images as a query. Note that the generation of theground-truth is an interactive process, in which we manuallycheck the retrieved list if images from the Distracter Set aretop-ranked and move them to the Basic Set if they are near-duplicates. This process has been done for several rounds,so as to avoid the case (to the largest extent) that a near-duplicate image is included in the Distracter Set. All imagesare normalized to fit an 800× 800 pixel box while preservingtheir aspect ratios. To the best of our knowledge, this is thedataset with the largest number of near duplicate images sofar, which can be downloaded from our demo webpage5 (willbe open to public upon acceptance).

In the experiments, we use each of the near-duplicate imagesas a query. The retrieved images are ranked according totheir similarities to the query. The search performance is thenevaluated with mean average precision (MAP ), where AP isdefined as

AP =1

min (R,n)

n∑j=1

Rj

jIj (9)

where R is the number of duplicate images to a query, Rj isthe number of duplicate images in the top-j retrieved images,and Ij = 1 if the image ranked at jth position is a near-duplicate (ND) and 0 otherwise. We set n=100, while MAPis calculated as the mean AP of the 12,901 queries.

B. Collaborative vs. Individual Representations

We employ two baselines in the experiment which extractLIPs with

• SIFT [6] based method, which is considered one ofthe most popular methods, where difference of Gaussians(DoG) is used for detecting the LIPs while SIFT featureis used for description;

• Color-Moment (CM ) based method (cf. Section IV-C),which extracts patches (of a fixed size of 20 × 20pixels) for each image and uses color-moment feature fordescription. Given the fact that all images are normalizedto fit an 800 × 800 pixel box during the experiments,the CM features are densely sampled, compared to thesparsely sampled SIFT features.

Both are implemented following the LIPs+BoW frameworkin Figure 2. To investigate the effect of enforcing contextualconsistency in NDR, we modify the two baselines by furtherimposing the contextual consistency constraint during quanti-zation, resulting in two new methods SIFTCC and CMCC

where the superscripts have been added to distinguish from thebaseline versions. The Training Set is used to learn the visualword collocations (i.e., co-occurrence patterns) for SIFTCC

100k 200k 300k 400k 500k 600k 700k 800k 900k 1000k0.1

0.15

0.2

0.25

0.3

0.35

0.4

Dataset Scale

Me

an

Ave

rag

e P

recis

ion

(M

AP

)

SIFT

CM

SIFTCC

CMCC

Fig. 4: Performance comparison over different dataset scales.

and CMCC . In addition, to investigate the performance overdifferent scales of datasets, our experiment starts by using thefully annotated Basic Set as a basis and randomly addingsamples from the Distracter Set to generate larger sets. Dueto a large number of experiments required for investigating theeffects of the features and parameters to the performances, wetemporarily fix the vocabulary at a small size 2k to speed upthe process. The performances of using larger vocabulary sizeswill be reported and discussed in Section V-C.

The results with MAP at the top-100 retrieved images areshown in Figure 4. It is easy to see that by further consideringcontextual consistency, the performance of the two baselineshave been improved significantly and the improvements arestable (ranging from 20.87% to 61.91%) over different datasetscales. Statistics on the Basic Set when dataset scale equals100k show that SIFTCC (CMCC) improves 6,164 (6,492)out of the 12,901 queries of from the baseline SIFT (CM ),while leaving 5,466 (4,802) queries unchanged and only asmall portion of 1,271 (1,617) queries with slight drops.This has confirmed the advantage of considering contextualconsistency in NDR, and is also consistent to our analy-sis in Section IV that enforcing contextual consistency inquantization can improve the quality of the LIPs-to-Visual-Words mapping and thus boosts the performance. To graspan in-depth understanding of the method, we further conductseveral experiments to investigate its impacts of/on features,parameters, and the ranking list.

1) The impacts of features: In Figure 4, we can see that,benefiting from the contextual consistency, the improvement ofCMCC from CM (47.18%±11.73) is much more significantthan that of SIFTCC from SIFT (38.20%±10.95). Theseresults have indeed further revealed the advantage of thecollaborative representation gained by considering contextualconsistency. Color moment are usually extracted from patchesof a grid-divided image so that features from those patches canbe concatenated to represent color patterns in the image withrespect to spatial relation given by the grid. However, the wayof using color-moment in CM without considering contextualconsistency has broken the spatial relations among patches

9

Query Ranked List

Fig. 5: Examples of the query images and the retrieved items by CMCC .

because each patch is considered individually in the LIP-s+BoW framework. This makes CM just a color histogram-like feature which is weak and lacks of representation powerwhen compared to SIFT .

However, when contextual consistency is considered inCMCC , spatial relations are taken into consideration, becausethe BQP quantization (cf. Section IV-B) will force the color-moment features of neighboring patches to be consistentwith the co-occurrence patterns learned on the Training Set.Furthermore, the way of using visual word collocations toindirectly represent the spatial patterns can be more flexiblethan the conventional grid dependent method, because thefixed grid is not easy to be scale or rotation invariant. Bycontrast, the improvement of SIFTCC from SIFT is notas significant as that of CMCC , because SIFT is inherentlyscale and rotation invariant, and with strong representativenessfor local patterns. Therefore, the room for improving SIFTis comparably limited.

To give an intuitive impression on the performance ofCMCC on the merchandize retrieval, in Figure 5, we showseveral query images and the corresponding ranked lists re-turned by CMCC . We can see that CMCC is rotation invariantand is capable of tolerating small to moderate changes inscale. The rotation invariance is straightforwardly resultingfrom the definition of the color-moment feature, which is

inherently a rotation invariant feature at patch-level. At image-level, since CMCC models the inter-patch relationship withthe co-occurrence among patches (instead of the strict spatialconstraint), it is also invariant to the rotation. Furthermore,although CMCC is not a scale invariant feature, it can stilltolerate small changes in scale, thanks to the relatively simpletextures in most of the merchandize images.

2) The impact on ranking: To investigate how the perfor-mance is improved by considering contextual consistency, weplot the change of APs on the Basic Set in Figure 6. Itis easy to see that the performance of a method is indeeddetermined by the number of NDs that it retrieved at the top-30 of the ranked list. This is an indication that most of the NDsdistribute within the section of top-1 to 30 of the ranked lists.Furthermore, we can see that the APs of SIFTCC and CMCC

of this section drop more dramatically than those of SIFTand CM , which is another indication that the improvementsbrought by considering contextual consistency has its impactmainly on the top-30 items of the ranked lists, where theNDs are boosted to the top positions. This is a highly desiredcharacteristics, because users usually will or can access onlythe top-30 retrieved items in most of the applications (e.g.,on the first page of the retrieved items in the e-Commercewebsites, or on their smart phones where the screen size istoo limited to display too many items).

10

10 20 30 40 50 60 70 80 90 100

0.35

0.4

0.45

0.5

0.55

0.6

top−k

Mea

n A

vera

ge P

reci

sion

(M

AP

)

CMSIFT

CMCC

SIFTCC

Fig. 6: Evaluation of MAP at top-k of the ranked lists.

5 10 15 20 25 30 35 400.3

0.32

0.34

0.36

0.38

0.4

0.42

0.44

Size of Neighboring Area (d)

Mea

n A

vera

ge P

reci

sion

(M

AP

)

SIFTCC

CMCC

Fig. 7: Evaluation of MAP over the change of the size of theneighboring area (d, in pixels).

3) The impacts of the parameters: In case that the vo-cabulary size is fixed, there are only two parameters in ourapproach, namely the radius d (in pixels) which defines thesize of the neighboring area, and the number of visual wordcandidates for each LIP (denoted as #can hereafter6).

Figure 7 shows the performances of SIFTCC and CMCC

over the changes of d on the Basic Set, where we can see thatwith the growth of d, the performance of SIFTCC (CMCC)steps up continuously and is approaching stable at the point 30(25)7. This is due to the fact that, while the d is increasing, theoverlaps among neighboring areas of LIPs are also enlarged,

6Note the parameter has been denoted as k in Section IV-B for a bettermathematical presentation. Here #can is used to make its meaning of visualword candidates more intuitive.

7Note that in SIFTCC , two words have the neighborhood relationshiponly when the corresponding LIPs fall into a circle of radius d (pixels), whilein CMCC , the relationship is dependent on whether the centers of the twopatches fall into the circle.

1 2 3 4 5 6 7 8 9 100.3

0.32

0.34

0.36

0.38

0.4

0.42

0.44

0.46

Number of Candidates (#can)

Mea

n A

vera

ge P

reci

sion

(M

AP

)

SIFTCC

CMCC

Fig. 8: Evaluation of MAP over the number of the candidatesfor each LIP.

so that, at first, LIPs within the same neighboring areas areconnected to form patterns which model the image moreaccurately. After that, while d increases further, the newlyformed patterns are also connected, so that relations amongLIPs not residing the same neighboring areas can be modeledindirectly, benefiting from the transitivity. Therefore, all theLIPs in an image can work collaboratively, resulting in a betterrepresentation. However, this will not bring further gain for theperformance when d crosses certain threshold, since all LIPshas already been connected as a whole. Moreover, we cansee that the performance of CMCC converges at smaller dthan SIFTCC . This is because CMCC (i.e., image blocksor patches) are tightly connected when we divide the image,while LIPs of SIFTCC (founded by LIP detector) are sparselyscattered.

Figure 8 shows the performances of SIFTCC and CMCC

over the number of visual word candidates for each LIP(#can), where the performances of the two methods areapproaching a stable plateau after crossing the thresholds,3 and 5, respectively. Generally, the more the candidatesare, the higher probability that the best visual word canbe found after the BQP quantization. However, the optimalnumber of candidates is indeed dependent on the variance ofcorresponding feature, because the larger the variance of thefeature is, the more candidates need to be selected to makesure the best visual word is included. This in fact explains whySIFTCC converges at smaller number of candidates than thatof CMCC , because the SIFT is claimed to be scale, rotationinvariant, which we cannot guarantee with CM .

We have also conducted experiments for finding the bestcombination of the three parameters (i.e., the vocabularysize, d and #can) by using Grid Search. The experimentalresults show that CMCC (SIFTCC) converges when thevocabulary size, d and #can are set at 15K, 25 and 5(25K, 30 and 3), respectively. These values will be used inthe subsequent sections as default settings for the parametersunless otherwise specified. Furthermore, the three parameters

11

are observed independent of each other, because the changeof the value for one parameter will not affect the conclusionsmade for the other parameters. This gives us advantage for theinvestigations.

C. Comparison with the State-of-the-art

To investigate the accuracy and efficiency of the proposedmethods, we compare SIFTCC and CMCC to several state-of-the-art NDR methods in the literature:

• Hamming Embedding (HE) [24], which assigns hammingcodes to LIPs as binary signatures for addressing thevisual ambiguity introduced by soft-weighting (SOFT)[27] so as to improve the quality of the quantization andspeed up the matching process at the same time;

• Weak Geometric Consistency (WGC) [13], which verifiesthe weak or partial geometric consistency between thequery and target images for re-ranking;

• The enhanced WGC (EWGC) [14], which enhances thegeometric consistency based reranking by further includ-ing translation information into WGC;

• Synonym-based Query Expansion (Syn-Expan) [30]8,which defines the visual words that frequently co-occurin the same circular regions (cf. Section IV) as synonymsto each other and uses synonyms to conduct queryexpansion for improving the performance of LIPs+BoWframework;

• Delaunay Triangulation (DT) [15], the method reportedwith the highest performance in literature, which proposesto model the geometric relations among visual words ina more principled way by using Delaunay Triangulation,and thus makes the mapping between images more robust.The “burstiness” of visual words (BUR) [22] is alsoconsidered in this method.

According to our study on Basic Set, the performances of mostmethods are approaching optimal when the vocabulary is setto the range of 10k to 40k (no significant improvement when alarger vocabulary is used). Therefore, to be fair and consistent,for the experiments hereafter, we unify the vocabulary size to20k for all the methods under investigation. Note the size is10 times larger than the one we used in Section V-B (i.e.,2k). Furthermore, to ease the discussion, we add a subscriptfor a method to indicate its vocabulary size (e.g., SIFTCC

20k ).All methods are implemented in C++ and all the experimentsare conducted on a station with Intel Xeon(R) 2.67G Hz and30G memory. The performances of all methods are optimized(with the condition that the vocabulary size equals to 20k )with the configurations summarized in Table II. Note that toinvestigate the effects of BQP and Synonyms (i.e., differentways of using spatial context), we have not combined HEand other advanced techniques into Syn-Expan. This makesthe performance of Syn-Expan lower than those reported in[30]. To fully investigate the nature of these methods, wecompare them on three datasets, 1) Basic Set, the datasetwith the largest number of fully annotated queries so far. It isemployed for investigating the basic characteristics of each

8We thank Drs. Zhang Lei and Cai Rui, the authors of [30], for sharing thebinary code and helping with the experiments.

TABLE II: Configurations of methods under investigation.

Method Configuration FeatureSIFTCC

20k BoW+BQP SIFTSIFT20k BoW SIFTCMCC

20k BoW+BQP Color-MomentCM20k BoW Color-MomentHE BoW+SOFT+HE SIFTWGC BoW+SOFT+HE+WGC SIFTEWGC BoW+SOFT+HE+EWGC SIFTSyn-Expan BoW+Synonyms SIFTDT BoW+SOFT+HE+DT+BUR SIFT

method; 2) Kentucky [37], a dataset similar to Basic Setwhich is composed of mostly product images and has beenpopularly employed in previous studies. It is employed fortesting the generalizability of each method; and 3) Oxford[38], which is another standard benchmark which is composedof building images. It is employed for conducting the cross-domain experiments.

1) General Comparison on Basic Set: The performancecomparison of accuracy (MAP) and efficiency is shown in Ta-ble III. In terms of accuracy, SIFTCC

20k outperforms all conven-tional approaches, while, surprisingly, CMCC

20k which employsmuch simpler feature than SIFT outperforms SIFT 20k, Syn-Expan and EWGC, and demonstrates comparable performanceto WGC and DT. In addition, by comparing the performanceof SIFT20k (MAP: 0.4080) to SIFTCC (MAP: 0.4332) (with2k vocabulary) or CM20k (MAP: 0.3894) to CMCC (MAP:0.4081), it is clear that with the help of BQP quantization,we can reduce the vocabulary size by 1,000% times withoutlosing accuracy. Figure 9 shows a more detailed comparison ofMAP over top-k, where SIFTCC

20k and CMCC20k ’s superiorities

over other methods are more evidential at the range of top-10 retrieved images. Given the fact that there are on average7.22 NDs for each query, this indicates that most of the NDscan be located at top-10. As mentioned in Section V-B, thisis a highly desired property in merchandize image retrieval.Moreover, with simply color-comment feature, which was con-ventionally recognized as a much weaker feature than SIFT,CMCC

20k earns a surprising performance without employing anyadvanced techniques (e.g., geometric constraint or hammingembedding). While this has again confirmed the advantageof the proposed BQP quantization, we should see that thisis also due to the specificity of the merchandize images. Ascolor is an important feature for most of the products, thephotographers usually try their best to maintain real color ofthe product when taking the pictures. However, most of thelocal feature (e.g., SIFT, SURF) has ignored the color, makingtheir representations discount a lot. Therefore, color momentappears a better choice for such images.

In terms of time cost per query, as shown in Table III,SIFTCC

20k and CMCC20k are obviously more efficient than con-

ventional approaches, while demonstrating comparable accu-racy. It is worth mentioning that by including additional BQPquantization on SIFT20k and CM20k respectively, the timecost for SIFTCC

20k and CMCC20k have not been increased but

rather decreased. The reason is that with the BQP quantization,the errors/typos (i.e., the mis-assigned visual words) have beenremoved, resulting in a smaller and sparser set of visual words

12

TABLE III: Performance comparison with the state-of-the-art on Basic Set. The best results are bold.

SIFTCC20k CMCC

20k SIFT20k CM20k WGC EWGC HE Syn-Expan DTMean Average Precision (MAP) 0.5578 0.5215 0.4080 0.3894 0.5404 0.5144 0.5352 0.4834 0.5406Time Cost (milliscond/query) 231 128 246 134 1895 1604 705 367 3411

TABLE IV: Performance comparison with the state-of-the-art on Kentucky. The best results are bold.

SIFTCC20k CMCC


TABLE V: Performance comparison with the state-of-the-art on Oxford. The best results are bold.

SIFTCC20k CMCC


10 20 30 40 50 60 70 80 90 1000.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

top-k

Me

an

Ave

rag

e P

recis

ion

(M

AP

)

CM20k

SIFT20k

CMCC

20k

SIFTCC

20k

EWGC

WGC

HE

DT

Syn-Expan

Fig. 9: Evaluation of MAP at the top-k of the ranked lists.

for representing each image, and thus saves the computationalcost. For example, in CMCC , the sparsity of the CM featurevectors has been improved from 15.07% to 11.24%, whichcan save about 1/3 of the time cost for calculating thesimilarities among feature vectors (the most expensive processin BoW framework since numerous times of multiplicationsare needed for computing the inner products). The saved cost islarger than the additional cost for BQP (only 27 milliseconds)and thus reduces the total time cost per query. Furthermore,considering both accuracy and efficiency, we can see that undercertain scenario like merchandize image retrieval, CMCC

20k

can be used to replace some conventional methods based oncomplicated SIFT feature (e.g., to replace EWGC so as toobtain comparable performance while reducing the time costby 9,202%). In terms of space complexity, one may haveconcern on the big matrix I that stores the collocative relationsamong words. Our experimental results show that the sparsityof the matrix is less than 10%, because only small amountof the word-pairs can pass the X 2 test to be recognizedas collocations. Therefore, at most 40M of the memory isneeded for the matrix (in floating point numbers) with a 20kvocabulary.

2) Comparison of Generalizability on Kentucky: To test thegeneralizability of each method, we extend the performancecomparison to Kentucky benchmark [37], which consists of10,200 product images forming 2,550 ND groups (4 images foreach group). We use each of the images as a query, resultingin 10,200 queries. The Training Set is used for training inSIFTCC

20k , CMCC20k , and Syn-Expan. The results are shown

in Table IV, where the performances of the methods underinvestigation are basically consistent with those on the BasicSet. This is not surprising, because the product images ofKentucky are naturally similar to the merchandize images ofBasic Set. However, CMCC

20k obtains the highest performancethis time, even better than SIFTCC

20k , WGC and DT in termsof MAP. The reason is that the scale variance on Kentuckyis not as serious as that on Basic Set, because the sameobject usually occupies the similar portion of an image andthe images of Kentucky are all with the same size. This helpsthe color-moment-based CMCC

20k bypass the requirement forscale invariant (i.e., its biggest disadvantage compared withthe SIFT-based features), and at the same time fully exhibitits strength in merchandize image retrieval (as we have learnton Basic Set). Due to the same reason, the performances ofspatial verification-based methods (e.g., WGC, EWGC, DT)are not as good as expected, indicated by the fact that thereare no significant improvements observed from their MAPswhen compared to that of HE. However, this observation isconsistent with that reported in [13].

3) Cross-Domain Experiments on Oxford: To investigatewhether the BQP works on datasets other than merchandizeimages, we conduct cross-domain experiments on Oxford[38], which is another standard benchmark consisting of 5,062images of particular Oxford landmarks, with 55 query imagesand manually labeled ground truth. An additional set of100,071 images (Flickr 100k) also provided by the authorsof Oxford [38] is used for training in SIFTCC

20k , CMCC20k ,

and Syn-Expan. The results are shown in Table V, wherethe performance of CMCC

20k drops significantly due to factthat compared to Basic Set and Kentucky, the scale varianceissue becomes much more serious on Oxford. However, byenforcing contextual consistency with BQP, SIFTCC

20k stilloutperforms some state-of-the-art methods such as HE, Syn-Expan, EWGC, while approaching WGC and DT, even without

13

employing any advanced techniques like hamming embeddingand soft-assignment.

VI. CONCLUSIONS

In this paper, we have studied the use of contextual consis-tency constraint for NDR to fulfill the critical requirement forresponse time when searching merchandize images. We haveextended the concept of “collocation” in text domain to NDRfor modeling the contextual relations among visual words,on the basis of which we use binary quadratic programming(BQP) to enforce the visual words selected for representingan image to be contextually consistent to each other so asto improve the quality of the quantization. The experimentalresults have validated the effectiveness and efficiency of theproposed method.

In terms of online application, one of the most favor-able findings in this paper is that, for merchandize imageretrieval, the conventional local features (e.g., SIFT) can bereplaced by patch-wise feature based on color-moment, whichcan significantly improve the efficiency. While encouraging,we should see that the patch-wise color-moment feature isnot scale invariant. It may cause problem when the scalechange exceeds a certain degree (even this rarely happens formerchandize images). Further incorporating a pyramid basedmatching process into the method may address the problem.We will investigate this in our future work.

REFERENCES

[1] J. Sivic and A. Zisserman, “Video google: A text retrievalapproach to object matching in videos,” in IEEE InternationalConference on Computer Vision, 2003.

[2] O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman, “Totalrecall: Automatic query expansion with a generative featuremodel for object retrieval,” in IEEE International Conferenceon Computer Vision, 2007, pp. 1–8.

[3] D. Nister and H. Stewenius, “Scalable recognition with avocabulary tree,” in IEEE Conference on Computer Vision andPattern Recognition, 2006, pp. 2161–2168.

[4] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman,“Object retrieval with large vocabularies and fast spatial match-ing,” in IEEE Conference on Computer Vision and PatternRecognition, 2007.

[5] H. Jegou, H. Hedi, and C. Schmid, “A contextual dissimilaritymeasure for accurate and efficient image search,” in IEEEConference on Computer Vision and Pattern Recognition, 2007,pp. 1–8.

[6] D. G. Lowe, “Distinctive image features from scale-invariantkeypoints,” International Journal of Computer Vision, vol. 60,no. 2, pp. 91–110, 2004.

[7] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-uprobust features (SURF),” Computer Vision and Image Under-standing, vol. 110, no. 3, pp. 346–359, 2008.

[8] J. Yuan, L.-Y. Duan, Q. Tian, and C. Xu, “Fast and robust shortvideo clip search using an index structure,” in ACM SIGMMInternational Workshop on Multimedia Information Retrieval,2004, pp. 61–68.

[9] E. Y. Chang, J. Z. Wang, C. Li, and G. Wiederhold, “Rime:A replicated image detector for the world-wide web,” in SPIESymposium of Voice, Video, and Data Communications, 1998,pp. 58–67.

[10] A. H. A, K. ho Hyun B, and R. B. A, “Comparison of sequencematching techniques for video copy detection,” in Conferenceon Storage and Retrieval for Media Databases, 2002.

[11] R. Fergus, P. Perona, and A. Zisserman, “Object class recog-nition by unsupervised scale-invariant learning,” in IEEE Con-ference on Computer Vision and Pattern Recognition, 2003, pp.264–271.

[12] F. Wang and C.-W. Ngo, “Rushes video summarization byobject and event understanding,” in TRECVID BBC RushesSummarization Workshop at ACM Multimedia, 2007, pp. 25–29.

[13] H. Jegou, M. Douze, and C. Schmid, “Improving bag-of-features for large scale image search,” International Journal ofComputer Vision, vol. 87, no. 3, pp. 316–336, 2010.

[14] W.-L. Zhao, X. Wu, and C.-W. Ngo, “On the annotation of webvideos by efficient near-duplicate search,” IEEE Transactions onMultimedia, vol. 12, no. 5, 2010.

[15] W. Zhang, L. Pang, and C.-W. Ngo, “Snap-and-ask: answer-ing multimodal question by naming visual instance,” in ACMInternational Conference on Multimedia, 2012, pp. 609–618.

[16] D.-Q. Zhang and S.-F. Chang, “Detecting image near-duplicateby stochastic attributed relational graph matching with learn-ing,” in ACM International Conference on Multimedia, 2004,pp. 877–884.

[17] Y. Ke, R. Sukthankar, and L. Huston, “An efficient parts-based near-duplicate and sub-image retrieval system,” in ACMInternational Conference on Multimedia, 2004, pp. 869–876.

[18] Z. Wu, Q. Ke, M. Isard, and J. Sun, “Bundling features for largescale partial-duplicate web image search,” in IEEE Conferenceon Computer Vision and Pattern Recognition, 2009, pp. 25–32.

[19] W. Zhou, Y. Lu, H. Li, Y. Song, and Q. Tian, “Spatial codingfor large scale partial-duplicate web image search,” in ACMInternational Conference on Multimedia, 2010, pp. 511–520.

[20] A. Joly, O. Buisson, and C. Frelicot, “Content-based copyretrieval using distortion-based probabilistic similarity search,”IEEE Transactions on Multimedia, vol. 9, no. 2, pp. 293–306,2007.

[21] M. S. Extremal, J. Matas, O. Chum, M. Urban, and T. Pajdla,“Robust wide baseline stereo from,” in British Machine VisionConference, 2002, pp. 384–393.

[22] H. Jegou, M. Douze, and C. Schmid, “On the burstiness ofvisual elements,” in IEEE Conference on Computer Vision andPattern Recognition, 2009, pp. 1169–1176.

[23] P. Jain, B. Kulis, and K. Grauman, “Fast image search forlearned metrics,” in IEEE Conference on Computer Vision andPattern Recognition, 2008.

[24] H. Jegou, M. Douze, and C. Schmid, “Hamming embeddingand weak geometric consistency for large scale image search,”in European Conference on Computer Vision: Part I, 2008, pp.304–317.

[25] O. Chum, J. Philbin, M. Isard, and A. Zisserman, “Scalablenear identical image and shot detection,” in ACM InternationalConference on Image and Video Retrieval, 2007.

[26] O. Chum, J. Philbin, and A. Zisserman, “Near duplicate imagedetection: min-hash and tf-idf weighting,” in British MachineVision Conference, 2008.

[27] Y.-G. Jiang, C.-W. Ngo, and J. Yang, “Towards optimal bag-of-features for object categorization and semantic video retrieval,”in ACM International Conference on Image and Video Retrieval,2007, pp. 494–501.

[28] J. C. van Gemert, C. J. Veenman, A. W. M. Smeulders, and J.-M. Geusebroek, “Visual word ambiguity,” IEEE Transactionson Pattern Analysis and Machine Intelligence, vol. 32, no. 7,pp. 1271–1283, 2010.

[29] R. A. Baeza-Yates and B. Ribeiro-Neto, Modern InformationRetrieval. Boston, MA, USA: Addison-Wesley LongmanPublishing Co., Inc., 1999.

[30] W. Tang, R. Cai, Z. Li, and L. Zhang, “Contextual synonymdictionary for visual object retrieval,” in ACM internationalconference on Multimedia, 2011, pp. 503–512.

[31] D. Pearce, “A comparative evaluation of collocation extrac-tion techniques,” in International Conference on Language

14

Resources and Evaluation, 2002.[32] Y. Jiang, “Randomized visual phrases for object search,” in

IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Washington, DC, USA, 2012, pp. 3100–3107.

[33] J. Yuan, Y. Wu, and M. Yang, “Discovery of collocation pat-terns: from visual words to visual phrases,” in IEEE Conferenceon Computer Vision and Pattern Recognition, 2007.

[34] Y. Huang, S. Shekhar, and H. Xiong, “Discovering collocationpatterns from spatial data sets: A general approach,” IEEETransactions on Knowledge and Data Engineering, vol. 16,no. 12, pp. 1472–1485, 2004.

[35] G. L. Nemhauser and L. A. Wolsey, Integer and combinatorialoptimization. New York, NY, USA: Wiley-Interscience, 1988.

[36] M. A. Stricker and M. Orengo, “Similarity of color images,”in IS&T/SPIE’s Symposium on Electronic Imaging: Science &Technology. International Society for Optics and Photonics,1995, pp. 381–392.

[37] D. Nister and H. Stewenius, “Scalable recognition with avocabulary tree,” in IEEE Conference on Computer Vision andPattern Recognition, 2006.

[38] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman,“Object retrieval with large vocabularies and fast spatial match-ing,” in IEEE Conference on Computer Vision and PatternRecognition, 2007.

Xiao-Yong Wei (M’10) is an Associate Professorwith the College of Computer Science, Chengdu,Sichuan University, China. His research interestsinclude multimedia retrieval, data mining, and ma-chine learning. He received his Ph.D. degree in Com-puter Science from City University of Hong Kongin 2009. He is one of the founding members of theVIREO multimedia retrieval group, City Universityof Hong Kong. He was a Senior Research Associatein Dept. of Computer Science and Dept. of Chinese,Linguistics and Translation of City University of

Hong Kong in 2009 and 2010, respectively. He had worked as a Managerof Software Dept. at Para Telecom Ltd., China, from 2000 to 2003.

Zhen-Qun Yang is a Ph.D candidate in the Collegeof Computer Science, Chengdu, Sichuan University,China. She was a Research Assistant with the Dept.of Computer Science, City University of Kong Kongfrom 2007 to 2008 and a Senior Research Assistantwith the Dept. of Chinese, Linguistics and Trans-lation of City University of Hong Kong in 2009.Her research interests include multimedia retrieval,image processing and pattern recognition, and neuralnetworks.

Chong-Wah Ngo (M’02) received the B.Sc.and M.Sc. degrees in computer engineering fromNanyang Technological University, Singapore, andthe Ph.D. degree in computer science from the HongKong University of Science and Technology, HongKong.

He was a Post-Doctoral Scholar with the Beck-man Institute, University of Illinois at Urbana-Champaign, Champaign. He was a Visiting Re-searcher with Microsoft Research Asia. He is cur-rently an Associate Professor with the Department of

Computer Science, City University of Hong Kong, Kowloon, Hong Kong. Hiscurrent research interests include large-scale multimedia information retrieval,video computing, and multimedia mining.

Dr. Ngo is currently an Associate Editor of the IEEE Transactions On Multi-media. He is the Conference Co-Chair of the ACM International Conferenceon Multimedia Retrieval 2015 and Pacific Rim Conference on Multimedia2014; and the Program Co-Chair of the ACM Multimedia Modeling Confer-ence 2012 and the ACM International Conference on Multimedia Retrieval2012; and the Area Chair of the ACM Multimedia 2012. He was the Chairmanof the ACM (Hong Kong Chapter) from 2008 to 2009.

Wei Zhang received his B.Eng degree from Schoolof Computer Software, the M.Eng degree fromSchool of Computer Science and Technology inTianjin University, Tianjin, China, in 2008 and 2011,respectively. He is now a Ph.D candidate with De-partment of Computer Science in City University ofHong Kong, Kowloon, Hong Kong.

He was a former member of CV-TJU lab in Tian-jin University from 2008 to 2011. He is now with theVIREO group in City University of Hong Kong. Hisresearch interests include large scale video retrieval

and digital forensic analysis.

visual typo correction by collocative optimization

Documents