Slide 1

Easy Samples First:Self-paced Reranking for Zero-Example Multimedia SearchLu Jiang1, Deyu Meng2, Teruko Mitamura1, Alexander G. Hauptmann11 School of Computer Science, Carnegie Mellon University2School of Mathematics and Statistics, Xi'an Jiaotong University

1PeopleCMU Informedia Team

Teruko MitamuraDeyu MengAlexander G. Hauptmann

Acknowledgement This work was partially supported by Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center contract number D11PC20068. Deyu Meng was partially supported by 973 Program of China (3202013CB329404) and the NSFC project (61373114). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government.

3OutlineBackgroundRelated WorkSelf-Paced Reranking (SPaR)Experiment ResultsConclusions

4OutlineBackgroundRelated WorkSelf-Paced Reranking (SPaR)Experiment ResultsConclusions

5Zero-Example SearchZero-Example Search (also known as 0Ex) represents a multimedia search condition where zero relevant examples are provided.An example: TRECVID Multimedia Event Detection (MED). The task is very challenging.Detect every-day event in Internet videosBirthday PartyChanging a vehicle tireWedding ceremonyContent-based search. No textual metadata (title/description) is available.

Detect the occurrence of a main event occurring in a video clip, e.g. Birthday Party, Marriage proposal, Repairing an appliance.Content-based search. No textual metadata (title) is available.Multi-modal search (visual concepts/speech/OCR).Challenging (complex events in amateur Internet videos)

6Zero-Example SearchEvent of Interest: Birthday PartyELAMP Prototype System

Zero-Example SearchEvent of Interest: Birthday PartyELAMP Prototype System

Birthday Cake, Kids, Gift

Zero-Example SearchEvent of Interest: Birthday PartyELAMP Prototype System

Birthday Cake, Kids, GiftHappy Birthday, Cheering

Zero-Example SearchEvent of Interest: Birthday PartyELAMP Prototype System

Birthday Cake, Kids, GiftHappy Birthday, Cheering

Birthday Party

Zero-Example SearchEvent of Interest: Birthday PartyELAMP Prototype System

Birthday Cake, Kids, GiftHappy Birthday, Cheering

Birthday Party0Ex SystemRerankingIntuition: initial ranked result is noisy.Refined by the multimodal info residing in the top ranked videos/images.

RerankingOutlineBackgroundRelated WorkSelf-Paced Reranking (SPaR)Experiment ResultsConclusions

13Related WorkCategorization of reranking methods:Classification-based (Yan et al. 2003) (Hauptmann et al. 2008)(Jiang et al. 2014)Clustering-based(Hsu et al. 2007)LETOR(LEarning TO Rank)-based(Liu et al. 2008) (Tian et al. 2008) (Tian et al. 2011)Graph-based(Hsu et al. 2007) (Nie et al. 2012)R. Yan, A. G. Hauptmann, and R. Jin. Multimedia search with pseudo-relevance feedback. In CVIR, 2003.A. G. Hauptmann, M. G. Christel, and R. Yan. Video retrieval based on semantic concepts. Proceedings of the IEEE, 96(4):602622, 2008.L. Jiang, T. Mitamura, S.-I. Yu, and A. G. Hauptmann. Zero-example event search using multimodal pseudo relevance feedback. In ICMR, 2014W. H. Hsu, L. S. Kennedy, and S.-F. Chang. Video search reranking through random walk over document-level context graph. In Multimedia, 2007.Y. Liu, T. Mei, X.-S. Hua, J. Tang, X. Wu, and S. Li. Learning to video search rerank via pseudo preference feedback. In ICME, 2008.X. Tian, Y. Lu, L. Yang, and Q. Tian. Learning to judge image search results. In Multimedia, 2011.X. Tian, L. Yang, J. Wang, Y. Yang, X. Wu, and X.-S. Hua. Bayesian video search reranking. In Multimedia, 2008.W. H. Hsu, L. S. Kennedy, and S.-F. Chang. Video search reranking through random walk over document-level context graph. In Multimedia, 2007.L. Nie, S. Yan, M. Wang, R. Hong, and T.-S. Chua. Harvesting visual concepts for image search with complex queries. In Multimedia, 2012.Generic Reranking Algorithm

Generic Reranking Algorithm

Pseudo labels: assumed (hidden) labels for samples.Zero-example: ground truth label unknown.Generic Reranking Algorithm

Iteration: 1

Treat top-ranked videos as pseudo-positives.Treat bottom-ranked or random-selected videos as pseudo-negatives.

17Generic Reranking Algorithm

Iteration: 1

Iteration: 2IntuitionExisting methods assign equal weights to pseudo samples.Intuition: samples ranked at the top are generally more relevant than those ranked lower.Our approach: learn the weight together with the reranking model.

19Generic Reranking AlgorithmQuestions:Why the reranking algorithm performs iteratively?Does the process converge? If so, to where?Does the arbitrarily predefined weighting scheme converge?

Generic Reranking AlgorithmQuestions:Why the reranking algorithm performs iteratively?Does the process converge? If so, to where?Does the arbitrarily predefined weighting scheme converge?

Generic Reranking AlgorithmQuestions:Why the reranking algorithm performs iteratively?Does the process converge? If so, to where?Does the arbitrarily predefined weighting scheme converge?

OutlineBackgroundRelated WorkSelf-Paced Reranking (SPaR)Experiment ResultsConclusions

23Self-paced LearningCurriculum Learning (Bengio et al. 2009) or self-paced learning (Kumar et al 2010) is a recently proposed learning paradigm that is inspired by the learning process of humans and animals.The samples are not learned randomly but organized in a meaningful order which illustrates from easy to gradually more complex ones.

Prof. Koller

Prof. BengioY. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In ICML, 2009.M. P. Kumar, B. Packer, and D. Koller. Self-paced learning forlatent variable models. In NIPS, pages 11891197, 2010.It learns samples iteratively.It examines the easiness of each sample based on what it has already learned to be used in the subsequent iterations.

24Self-paced LearningEasy samples to complex samples.Easy sample smaller loss to the already learned model.Complex sample bigger loss to the already learned model.

AgeSelf-paced LearningIn the context of reranking : easy samples are the top-ranked videos that have smaller loss.

AgeRanked list of iteration 1Ranked list of iteration nSelf-paced Reranking (SPaR)We propose a novel framework named Self-Paced Reranking (SPaR) pronounced as /sp/.Inspired by the self-paced learning theory.Formulate the problem as a concise optimization problem.

*Images from http://en.wikipedia.org/wiki/Hot_springSelf-paced Reranking (SPaR)The propose model:

Reranking models for each modality.The weight for each sample.The pseudo label.Self-paced Reranking (SPaR)The propose model:

loss-functionconstraintsregularizerThe loss in the reranking model is discounted by a weight.

Reranking models for each modality.The weight for each sample.The pseudo label.29Self-paced Reranking (SPaR)The propose model:

regularizerFor example the Loss in the SVM model.

Reranking models for each modality.The weight for each sample.The pseudo label.Self-paced Reranking (SPaR)The propose model:

loss-functionconstraints is the total number of modality.

Reranking models for each modality.The weight for each sample.The pseudo label.

is the self-paced function in self-paced learning.

The self-paced is implemented by a regularizer.Physically corresponds to learning schemes that human use to learn different tasks.The regularizer is general and independent of the loss function.

31Self-paced Function

is the age parameter in self-paced learning.Physically it corresponds to the age of the learner.The definition which provides an axiom for self-paced learning.Convex function.Self-paced Function

We propose the definition which provides an axiom for self-paced learning.Favors easy samples.

is the age parameter in self-paced learning.Physically it corresponds to the age of the learner.Self-paced Function

We propose the definition which provides an axiom for self-paced learning.When the model is younguse less samples;When the model is matureuse more;

is the age parameter in self-paced learning.Physically it corresponds to the age of the learner.Self-paced FunctionExisting self-paced functions only support binary weighting (Kumar et al 2010).

We argument the weight schemes and proposes the following soft weighting.

M. P. Kumar, B. Packer, and D. Koller. Self-paced learning forlatent variable models. In NIPS, pages 11891197, 2010.Binary weightingLinear weightingLogarithmic weightingMixture weightingSelf-paced FunctionExisting self-paced functions only support binary weighting (Kumar et al 2010).

Binary weightingLinear weightingLogarithmic weightingMixture weighting

Reranking in Optimization and Conventional Perspective

CCM (Cyclic Coordinate Method) is used to solve the problem.Fixing one variable and optimizing the other variables.Reranking in Optimization and Conventional Perspective

Optimization perspective theoretical justificationsConventional perspective offers practical lessons Reranking is a self-paced learning process.Optimization perspectiveConventional perspective Reranking in Optimization and Conventional Perspective

Q1: Why the reranking algorithm performs iteratively?A: Self-paced learning mimicking human and animal learning process (from easy to complex examples).Q2: Does the process converge? If so, to where?A: Yes, to the local optimum. See the theorem in our paper.Q3: Does the arbitrarily predefined weighting scheme converge?A: No, but the weights by self-paced function guarantees the convergence.Reranking in Optimization and Conventional Perspective

Q1: Why the reranking algorithm performs iteratively?A: Self-paced learning mimicking human and animal learning process (from easy to complex examples).Q2: Does the process converge? If so, to where?A: Yes, to the local optimum. See the theorem in our paper.Q3: Does the arbitrarily predefined weighting scheme converge?A: No, but the weights by self-paced function guarantees the convergence.Reranking in Optimization and Conventional Perspective

Q1: Why the reranking algorithm performs iteratively?A: Self-paced learning mimicking human and animal learning process (from easy to complex examples).Q2: Does the process converge? If so, to where?A: Yes, to the local optimum. See the theorem in our paper.Q3: Does the arbitrarily predefined weighting scheme converge?A: No, but the weights by self-paced function guarantees the convergence.

OutlineBackgroundRelated WorkSelf-Paced Reranking (SPaR)Experiment ResultsConclusions

42TRECVID Multimedia Event DetectionDataset: MED13Test (around 34,000 videos) on 20 predefined events. Test on the NISTs split (25,000 videos).Evaluated by Mean Average Precision.Four types of high-level features:ASR, OCR, SIN, and ImageNet DCNNTwo types of low-level features:Dense trajectory and MFCCConfigurations:Mixture self-paced function Starting values obtained by MMPRFSetting age parameter to include certain number of samples.Results on MED13Test

By far the best MAP of the 0Ex task reported on the dataset!Outperforms MMPRF on 15/20 events.Outperforms MMPRF on 15/20 events.

44Comparison of top-ranked videos

Comparison of top-ranked videos

TRECVID MED 2014Very challenging task:Search over MED14Eval Full (200K videos)The ground-truth label is unavailable.Can only submit one run. Ad-Hoc queries (events) are unknown to the system.

SPaR yields outstanding improvements for TRECVID MED14 000Ex and 010Ex condition!Take no more than 60 seconds/query on a workstation.Cost-effective Method!Web Query DatasetWeb image (353 queries over 71,478 images)Densely sampled SIFT are extracted.Parameters are tuned on a validation set.Mixture self-paced function is used.

SPaR also works for image reranking (single modality)DiscussionsTwo scenarios where SPaR fails:Initial top-ranked videos are completely off-topic.Features used in reranking are not discriminative to the queries. Sensitive to random starting values Initializing by existing reranking algorithms such as MMPRF/CPRF.Tuning the age parameter by the statistics collected from the ranked samples. as opposed to absolute values.OutlineMotivationRelated WorkJensen - Shannon TilingExperiment ResultsConclusions

50SummaryA few messages to take away from this talk:Reranking follows the self-paced learning process.SPaR is a novel and general framework with theoretical backgrounds for multimodal reranking.SPaR achieves by far the best result on the Multimedia Event Detection zero-example search.

Thank You.Q&A?

Appendix

Relation to Existing Reranking MethodsSPaRLETOR-basedMMPRFLearning to rank modelClassifierY. Liu, T. Mei, X.-S. Hua, J. Tang, X. Wu, and S. Li. Learning to video search rerank via pseudo preference feedback. In ICME, 2008.L. Jiang, T. Mitamura, S.-I. Yu, and A. G. Hauptmann. Zero-example event search using multimodal pseudo relevance feedback. In ICMR, 2014(Jiang et al. 2014)(Liu et al. 2008)Relation to Existing Reranking MethodsSPaRLETOR-basedMMPRFCPRFLearning to rank modelClassifierSingle modality(Yan et al. 2003)Y. Liu, T. Mei, X.-S. Hua, J. Tang, X. Wu, and S. Li. Learning to video search rerank via pseudo preference feedback. In ICME, 2008.L. Jiang, T. Mitamura, S.-I. Yu, and A. G. Hauptmann. Zero-example event search using multimodal pseudo relevance feedback. In ICMR, 2014R. Yan, A. G. Hauptmann, and R. Jin. Multimedia search with pseudo-relevance feedback. In CVIR, 2003.(Jiang et al. 2014)(Liu et al. 2008)SPaR for Practitioners1. Pick a self-paced functionBinary/Linear/Logarithmic/Mixture weighting .2. Pick a favorite reranking modelSVM*/Logistic Regression/Learning to Rank.3. Get reasonable starting values Initializing by existing reranking algorithms.*weighted sample LibSVM http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#weights_for_data_instances

SPaR for PractitionersIterate the following steps:Training a reranking model using the pseudo samples.Selecting pseudo positive samples and their weights by self-paced function. Selecting some pseudo negative samples randomly.Changing the model age to include more positive samples for the next iteration (setting to include certain number of examples).

Cyclic Coordinate Algorithm The propose model:

Reranking models for each modality.The weight for each sample.The pseudo label.Algorithm (Cyclic Coordinate Method):Fix , optimize

Fix optimize

Fix optimize

Change the age parameter to include more samples.

Using the existing off-the-shelf algorithm.

Enumerating binary labels.

Selecting samples and their weights for the next iterationnull2037.5502null2742.8591null3578.7811

2

3

4

...

n-1

n-2

n-3

n-4

n-5

n

+

+

Reranking Model

-

-

1

2

3

4

...

n-1

n-2

n-3

n-4

n-5

n

+

+

Reranking Model

-

-

1

2

3

4

...

n-1

n-2

n-3

n-4

n-5

n

+

+

Reranking Model

-

-

-

-

+

1

2

3

4

+

+

...

5

6

7

8

Ranked List

1

2

3

4

+

+

+

+

...

+

+

+

5

6

7

8

Ranked List

1

2

3

4

+

+

...

5

6

7

8

Ranked List

+

+

Reranking Model

-

-

-

-

+