Principled Hybrids of Generative and Discriminative Models

Julia A. LasserreUniversity of Cambridge

Cambridge, UKjal62@cam.ac.uk

Christopher M. BishopMicrosoft Research

Cambridge, UKcmbishop@microsoft.com

Thomas P. MinkaMicrosoft Research

Cambridge, UKminka@microsoft.com

Abstract

When labelled training data is plentiful, discriminativetechniques are widely used since they give excellent gen-eralization performance. However, for large-scale appli-cations such as object recognition, hand labelling of datais expensive, and there is much interest in semi-supervisedtechniques based on generative models in which the ma-jority of the training data is unlabelled. Although the gen-eralization performance of generative models can often beimproved by ‘training them discriminatively’, they can thenno longer make use of unlabelled data. In an attempt togain the benefit of both generative and discriminative ap-proaches, heuristic procedure have been proposed [2, 3]which interpolate between these two extremes by taking aconvex combination of the generative and discriminativeobjective functions.

In this paper we adopt a new perspective which says thatthere is only one correct way to train a given model, andthat a ‘discriminatively trained’ generative model is funda-mentally a new model [7]. From this viewpoint, generativeand discriminative models correspond to specific choicesfor the prior over parameters. As well as giving a prin-cipled interpretation of ‘discriminative training’, this ap-proach opens door to very general ways of interpolating be-tween generative and discriminative extremes through alter-native choices of prior. We illustrate this framework usingboth synthetic data and a practical example in the domain ofmulti-class object recognition. Our results show that, whenthe supply of labelled training data is limited, the optimumperformance corresponds to a balance between the purelygenerative and the purely discriminative.

1. Introduction

Machine learning techniques are now widely used incomputer vision. In many applications the goal is to takea vector x of input features and to assign it to one of a num-ber of alternative classes labelled by a vector c (for instance,if we have C classes, then c might be a C-dimensional bi-nary vector in which all elements are zero except the one

corresponding to the class). Throughout this paper we willhave in mind the problem of object recognition, in which xcorresponds to an image (or a region within an image) and crepresents the categories of object (or objects) present in theimage, although the techniques and conclusions presentedare much more widely applicable.

In the simplest scenario, we are given a training data setcomprising N images X = {x1, . . . ,xN} together withcorresponding labels C = {c1, . . . , cN}, in which we as-sume that the images, and their labels, are drawn indepen-dently from the same fixed distribution. Our goal is to pre-dict the class c for a new input vector x, and so we requirethe conditional distribution

p(c|x,X,C). (1)

To determine this distribution we introduce a paramet-ric model governed by a set of parameters θ. In a dis-criminative approach we define the conditional distributionp(c|x,θ), where θ are the parameters of the model. Thelikelihood function is then given by

L(θ) = p(C|X,θ) =N∏

n=1

p(cn|xn,θ). (2)

The likelihood function can be combined with a prior p(θ),to give a joint distribution

p(θ,C|X) = p(θ)L(θ) (3)

from which we can obtain the posterior distribution by nor-malizing

p(θ|X,C) =p(θ)L(θ)p(C|X)

(4)

where

p(C|X) =∫

p(θ)L(θ) dθ. (5)

Predictions for new inputs are then made by marginalizingthe predictive distribution with respect to θ weighted by theposterior distribution

p(c|x,X,C) =∫

p(c|x,θ)p(θ|X,C) dθ. (6)

In practice this marginalization, as well as the normalizationin (5), are rarely tractable and so approximation, schemessuch as variational inference, must be used. If training datais plentiful a point estimate for θ can be made by maximiz-ing the posterior distribution to give θMAP, and the predic-tive distribution then estimated using

p(c|x,X,C) � p(c|x,θMAP). (7)

Note that maximizing the posterior distribution (4) is equiv-alent to maximizing the joint distribution (3) since these dif-fer only by a multiplicative constant. In practice, we typi-cally take the logarithm before maximizing as this gives riseto both analytical and numerical simplifications. If we con-sider a prior distribution p(θ) which is constant over theregion in which the likelihood function is large, then max-imizing the posterior distribution is equivalent to maximiz-ing the likelihood. In all cases, however, the key quantityfor model training is the likelihood function L(θ). Discrim-inative methods give good predictive performance and havebeen widely used in many applications.

In recent years there has been growing interest in a com-plementary approach based on generative models, whichdefine a joint distribution p(x, c|θ) over both input vec-tors and class labels [4]. One of the motivations is that incomplex problems such as object recognition, where thereis huge variability in the range of possible input vectors, itmay be difficult or impossible to provide enough labelledtraining examples, and so there is increasing use of semi-supervised learning in which the labelled training examplesare augmented with a much larger quantity of unlabelledexamples. A discriminative model cannot make use of theunlabelled data, as we shall see, and so in this case we needto consider a generative approach.

The complementary properties of generative and dis-criminative models have led a number of authors to seekmethods which combine their strengths. In particular, therehas been much interest in ‘discriminative training’ of gen-erative models [2, 3, 12] with a view to improving classi-fication accuracy. This approach has been widely used inspeech recognition with great success [5] where generativehidden Markov models are trained by optimizing the pre-dictive conditional distribution. As we shall see later, thisform of training can lead to improved performance by com-pensating for model mis-specification, that is differencesbetween the true distribution of the process which gener-ates the data, and the distribution specified by the model.However, as we have noted, discriminative training cannottake advantage of unlabelled data. In particular, Ng et al.[8] show that logistic regression (the discriminative counter-part of a Naive Bayes generative model) works better thanits generative counterpart, but only for a large number oftraining datapoints (large depending on the complexity ofthe problem), which confirms the need for using unlabelleddata.

Recently several authors [2, 3] have proposed hybridsof the generative and discriminative approaches in which amodel is trained by optimizing a convex combination of thegenerative and discriminative log likelihood functions. Al-though the motivation for this procedure was heuristic, itwas sometimes found that the best predictive performancewas obtained for intermediate regimes in between the dis-criminative and generative limits.

In this paper we develop a novel viewpoint [7] whichsays that, for a given model, there is a unique likelihoodfunction and hence there is only one correct way to trainit. The ‘discriminative training’ of a generative model is in-stead interpreted in terms of standard training of a differentmodel, corresponding to a different choice of distribution.This removes the apparently ad-hoc choice for the trainingcriterion, so that all models are trained according to the prin-ciples of statistical inference. Furthermore, by introducinga constraint between the parameters of this model, throughthe choice of prior, the original generative model can be re-covered.

As well as giving a novel interpretation for ‘discrimina-tive training’ of generative models, this viewpoint opens thedoor to principled blending of generative and discriminativeapproaches by introducing priors having a soft constraintamongst the parameters. The strength of this constrainttherefore governs the balance between generative and dis-criminative.

In Section 2 we give a detailed discussion of the new in-terpretation of discriminative training for generative mod-els, and in Section 3 we illustrate the advantages of blend-ing between generative and discriminative viewpoints usinga synthetic example in which the role of unlabelled data andof model mis-specification becomes clear. In Section 4 weshow that this approach can be applied to a large scale prob-lem in computer vision concerned with object recognition inimages, and finally we draw some conclusions in Section 5.

2. A New View of ‘Discriminative Training’

A parametric generative model is defined by specifyingthe joint distribution p(x, c|θ) of the input vector x and theclass label c, conditioned on a set of parameters θ. Typ-ically this is done by defining a prior probability for theclasses p(c|π) along with a class-conditional density foreach class p(x|c,λ), so that

p(x, c|θ) = p(c|π)p(x|c,λ) (8)

where θ = {π,λ}. Since the data points are assumed to beindependent, the joint distribution is given by

LG(θ) = p(X,C,θ) = p(θ)N∏

n=1

p(xn, cn|θ). (9)

This can be maximized to determine the most proba-ble (MAP) value of θ. Again, since p(X,C,θ) =

p(θ|X,C)p(X,C), this is equivalent to maximizing theposterior distribution p(θ|X,C).

In order to improve the predictive performance of gen-erative models it has been proposed to use ‘discriminativetraining’ [12] which involves maximizing

LD(θ) = p(C,θ|X) = p(θ)N∏

n=1

p(cn|xn,θ) (10)

in which we are conditioning on the input vectors instead ofmodelling their distribution. Here we have used

p(c|x,θ) =p(x, c|θ)∑c′ p(x, c′|θ)

. (11)

Note that (10) is not the joint distribution for the originalmodel defined by (9), and so does not correspond to MAPfor this model. The terminology of ‘discriminative training’is therefore misleading, since for a given model there is onlyone correct way to train it. It is not the training methodwhich has changed, but the model itself.

This concept of discriminative training has been takena stage further [2, 3] by maximizing a function given by aconvex combination of (9) and (10) of the form

α lnLD(θ) + (1 − α) ln LG(θ) (12)

where 0 � α � 1, so as to interpolate between generative(α = 0) and discriminative (α = 1) approaches. Unfor-tunately, this criterion was not derived by maximizing thedistribution of a well-defined model.

Following [7] we therefore propose an alternative viewof discriminative training, which will lead to an elegantframework for blending generative and discriminative ap-proaches. Consider a model which contains an additionalindependent set of parameters θ = {π, λ} in addition tothe parameters θ = {π,λ}, in which the likelihood func-tion is given by

q(x, c|θ, θ) = p(c|x,θ)p(x|θ) (13)

wherep(x|θ) =

∑c′

p(x, c′|θ). (14)

Here p(c|x,θ) is defined by (11), while p(x, c|θ) has inde-pendent parameters θ.

The model is completed by defining a prior p(θ, θ) overthe model parameters, giving a joint distribution of the form

q(X,C,θ, θ) = p(θ, θ)N∏

n=1

p(cn|xn,θ)p(xn|θ). (15)

Now suppose we consider a special case in which theprior factorizes, so that

p(θ, θ) = p(θ)p(θ). (16)

We then determine optimal values for the parameters θ andθ in the usual way by maximizing (15), which now takesthe form[

p(θ)N∏

n=1

p(cn|xn,θ)

] [p(θ)

N∏n=1

p(xn|θ)

]. (17)

We see that the resulting value of θ will be identical to thatfound by maximizing (11), since it is the same functionwhich is being maximized. Since it is θ and not θ whichdetermines the predictive distribution p(c|x,θ) we see thatthis model is equivalent in its predictions to the ‘discrimi-natively trained’ generative model. This gives a consistentview of training in which we always maximize the joint dis-tribution, and the distinction between generative and dis-criminative training lies in the choice of model.

The relationship between the generative model and thediscriminative model is illustrated using directed graphs inFigure 1.

x

c c

xλ

π

N images N images

p q

θ = {λ , π }’ ’ ’

θ = {λ, π}

Figure 1. Probabilistic directed graphs, showing on the left, theoriginal generative model, and on the right the corresponding dis-criminative model.

Now suppose instead that we consider a prior which en-forces equality between the two sets of parameters

p(θ, θ) = p(θ)δ(θ − θ). (18)

Then we can set θ = θ in (13) from which we recover theoriginal generative model p(x, c|θ). Thus we have a sin-gle class of distributions in which the discriminative modelcorresponds to independence in the prior, and the generativemodel corresponds to an equality constraint in the prior.

2.1. Blending Generative and Discriminative

Clearly we can now blend between the generative anddiscriminative extremes by considering priors which im-pose a soft constraint between θ and θ. Why should wewish to do this?

First of all, we note that the reason why ‘discriminativetraining’ might give better results than direct use of the gen-erative model, is that (15) is more flexible than (9) sinceit relaxes the implicit constraint θ = θ. Of course, if thegenerative model were a perfect representation of reality (inother words the data really came from the model) then in-creasing the flexibility of the model would lead to poorer

results. Any improvement from the discriminative approachmust therefore be the result of a mis-match between themodel and the true distribution of the (process which gener-ates the) data. In other words, the benefit of ‘discriminativetraining’ is dependent on model mis-specification.

Conversely, the benefit of the generative approach is thatit can make use of unlabelled data to augment the labelledtraining set. Suppose we have a data set comprising a setof inputs XL for which we have corresponding labels CL,together with a set of inputs XU for which we have no la-bels. For the correctly trained generative model, the func-tion which is maximized is given by

p(θ)∏n∈L

p(xn, cn|θ)∏

m∈U

p(xm|θ) (19)

where p(x|θ) is defined by

p(x|θ) =∑c′

p(x, c′|θ). (20)

We see that the unlabelled data influences the choice of θand hence affects the predictions of the model. By contrast,for the ‘discriminatively trained’ generative model the func-tion which is now optimized is again the product of the priorand the likelihood function and so takes the form

p(θ)∏n∈L

p(xc|xn,θ) (21)

and we see that the unlabelled data plays no role. Thus,in order to make use of unlabelled data we cannot use adiscriminative approach.

Now let us consider how a combination of labelled andunlabelled data can be exploited from the perspective of ournew approach defined by (15), for which the joint distribu-tion becomes

q(XL,CL,XU,θ, θ) = p(θ, θ)[∏n∈L

p(cn|xn,θ)p(xn|θ)

] [ ∏m∈U

p(xm|θ)

](22)

We see that the unlabelled data (as well as the labelled data)influences the parameters θ which in turn influence θ viathe soft constraint imposed by the prior.

In general, if the model is not a perfect representationof reality, and if we have unlabelled data available, then wewould expect the optimal balance to lie neither at the purelygenerative extreme nor at the purely discriminative extreme.

As a simple example of a prior which interpolatessmoothly between the generative and discriminative limits,consider the class of priors of the form

p(θ, θ) ∝ p(θ)p(θ)1σ

exp{− 1

2σ2‖θ − θ‖2.

}(23)

If desired, we can relate σ to an α like parameter by defininga map from (0, 1) to (0,∞), for example using

σ(α) =(

α

1 − α

)2

. (24)

For α → 0 we have σ → 0, and we obtain a hard constraintof the form (18) which corresponds to the generative model.Conversely for α → 1 we have σ → ∞ and we obtain anindependence prior of the form (16) which corresponds tothe discriminative model.

3. Illustration

We now illustrate the new framework for blending be-tween generative and discriminative approaches using anexample based on synthetic data. This is chosen to be assimple as possible, and so involves data vectors xn whichlive in a two-dimensional Euclidean space for easy visual-ization, and which belong to one of two classes. Data fromeach class is generated from a Gaussian distribution as illus-trated in Figure 2. Here the scales on the axes are equal, and

−2 0 2 4 6 8

−1

0

1

2

3

4

5

6

7

Data distribution

Figure 2. Synthetic training data, shown as red crosses and bluedots, together with contours of probability density for each of thetwo classes. Two points from each class are labelled (indicated bycircles around the data points).

so we see that the class-conditional densities are elongatedin the horizontal direction.

We now consider a continuum of models which inter-polate between purely generative and purely discriminative.To define this model we consider the generative limit, andrepresent each class-conditional density using an isotropicGaussian distribution. Since this does not capture the hori-zontally elongated nature of the true class distributions, thisrepresents a form of model mis-specification. The parame-ters of the model are the means and variances of the Gaus-sians for each class, along with the class prior probabilities.

We consider a prior of the form (23) in which σ(α)is defined by (24). Here we choose p(θ, θ|α) =p(θ) N(θ|θ, σ(α)), where p(θ) are the usual congugate pri-ors (a gaussian prior for the means, a gamma prior for thevariances, and a dirichlet for the class priors). This resultsin a proper prior.

The training data set comprises 200 points from eachclass, of which just 2 from each class are labelled, and the

test set comprises 200 points all of which are labelled. Ex-periments are run 10 times with differing random initializa-tions and the results used to computer a mean and varianceover the test set classification, which are shown by ‘errorbars’ in Figure 3. We see that the best generalization occurs

0 0.2 0.4 0.6 0.8 140

50

60

70

80

90

100Influence of α on the classification performance

generative − 0 ≤ α ≤ 1 − discriminative

Cla

ssif

icat

on p

erfo

rman

ce in

%

Figure 3. Plot of the percentage of correctly classified points onthe test set versus α for the synthetic data problem.

for values of α intermediate between the generative and dis-criminative extremes.

To gain insight into this behaviour we can plot the con-tours of density for each class corresponding to differentvalues of α, as shown in Figure 4. We see that a purely gen-

← point 1point 2 →point 1 → ← point 2

α = 0 − generative case

← point 1point 2 →point 1 → ← point 2

α = 0.4

← point 1point 2 →point 1 → ← point 2

α = 0.6

← point 1point 2 →point 1 → ← point 2

α = 0.7

← point 1point 2 →point 1 → ← point 2

α = 0.8

← point 1point 2 →point 1 → ← point 2

α = 1 − discriminative case

Figure 4. Results of fitting an isotropic Gaussian model to the syn-thetic data for various values of α. The top left shows α = 0(generative case) while the bottom right shows α = 1 (discrimina-tive case). The yellow area corresponds to points that are assignedto the red class.

erative model is strongly influenced by modelling the den-sity of the data and so gives a decision boundary which is

orthogonal to the correct one. Conversely a purely discrim-inative model attends only to the labelled data points andso misses useful information about the horizontal elonga-tion of the true class-conditional densities which is presentin the unlabelled data.

4. Object Recognition

We now apply our approach to a realistic application in-volving object recognition in static images. This is a widelystudied problem which has been tackled using a range ofdifferent discriminative and generative models. The longterm goal of such research is to achieve near human levelsof recognition accuracy across thousands of object classesin the presence of wide variations in location, scale, ori-entation and lighting, as well as changes due to intra-classvariability and occlusion.

4.1. The data

We used 8 different classes: airplanes, bikes, cows,faces, horses, leaves, motorbikes, sheep. The cowsand sheep images come from Microsoft Research(http://www.research.microsoft.com/mlp), the airplanes,faces, leaves and motorbikes images come from the Caltechdatabase (http://www.robots.ox.ac.uk/∼vgg/data), thebikes images were downloaded from the Technical Uni-versity of Graz (http://www.emt.tugraz.at/∼pinz/data),and the horses images were downloaded fromthe Mathematical Sciences Research Institute(http://www.msri.org/people/members/eranb). Togetherthese images exhibit a wide variety of poses, colours, andillumination, as illustrated by the sample images shown inFigure 5.

Each image contains one or more objects from a particu-lar class, and the goal is to build a true multi-class classifierin which each image is assigned to one of the classes (ratherthan simply classifying each class separately versus the rest,which would be a much simpler problem).

All images were re-scaled to 300×200, and raw patchesof size 48×48 were extracted on a regular grid of size 24×24 (i.e. every 24th pixel).

4.2. The features

Our features are taken from [11], in which the originalRGB images are first converted into the CIE (L, a, b) colourspace [6]. Each image is then convolved with 17 filters,and the set of corresponding pixels from each of the filteredimages represents a 17-dimensional vector. All these fea-ture vectors are clustered using K-means with K = 100.Since this large value of K is computationally costly in laterstages of processing, PCA is used to give a 15-dimensionalfeature vector. Winn et al. [11] use a more powerful tech-nique to reduce the number of features, but since this is asupervised method based on fully labelled training data, we

Figure 5. Sample images from the training set.

did not re-implement it here. The cluster centers obtainedthrough K-means are called textons [10].

The filters are quite standard: the first three filters areobtained by scaling a Gaussian filter, and are applied to eachchannel of the colour image, which gives 3×3 = 9 responseimages. Then a Laplacian filter is applied to the L channel,at 4 different scales, which gives 4 more response images.Finally 2 DoG (difference of Gaussians) filters (one alongeach direction) are applied to the L channel, at 2 differentscales, giving another 4 responses.

From these response images, we extract every pixel on a4 × 4 grid, and apply K-means to obtain K textons. Noweach patch will be represented by a histogram of these tex-tons, i.e. by a K-dimensional vector containing the propor-tion of each texton. Textons were obtained from 25 trainingimages per class (half of the training set). Note that the tex-

ton features are found using only unlabelled data. Thesevectors are then reduced using PCA to a dimensionality of15.

4.3. The model

We consider the generative model introduced in [9],which we now briefly describe. Each image is representedby a feature vector xn, where n = 1, . . . , N , and N is thetotal number of images. Each vector comprises a set of Jpatch vectors x = {xnj} where j = 1, . . . , J . We assumethat each patch belongs to one and only one of the classes,or to a separate ‘background’ class, so that each patch canbe characterized by a binary vector τnj coded so that allelements of τnj are zero except the element correspondingto the class. We use cn to denote the image label vectorfor image n with independent components cnk ∈ {0, 1} inwhich k = 1, . . . C labels the class.

The overall joint distribution for the model can be rep-resented as a directed graph, as shown in Figure 6. We

J

xnjxnj

cncn

�

�

�

N

�nj�nj

Figure 6. The generative model for object recognition expressed asa directed acyclic graph, for unlabelled images, in which the boxesdenote ‘plates’ (i.e. independent replicated copies). Only the patchfeature vectors {xnj} are observed, corresponding to the shadednode. The image class labels cn and patch class labels τ nj arelatent variables.

can therefore characterize the model completely in terms ofthe conditional probabilities p(c), p(τ |c) and p(x|τ ). Thismodel is most easily explained generatively, that is, we de-scribe the procedure for generating a set of observed featurevectors from the model.

First we choose the overall class of the image accord-ing to some prior probability parameters ψk where k =1, . . . , C, and 0 � ψk � 1, with

∑k ψk = 1, so that

p(c) =C∏

k=1

ψck

k . (25)

Given the overall class for the image, each patch is thendrawn from either one of the foreground classes or the back-ground (k = C + 1) class. The probability of generating apatch from a particular class is governed by a set of param-eters πk, one for each class, such that πk � 0, constrainedby the subset of classes actually present in the image. Thus

p(τ j |c) =

(C+1∑l=1

clπl

)−1 C+1∏k=1

(ckπk)τjk . (26)

Note that there is an overall undetermined scale to theseparameters, which may be removed by fixing one of them,e.g. πC+1 = 1.

For each class, the distribution of the patch feature vectorx is governed by a separate mixture of Gaussians which wedenote by

p(x|τ j) =C+1∏k=1

φk(xj ;λk)τjk (27)

where λk denotes the set of parameters (means, covari-ances and mixing coefficients) associated with this mixturemodel.

If we assume N independent images, and for image nwe have J patches drawn independently, then the joint dis-tribution of all random variables is

N∏n=1

p(cn)J∏

j=1

p(xnj |τnj)p(τnj |cn)

. (28)

Here we are assuming that each image has the same num-ber J of patches, though this restriction is easily relaxed ifrequired.

The graph shown in Figure 6 corresponds to unlabelledimages in which only the feature vectors {xnj} are ob-served, with both the image category and the classes of eachof the patches being latent variables. It is also possible toconsider images which are ‘weakly labelled’, that is eachimage is labelled according to the category of object presentin the image. This corresponds to the graphical model ofFigure 7 in which the node cn is shaded. Of course, for agiven size of data set, better performance is expected if allof the images are ‘strongly labelled’, that is segmented im-ages in which the region occupied by the object or objects isknown so that the patch labels τnj become observed vari-ables. The graphical model for a set of strongly labelledimages is also shown in Figure 7. Strong labelling requireshand segmentation of images, and so is a time consumingand expensive process as compared with collection of theimages themselves. For a given level of effort it will alwaysbe possible to collect many unlabelled or weakly labelledimages for the same cost as a single strongly labelled im-age. Since the variability of natural images and objects isso vast we will always be operating in a regime in which

J

xnjxnj

cncn

�

�

�

N

�nj�nj

J

xnjxnj

cncn

�

�

�

N

�nj�nj

Figure 7. Graphical models corresponding to Figure 6 for weaklylabelled images (left) and strongly labelled images (right).

the size of our data sets is statistically small (though theywill often be computationally large).

For this reason there is great interest in augmenting ex-pensive strongly labelled images with lots of cheap weaklylabelled or unlabelled images in order to better character-ize the different forms of variability. Although the twostage hierarchical model shown in Figure 6 appears tobe more complicated than in the simple example shownin Figure 1, it does in fact fall within the same frame-work. In particular, for labelled images the observed datais {xn, cn, τnj}, while for ‘unlabelled’ images only {xn}are observed. The experiments described here could readilybe extended to consider arbitrary combinations of stronglylabelled, weakly labelled and unlabelled images if desired.

If we let θ = {ψk, πk,λk} denote the full set of parame-ters in the model, then we can consider a model of the form(22) in which the prior is given by (23) with σ(α) definedby (24), and the terms p(θ) and p(θ) taken to be constant.

We use conjugate gradients to optimize the parameters.Due to lack of space we do not write down all the deriva-tives of the log likelihood function required by the conju-gate gradient algorithm. However, the correctness of themathematical derivation of these gradients, as well as theirnumerical implementation, can easily be verified by com-parison against numerical differentiation [1]. The conju-gate gradients is the most used technique when it comesto blending generative and discriminative models, thanks toits flexibility. Indeed, because of the discriminative com-ponent p(cn|xn,θ) which contains a normalising factor, analgorithm such as EM would require much more work, asnothing is directly tractable anymore. However, a compari-son of the two methods is currently being investigated.

4.4. Results

We use 50 training images per class (giving 400 trainingimages in total) of which 5 images per class (a total of 40)were fully labelled i.e. both the image and the individualpatches have class labels. All the other images are left to-tally unlabelled, i.e. not even the category they belong to isgiven. Note that this kind of training data is 1) very cheap

to get and 2) very unusual for a discriminative model. Thetest set consists of 100 images per class (giving a total of800 images), the task is to label each image.

Experiments are run 5 times with differing random ini-tializations and the results used to compute a mean and vari-ance over the test set classification, which are shown by‘error bars’ in Figure 8. Note that, since there are 8 bal-

0 0.2 0.4 0.6 0.8 1

30

35

40

Influence of α on the classification performance

generative − 0 ≤ α ≤ 1 − discriminative

Cla

ssif

icat

on p

erfo

rman

ce in

%

Figure 8. Influence of the term α on the test set classification per-formance.

anced classes, random guessing would give 12.5% correcton average. Again we see that the best performance is ob-tained with a blend between generative and discriminativeextremes.

5. Conclusions

In this paper we have shown that ‘discriminative train-ing’ for generative models can be re-cast in terms of stan-dard training methods applied to a modified model. Thisnew viewpoint opens the door to a wide range of new mod-els which interpolate smoothly between generative and dis-criminative approaches and which can benefit from the ad-vantages of both. The main drawback of this framework isthat the number of parameters in the model is doubled lead-ing to greater computational cost.

Although we have focussed on a specific applicationin computer vision concerned with object recognition, thetechniques proposed here have very wide applicability.

A principled approach to combining generative and dis-criminative approaches not only gives a more satisfyingfoundation for the development of new models, but it alsobrings practical benefits. In particular, the parameter αwhich governs the trade-off between generative and dis-criminative is now a hyper-parameter within a well definedprobabilistic model which is trained using the (unique) cor-rect likelihood function. In a Bayesian setting the value ofthis hyper-parameter can therefore be optimized by maxi-mizing the marginal likelihood in which the model parame-ters have been integrated out, thereby allowing the optimal

trade-off between generative and discriminative limits to bedetermined entirely from the training data without recourseto cross-validation [1]. This extension of the work describedhere is currently being investigated.

References

[1] C. M. Bishop. Neural Networks for Pattern Recognition. Ox-ford University Press, 1995. 7, 8

[2] G. Bouchard and B. Triggs. The trade-off between gener-ative and discriminative classifiers. In IASC 16th Interna-tional Symposium on Computational Statistics, pages 721–728, Prague, Czech Republic, august 2004. 1, 2, 3

[3] A. Holub and P. Perona. A discriminative framework formodelling object classes. In IEEE Conference on Com-puter Vision and Pattern Recognition, San Diego (Califor-nia), USA, june 2005. IEEE Computer Society. 1, 2, 3

[4] T. Jebara. Machine Learning: Discriminative and Genera-tive. Kluwer, 2004. 2

[5] S. Kapadia. Discriminative Training of Hidden Markov Mod-els. Phd, University of Cambridge, Cambridge, U.K., March1998. 2

[6] J. M. Kasson and W. Plouffe. An analysis of selectedcomputer inter-change color spaces. ACM Transactions onGraphics, 11:373–405, 1992. 5

[7] T. Minka. Discriminative models, not discriminative train-ing. Technical report, Microsoft Research, Cambridge, UK,2005. 1, 2, 3

[8] A. Y. Ng and M. I. Jordan. On discriminative vs generativeclassifiers: A comparison of logistic regression and naivebayes. In Neural Information Processing Systems, pages841–848, Vancouver, Canada, december 2001. MIT Press.2

[9] I. Ulusoy and C. M. Bishop. Generative versus discrimi-native models for object recognition. In Proceedings IEEEInternational Conference on Computer Vision and PatternRecognition, CVPR., San Diego, 2005. 6

[10] M. Varma and A. Zisserman. A statistical approach to textureclassification from single images. In IJCV, volume 62, pages61–81, 2005. 6

[11] J. Winn, A. Criminisi, and T. Minka. Object categorizationby learned universal visual dictionary. In IEEE InternationalConference on Computer Vision, Beijing, China, 2005. IEEEComputer Society. 5

[12] O. Yakhnenko, A. Silvescu, and V. Honavar. Discrimina-tively trained markov model for sequence classification. In5th IEEE International Conference on Data Mining, Hous-ton (Texas), USA, november 2005. IEEE Computer Society.2, 3