atex class files, vol. 14, no. 8, august 2015 1 general...

1520-9210 (c) 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMM.2017.2724843, IEEETransactions on Multimedia

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1

General Knowledge Embedded ImageRepresentation Learning

Peng Cui, Member, IEEE, Shaowei Liu and Wenwu Zhu, Fellow, IEEE

Abstract—Image representation learning is a fundamental problem in understanding semantics of images. However, traditionalclassification-based representation learning methods face the noisy and incomplete problem of the supervisory labels. In this paper, wepropose a General Knowledge Base Embedded Image Representation Learning approach, which uses general knowledge graph,which is a multi-type relational knowledge graph consisting of human commonsense beyond image space, as external semanticresource to capture the relations of concepts in image representation learning. A Relational Regularized Regression CNN (R3CNN)model is designed to jointly optimize the image representation learning problem and knowledge graph embedding problem. In thismanner, the learnt representation can capture not only labeled tags but also related concepts of images, which involves more preciseand complete semantics. Comprehensive experiments are conducted to investigate the effectiveness and transferability of ourapproach in tag prediction task, zero-shot tag inference task, and content based image retrieval task. The experimental resultsdemonstrate that the proposed approach performs significantly better than the existing representation learning methods. Finally,observation of the learnt relations show that our approach can somehow refine the knowledge base to describe images and label theimages with structured tags.

Index Terms—image representation learning, knowledge base, multi-relational graph embedding

F

1 INTRODUCTIONImage representation is a fundamental problem in variousimage applications. In recent years, we have witnessed afast advancement of image representation learning fromhand-crafted features to learning image representation fromscratch through deep neural network models. As deep mod-els are trained in an end-to-end fashion, the learned imagerepresentations can perform much better than hand-craftedfeatures in the target applications, as long as the trainingdata is of sufficient quality and quantity. The side effect,however, is that the goodness of the learned representationsheavily depends on the input training data, especially thesupervisory information of the target application. If the goalis to learn image representations to bridge the semantic gap,the completeness, preciseness and richness of the semanticlabels of images are decisive to the capability of the learnedimage representations to bridge the pixel data and seman-tics.

In existing image representation learning works, expertannotated semantic labels (e.g. ImageNet) are commonlyused as the supervisory information. But the completenessand the quantity of these expert labels cannot be guaranteeddue to expensive and limited human labors. Recently, apaucity of works start to exploit social tags (e.g. in Flickr)as the supervisory information. Although the problem ofcompleteness is alleviated to some extent, the noisy andimprecise tags seriously affect the quality of the learnedrepresentations. In order to tradeoff the completeness andpreciseness, [7], [17] consider to exploit the “isA” relations

• P. Cui, S. Liu and W. Zhu are with the Department of Computer Science,Tsinghua University, Beijing, China.E-mail: [email protected]

Manuscript received April 19, 2005; revised August 26, 2015.

in image knowledge graph ImageNet to extend the labelsof images. However, very small improvement was achievedover the methods without extending labels. One plausiblereason is that the richness of ImageNet is quite limited andthus cannot bring much additional information. Meanwhile,most images in ImageNet are with high-quality and singlesalient object, which lead the learned image representationshard to apply in wild image applications. How to learnimage representations from data guided by supervisoryinformation with satisfactory completeness, preciseness andrichness, is still an open problem.

In this paper, we explore the possibility of embeddinggeneral knowledge graph into the representation learning ofwild images with multiple tags. Here the general knowledgegraph is referred to the multi-type relational knowledgebases consisting of concept-level common-senses definedbeyond image space. Let us take ConceptNet [23], a generalcommonsense knowledge base, as a representative. Thereare 3.4 million concepts in English in total, and 56 kindsof relations are included, such as “IsA”, “AtLocation”,“PartOf”. By properly embedding these relational knowl-edge into image representations, we can get two notablebenefits. First, the traditional supervisory information, liketags, can be extended, filtered and relationally structured,and thus can address the problem of completeness, precise-ness and richness in supervisory information. Additionally,through end-to-end training in deep models, the learnedimage representations can well support relational reasoningif the supervisory information consists of relational tags.That is, we can predict precise tags for an image withrelational structure. In contrast with the traditional flat-structured tags, the relational structured tags have largerpotential in deeply understanding images and inferring userintents in different semantic levels or aspects.




However, we face the following challenges in embed-ding multi-type relational knowledge into image represen-tations:

(1) The semantic spaces of images and the conceptsin knowledge graph are not consistent. We cannot simplylearn the image representations and concept embeddingsindependently and then bridge them together because itis very difficult to make the semantic spaces of imagesconsistent to the concepts in knowledge graph. First, thereare a lot of relations in knowledge graph that cannot be usedto describe images, thus they are not helpful in knowledgeembedding. Second, if the existed image representation haslost some semantic information, bringing it to knowledgeembedding will confuse the model. Thus, we need to jointlyoptimize image representation learning problem and knowl-edge graph embedding problem in a framework.

(2) The multiple relations are not equally effective inimage representation learning. Different from traditionalsemantic hierarchy, there are multiple types of relations ina knowledge graph. However, they are not equally effectiveto describe images. For example, the relations “CausedBy”and “MotivatedByGoal” can hardly be applied to imagetagging. Thus, we cannot simply conduct image annotationand then extend related tags. To solve this problem, we needto consider the relations and their corresponding visualcontents at the same time.

(3) Both symmetric and asymmetric relations need to beincorporated. Knowledge graph is a directed graph, whichincludes asymmetric relations, such as “IsA”, “MadeOf”,and symmetric relations, such as “RelatedTo” and “Syn-onym”. We need to consider the types of relations andincorporate all of them.

In order to address the above challenges, we propose anew framework to embed the multiple relational multi-typerelational knowledge into image representations, as shownin Figure 1. In our approach, we design a Relational Regular-ized Regression Convolutional Neural Network (R3CNN),which jointly optimize the multi-label image representa-tion learning problem and knowledge graph embeddingproblem. In image representation learning task, a regressionCNN is adopted to learn image representation from image-tag data. In knowledge graph embedding task, we map aconcept in knowledge graph into multiple hidden spacesfor different types of relations. The mapping functions of leftconcept and right concept in a relation are different, whichmakes the distance of two concepts asymmetric. Therefore,both of visual contents of images and relations in knowledgegraph are involved in the model through joint optimization.The learnt representation can capture the information ofnot only labeled tags, but also relations of concepts inknowledge graph. In addition, for zero-shot tags unseen intraining data, we can easily infer their representations basedon the existed tag embeddings in knowledge graph, andthen predict its relevance to the images.

It is worthy to highlight the contributions of the pro-posed approach as follows:

(1) In order to address the problem of completeness, pre-ciseness and richness of supervisory information in imagerepresentation learning, we propose a new framework toembed knowledge graph into image representations.

(2) We propose a novel R3CNN model to jointly optimizethe multi-label image representation learning problem andknowledge graph embedding problem in a deep model,where both symmetric and asymmetric relations are incor-porated.

(3) Extensive experiments are conducted to demonstratethe effectiveness of the proposed method in tag prediction,zero-shot tag inference, and content-based image retrievaltasks. The results demonstrate that our method significantlyoutperforms state-of-the-art image representation learningmethods.

The rest of the paper is organized as follows: Section2 gives a brief comparison of related works. Section 3introduces some statics and observations of data and definesour problem. In Section 4, we present the proposed R3CNNmodel and introduce the applications of our approach.Then, the experimental results are reported to demonstratethe effectiveness of our approach in Section 5. Finally, Sec-tion 6 summarizes the paper.

2 RELATED WORK2.1 Deep Model based Image Representation LearningTo bridge the gap between low-level features and high-level semantics, deep model based image representationlearning methods show their superiority to traditional hand-crafted features. In these methods, Convolutional NeuralNetwork (CNN) [15], [22] is widely used and proved to bevery effective. These works usually use image classificationdataset with single label(such as ImageNet). To date, thisproblem is well solved and its top-5 error is reduced toless than 5% [11]. However, CNN based representationlearning methods rely on the supervisory information inthe dataset very much. The completeness and richness ofsupervisory information determines the effectiveness of thelearned representations. The concept of the salient object isnot enough to cover the all the semantic information in animage due to the lack of motion, attributes, background, etc.In recent years, some researchers explore to learn image rep-resentation based on the corresponding textual information(e.g. tags and sentences) because such textual informationcan cover more semantics in an image. Some of these workstreat each tag as an independent classifier, thus convertthis problem into a multi-label classification problem [1],[8], [13], [32]. Also, [25], [30] [26] use multimodal methodsto bridge the visual modality and text modality, and [28]proposes a novel deep relative attributes (DRA) algorithmto learn visual features. In order to alleviate the intentiongap problem in image applications, [19] proposes an Asym-metric Multi-task CNN model to embed social signals intoimage representations. However, all the above solutions facethe the noisy and incomplete problems of the wild textualdata. Intuitively, only using image-text data cannot solvethis problem because such wild data cannot provide enoughclues to filter the noisy tags and complete the missing tags.

2.2 Image Understanding with Extra Semantic Re-sourcesImage annotation with noisy tags is a well-studied problem.Traditional methods usually use nearest-neighbor-based ap-proaches [2] [9], [18], where the semantic meaning of the




pumpkin

squirrel

cat

dessert

walk

Images with tags

General knowledge base

Relational Regularized Regression CNN

image

representation

lion

squirrel

nature

fruit

colorful …

Inference of Tags and

Relations

Relation 1

Multi-label image

representation learning

Multiple

projection

matrices

. . .

tag representation

Knowledge graph embedding

Relation 2 Relation 3

tags

Fig. 1. The framework of general knowledge embedded image representation learning.

words is ignored. In recent years, researchers explore toconsider the relation of labels with extra semantic resourcesto capture the relation labels. Visual-semantic embeddingmodels [7], [17] to embed visual ontology ImageNet intoimage classification task. It considers the ontology of the1000 classes in ImageNet to make the results more seman-tically reasonable and can be used for zero-shot prediction(predict unseen categories). However, ImageNet ontologycannot include all the semantic relations of the concepts,such as property and location. To import general worldknowledge into image understanding, Xie et al. uses thegeneral knowledge base ConceptNet as semantic resourcesto learn tags relations for image tagging [27]. However, inthis work, the tag relations is evaluated by simple similarityin a common space, which cannot reflect the property ofmulti-relation and single-direction in knowledge base. Im etal. uses DBPedia [4] to extend the annotation results basedon the existed tags to improve recall [12]. [31] constructs asemantic-visual knowledge base to encode the rich event-centric concepts to enhance video event recognition. [20]proposes a cross-domain learning method to classify webmultimedia objects by transferring the correlation knowl-edge among different information sources. However, forvision tasks, there are a lot of useless relations in knowledgebase. Besides, there are a lot of ambiguous tags. Thus, tagextension should consider the visual content, which is betterto be conducted at representation learning stage rather thantagging stage. More recently, knowledge graph embeddingand their combination with visual information arouse someresearch interest [5] [10]. [29] borrows the idea of relationrepresentation in knowledge graph and propose a novelvisual translation network for visual relation detection. Thispaper differs in that we exploit the semantic relationships(e.g. UsedFor, IsA) among labels for representation learningand tagging, while these works exploit the visual relation-ships (e.g. ride-on, walk-to) among object labels for imagecontent understanding.

3 PRELIMINARY STUDY

In this section, we preliminarily demonstrate the feasibilityof exploiting general knowledge graph in image represen-tation learning by data statistics, and then give the formalproblem definition.

3.1 Statistics and Observation

In this work, we use two data resources, a visual resourceNUSWIDE and a semantic resource ConceptNet. NUSWIDEincludes 269,648 images crawled in Flickr 1 with 425,001 tagslabeled by users. ConceptNet is a knowledge graph consist-ing of general common sense knowledge in multi-language.There are 3.4 million concepts ,56 kinds of relations and11 millon relations in total. We conduct statistics on theabove resources to answer the following two questions: 1)Whether these two resources can be bridged? 2) Can therelations in ConceptNet help to understand the tag relationsin describing images?

To answer the first question, we give statistics on howmany shared tags2 occur in the two resources. In the 425, 001tags in NUSWIDE, there are 92, 595 tags also occur in Con-ceptNet. Figure 2(a) illustrates the distribution of number oftags versus the occurrence frequency in NUSWIDE amongall tags in NUSWIDE and the shared tags of NUSWIDEand ConceptNet. We can observe that the higher occurrencefrequency of tags, the higher possibility of sharing betweenNUSWIDE and ConceptNet. For the tags that occur morethan 50 times in NUSWIDE, more than 50% of them alsooccur in ConceptNet. Thus, ConceptNet can encompassmost of the common tags in NUSWIDE, and the shared tagscan play the role of a bridge between these two resources.

For the second question, we use the most typical tagrelation in NUSWIDE, i.e., co-occurrence relation to observe

1. www.flickr.com2. A word is called tag in NUSWIDE, and concept in ConceptNet. In

this paper, we denote a shared entity as tag or label without confusion.




20 40 60 800.0

5.0

10.0

15.0nu

mbe

r(*10

3 )

frequency

tags in NUSWIDE overlapped tags

(a)

269124

893383

235124 89

54 46 41 31

260

0 50 100100

101

102

103

104

105

106

numbe

r(log

)

co-occurrence frequency

(b)

Fig. 2. Statistics of overlapped tags and relations in NUSWIDE andConceptNet.

whether it is enough to describe tag relations without Con-ceptNet. Due to the fact that ConceptNet has multiple typesof relations, we use the relation types “IsA”, “AtLocation”,and “InstanceOf” as representation, which are very usefulfor image tagging in our observation. For any two over-lapped tags, if they have at least one of the above relationsin ConcepNet, we conduct count their co-occur frequency inNUSWIDE. The statistical histogram is illustrated in Figure2(b). We can observe that most of the relational pairs do notoccur in ConceptNet. It indicates that co-occurrence relationcannot include the above three relations in ConceptNet.Besides, There are 33, 401 tag pairs occur more than 100times in NUSWIDE. While only 3, 412 of them occur inConceptNet. It indicates that the relations in ConceptNetand co-occurrence in NUSWIDE are complementary.

Based on the above statistics and observations, we canfind that bridging NUSWIDE and ConceptNet together andincorporating the relations of these resources can help usbetter understand image semantics.

3.2 Notation and Problem DefinationOur target is to jointly learn image representation xi forimage i and tag representation wj for tag j, which cansatisfy the requirements of two tasks: multi-label imagerepresentation learning and knowledge graph embedding.Definition 1. (Image Representation Learning) In image

representation task, out goal is to learn xi, wi to makeyij = f(xi, wj) for a given function f(·), where yij is theground truth that denote whether tag j belongs to imagei.

Definition 2. (General Knowledge Graph) A general knowl-edge graph is a multi-type relational graph whose ver-tices are concepts (one or several words), and edges arethe types of relations between two tags. A relation isrepresented as a triplet < i, j, p >, which means concepti and concept j has relation type p.

To be noted that if the relation type p is symmetric suchas “RelatedTo”, both of < i, j, p > and < j, i, p > willoccur in our knowledge graph. Thus, we can simply regardthe knowledge graph as a directed graph can consider theasymmetric relations.Definition 3. (Knowledge Graph embedding) Let dpij =

gp(wi, wj) denote the distance of concept i and j in thespace p. Knowledge graph embedding task is to find the

optimal concept embedding wi for tag i, which makesthe distance dpij consistent to the relations knowledgegraph, i.e., dpij is small when < i, j, p > exists in theknowledge graph, and vise versa.

In image representation learning and knowledge graphembedding, tag representation wi is a common variable,which bridges these two tasks together for joint optimiza-tion.

4 RELATIONAL REGULARIZED REGRESSIONCNNIn this section, we introduce the proposed model R3CNN,where the image representation learning and knowledgegraph embedding are jointly optimized.

4.1 Multi-label Image Representation Learning

Most existing CNN models are designed for single-labelclassification task, which cannot be directly applied to multi-label image representation learning problem. Let us takeAlexNet [15] as an example. In AlexNet, the activationfunction for fc7 (the last fully connection) layer is softmax.Softmax is a function considering multiple tags as a jointprobabilistic distribution, which makes different tags mutu-ally exclusive. As wild images are often labeled with noisyand incomplete tags, we can not know the real distributionof multiple labels. Using the observed noisy tags insteadwith softmax function will lead the learned distribution tohave a large discrepancy with the real distribution. There-fore, the softmax function cannot work well in multi-labelimage representation problem. To solve this problem, wepropose to use sigmoid function as in [16] to replace softmaxfunction, and regard each tag to be independent. In this way,the influence of noisy and incomplete tags will be reducedin the objective function by reducing the weight of negativesamples. In addition, the original loss function in AlexNetis based on cross entropy, which is used to measure thedistance of two probabilistic distribution. In our case, asthe output is not a joint distribution of multi-labels butindependent probability for each label, we use L2 loss asour loss function. In this model, the structure of the first 10layers is the same with AlexNet, including 5 convolutionlayers, 3 pooling layers, and 2 fully connection layers. Then,we use sigmoid function as the activation function of fc7layer and adopt Euclidean loss (L2 loss) instead of crossentropy . Here the loss function is:

L1 =N∑i=1

(∑yij=1

1

2(yij − tij)

2 + α∑yij=0

1

2(yij − tij)

2), (1)

where N is the number of images, tij is the ground truthof tags where yij = 1 denotes images i contains tag j, andα is the parameter which balances the positive samples andnegative samples. y is the output of the fc7 layer, which iscalculated by:

yi = σ(Wx(7)i + b), (2)

where x(7)i is the input of fc7 layer, W and b are the pa-

rameters, and σ is the sigmoid activation function σ(xi) =




11+e−xi

. By replacing [x(7)i , 1] by new xi and [W, b] by new

W , Equation 2 can be written as:

yi = σ(Wxi) (3)

In the rest of this paper, we use this formulation instead ofEquation 2.

In Equation 3, each tag can be regarded as an indepen-dent classifier. Thus, for a single tag j, Equation 2 can bewritten as:

yij = σ(wj · xi), (4)

where wj is the j-th row of the weight matrix W . Then, xi isthe representation of image i and wj is the representation oftag j. Thus, the final output yij is determined by the innerproduct of the representations of image i and tag j.

4.2 General Knowledge Graph Embedding

Graph embedding is to represent nodes in a hidden vectorspace and maintain the graph edges by distance metricin vector space. As general knowledge graphs consists ofmultiple relation types, and these relation types have het-erogeneous characteristics, we cannot use one vector space,as usual, to reflect these multiple relation types. Also, thereare both symmetric and asymmetric relations in knowledgegraphs. Traditional similarity metric is inadequate to main-tain direct relations.

In this paper, we use multiple hidden spaces to maintainmultiple types of relations. To address the problem of asym-metric relations, we generate two mappings from originalconcept representations to the hidden space p: Lp for theleft concept, and Rp for the right concept. Then, we calculatethe distance of concept wi and wj in hidden space p by theEuclidean distance:

dpij = (Lpwi −Rpwj)T (Lpwi −Rpwj), (5)

where Lp and Rp are k × m projection matrices (m is thedimension of concept representation and k is the dimensionof hidden space). Then, the target of knowledge base em-bedding learning is to learn mappings Lp and Rp for eachrelation p, which make dpij small if concepts i and j haverelation p in knowledge graph, and vise versa. In order toreduce down the dimensionality of the embedding space,we impose orthogonal constraints on Lp and Rp as in [25]which demonstrates that imposing orthogonal constraint onthe mapping functions can result in orthogonal embeddingrepresentations. In this way, the redundancy of differentembedding dimensions can be largely reduced, leading tofewer required embedding dimensions. Therefore, the lossfunction is defined as:

L2 = − 1

|E|∑

eij∈Ep

(dpij)2 +

γ

N2 − |E|∑

eij /∈Ep

(dpij)2

+ λ∑p

(||LTp Lp − I||2F + ||RT

p Rp − I||2F ),(6)

where |E| is the number of edges in knowledge graph, Nis the number of vertices (tags) in the knowledge graph,eij ∈ Ep means the edge from i to j is the pth type ofrelation.

4.3 Joint OptimizationIn order to jointly optimize image representation learningand knowledge graph embedding, we combine the lossfunction in Regression CNN (Equation 1) and the loss forknowledge base (Equation 6) to formulate the final lossfunction of the proposed R3CNN:

L = L1 + βL2, (7)

where β is the trade-off parameter to balance image taggingand the constraints of knowledge base embedding. By usingthe tag representation wi as bridge for joint optimization, thelearnt image representation can capture more precise andcomplete semantics, and the tag representation can capturenot only semantic relations but also visual similarities.

In the loss function 7, there are two groups of variables:1) The parameters in Regularized Regression CNN (weightsand biases of each layer) 2) The projection matrices Lp

and Rp. The parameters in Regression CNN are usuallyoptimized using back propagation with stochastic gradientdescent (SGD). Lp and Rp can be optimized by gradientdescent. Therefore, we can iteratively optimize these twogroups of parameters. First, in Regression CNN, the weightsW of output layer is special because it occurs in both twoterms in the loss function. Its gradient is calculated as:

∂L∂wi

=

N∑j=1

yji(1− σ(wi · x(7)j )) + α(1− yji)σ(wi · x(7)

j )

− 2β

|E|∑

eij∈Ep

LTp (Lpwi −Rpwj)

+2βγ

N2 − |E|∑

eij /∈Ep

LTp (Lpwi −Rpwj).

(8)

The other parameters in Regression CNN can be easilycalculated by back propagation. After the parameters inRegression CNN are converged, we optimize Lp and Rp

using gradient descent:

∂L∂Lp

= − 2β

|E|∑p

∑eij∈Ep

(Lpwi −Rpwj)wTi

+2βγ

N2 − |E|∑p

∑eij /∈Ep

(Lpwi −Rpwj)wTi

+ 4βλ∑p

Lp(LTp Lp − I),

(9)

similarly,

∂L∂Rp

= − 2β

|E|∑p

∑eij∈Ep

(Rpwj − Lpwi)wTj

+2βγ

N2 − |E|∑p

∑eij /∈Ep

(Rpwj − Lpwi)wTj

+ 4βλ∑p

Rp(RTp Rp − I).

(10)

We alternately optimize all these parameters until they areconverged. To be noted that when updating W , the allof the parameters in CNN are also updated through backpropagation.




4.4 Algorithm and Complexity

We summarize the algorithm general knowledge embeddedimage representation learning as described in Algorithm 1.

Algorithm 1: General Knowledge embedded ImageRepresentation Learning

Require: Image-tag data with visual contents;Knowledge base data in triplet form.

Ensure: The R3CNN model that includes tag presentationsand can be used to extract image representations.

1: Initialize the weights in R3CNN using the trainedAlexNet model in ImageNet.

2: Fine tune R3CNN using image-tag data, obtain theweights of the last layer W .

3: Initial Lp and Rp with random value.4: repeat5: Use gradient descent to find optimal Lp and Rp w.r.t.

loss function (Equation 6) and gradients (Equation 9and 10).

6: Fine tune the Regularized Regression CNN based onthe loss function (Equation 7).

7: until All variables converge.

The complexity of optimizing Lp and Rp in an iterationin total is O(N2pkm), where N is the number of tags,p is the number of relations types, k is the dimension ofhidden space and m is the dimension of tag representation.The complexity of optimizing W for each epoch (a wholeiteration in CNN) is O(n(pm2+N2)), where n is the numberof training samples. The complexity of optimizing the wholeRegression CNN is difficult to analysis. But in our problempm2+N2 is much smaller than the whole scale of the param-eters in CNN. Thus, optimizing the proposed RegularizedRegression CNN costs comparable time to AlexNet (lessthan 2 times in practice). Besides, the time cost in optimizingLp and Rp is much less than tuning the CNN. Overall, thetime cost of our method is linear to the time cost of tuningan AlexNet, which is acceptable for most GPU computingenvironments.

4.5 Applications

The learned image representations can be used in the fol-lowing application scenarios:

(1) Tag prediction. In our model, we can obtain therepresentation of each tag and for any image, we can extractits features using the activation of fc7 layer in our R3CNNmodel. Thus, we can use Equation 4 to predict the probabil-ity of a tag belonging to an image.

(2) Zero-shot tag inference. Tags of images alwaysevolve over time. It is impossible to cover all of the tagsin a given learning framework. For new tags, it takes a lotof time to learn new classifiers. It is important to investigatethe problem of inferring unseen tags without training forimages, that is zero-shot tagging problem. In our method,we can address this problem by exploiting knowledge graphto infer the representation of the new tags if these tagshave relations to the trained tags for images, that is. Fora new tag wi, suppose that it has triplet < i, j, p > in the

knowledge graph and wj has been trained in our method.From Equation 6, we know that a converged solutionshould have Lpwi ≈ Rpwj . Besides, the regularizer requiresthat:LT

p Lp ≈ I . Therefore, we have wi ≈ LTp Rpwj . When

wi occurs on the right of a relation, the result is similar. Byaveraging all the relations, we have:

wi ≈1

Zi(∑

eij∈Ep

LTp Rpwj +

∑eji∈Ep

RTp Lpwj), (11)

where Zi is the total number of edges that are related to i fornormalization. After we calculate wi by Equation 11, we caninfer that whether image i should have tag j by Equation4. This procedure can be calculated very fast without anytraining process.

(3) Content based image retrieval. Although we usethe image-tag as supervisory information to learn imagerepresentations, we argue that the learnt representationscan also be applied into other image applications, such astypical content based image retrieval. We can extract imagefeatures using the proposed R3CNN model. Due to the factthat the higher layers are more close to semantics, we usethe activation of fc6 or fc7 layer as image features. Then,nearest neighbor based methods can be used for contentbased image retrieval task.

5 EXPERIMENTS5.1 Experimental Settings

We evaluate our approach in three applications, includingtag prediction, zero-shot tag inference and CBIR. Due to thefact that our approach is a general representation learningmethod to capture image semantics, we select the state-of-the-art representation learning approaches with or withoutextra semantic resources as baselines.

5.1.1 DatasetsThere are two datasets in our experiments. One isNUSWIDE [6] for image tagging and the other is Concept-Net [23] for knowledge base embedding.

NUSWIDE contains 269, 648 images with 425, 001 tags,which are collected from Flickr. ConceptNet is a generalknowledge base with multiple languages. In our experi-ments, we only select concepts from ConceptNet53 in En-glish language, which has millions of concepts and 56 typesof relations. In NUSWIDE and ConceptNet, there are 92, 595common tags (including words and phrases). To bridgethem together, we sampled 1, 000 overlapped tags withthe highest frequency. And then, we select the images inNUSWIDE, which has at least three selected tags. Afterfiltering the images that cannot be crawled from Flickr atpresent, there are 86, 035 images left. Referring to [27], wefirst filter some relations that are intuitively not recognizablein images (e.g. Causedby, HasSubevent) and some negativerelations (e.g. NotIsA, Antonym). Finally, there are 15 typesof relations selected, including IsA, HasA, RelatedTo, UsedFor,AtLocation,DefinedAs, InstanceOf, PartOf, HasProperty, Capa-bleOf,SymbolOf, LocatedNear, ReceivesAction, MadeOf and Syn-onym. Compared to [27], we add Synonym because we found

3. http://conceptnet5.media.mit.edu




it is positive in our experiments. There are 27, 440 edgesin these 15 types among the 1000 selected concepts. Insummary, the scale of our dataset is shown in Table 1. Inour experiments, we randomly sampled 50, 000 images fortraining and rest 36, 035 images for testing.

TABLE 1The scale of the dataset.

tags images relation types relationsnumber 1000 86,035 15 27,440

5.1.2 Model ImplementationWe train the R3CNN using the training dataset followingalgorithm in Section 4.4. In training data, each image isresized to 224 × 224 with horizontal flipping. FollowingAlexNet [15], dropout with probability 0.5 is used in fc6and fc7 layers in training process. When fine-tuning theR3CNN, we fix the weights of first 3 convolutional layersby setting the learning rate to be 0; set learning rate of thelast regression layer to be 0.1; and the learning rate of otherlayers to be 0.01. The learning rates are decreased by 0.1times after 10, 000 iterations. When fine-tuning the Regu-larized Regression CNN with knowledge base embeddings,we also fix the first 3 convolutional layers and set the otherlearning rates to be 0.01. Our model is implemented bymatlab toolbox MatConvnet [24]. It is trained on a singleGeForce Tesla K40 GPU with 12GB memory.

In the loss function Equation 7, there are 5 parameters:α, β, γ, λ, and the dimension of projection hidden space k.In our experiments, we tune these parameters alternativelyby grid search. We get an acceptable performance when γ =0.5, λ = 0.1, k = 100 and β = 0.02. When k is larger than100, the performance improves a little, but it costs moretime.

5.2 Tag Prediction

We use image-tag associations in NUSWIDE dataset asground truth and use 50, 000/36, 035 images for train-ing/testing. After training, we use the probability producedby Equation 4 to rank the 1, 000 tags for a given image.We use Precision and MAP (Mean Average Precision) toevaluate the ranking performance.

We use the following image tagging methods as base-lines:

• Neighbor Voting (NV) [18] It is a nearest neighborbased method, which evaluates tag relevance foran image w.r.t the tags of its nearest neighbors. Inour experiments, we use: 1) Bag-of-Words featureprovided by NUSWIDE (denoted as NV-BoW) and2) the CNN feature extracted by AlexNet fc7 layerfor nearest neighbor selection (denoted as NV-CNN).We use 100 as the number of nearest neighbors.

• SVM We use SVM as the classifiers of the tags basedon CNN features from AlexNet.

• Regression CNN (RegCNN) We use the RegressionCNN introduced in Section 4 without knowledgebase embedding. It can be regarded as a fine-tuneddeep model for image tagging.

0 20 40 60 80 1000

2000

4000

6000

num

ber o

f rel

atio

ns

Distance (1e-3)

Fig. 4. Histogram of the relations on distance regions.

• Linked Tag (LinkTag) [12] This method uses DBPe-dia to extend the tags. In our experiments, we useConceptNet instead.

• DeViSE [7] DeViSE first learns words embeddingsand then learns a CNN model which can trans-fer the visual representations to word embeddings.This method is designed for ImageNet classification.When implementing the algorithm, in order to makethe learned representations fitting for our data, weuse the tags in NUSWIDE and ConceptNet instead.As in [7], we use the ConceptNet as an indirectedsimilarity graph.

The comparison with these baselines can demonstratethe advantages of our methods in different aspects. The firstthree methods do not use external semantic resources: NV isknn-based, SVM is a shallow-structured classification modeland RegCNN is a deep-structured classification model. Therest two methods exploit knowledge base, LinkTag fuse tagsand knowledge base in the late stage rather than represen-tation stage. DeViSE is a deep model based method withsemantic ontology embedding for classification. However, itlearns word embedding and CNN independently withoutjoint optimization, and it is based on semantic ontology(WordNet) rather than multiple kinds of relational knowl-edge.

The experimental results are shown in Table 2. We havethe following observations:

(1) Our method performs significantly better than base-line methods in all cases. Especially, it has 26% improvementin MAP.

(2) The improvement of R3CNN over DeVISE demon-strates that directly mapping image space to general knowl-edge space is inadequate, due to the inconsistency of thesetwo spaces. The relation regularization imposed to deepmodel can help to solve this problem. Also, exploiting dif-ferent kinds of relations rather than treating them uniformlycan bring much more additional useful information to imagespace.

(3) LinkTag performs much worse than DeVISE, indicat-ing that the late fusion of image labels and knowledge spacewill bring many erroneous labels that are not related withthe image content. Embedding the knowledge into imagerepresentation stage is an effective way.

(4) The RegCNN performs better than SVM, NV-CNNand NV-BoW, demonstrating the advantages of end-to-endrepresentation learning over hand-crafted features and theAlexNet representations cannot work well in wild images.




laptop

piano

narro

wkic

kmea

teg

gbre

adinj

uryco

ach

mattoo

thclo

th flat

cry

conta

iner

TVshow

laugh gif

tink

rock m

usic

mean

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16MAP

LinkedTag DeVISE R3CNN retrained R3CNN

Fig. 3. MAP of zero-shot tag inference task, including 20 example tags and the average of 100 unseen tags.

TABLE 2Performance of tag prediction task on NUSWIDE dataset.

NV-BoW NV-CNN SVM RegCNN LinkTag DeViSE R3CNNMAP 0.0651 0.1627 0.1726 0.1890 0.0932 0.1863 0.2388P@5 0.1030 0.3000 0.3182 0.3854 0.1891 0.4835 0.5587P@10 0.0890 0.2758 0.3163 0.2970 0.1456 0.3001 0.3951P@50 0.0704 0.1206 0.1805 0.2071 0.0610 0.1900 0.2109

Useless relations: Omissive relations:

Concept1 Concept2 Relation Concept1 Concept2 Relation

holiday book InstanceOf love action RelatedTo

nature artist IsA cat dog isA

country field RelatedTo Paris capital AtLocation

stop fast RelatedTo airplane large HasProperty

space cold HasProperty police tourist RelatedTo

sheep female RelatedTo apple phone isA

fence metal MadeOf job work InstanceOf

cat feline isA sunset sky RelatedTo

cemetery house part of park warm HasProperty

Fig. 5. Showcase of “useless relations” and “omissive relations”.

5.3 Zero-shot Tag InferenceWe randomly sampled 100 unseen tags with the require-ment that each one should occur in both NUSWIDE andConceptNet and have at least 5 relations with the seentags. For each unseen tag, we use Equation 4 to evaluateits relevance to the testing image, and give it a rankingscore. The images that are eventually labeled by the unseentag are ground truth. Then, we use MAP to evaluatethe performances. Zero-shot problem cannot be solved byclassification basedl methods because there is no trainingprocess for the unseen tags. Therefore, only LinkTag andDeViSE can be used as baselines.

We randomly select 20 unseen tags and report the per-formances of different methods on these selected tags inFigure 3. From left to right, the unseen tags are ordered bythe number of relations do they have with the trained tags.We also report the average performances for the 100 unseentags at the last. From the results, we have the followingobservations.

(1) Our method consistently and significantly outper-form the baselines in all these unseen tags. On average,

our method has more than 100% relative improvement overother baselines in MAP.

(2) The more relations do the unseen tag has with thetrained tags in ConceptNet, the higher MAP and relativeimprovement can be achieved by our method. This demon-strates the values of general knowledge in benefiting imagerepresentation learning.

(3) In addition, we retrain the R3CNN model with thedata including unseen tag information. Then we use theretrained R3CNN to predict the unseen tags for images, andreport the prediction performance in the last column in eachgroup. Here we use this performance as the higher boundof unseen tag inference. We can see that, in average, theperformance in inferring unseen tags by our method canget about 70% of the performance in predicting trained tags.Also, the more relations an unseen tag has with trained tagsin ConceptNet, the smaller the margin becomes betweenunseen tag inference and trained tag prediction in ourmethod. This once again demonstrates the importance ofintroducing general knowledge graph into image represen-tation learning.

5.4 Content based Image Retrieval

In order to demonstrate that the learned representations canbe widely used in multiple application scenarios with goodgeneralization ability, we evaluate the proposed approachin content-based image retrieval task.

Holidays [14] is a commonly used dataset in contentbased image retrieval tasks, which contains 1491 high res-olution personal holiday photos. 500 images are used asqueries and the remaining 991 images are labeled as groundtruth. For each query image, we use Euclidean distance asthe metric to rank the testing images. Then, we evaluate the




TABLE 3Performance of CBIR on Holidays.

BoW AlexNet RegCNN DeViSE R3CNN-fc6 R3CNN-fc7MAP 0.540 [3] 0.642 [21] 0.6709 0.6876 0.7254 0.7445

ranking results by MAP. There are four baselines in this ex-periment, including two typical representations, includingBoW (Bag-of-Words based on SIFT discriptors) and AlexNet(CNN feature extracted from AlexNet), RegCNN in Section4.1 for multi-label image representation learning (note thatRegCNN is eventually a fine-tuned AlexNet by NUS-WIDEin a multi-label setting), and DeViSE (the representationsderived from DeVISE). In our method, we have two vari-ants: R3CNN-fc6 and R3CNN-fc7. R3CNN-fc6 correspondsto fc6 layer (the layer before the last fully connected layer)and R3CNN-fc7 corresponds to the last fully connected fc7layer.

The experimental results are shown in Table 3. Theresults of BoW and AlexNet are respectively reported by[3] and [21]. We can see that both of the R3CNN-fc6 andR3CNN-fc7 representations perform better than baselinerepresentations. This demonstrates that the representationlearned from our model has good generalization abilityin different datasets and different application scenarios.The result that R3CNN-fc7 performs better than R3CNN-fc6 indicates that the higher-level representations performbetter when transferring to a different dataset. A plausiblereason is that the higher-level representations are closer tosemantics and thus have better transferability across thesetwo datasets.

5.5 Insight Analysis

Till now, we have demonstrated the advantages and effec-tiveness of embedding general relational knowledge intoimage representation learning. In this section, we will fur-ther analyze how the image representation learning andknowledge graph embedding reenforce each other, andwhat kinds of relational knowledge can benefit the imagerepresentation learning.

In our method, the image representation learning andknowledge graph embedding are jointly optimized. Thusfor any pair of concepts i and j, there are two sources ofconstraints on the similarity of their representation. If theyare related in ConceptNet with relation type p, they shouldshare similar representation in space p through knowledgegraph embedding. However, if the images with tag i andimages with tag j are visually different, their representationsshould be quite different in image representation learning.After the model converges, the distance of the output repre-sentations of wi and wj in p− th space dpij can be calculatedby equation 5, and the distance can tell us: 1) If concept iand j are related in ConceptNet with relation type p, butdpij is relatively large, that means the knowledge < i, j, p >cannot be reflected or supported in image space. 2) If i andj are not related in ConceptNet with relation type p, but dpijis relatively small, that means the knowledge < i, j, p >might be an image-specific knowledge and can possiblysupplement ConceptNet. Thus, we temporarily define that:

IsAHasA

RelatedTo

UsedFor

AtLocation

DefinedAs

InstanceOf

PartOf

HasProperty

CapableOf

SymbolOf

LocatedNear

ReceivesAction

MadeOf

Synonym

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Distanc

e

Fig. 6. Average distance for each relation type.

• if the distance dpij > δ, and relation ⟨ci, cj , rp⟩ existsin ConceptNet, this is a “useless” relation in imagerepresentation learning;

• if the distance dpij < ϵ, and relation ⟨ci, cj , rp⟩ doesnot exist in ConceptNet, this is an “omissive” relationin the knowledge base.

Here δ and ϵ are two parameters, which are determinedaccording to the distribution of the distances.

For the 27, 440 relations in our training data, we calculatetheir histogram on distance regions, which is shown inFigure 4. According to the distribution of the distances, wedefine δ = 0.08 and ϵ = 0.03. Then, we select some rep-resentative “useless” relations and “omissive” relations asshowcase in Figure 5. We can observe that useless relationsare usually caused by: 1) word ambiguity, such as “holidayis an instance of book”; 2) abstract or invisible concepts,such as “stop is related to fast”; 3) true but rarely-used tags,such as “cat is a feline”. For omissive relations, we just selectthe space which makes them the nearest. From the results,we can observe that, except several failed cases, most ofthe pairs of tags in omissive relations are truly related, andthere are many reasonable relations that can supplementConceptNet.

In addition, we also calculate the average distance ofeach relation type (as shown in Figure 6 to observe whichtype is the most helpful in image representation learning.From the results, we can observe that ”AtLocation”, “isA”and “InstanceOf” are the most important relation types inimage representation learning, which is quite reasonable. Incontrast, “MadeOf” and “SymbolOf” are the most uselessrelation types. The main reason is that such relations canhardly be reflected in image space. For example, “paperis made of wood” is a common knowledge. However, werarely tag “wood” on a picture of papers.

By properly embedding the general relational knowl-edge into image representation learning, the learnt repre-sentation is endowed with the ability of reflecting the struc-tured semantics. We further explore this merit and evaluatewhether the learnt representation can generate structuredtags for images. We first predict tags for an image as insection 5.2. Then, for the top tags, we use the previous




Image

Tags in Flickr

blue, europe, aircraft,

canon, airport, lamp,

sunset, blackwhite,

hangar

pet, toy, dog, animal,

garden

beautiful, reef, fish,

tropical, aquarium,

crab, marine, coral

Our results

Fig. 7. A showcase of structured image tagging results. The first tworows are successful cases and the last row is a failure case.

definition: if dpij < 0.03, we think tag i and tag j haverelation p. Figure 7 is a showcase of our results. The firsttwo rows are successful cases and the last row is a failedcase. From these examples, we can see the contrast betweentraditional tags and the structured tags generated by ourmethod. It is obvious that the structured tags with relationalknowledge can significantly help to understand the imagecontent. Also, it can well support reasoning on the images,which is crucial in various image applications.

6 CONCLUSION

In this paper, we propose a novel approach, which embedsthe relations in general knowledge graph into multi-labelimage representation learning task to make the learnt rep-resentation involving more precise and complete seman-tics of images. A Relational Regularized Regression CNN(R3CNN) model is designed to jointly optimize the imagerepresentation learning problem and knowledge graph em-bedding problem. In image representation learning prob-lem, a Regression CNN model is adopted to learn imagerepresentation from multi-label image-tag data. In knowl-edge graph embedding task, we map a concept in knowl-edge graph into multiple hidden spaces and consider theasymmetric relations. The experimental results in tag pre-diction and zero-shot tag inference tasks demonstrate thatour approach can better embed the general knowledge inrepresentation learning. Experiments in CBIR show thatthe learnt representation is transferable to other semanticrelated tasks and performs better than traditional deepfeatures. The observations of the learnt tag relations demon-strate the potential of our approach in building vision-basedconcept relationships.

In this work, we just give some simple observations ofthe learnt relationships. In the future, we can go furtherin building vision-based concept relations based on theuser tagging behavior and discover the difference betweenlanguage-level knowledge and vision-level knowledge.

7 ACKNOWLEDGEMENTThis work was supported by National Program on KeyBasic Research Project, No. 2015CB352300; National NaturalScience Foundation of China, No. 61370022, No. 61521002,No. 61531006, No. 61472444 and No. 61210008. Thanks forthe research fund of Tsinghua-Tencent Joint Laboratory forInternet Innovation Technology, and the Young Elite Scien-tist Sponsorship Program by CAST.

REFERENCES

[1] A. H. Abdulnabi, G. Wang, J. Lu, and K. Jia. Multi-task cnnmodel for attribute prediction. Multimedia, IEEE Transactions on,17(11):1949–1959, 2015.

[2] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Label-embedding for attribute-based classification. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages819–826, 2013.

[3] R. Arandjelovic and A. Zisserman. All about vlad. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition,pages 1578–1585, 2013.

[4] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, andZ. Ives. Dbpedia: A nucleus for a web of open data. Springer, 2007.

[5] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, andO. Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems,pages 2787–2795, 2013.

[6] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng. Nus-wide: a real-world web image database from national universityof singapore. In CIVR, page 48. ACM, 2009.

[7] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov,et al. Devise: A deep visual-semantic embedding model. InAdvances in Neural Information Processing Systems, pages 2121–2129,2013.

[8] Y. Gong, Y. Jia, T. Leung, A. Toshev, and S. Ioffe. Deep convo-lutional ranking for multilabel image annotation. arXiv preprintarXiv:1312.4894, 2013.

[9] M. Guillaumin, T. Mensink, J. Verbeek, and C. Schmid. Tagprop:Discriminative metric learning in nearest neighbor models forimage auto-annotation. In ICCV, pages 309–316. IEEE, 2009.

[10] K. Guu, J. Miller, and P. Liang. Traversing knowledge graphs invector space. arXiv preprint arXiv:1506.01094, 2015.

[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning forimage recognition. arXiv preprint arXiv:1512.03385, 2015.

[12] D.-H. Im and G.-D. Park. Linked tag: image annotation usingsemantic relationships between image tags. Multimedia Tools andApplications, 74(7):2273–2287, 2015.

[13] H. Izadinia, B. C. Russell, A. Farhadi, M. D. Hoffman, andA. Hertzmann. Deep classifiers from image tags in the wild. InProceedings of the 2015 Workshop on Community-Organized Multi-modal Mining: Opportunities for Novel Solutions, pages 13–18. ACM,2015.

[14] H. Jegou, M. Douze, and C. Schmid. Hamming embedding andweak geometric consistency for large scale image search. InComputer Vision–ECCV 2008, pages 304–317. Springer, 2008.

[15] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classifi-cation with deep convolutional neural networks. In NIPS, pages1097–1105, 2012.

[16] Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Muller. Efficientbackprop. In Neural networks: Tricks of the trade, pages 9–48.Springer, 2012.

[17] X. Li, S. Liao, W. Lan, X. Du, and G. Yang. Zero-shot image taggingby hierarchical semantic embedding. In Proceedings of the 38thInternational ACM SIGIR Conference on Research and Developmentin Information Retrieval, pages 879–882. ACM, 2015.

[18] X. Li, C. G. Snoek, and M. Worring. Learning social tag relevanceby neighbor voting. Multimedia, IEEE Transactions on, 11(7):1310–1322, 2009.

[19] S. Liu, P. Cui, W. Zhu, and S. Yang. Learning socially embeddedvisual representation from scratch. In Proceedings of the 23rd ACMInternational Conference on Multimedia, MM ’15, pages 109–118,New York, NY, USA, 2015. ACM.

[20] W. Lu, J. Li, T. Li, W. Guo, H. Zhang, and J. Guo. Web multimediaobject classification using cross-domain correlation knowledge.IEEE transactions on multimedia, 15(8):1920–1929, 2013.




[21] A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnnfeatures off-the-shelf: an astounding baseline for recognition. InProceedings of the IEEE Conference on Computer Vision and PatternRecognition Workshops, pages 806–813, 2014.

[22] K. Simonyan and A. Zisserman. Very deep convolutional net-works for large-scale image recognition. arXiv preprint, 2014.

[23] R. Speer and C. Havasi. Conceptnet 5: A large semantic networkfor relational knowledge. In The Peoples Web Meets NLP, pages161–176. Springer, 2013.

[24] A. Vedaldi and K. Lenc. Matconvnet: Convolutional neural net-works for matlab. In ACM Multimedia, pages 689–692. ACM, 2015.

[25] D. Wang, P. Cui, M. Ou, and W. Zhu. Deep multimodal hashingwith orthogonal regularization. In Proceedings of the 24th Interna-tional Conference on Artificial Intelligence, pages 2291–2297. AAAIPress, 2015.

[26] D. Wang, P. Cui, M. Ou, and W. Zhu. Learning compact hash codesfor multimodal representations using orthogonal deep structure.IEEE Transactions on Multimedia, 17(9):1404–1416, 2015.

[27] L. Xie and X. He. Picture tags and world knowledge: Learning tagrelations from visual semantic sources. In Proceedings of the 21stACM international conference on Multimedia, pages 967–976. ACM,2013.

[28] X. Yang, T. Zhang, C. Xu, S. Yan, M. S. Hossain, and A. Ghoneim.Deep relative attributes. IEEE Transactions on Multimedia,18(9):1832–1842, 2016.

[29] H. Zhang, Z. Kyaw, S.-F. Chang, and T.-S. Chua. Visual translationembedding network for visual relation detection. arXiv preprintarXiv:1702.08319, 2017.

[30] H. Zhang, X. Shang, H. Luan, Y. Yang, and T.-S. Chua. Learningfeatures from large-scale, noisy and social image-tag collection. InACM Multimedia, pages 1079–1082. ACM, 2015.

[31] X. Zhang, Y. Yang, Y. Zhang, H. Luan, J. Li, H. Zhang, and T.-S. Chua. Enhancing video event recognition using automaticallyconstructed semantic-visual knowledge base. IEEE Transactions onMultimedia, 17(9):1562–1575, 2015.

[32] F. Zhao, Y. Huang, L. Wang, and T. Tan. Deep semantic rankingbased hashing for multi-label image retrieval. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages1556–1564, 2015.

atex class files, vol. 14, no. 8, august 2015 1 general...

Documents