Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 1021–1029,Suntec, Singapore, 2-7 August 2009. c©2009 ACL and AFNLP

Unsupervised Relation Extraction by Mining Wikipedia Texts UsingInformation from the Web

Yulan Yan, Naoaki Okazaki, Yutaka Matsuo, Zhenglu Yang and Mitsuru IshizukaThe University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan

yulan@mi.ci.i.u-tokyo.ac.jpokazaki@is.s.u-tokyo.ac.jp

matsuo@biz-model.t.utokyo.ac.jpyangzl@tkl.iis.u-tokyo.ac.jp

ishizuka@i.u-tokyo.ac.jp

Abstract

This paper presents an unsupervised rela-tion extraction method for discovering andenhancing relations in which a specifiedconcept in Wikipedia participates. Usingrespective characteristics of Wikipedia ar-ticles and Web corpus, we develop a clus-tering approach based on combinations ofpatterns: dependency patterns from depen-dency analysis of texts in Wikipedia, andsurface patterns generated from highly re-dundant information related to the Web.Evaluations of the proposed approach ontwo different domains demonstrate the su-periority of the pattern combination overexisting approaches. Fundamentally, ourmethod demonstrates how deep linguisticpatterns contribute complementarily withWeb surface patterns to the generation ofvarious relations.

1 Introduction

Machine learning approaches for relation extrac-tion tasks require substantial human effort, partic-ularly when applied to the broad range of docu-ments, entities, and relations existing on the Web.Even with semi-supervised approaches, which usea large unlabeled corpus, manual construction of asmall set of seeds known as true instances of thetarget entity or relation is susceptible to arbitraryhuman decisions. Consequently, a need exists fordevelopment of semantic information-retrieval al-gorithms that can operate in a manner that is asunsupervised as possible.

Currently, the leading methods in unsupervisedinformation extraction collect redundancy infor-mation from a local corpus or use the Web as acorpus (Pantel and Pennacchiotti, 2006); (Bankoet al., 2007); (Bollegala et al., 2007): (Fan etal., 2008); (Davidov and Rappoport, 2008). The

standard process is to scan or search the cor-pus to collect co-occurrences of word pairs withstrings between them, and then to calculate termco-occurrence or generate surface patterns. Themethod is used widely. However, even when pat-terns are generated from well-written texts, fre-quent pattern mining is non-trivial because thenumber of unique patterns is loose, but many pat-terns are non-discriminative and correlated. Asalient challenge and research interest for frequentpattern mining is abstraction away from differentsurface realizations of semantic relations to dis-cover discriminative patterns efficiently.

Linguistic analysis is another effective tech-nology for semantic relation extraction, as de-scribed in many reports such as (Kambhatla,2004); (Bunescu and Mooney, 2005); (Harabagiuet al., 2005); (Nguyen et al., 2007). Currently, lin-guistic approaches for semantic relation extractionare mostly supervised, relying on pre-specificationof the desired relation or initial seed words or pat-terns from hand-coding. The common process isto generate linguistic features based on analyses ofthe syntactic features, dependency, or shallow se-mantic structure of text. Then the system is trainedto identify entity pairs that assume a relation andto classify them into pre-defined relations. The ad-vantage of these methods is that they use linguistictechnologies to learn semantic information fromdifferent surface expressions.

As described herein, we consider integratinglinguistic analysis with Web frequency informa-tion to improve the performance of unsupervisedrelation extraction. As (Banko et al., 2007)reported, “deep” linguistic technology presentsproblems when applied to heterogeneous text onthe Web. Therefore, we do not parse informa-tion from the Web corpus, but from well writtentexts. Particularly, we specifically examine unsu-pervised relation extraction from existing texts ofWikipedia articles. Wikipedia resources of a fun-

1021

damental type are of concepts (e.g., representedby Wikipedia articles as a special case) and theirmutual relations. We propose our method, whichgroups concept pairs into several clusters based onthe similarity of their contexts. Contexts are col-lected as patterns of two kinds: dependency pat-terns from dependency analysis of sentences inWikipedia, and surface patterns generated fromhighly redundant information from the Web.

The main contributions of this paper are as fol-lows:

• Using characteristics of Wikipedia articlesand the Web corpus respectively, our studyyields an example of bridging the gap sep-arating “deep” linguistic technology and re-dundant Web information for InformationExtraction tasks.

• Our experimental results reveal that relationsare extractable with good precision usinglinguistic patterns, whereas surface patternsfrom Web frequency information contributegreatly to the coverage of relation extraction.

• The combination of these patterns producesa clustering method to achieve high pre-cision for different Information Extractionapplications, especially for bootstrapping ahigh-recall semi-supervised relation extrac-tion system.

2 Related Work

(Hasegawa et al., 2004) introduced a method fordiscovering a relation by clustering pairs of co-occurring entities represented as vectors of con-text features. They used a simple representationof contexts; the features were words in sentencesbetween the entities of the candidate pairs.

(Turney, 2006) presented an unsupervised algo-rithm for mining the Web for patterns expressingimplicit semantic relations. Given a word pair, theoutput list of lexicon-syntactic patterns was rankedby pertinence, which showed how well each pat-tern expresses the relations between word pairs.

(Davidov et al., 2007) proposed a method forunsupervised discovery of concept specific rela-tions, requiring initial word seeds. That methodused pattern clusters to define general relations,specific to a given concept. (Davidov and Rap-poport, 2008) presented an approach to discoverand represent general relations present in an arbi-trary corpus. That approach incorporated a fully

unsupervised algorithm for pattern cluster discov-ery, which searches, clusters, and merges high-frequency patterns around randomly selected con-cepts.

The field of Unsupervised Relation Identifica-tion (URI)—the task of automatically discover-ing interesting relations between entities in largetext corpora—was introduced by (Hasegawa etal., 2004). Relations are discovered by cluster-ing pairs of co-occurring entities represented asvectors of context features. (Rosenfeld and Feld-man, 2006) showed that the clusters discovered byURI are useful for seeding a semi-supervised rela-tion extraction system. To compare different clus-tering algorithms, feature extraction and selectionmethod, (Rosenfeld and Feldman, 2007) presenteda URI system that used surface patterns of twokinds: patterns that test two entities together andpatterns that test either of two entities.

In this paper, we propose an unsupervised rela-tion extraction method that combines patterns oftwo types: surface patterns and dependency pat-terns. Surface patterns are generated from the Webcorpus to provide redundancy information for re-lation extraction. In addition, to obtain seman-tic information for concept pairs, we generate de-pendency patterns to abstract away from differentsurface realizations of semantic relations. Depen-dency patterns are expected to be more accurateand less spam-prone than surface patterns from theWeb corpus. Surface patterns from redundancyWeb information are expected to address the datasparseness problem. Wikipedia is currently widelyused information extraction as a local corpus; theWeb is used as a global corpus.

3 Characteristics of Wikipedia articles

Wikipedia, unlike the whole Web corpus, hasseveral characteristics that markedly facilitate in-formation extraction. First, as an earlier report(Giles, 2005) explained, Wikipedia articles aremuch cleaner than typical Web pages. Becausethe quality is not so different from standard writ-ten English, we can use “deep” linguistic tech-nologies, such as syntactic or dependency parsing.Secondly, Wikipedia articles are heavily cross-linked, in a manner resembling cross-linking ofthe Web pages. (Gabrilovich and Markovitch,2006) assumed that these links encode numerousinteresting relations among concepts, and that theyprovide an important source of information in ad-

1022

dition to the article texts.To establish the background for this paper, we

start by defining the problem under consideration:relation extraction from Wikipedia. We use the en-cyclopedic nature of the corpus by specifically ex-amining the relation extraction between the enti-tled concept (ec) and a related concept (rc), whichare described in anchor text in this article. A com-mon assumption is that, when investigating the se-mantics in articles such as those in Wikipedia (e.g.semantic Wikipedia (Volkel et al., 2006)), key in-formation related to a concept described on a pagep lies within the set of links l(p) on that page; par-ticularly, it is likely that a salient semantic relationr exists between p and a related page p′ ∈ l(p).

Given the scenario we described along withearlier related works, the challenges we face arethese: 1) enumerating all potential relation typesof interest for extraction is highly problematic forcorpora as large and varied as Wikipedia; 2) train-ing data or seed data are difficult to label. Consid-ering (Davidov and Rappoport, 2008), which de-scribes work to get the target word and relationcluster given a single (‘hook’) word, their methoddepends mainly on frequency information fromthe Web to obtain a target and clusters. Attempt-ing to improve the performance, our solution forthese challenges is to combine frequency informa-tion from the Web and the “high quality” charac-teristic of Wikipedia text.

4 Pattern Combination Method forRelation Extraction

With the scene and challenges stated, we propose asolution in the following way. The intuitive idea isthat we integrate linguistic technologies on high-quality text in Wikipedia and Web mining tech-nologies on a large-scale Web corpus. In this sec-tion, we first provide an overview of our methodalong with the function of the main modules. Sub-sequently, we explain each module in the methodin detail.

4.1 Overview of the MethodGiven a set of Wikipedia articles as input, ourmethod outputs a list of concept pairs for each ar-ticle with a relation label assigned to each conceptpair. Briefly, the proposed approach has four mainmodules, as depicted in Fig. 1.

• Text Preprocessor and Concept Pair Col-lector preprocesses Wikipedia articles to

Wikipedia articlesPreprocessorConcept pair collectionSentence filtering Web context collectorWeb ContextTi = t1, t2…tnPi = p1,p2…pnDependency pattern Extractor

n1i,…n1j …ni2i, ..n2j ni,…nj…surface clusteringdepend clusteringRelation listOutput: relations for each article

input:

Eric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovellEric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovellEric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovell ......…………......………… ......…………Tyco becomingjoined comp:CEOobj: cc:joinedobj:subj:joinedobj: cc: Clustering approach

Wikipedia articlesPreprocessorConcept pair collectionSentence filtering Web context collectorWeb ContextTi = t1, t2…tnPi = p1,p2…pnDependency pattern Extractor

n1i,…n1j …ni2i, ..n2j ni,…nj…surface clusteringdepend clusteringRelation listOutput: relations for each article

input:

Eric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovellEric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovellEric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovellEric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovellEric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovellEric Emerson SchmidtCEOa-member-ofBorn GoogleBoard of DirectorsWashington, D.C.Is-a chairmanNovell ......…………......………… ......…………Tyco becomingjoined comp:CEOobj: cc:joinedobj:subj:joinedobj: cc: Tyco becomingjoined comp:CEOobj: cc:joinedobj:subj:joinedobj: cc: Clustering approach

Figure 1: Framework of the proposed approach

split text and filter sentences. It outputs con-cept pairs, each of which has an accompany-ing sentence.

• Web Context Collector collects context in-formation from the Web and generates rankedrelational terms and surface patterns for eachconcept pair.

• Dependency Pattern Extractor generatesdependency patterns for each concept pairfrom corresponding sentences in Wikipediaarticles.

• Clustering Algorithm clusters concept pairsbased on their context. It consists of the twosub-modules described below.

– Depend Clustering, which merges con-cept pairs using dependency patternsalone, aiming at obtaining clusters ofconcept pairs with good precision;

– Surface Clustering, which clustersconcept pairs using surface patternsbased on the resultant clusters of dependclustering. The aim is to merge moreconcept pairs into existing clusters withsurface patterns to improve the coverageof clusters.

1023

4.2 Text Preprocessor and Concept PairCollector

This module pre-processes Wikipedia article textsto collect concept pairs and corresponding sen-tences. Given a concept described in a Wikipediaarticle, our idea of preprocessing executes initialconsideration of all anchor-text concepts linkingto other Wikipedia articles in the article as relatedconcepts that might share a semantic relation withthe entitled concept. The link structure, more par-ticularly, the structure of outgoing links, providesa simple mechanism for identifying relevant arti-cles. We split text into sentences and select sen-tences containing one reference of an entitled con-cept and one of the linked texts for the dependencypattern extractor module.

4.3 Web Context CollectorQuerying a concept pair using a search engine(Google), we characterize the semantic relationbetween the pair by leveraging the vast size of theWeb. Our hypothesis is that there exist some keyterms and patterns that provide clues to the rela-tions between pairs. From the snippets retrievedby the search engine, we extract relational infor-mation of two kinds: ranked relational terms askeywords and surface patterns. Here surface pat-terns are generated with support of ranked rela-tional terms.

4.3.1 Relational Term RankingTo collect relational terms as indicators for eachconcept pair, we look for verbs and nouns fromqualified sentences in the snippets instead of sim-ply finding verbs. Using only verbs as relationalterms might engender the loss of various importantrelations, e.g. noun relations “CEO”, “founder”between a person and a company. Therefore, foreach concept pair, a list of relational terms is col-lected. Then all the collected terms of all conceptpairs are combined and ranked using an entropy-based algorithm which is described in (Chen et al.,2005). With their algorithm, the importance ofterms can be assessed using the entropy criterion,which is based on the assumption that a term is ir-relevant if its presence obscures the separability ofthe dataset. After the ranking, we obtain a globalranked list of relational terms Tall for the wholedataset (all the concept pairs). For each conceptpair, a local list of relational terms Tcp is sorted ac-cording to the terms’ order in Tall. Then from therelational term list Tcp, a keyword tcp is selected

Table 1: Surface patterns for a concept pair

Pattern Patternec ceo rc rc found ecceo rc found ec rc succeed as ceo of ecrc be ceo of ec ec ceo of rcec assign rc as ceo ec found by ceo rcceo of ec rc ec found in by rc

for each concept pair cp as the first term appearingin the term list Tcp. Keyword tcp will be used toinitialize the clustering algorithm in Section 4.5.1.

4.3.2 Surface Pattern GenerationBecause simply taking the entire string betweentwo concept words captures an excess of extra-neous and incoherent information, we use Tcp ofeach concept pair as a key for surface pattern gen-eration. We classified words into Content Words(CWs) and Functional Words (FWs). From eachsnippet sentence, the entitled concept, related con-cept, or the keyword kcp is considered to be a Con-tent Word (CW). Our idea of obtaining FWs is tolook for verbs, nouns, prepositions, and coordinat-ing conjunctions that can help make explicit thehidden relations between the target nouns.

Surface patterns have the following generalform.

[CW1] Infix1 [CW2] Infix2 [CW3] (1)

Therein, Infix1 and Infix2 respectively con-tain only and any number of FWs. A pattern ex-ample is “ec assign rc as ceo (keyword)”. All gen-erated patterns are sorted by their frequency, andall occurrences of the entitled concept and relatedconcept are replaced with “ec” and “rc”, respec-tively for pattern matching of different conceptpairs.

Table 1 presents examples of surface patternsfor a sample concept pair. Pattern windows arebounded by CWs to obtain patterns more preciselybecause 1) if we use only the string between twoconcepts, it may not contain some important re-lational information, such as “ceo ec resign rc”in Table 1; 2) if we generate patterns by settinga windows surrounding two concepts, the numberof unique patterns is often exponential.

4.4 Dependency Pattern Extractor

In this section, we describe how to obtain depen-dency patterns for relation clustering. After pre-processing, selected sentences that contain at least

1024

one mention of an entitled concept or related con-cept are parsed into dependency structures. We de-fine dependency patterns as sub-paths of the short-est dependency path between a concept pair fortwo reasons. One is that the shortest path de-pendency kernels outperform dependency tree ker-nels by offering a highly condensed representationof the information needed to assess their relation(Bunescu and Mooney, 2005). The other reasonis that embedded structures of the linguistic repre-sentation are important for obtaining good cover-age of the pattern acquisition, as explained in (Cu-lotta and Sorensen, 2005); (Zhang et al., 2006).The process of inducing dependency patterns hastwo steps.

1. Shortest dependency path inducement. Fromthe original dependency tree structure by parsingthe selected sentence for each concept pair, wefirst induce the shortest dependency path with theentitled concept and related concept.

2. Dependency pattern generation. We usea frequent tree-mining algorithm (Zaki, 2002) togenerate sub-paths as dependency patterns fromthe shortest dependency path for relation cluster-ing.

4.5 Clustering Algorithm for RelationExtraction

In this subsection, we present a clustering algo-rithm that merges concept pairs based on depen-dency patterns and surface patterns. The algorithmis based on k-means clustering for relation cluster-ing.

The dependency pattern has the properties ofbeing more accurate, but the Web context has theadvantage of containing much more redundant in-formation than Wikipedia. Our idea of conceptpair clustering is a two-step clustering process:first it clusters concept pairs into clusters withgood precision using dependency patterns; then itimproves the coverage of the clusters using surfacepatterns.

4.5.1 Initial Centroid Selection and DistanceFunction Definition

The standard k-means algorithm is affected bythe choice of seeds and the number of clustersk. However, as we claimed in the Introduc-tion section, because we aim to extract relationsfrom Wikipedia articles in an unsupervised man-ner, cluster number k is unknown and no goodcentroids can be predicted. As described in this

paper, we select centroids based on the keywordtcp of each concept pair.

First of all, all concept pairs are grouped bytheir keywords tcp. Let G = {G1, G2, ...Gn}be the resultant groups, where each Gi ={cpi1, cpi2, ...} identify a group of concept pairssharing the same keyword tcp (such as “CEO”).We rank all the groups by their number of conceptpairs and then choose the top k groups. Then acentroid ci is selected for each group Gi by Eq. 2.

ci = arg maxcp∈Gi

|{cpij |(dis1(cpij , cp)+

λ ∗ dis2(cpij , cp)) <= Dz, 1 ≤ j ≤ |Gi|}| (2)

We assume a centroid for each group to be theconcept pair which has the most other conceptpairs in the same group that have distance lessthan Dz with it. Also, Dz is a threshold to avoidnoisy concept pairs: we assign it 1/3. To balancethe contribution between dependency patterns andsurface patterns, λ is used. The distance functionto calculate the distance between dependency pat-tern sets DPi, DPj of two concept pairs cpi andcpj is dis1. The distance is decided by the numberof overlapped dependency patterns with Eq. 3.

dis1(cpi, cpj) = 1− |DPi ∩DPj |√(|DPi| ∗ |DPj |)

(3)

Actually, dis2 is the distance function to calcu-late distance between two surface pattern sets oftwo concept pairs. To compute the distance oversurface patterns, we implement the distance func-tion dis2(cpi, cpj) in Fig. 2.

Algorithm 1: distance function dis2(cpi, cpj)Input: SP1 = {sp11, ..., sp1m}(surface patterns of

cpi)SP2 = {sp21, ..., sp2n} (surface patterns of cpj)Output: dis (distance between SP1 and SP2)define a m× n distance matrix A:{Aij =

LD(sp1i,sp2j)

Max(|sp1i|,|sp2j |) , 1≤i≤m; 1≤j≤n};dis ← 0for min(m, n) times do

(x, y) ← argmin0<i<m;0<j<nAij ;dis ← dis + Axy/min(m, n);Ax∗ ← 1; A∗y ← 1;

return dis

Figure 2: Distance function over surface patterns

As shown in Fig. 2, the distance algorithm per-forms as: firstly it defines a m×n distance matrixA, then repeatedly selects two nearest sequencesand sums up their distances. While computing

1025

dis2, we use the Levenshtein distance LD to mea-sure the difference of two surface patterns. TheLevenshtein distance is a metric for measuring theamount of difference between two sequences (i.e.,the so-called edit distance). Each generated sur-face pattern is a sequence of words. The distanceof two surface patterns is defined as the fraction ofthe LD value to the length of the longer sequence.

For estimating the number of clusters k, we ap-ply the stability-based criteria from (Chen et al.,2005) to decide the number of optimal clusters kautomatically.

4.5.2 Concept Pair Clustering withDependency Patterns

Given the initial seed concept pairs and clusternumber k, this stage merges concept pairs over de-pendency patterns into k clusters. Each conceptpair cpi has a set of dependency patterns DPi. Wecalculate distances between two pairs cpi and cpj

using above the function dis1(cpi, cpj). The clus-tering algorithm is portrayed in Fig. 3. The pro-cess of depend clustering is to assign each conceptpair to the cluster with the closest centroid andthen recomputing each centroid based on the cur-rent members of its cluster. As shown in Figure 3,this is done iteratively by repeating both two stepsuntil a stopping criterion is met. We apply the ter-mination condition as: centroids do not change be-tween iterations.

Algorithm 2: Depend ClusteringInput: I = {cp1, ..., cpn}(all concept pairs)

C = {c1, ..., ck} (k initial centroids)Output: Md : I → C (cluster membership)

Ir (rest of concept pairs not clustered)Cd = {c1, ..., ck} (recomputed centroids)

while stopping criterion has not been met dofor each cpi ∈ I do

if mins∈1..k dis1(cpi, cs) <= Dl thenMd(cpi) ← argmins∈1..k dis1(cpi, cs)

elseMd(cpi) ← 0

for each j ∈ {1..k} dorecompute cj as the centroid of

{cpi|mloc(cpi) = j}Ir ← C0

return C and Cd

Figure 3: Clustering with dependency patterns

Because many concept pairs are scattered anddo not belong to any of the top k clusters, wefilter concept pairs with distance larger than Dl

with the seed concept pairs. Such concept pairs

ST1

ST3 ST4

ST2

Text3: RC was hired as EC’s CEO Text4: EC assign RC as CEO

Text1: the CEO of EC is RC Text2: RC is the CEO of ECST1

ST3 ST4

ST2

Text3: RC was hired as EC’s CEO Text4: EC assign RC as CEO

Text1: the CEO of EC is RC Text2: RC is the CEO of EC

Figure 4: Example showing why surface cluster-ing is needed

are stored in C0. We named the cluster of conceptpairs Ir which are left to be clustered in the nextstep of clustering. After this step, concept pairswith similar dependency patterns are merged intosame clusters, see Fig. 4 (ST1, ST2).

4.5.3 Concept Pair Clustering with SurfacePatterns

A salient difficulty posed by dependency patternclustering is that concept pairs of the same se-mantic relation cannot be merged if they are ex-pressed in different dependency structures. Fig-ure 4 presents an example demonstrating why weperform surface pattern clustering. As depictedin Fig. 4, ST1, ST2, ST3, and ST4 are depen-dency structures for four concept pairs that shouldbe classified as the same relation “CEO”. HoweverST3 and ST4 can not be merged with ST1 andST2 using the dependency patterns because theirdependency structures are too diverse to share suf-ficient dependency patterns.

In this step, we use surface patterns to mergemore concept pairs for each cluster to improve thecoverage. Figure 5 portrays the algorithm. Weassume that each concept pair has a set of sur-face patterns from the Web context collector mod-ule. As shown in Figure 5, surface clustering isdone iteratively by repeating two steps until a stop-ping criterion is met: using the distance functiondis2 explained in the preceding section, assigneach concept pair to the cluster with the closestcentroid and recomputing each centroid based onthe current members of its cluster. We apply thesame termination condition as depend clustering.

1026

Additionally, we filter concept pairs with distancegreater than Dg with the centroid concept pairs.

Algorithm 3: Surface ClusteringInput: Ir (rest of concept pairs)

Cd = {c1, ..., ck} (initial centroids)Output: Ms : Ir → C (cluster membership)

Cs = {c1, ..., ck} (final centroids)while stopping criterion has not been met do

for each cpi ∈ Ir doif mins∈1..k dis2(cpi, cs) <= Dg then

Ms(cpi) ← argmins∈1..k dis2(cpi, cs)else

Ms(cpi) ← 0

for each j ∈ 1..k dorecompute cj as the centroid of cluster

{cpi|Md(cpi) = j ∨Ms(cpi) = j}return clusters C

Figure 5: Clustering with surface patterns

Finally we have k clusters of concept pairs, eachof which has a centroid concept pair. To attacha single relation label to each cluster, we use thecentroid concept pair.

5 Experiments

We apply our algorithm to two categories inWikipedia: “American chief executives” and“Companies”. Both categories are well definedand closed. We conduct experiments for extract-ing various relations and for measuring the qualityof these relations in terms of precision and cover-age. We use coverage as an evaluation instead ofusing recall as a measure. The coverage is used toevaluate all correctly extracted concept pairs. It isdefined as the fraction of all the correctly extractedconcept pairs to the whole set of concept pairs. Tobalance between precision and coverage of clus-tering, we integrate two parameters: Dl, Dg.

We downloaded the Wikipedia dump as of De-cember 3, 2008. The performance of the pro-posed method is evaluated using different patterntypes: dependency patterns, surface patterns, andtheir combination. We compare our method with(Rosenfeld and Feldman, 2007)’s URI method.Their algorithm outperformed that presented in theearlier work using surface features of two kinds forunsupervised relation extraction: features that testtwo entities together and features that test only oneentity each. For comparison, we use a k-meansclustering algorithm using the same cluster num-ber k.

Table 2: Results for the category: “American chiefexecutives”

method Existing method Proposed method(Rosenfeld et al.) (Our method)

Relation # Ins. pre # Ins. pre(sample)chairman 434 63.52 547 68.37(x be chairman of y)ceo 396 73.74 423 77.54(x be ceo of y)bear 138 83.33 276 86.96(x be bear in y)attend 225 67.11 313 70.28(x attend y)member 14 85.71 175 91.43(x be member of y)receive 97 67.97 117 73.53(x receive y)graduate 18 83.33 92 88.04(x graduate from y)degree 5 80.00 78 82.05(x obtain y degree)marry 55 41.67 74 61.25(x marry y)earn 23 86.96 51 88.24(x earn y)award 23 43.47 46 84.78(x won y award)hold 5 80.00 37 72.97(x hold y degree)become 35 74.29 37 81.08(x become y)director 24 67.35 29 79.31(x be director of y)die 18 77.78 19 84.21(x die in y)all 1510 68.27 2314 75.63

5.1 Wikipedia Category: “American chiefexecutives”

We choose appropriate Dl(concept pair filter independ clustering) and Dg(concept pair filter insurface clustering) in a development set. To bal-ance precision and coverage, we set 1/3 for bothDl and Dg.

The 526 articles in this category are used forevaluation. We obtain 7310 concept pairs fromthe articles as our dataset. The top 18 groups arechosen to obtain the centroid concept pairs. Ofthese, 15 binary relations are the clearly identifi-able relations shown in Table 2, where # Ins. rep-resents the number of concept pairs clustered us-ing each method, and pre denotes the precision ofeach cluster.

The proposed approach shows higher precisionand better coverage than URI in Table 2. Thisresult demonstrates that adding dependency pat-terns from linguistic analysis contributes more tothe precision and coverage of the clustering taskthan the sole use of surface patterns.

1027

Table 3: Performance of different pattern typesPattern type #Instance Precision Coveragedependency 1127 84.29 13.00%surface 1510 68.27 14.10%Combined 2314 75.63 23.94%

Table 4: Results for the category: “Companies”Method Existing method Proposed method

(Rosenfeld et al.) (Our method)Relation # Ins. pre # Ins. pre(sample)found 82 75.61 163 84.05(found x in y)base 82 76.83 122 82.79(x be base in y)headquarter 23 86.97 120 89.34(x be headquarter in y)service 37 51.35 108 69.44(x offer y service)store 113 77.88 88 72.72(x open store in y)acquire 59 62.71 70 64.28(x acquire y)list 51 64.71 67 70.15(x list on y)product 25 76.00 57 77.19(x produce y)CEO 37 64.86 39 66.67(ceo x found y)buy 53 62.26 37 56.76(x buy y)establish 35 82.86 26 80.77(x be establish in y)locate 14 50.00 24 75.00(x be locate in y)all 685 71.03 1039 76.87

To examine the contribution of dependency pat-terns, we compare results obtained with patternsof different kinds. Table 3 shows the precision andcoverage scores. The best precision is achieved bydependency patterns. The precision is markedlybetter than that of surface patterns. However, thecoverage is worse than that by surface patterns. Aswe reported, many concept pairs are scattered anddo not belong to any of the top k clusters, the cov-erage is low.

5.2 Wikipedia Category: “Companies”

We also evaluate the performance for the “Com-panies” category. Instead of using all the arti-cles, we randomly select 434 articles for evalua-tion and 4073 concept pairs from the articles formour dataset for this category. We also set Dl andDg to 1/3. Then 28 groups are chosen. For eachgroup, a centroid concept pair is obtained. Finally,of 28 clusters, 25 binary relations are clearly iden-tifiable relations. Table 4 presents some relations.

Table 5: Performance of different pattern typesPattern type #Instance Precision Coveragedependency 551 82.58 11.17%surface 685 71.03 11.95%Combined 1039 76.87 19.61%

Our clustering algorithms use two filters Dl andDg to filter scattering concept pairs. In Table 4, wepresent that concept pairs are clustered with goodprecision. As in the first experiments, the combi-nation of dependency patterns and surface patternscontribute greatly to the precision and coverage.Table 5 shows that, using dependency patterns,the precision is the highest (82.58%), although thecoverage is the lowest.

All experimental results support our ideamainly in two aspects: 1) Dependency analysiscan abstract away from different surface realiza-tions of text. In addition, embedded structures ofthe dependency representation are important forobtaining a good coverage of the pattern acqui-sition. Furthermore, the precision is better thanthat of the string surface patterns from Web pagesof various kinds. 2) Surface patterns are used tomerge concept pairs with relations represented indifferent dependency structures with redundancyinformation from the vast size of Web pages. Us-ing surface patterns, more concept pairs are clus-tered, and the coverage is improved.

6 Conclusions

To discover a range of semantic relations froma large corpus, we present an unsupervised rela-tion extraction method using deep linguistic in-formation to alleviate surface and noisy surfacepatterns generated from a large corpus, and useWeb frequency information to ease the sparse-ness of linguistic information. We specifically ex-amine texts from Wikipedia articles. Relationsare gathered in an unsupervised way over pat-terns of two types: dependency patterns by parsingsentences in Wikipedia articles using a linguisticparser, and surface patterns from redundancy in-formation from the Web corpus using a search en-gine. We report our experimental results in com-parison to those of previous works. The resultsshow that the best performance arises from a com-bination of dependency patterns and surface pat-terns.

1028

ReferencesMichele Banko, Michael J. Cafarella, Stephen Soder-

land, Matt Broadhead and Oren Etzioni. 2007.Open information extraction from the Web. In Pro-ceedings of IJCAI-2007.

Danushka Bollegala, Yutaka Matsuo and MitsuruIshizuka. 2007. Measuring Semantic Similarity be-tween Words Using Web Search Engines. In Pro-ceedings of WWW-2007.

Razvan C. Bunescu and Raymond J. Mooney. 2005. Ashortest path dependency kernel for relation extrac-tion. In Proceedings of HLT/EMLNP-2005.

Jinxiu Chen, Donghong Ji, Chew Lim Tan andZhengyu Niu. 2005. Unsupervised Feature Se-lection for Relation Extraction. In Proceedings ofIJCNLP-2005.

Aron Culotta and Jeffrey Sorensen. 2004. Dependencytree kernels for relation extraction. In Proceedingsof the ACL-2004.

Dmitry Davidov, Ari Rappoport and Moshe Koppel.2007. Fully unsupervised discovery of concept-specific relationships by Web mining. In Proceed-ings of ACL-2007.

Dmitry Davidov and Ari Rappoport. 2008. Classifi-cation of Semantic Relationships between NominalsUsing Pattern Clusters. In Proceedings of ACL-2008.

Wei Fan, Kun Zhang, Hong Cheng, Jing Gao, XifengYan, Jiawei Han, Philip S. Yu and Olivier Ver-scheure. 2008. Direct Mining of Discriminative andEssential Frequent Patterns via Model-based SearchTree. In Proceedings of KDD-2008.

Evgeniy Gabrilovich and Shaul Markovitch. 2006.Overcoming the brittleness bottleneck usingwikipedia: Enhancing text categorization withencyclopedic knowledge. In Proceedings ofAAAI-2006.

Jim Giles. 2005. Internet encyclopaedias go head tohead. Nature 438:900C901.

Sanda Harabagiu, Cosmin Adrian Bejan and PaulMorarescu. 2005. Shallow semantics for relationextraction. In Proceedings of IJCAI-2005.

Takaaki Hasegawa, Satoshi Sekine and Ralph Grish-man. 2004. Discovering Relations among NamedEntities from Large Corpora. In Proceedings ofACL-2004.

Nanda Kambhatla. 2004. Combining lexical, syntacticand semantic features with maximum entropy mod-els. In Proceedings of ACL-2004.

Dat P.T. Nguyen, Yutaka Matsuo and Mitsuru Ishizuka.2007. Relation extraction from Wikipedia using sub-tree mining. In Proceedings of AAAI-2007.

Patrick Pantel and Marco Pennacchiotti. 2006.Espresso: Leveraging generic patterns for automat-ically harvesting semantic relations. In Proceedingsof ACL-2006.

Benjamin Rosenfeld and Ronen Feldman. 2006.URES: an Unsupervised Web Relation ExtractionSystem. In Proceedings of COLING/ACL-2006.

Benjamin Rosenfeld and Ronen Feldman. 2007. Clus-tering for Unsupervised Relation Identification. InProceedings of CIKM-2007.

Peter D. Turney. 2006. Expressing implicit seman-tic relations without supervision. In Proceedings ofACL-2006.

Max Volkel, Markus Krotzsch, Denny Vrandecic,Heiko Haller and Rudi Studer. 2006. Semanticwikipedia. In Proceedings of WWW-2006.

Mohammed J. Zaki. 2002. Efficiently mining frequenttrees in a forest. In Proceedings of SIGKDD-2002.

Min Zhang, Jie Zhang, Jian Su and Guodong Zhou.2006. A Composite Kernel to Extract Relations be-tween Entities with both Flat and Structured Fea-tures. In Proceedings of ACL-2006.

1029