patentminer: topic-driven patent analysis and...
TRANSCRIPT
PatentMiner: Topic-driven Patent Analysis and Mining
Jie Tang†, Bo Wang†, Yang Yang†, Po Hu†, Yanting Zhao†, Xinyu Yan†, Bo Gao†,Minlie Huang†, Peng Xu�, Weichang Li�, and Adam K. Usadi�
†Department of Computer Science and Technology, Tsinghua University, China�ExxonMobil Research and Engineering Company, New Jersey, USA
ABSTRACTPatenting is one of the most important ways to protect com-pany’s core business concepts and proprietary technologies.Analyzing large volume of patent data can uncover the po-tential competitive or collaborative relations among com-panies in certain areas, which can provide valuable infor-mation to develop strategies for intellectual property (IP),R&D, and marketing. In this paper, we present a noveltopic-driven patent analysis and mining system. Instead ofmerely searching over patent content, we focus on studyingthe heterogeneous patent network derived from the patentdatabase, which is represented by several types of objects(companies, inventors, and technical content) jointly evolv-ing over time. We design and implement a general topic-driven framework for analyzing and mining the heteroge-neous patent network. Specifically, we propose a dynamicprobabilistic model to characterize the topical evolution ofthese objects within the patent network. Based on this mod-eling framework, we derive several patent analytics toolsthat can be directly used for IP and R&D strategy plan-ning, including a heterogeneous network co-ranking method,a topic-level competitor evolution analysis algorithm, and amethod to summarize the search results. We evaluate theproposed methods on a real-world patent database. The ex-perimental results show that the proposed techniques clearlyoutperform the corresponding baseline methods.
Categories and Subject DescriptorsH.3.3 [Information Search and Retrieval]: Text Mining;H.2.8 [Database Management]: Database Applications
General TermsAlgorithms, Experimentation
KeywordsPatent analysis, Competitor analysis, Company ranking, So-cial network
1. INTRODUCTIONPatenting becomes one of the most important ways to pro-
tect company’s core business concepts and proprietary tech-nologies. Before a company entering the market of a newfield, a comprehensive patent landscape overview is alwaysnecessary, lest be faced with a flurry of third-party licens-ing opportunities or even lawsuits. Nowadays, the collec-tion and retrieval of patent publications is a critical compo-nent of a company’s intellectual property strategy. Withinmany companies (or organizations), patent analysts have theresponsibility to determine how patent information is bestmade available and how it is best used within their organiza-tions in a strategic manner. From the technical perspective,patent, as one of the few real indicators of future product re-leases, carries out the technical details of research of differentcompanies long before the product reaches the marketplace.Keeping being aware of novel technology development andcompetitors’ technological advancement also becomes moreand more important for a company to make the decision onmarketing and R&D strategies.
However, patent analysts in the 21st century now facemany challenges. They must search over the huge volume ofpatents to find relevant patents, to recognize potential com-petitors (or collaborators), and to identify inventors withsignificant impact. Despite a great deal of theoretical de-velopment in information retrieval and data mining tech-niques, advanced search tools for patent professionals arestill in their infancy. Most existing patent analysis systemssuch as Google Patent1, WikiPatent2, FreePatentsOnline3
only focus on the search function. A few other systemssuch as Patents4, PatentLens5, and PriorArtSearch6 providemore advanced analysis and mining capabilities. For exam-ple, the Patents system uses iBoogie to cluster the retrievedpatents. It also provides a forum for people to discuss dif-ferent patenting related problems. PatentLens provides afunction called Patent Landscapes, which lists a number of“White Papers” discussing technologies relevant to life sci-entists. However, the Technology Landscapes require exten-sive searching and analytical work by people skilled in bothscience and intellectual property, thus infeasible to scale upto various topics.
1http://www.google.com/patents2http://www.wikipatents.com/3http://www.freepatentsonline.com/4http://www.patents.com/5http://www.patentlens.net/6http://www.priorartsearch.com/
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. KDD’12, August 12–16, 2012, Beijing, China. Copyright 2012 ACM 978-1-4503-1462-6/12/08... $15.00.
1366
This paper reports the development of PatentMiner7,a novel topic-driven patent analysis and mining system.PatentMiner is designed for an in-depth analysis of patentactivity at the topic-level. The main unique characteristicsof the PatentMiner system that distinguish it from tradi-tional patent search systems are as follows: (1) topic-drivenmodeling; (2) heterogeneous network co-ranking; (3) intelli-gent competitive analysis; and (4) patent summarization.
1. Topic-driven modeling. The fundamental problem inmost existing systems is that all patents are simplymodeled based on keywords. In this work, we presenta probabilistic model to simultaneously model the top-ical aspects of different objects in the heterogeneouspatent network.
2. Heterogeneous network co-ranking. When a companyplans to enter a new market or brainstorm novel ideas,several typical questions are: what are the most activecompanies in this area? what are the most relevantpatents? and who are the most prolific inventors? Wepropose a heterogeneous co-ranking algorithm to ad-dress these questions.
3. Competitive analysis. It would be very helpful for acompany to make right business strategies by identify-ing who are its competitors and what is the trend of acompetitor’s technology development. We define fourmeasures to identify competitors, topic-level competi-tors, and competitors’ evolutionary pattern based onthe topic modeling results.
4. Patent summarization. It is always expensive to di-gest the large number of patents returned by a searchengine. We present a maximum coverage method toautomatically generate a summary for the results ofpatent search.
We conducted empirical evaluations of the proposed meth-ods. Experimental results show that our methods clearlyoutperform the baseline methods for addressing the aboveissues. Our technical contributions in this paper include: (1)a proposal of a probabilistic topic modeling approach, (2) aproposal of a heterogeneous co-ranking method for searchover the patent network, (3) a proposal of a topic-level com-petitive evolution analysis approach, and (4) a proposal ofa maximum coverage model for patent summarization.
2. OVERVIEWFigure 1 shows the architecture of the system. The system
consists of five major components:
1. Patent Network Extraction: The patent data containshuge amount of information. From the patent data,we derive a heterogeneous information network con-sisting of different types of objects such as companies,inventors, and patents.
2. Patent Network Storage: It provides storage andindexing for the extracted patent networking data.Specifically, for storage it employs MySQL; for index-ing, it employs the inverted file indexing method [20].
7http://pminer.org
Patent
USPTO
Patent
Japan
Patent
Germany
DataSources
IntegrationPatentNetworkExtraction
Object extraction
Patent
Great Britain
PatentNetworkStorage
StorageIndexing
Access interface
KnowledgeBase Metadata
Heterogeneous co-ranking
Patent summarization
Map-reduce
Distributed
Platform
Top companiesfinding
Prolific inventorsfinding
Influentialpatents finding
Patent
Canada
Data cleaning and processing
Probabilistic Topic Modeling
Competitive Analysis
Evolutionary competitor discoveryCompetitor discovery
Analysisand Mining
Concept extractionSummary generation
Figure 1: Architecture of PatentMiner.
3. Probabilistic Topic Modeling : It utilizes a generativeprobabilistic model to simultaneously model the dif-ferent types of objects. After modeling, each object isassociated with a topic distribution. The topic model-ing is the basis for the analysis and mining component.
4. Analysis and Mining : This is the most important com-ponent in the PatentMiner system, and our majortechnical contribution also lies in this component. Insummary, it provides three functions: heterogeneousco-ranking, competitive evolution analysis, and patentsummarization.
5. Distributed Platform: The back-end system is built onthe Map-reduce distributed platform [6], a program-ming platform for distributed processing of large datasets. For some analysis and mining tasks (such as topicmodel and competitive analysis), we implement thedistributed version of the proposed algorithms.
At present, we maintain an English patent database ex-tracted from USPTO.gov, which consists of nearly 4,000,000patents, 2,000,000 inventors, and 400,000 companies. Thesystem is very flexible and can be easily extended to multi-ple different sources.
Preliminaries Assume that a patent d contains a vec-tor wd of Nd words, in which each word wdi is chosenfrom a vocabulary of size V , and the patent d is devel-oped by a group of inventors ad and is owned by companycd, then a collection of M patents can be represented asD = {(a1, c1,w1), . . . , (aM , cM ,wM )}. Given a collection ofpatents D, we extract the inventor and company informa-tion from each patent, and derive a heterogeneous patentnetwork.
Definition 1. Heterogenous Patent Network. Theheterogeneous patent network can be represented as a graphcentered by the patent G = (Vd ∪ Va ∪ Vc, Eda ∪Edc ∪Edd′ ∪
1367
Table 1: Notations.Symbol Description
M Number of patentsW Number of unique words in patentsK Number of topicsA Number of inventorsC Number of companiesNd Number of words in patent d
d, c, a A patent, a company, and an inventor respectivelywdi The i-th word in patent dxdi The chosen inventor to be responsible for word wdi
zdi The topic assigned to the i-th word in patent dθa Multinomial on topics specific to inventor aφz Multinomial on words specific to topic zψc Multinomial on topics specific to company c
α, β, μ Dirichlet priors to the multinomials θ, φ, and ψ
Eac), where Vd includes all patents, Va includes all inven-tors, Vc includes all companies, and the edge (vd, va) ∈ Eda
(or briefly eda) suggests that there is a relationship betweenpatent vd and inventor va. Similarly we can define the otherrelationships edc, edd′ , and eac.
The heterogeneous patent network is comprised of threedifferent types of objects: inventors, companies, and patents.Each object may be associated with different topics. Forexample, a patent may talk about “web search” and “datamining”. The goal of topic modeling over the patentnetwork is to discover the latent topics associated witheach object. Each topic z is defined as a mixture ofwords and their probabilities belonging to the topic, i.e.,{(w1, P (w1|z)), · · · , (wN1, P (wN1|z))}. The definition canbe extended to other information sources. For exam-ple, we can extend the topic definition by companies, i.e.,{(c1, P (c1|z)), · · · , (cN1, P (cN1|z))}. After topic modeling,each object would be associated with a topic distribution,e.g., an inventor a is associated with {P (z|a)}z. By furtherconsidering the time information, a patent network can besegmented into a network sequence GS = (G1, G2, · · · , GT )according to the time-stamp associated with each object.Each sub network Gt in the sequence is comprised of ob-jects in the time window t, e.g., patents published at timet. In this work, the time-stamp is defined as the publishedyear of each patent. Table 1 lists the major notations.
3. MODELING PATENT NETWORKSeveral topic models have been proposed and successfully
applied to text mining tasks [1, 4, 9, 13, 18]. However, thesemodels can only model patent contents and are not able toincorporate the company and inventor information, whichcontains valuable information for topic modeling. More-over, they cannot model the time information. In this sec-tion, we will first present an Inventor-Company-Topic modelwhich leverages interdependencies between different objectsto learn the topic model, and then describe an extension ofthe model by combining time information.
3.1 Inventor-Company-Topic (ICT) ModelWe present an Inventor-Company-Topic (ICT) model,
which incorporates companies, inventors, and patents intoa unified probabilistic model. The basic idea is to describepatent writing in a generative process. The generative pro-cess can be described as follows: when preparing a patent
d, each inventor x ∈ ad would suggest what topics to beincluded in patent according to his expertise (the associ-ated topic distribution P (z|θx) or θxz); then the word wdi
is sampled from a suggested topic zdi by the inventor ac-cording to P (wdi|zdi) or φzdiwdi . In the generative process,all the suggested topics specific to the patent d are relevantto the own company cd. Aggregating topics of all patentsowned by company cd could constitute a topic distributionof the company P (z|ψc) or ψcz. Formally, we use a uniformdistribution to associate each patent with its inventors, usea multinomial distribution (with a prior α) to associate eachinventor with the topics, and use a similar multinomial dis-tribution (with a prior β) to associate each topic with words.The company-topic distribution is represented as a mixtureof topic distribution extracted from each patent (again canbe considered as a multinomial distribution with a differentprior μ). Finally, given a collection of patents D, we couldwrite its generative log-likelihood as:
L(D) =P (x, z,w, c|Θ,Φ,Ψ,a) =M∏d=1
Nd∏i=1
1
Ad×
K∏z=1
⎛⎝
A∏x=1
θmxzxz
W∏j=1
φnzwjzwj
C∏c=1
ψnzczc
⎞⎠ (1)
wheremxz is the number of times that topic z was associatedwith inventor x, nzwj is the number of times that word wj
is generated by topic z, nzc is the number of times thatcompany c is generated by topic z, Nd is the number ofwords in patent d, and Ad is the number of inventors forpatent d.
Learning the ICT model is to estimate the unknown pa-rameters in the ICT model. There are two sets of unknownparameters: (1) the distribution θ of A inventor-topics, thedistribution φ of K topic-words, and the distribution ψ ofK topic-companies; and (2) the corresponding topic zdi andinventor xdi for each word wdi. It is usually intractable todo exact inference in such a probabilistic model. A vari-ety of algorithms have been proposed to conduct approxi-mate inference, for example variational EM methods [1] andGibbs sampling [7]. We chose Gibbs sampling for its easeof implementation. Specifically, we calculate the posteriordistribution on z and then sample the topic for each word.Based on the sampling results, we could infer the distribu-tion θ, φ, and ψ. For the hyperparameters α, β, and μ, forsimplicity, we take fixed values (i.e., α = 50/K, β = 0.01,and μ = 0.01.
3.2 Dynamic ICT ModelThough the ICT model incorporates companies, inventors,
and patents together, it still cannot capture the temporal in-formation. We therefore propose a dynamic extension of theICT model. The general idea is that topic distribution of anobject (e.g., company) in adjacent time-stamps (e.g., twocontinuous years) should be similar. At each time-stamp,an ICT model is built, i.e., each object is associated witha topic distribution which will be used as a prior for theobject in the next time-stamp. The prior has a smoothingeffect to make the discovered topic models between adjacenttime-stamps similar to each other. The new model is re-ferred to as Dynamic Inventor-Company-Topic (DICT). Tosummarize, there are three smoothing requirements for thedynamic modeling:
• Inventor-topic smoothing. The topic distribution of
1368
an inventor should be smooth over time, aka Ω1 =∑z(θ
taz − θt−1
az )2 should be small, where θtaz indicatesthe probability of topic z given inventor a at time t.
• Company-topic smoothing. The topic distribution ofa company should be smooth over time, aka Ω2 =∑
z(ψtcz − ψt−1
cz )2 should be small.
• Topic smoothing. The topic distribution itself shouldbe smooth over time, aka Ω3 =
∑z(P (z)t −P (z)t−1)2
should be small.
Now, the problem is how to combine the three smooth-ing hypothesis into the ICT model. One strategy is to usea regularization framework to describe the three smoothingfactors as three regularization terms, and plug into the orig-inal log-likelihood function, thus we obtain a new objectivefunction:
O(D) = −L(D) + γ1Ω1 + γ2Ω2 + γ3Ω3 (2)
where γ1, γ2, and γ3 are three parameters to balance theimportances of different smoothing factors.
It is intractable to solve the new objective function. In ex-isting literatures, there are a few attempts to deal with theconstrained regularization framework using approximationalgorithms such as [24] and [11]. However, these methodscannot guarantee a convergence. In this paper, we consideran alternative method to solve the problem. Instead of plug-ging the regularization terms into the log-likelihood func-tion, we use the learned topic model of previous time-stampas the prior for the topic model of the current time-stamp.Specifically, we use the Gaussian distribution as the priordistribution, e.g. αt|αt−1 ∼ N (αt−1, δ
2I). In principle, wecan also consider other prior distributions, such as Dirichletdistribution or Gamma distribution. We tested a few dis-tributions and found that the Gaussian distribution has thebest performance. With a prior distribution, we can incor-porate the smoothing effect directly into the probabilisticgenerative process (as summarized in Algorithm 1).
To learn the Dynamic ICT model, we can still use theGibbs sampling algorithm for parameter estimation of theDICT model. In an analogous way, we first estimate theposterior probability of sampling the topic ztdi for each wordwt
di with time-stamp t, and then use the sampling resultsto infer θt, φt, and ψt. Specifically, with the learned topicmodel at time t, we can estimate the probability of a topicgiven an inventor θtxz, the probability of a word given atopic φt
zv, and the probability of a company given a topicψt
zc respectively by (Derivation is omitted for brevity.):
θtxz =mt
xz + αtz + τ(mt−1
xz + αt−1z )∑
z′ (mtx,z′ + αt
z′ ) + τ∑
z′ (mt−1xz′ + αt−1
z′ )(3)
φtzw =ntzw + βt
w + τ(nt−1zw + βt−1
w )∑w′ (nt
zw′ + βtw′ ) + τ
∑w′ (n
t−1zw′ + βt−1
w′ )(4)
ψtzc =
ntzc + μtc + τ(nt−1
zc + μt−1c )∑
c′ (ntzc′ + μt
c′ ) + τ∑
c′ (nt−1zc′ + μt−1
c′ )(5)
where τ is a parameter to control the influence of the topicmodel of the previous time on the topic model of to thecurrent time.
Initialize α0 = 50/K, β0 = 0.01, and μ0 = 0.01;foreach time-stamp t do
Draw αt|αt−1 ∼ N (αt−1, δ2I);Draw βt|βt−1 ∼ N (βt−1, σ2I);Draw μt|μt−1 ∼ N (μt−1, ε2I);For each topic zt, draw φtz and ψt
z respectively fromDirichlet prior βt and μt;foreach word wdi in patent d do
Draw an inventor xdi from ad uniformly;Draw a topic ztdi from a multinomial distribution
θtxdispecific to inventor xdi, where θt is generated
from the Dirichlet prior αt;Draw a word wt
di from multinomial φtzdi ;
Draw a company stamp ctdi from multinomial ψtzdi
;
endend
Algorithm 1: Probabilistic generative process in DICT.
4. HETEROGENEOUS CO-RANKINGThe unique requirements of searching over the heteroge-
neous patent network give rises to several challenging is-sues and make them different from general search engines.First, the information seeking practice [8] is not only aboutpatents, but also about other information sources, such ascompanies and inventors. In this spirit, a good patent searchengine should provide supports not only for patents, butalso for these information sources. Second, search over thepatent network typically requires much higher retrieval ac-curacy. Given a query, such as “data mining”, a user doesnot mean to find patents merely containing these two words.Her/his intention is to find patents on the data mining topic.Finally, these two issues are often intertwined.
Formally, given a heterogeneous patent network G =(V,E), and a query q = {w1, · · · , wn}, our objective is toleverage the power of both textual content (patent content)and the network information (relationships between differ-ent types of objects) in the patent network to obtain accu-rate ranking results for companies, inventors, and patents.
To deal with the ranking problem over the patent network,a straightforward method is to first represent each objectwith a bag of words, and then calculate the relevance scoreof each object with a given query q by using methods suchas language model [22] or vector space model [20]. We uselanguage model as the example to explain how to calculatethe relevance score. Language model is one of the state-of-the-art approaches for information retrieval. It interpretsthe relevance between a document and a query word as agenerative probability:
PLM (w|d) = Nd
Nd + λ· tf(w, d)
Nd+ (1− Nd
Nd + λ) · tf(w,D)
ND(6)
where Nd is the number of words in document (patent)d, tf(w, d) is the word frequency (i.e., occurring number)of word w in d, ND is the number of words in the entirecollection, and tf(w,D) is the word frequency of word win the collection D. λ is the Dirichlet smoothing factorand is commonly set according to the average documentlength in the collection [22]. Further, the probability of thedocument model d generating a query q can be defined asP (q|d) = Πw∈qP (w|d). For companies and inventors, wefirst combine all patents associated with each object andcreate a virtual document, and then use a similar formula
1369
to calculate the relevance score of company P (q|c) and in-ventor P (q|a).However, such a method is only based on keyword match-
ing and cannot leverage the topic modeling information. Totake advantage of the topic modeling results, we define an-other relevance score:
PICT (w|d, θ, φ) =K∑
z=1
Ad∑x=1
P (w|z)P (z|x)P (x|d) (7)
where P (w|z) = φzw, P (z|x) = θxz, and P (x|d) = 1Ad
. By
combining the two relevance scores, we have:
P (w|d) = PLM (w|d)× PICT (w|d) (8)
In practice, the learned topics by the topic model is usu-ally general to a given query while the language model is spe-cific to keywords in the query. Combining the two relevancescores achieves a balance between generality and specificity,thus could improve the ranking performance. However, thismethod still does not consider the network information. Wetherefore propose a heterogeneous co-ranking method to ad-dress this issue.
The basic idea of the heterogenous co-ranking method is topropagate the relevance score between the linked objects inthe network. The intuition behind the method is as follows:(a) a patent applied by inventors with higher expertise de-grees on a topic (query) is more likely to have a higher qual-ity (or impact); (b) a company who owns many high qualitypatents on a topic is more likely to be ranked higher; (c) aninventor who applies many patents with high impacts shouldbe ranked higher. Similar strategies have been also consid-ered in [23, 14] for academic search. Based on this intuition,we propose a two-stage method. In the first stage, we use Eq.8 to calculate the relevance score of each object to the givenquery q and select the top-ranked objects as candidates. Inthe second stage, we use the candidates to construct a het-erogeneous subgraph and perform a score propagation onthe subgraph. Finally, we use the new score to re-rank eachtype of objects. In the propagation, we calculate the newscore by (here we use company as the example):
rk[c] = (1−ξ1−ξ2)rk−1[c]+ξ1
|V ca |
∑d∈V c
a
rk−1[a]+ξ2
|V cd |
∑d∈V c
d
rk−1[d]
where rk[c] is the ranking score of company c after the k-steppropagation; the score is initialized by Eq. 8; V c
a denotes aset of inventors related to company c and V c
d denotes a setof patents owned by company c; ξ1 and ξ2 are two parame-ters to control the propagation. The number of propagationsteps reflects how we trust the network information. Settingk = 0 indicates that we only use the content information,thus the method degrades to Eq. 8; while setting k = ∞indicates that we only trust the network information, thusthe algorithm obtains a result similar to that of PageRank[12] on the heterogenous network.
5. COMPETITIVE ANALYSISThis component aims to quantitatively characterize the
competitive relations between companies. Based on themodeling results of the DICT model, we define four mea-sures to quantify the competitive relations.
Global competitor discovery If two companies competein their major areas, we call these two companies globalcompetitors of each other. For example, ExxonMobil andShell Oil are two global competitors: they are two energycompanies, competing in“oil exploration”, “oil refinery”, and“chemical”. Given a company c, we define the following mea-sures to rank its global competitors:
• Word-based similarity (WBS). It represents each com-pany by a vector of words, and ranks the competitorsbased on (Cosine) similarity between company c andeach candidate.
• Topic-based divergence (TBD). It represents each com-pany using the topic distribution {P (z|c)}z (or ψc),and ranks the competitors by the KL-divergence be-tween company c and each candidate c′, i.e.,
KL(ψc||ψc′ ) =K∑i=1
ψczi logψczi
ψc′zi.
• Probability-based correlation (PBC). It defines a cor-relation score to rank the candidate companies. For acompany c and a candidate competitor c′, the score isdefined as:
S(c, c′) =K∑i=1
p(zi|c)p(zi|c′) + η(ln(Mc)− ln(Mc′ ))2
where Mc is the number of patents by company c; ηis balance parameter; P (zi|c) can be obtained usingBayes rule based on ψzic. The second term is used toavoid noise.
Topic-level competitor discovery Two companies mayonly compete in one or a few specific areas. For example,Apple and Amazon may not be global competitors, but theycompete fiercely on“Tablet PC”. Given a company c, we aimto find its competitors on a specific topic z. The simplestway is to utilize the topic distribution associated with eachcompany. For two companies c and c′, if |P (c|z)−P (c′|z)| ≤τ , then we say that c and c′ are competitors on topic z.The method is referred to as distribution-based competitorfinding (DBC). However, this method ignores the correlationbetween topics. For example, companies who compete on“data mining” may also compete on “web search”, as thetwo topics have a strong correlation with each other. Toincorporate the topic correlation, we define a hybrid measure(HBC) as follows:
S(c, c′, z) = (ψzc − ψz′c)2 + η
∑z′ �=z
ρzz′ (ψz′c − ψz′c′ )2
where ρzz′ is the correlation between topic z and topic z′,could be calculated as the negative Kullback-Leibler diver-gence [22]: ρzz′ = −KL(φz||φ′
z).Evolutionary competitor discovery Companies maychange their IP and marketing strategies, thus the competi-tive relations between them would change over time as well.Based on the dynamic ICT model, we define the competitivedegree between two companies c and c′ at time t as:
S(c, c′, t) =t∑
t′=1
πtt′K∑i=1
p(c|zi, t′)p(c′|zi, t′)+η(ln(Mtc)−ln(Mt
c′ ))2
1370
where P (zi|c) can be obtained using Bayes rule based on
ψtzic. We use the parameter πtt′ (defined as e−|t−t′|) to
model the historic information in the above equation, whichmeans if two companies are competitors in a recent pasttime, it is also likely that they are competitors in the currenttime.
We can further extend the evolutionary competitor discov-ery to the topic level. The basic idea is to replace the innersummation in the above equation with the specific topic.
6. PATENT SUMMARIZATIONWhen a user performs a search in the patent mining sys-
tem, a large number of objects (patents, companies, andinventors) may be returned. It is always expensive for theuser to digest the large volume information. It is desirablethat the system can automatically generate a concise andinformative summary for the returned objects, so that theuser can quickly grasp a global picture before ‘click-and-view’ each object. A high-quality summary should satisfythe following requirements: (1) cover the most importantinformation in the returned objects, (2) be relevant to thequery as well, and (3) minimize the redundancy in the gen-erated summary.
To solve the patent summarization problem, we proposea maximum coverage method. The idea is to choose a set ofrepresentative sentences as the summary from the returnedobjects for a query. The method consists of three majorsteps. First, when the user issues a query, it retrieves rele-vant patents for the query. Second, it extracts concepts fromeach sentence in the patents. All the extracted conceptsform the knowledge space for the query and each sentence isalso represented by the extracted concepts. Finally, it em-ploys integer linear programming to find a set of sentenceswhose concepts maximally cover the knowledge space.Concept extraction In our maximum coverage method,concept is the basic unit to represent the knowledge. Aconcept can be a word, a phrase, a named entity (e.g.,Person or Location), or even a parsed syntax subtree ofa sentence. Each concept has an importance score to thequery. The schemes of concept scoring vary from simpleterm frequency to sophisticated machine learning methods.In our method, candidate concepts are selected using bi-grams and are weighted using TF ∗ IDF . Specifically, allpatents are preprocessed by (1) sentence splitting, (2) part-of-speech tagging, (3) stemming and phrase chunking. Thenwe collect all extracted noun phrases (obtained by phrasechunking) from the patents and further split them into bi-grams as candidate concepts. The next step is to score eachcandidate concept. In particular, we segment each patentinto five fields: Title, Abstract, Claim, Background, andOther. Each field has a weight to reflect its importance (weempirically set the field-weights as 2, 1.5, 1.5, 1.0 and 0.5respectively). Then, the importance score of each candi-date concept is defined as the sum of weighted frequencies,i.e., filed-weight×concept-frequency-in-the-field. Finally, wechoose the top 50 candidate concepts with the highest scoresto form the knowledge space of the given query.Summary generation An ideal summary should coveras many important and diverse concepts as possible. Theproblem can be formulated as a 0-1 knapsack problem. For-mally, it aims to maximize the total importance score of
concepts that form the knowledge space. In addition, weneed to consider several constraints. The first one is thesummary length: a user would not want to read a long sum-mary. Thus, we define a maximal number of words in theextracted summary, i.e., MaxLength. Another constraint isto maintain the logical consistency between variables. Forexample, if the summary contains an important concept,then it at least includes one sentence which contains thisconcept. By combining the objective and the constraintstogether, we define the following constrained optimizationproblem:
∑
i
ISi ∗ Ci
s.t.∑
j
Lj ∗ Sj ≤ MaxLength
Sj ∗ Occi,j ≤ Ci ∀i, j;∑
j
Sj ∗ Occi,j ≥ Ci ∀i
(9)
where Ci is an indicator to represent whether the i-th con-cept is included in the summary (Ci = 1) or not (Ci = 0),and ISi is the importance score of the i-th concept. In thefirst constraint, Lj is the number of words in the j-th sen-tence, and Si is an indicator of whether the j-th sentence isincluded in the summary. In the second and the third con-straints, Occi,j is an indicator of whether the i-th conceptoccurs in the j-th sentence.
We solve the above optimization problem using integerlinear programming (ILP). Specifically, we employ an opensource software, LP-Solve (http://lpsolve.sourceforge.net/).The obtained result is an optimal solution that maximizesthe objective function. Finally, we construct the summaryby selecting sentences with Sj = 1.
7. EXPERIMENTAL RESULTSWe evaluated the proposed methods in the context of
the PatentMiner system7, consisting of 3,880,211 patents,2,134,211 inventors, and 421,032 companies. We conductedthree experiments to evaluate the proposed methods: het-erogeneous co-ranking, competitor analysis, and patent sum-marization.
7.1 Results on Heterogeneous Co-RankingData Sets, Evaluation Measures, and Baselines Toqualitatively evaluate the proposed methods and comparewith existing methods, we collected a list of 50 popularqueries (e.g., “data mining”, “web search”). For each query,we independently request five annotators (two undergrad-uates, two PhD students, and one faculty) to provide hu-man judgement on the top (20) returned objects (compa-nies, patents, and inventors) by the system. The judgementis about relevant (Like) or irrelevant (DisLike). If there aremore than two annotators saying that an object is irrelevant,we remove the object from the returned list. As it is reallydifficult to judge the expertise for inventors even for human,we only perform the evaluation for companies and patents.We conducted the evaluation in terms of P@N (Precision fortop N results), mean average precision (MAP), and normal-ized discounted cumulative gain (NDCG) [2, 5].
We used language model (LM) as the baseline method.For language model, we used Eq. 6 to calculate the relevancebetween a query term and a patent and similar equations foran inventor/company, where an inventor is represented by
1371
Table 2: Average ranking performance for thepatents and companies. N@1 and N@5 indicate NDCG
for the top 1 or 5 results. HCR-1 indicates our proposed
HCR method with one step propagation.
Object Method P@1 P@5 MAP N@1 N@5
Patent
LM .7001 .6900 .6991 .7021 .6833
HCR-1 .7592 .7102 .7359 .7592 .7310
HCR-2 .7598 .7201 .7361 .7600 .7300
HCR-5 .7600 .7298 .7400 .7678 .7367
Company
LM .6931 .6790 .6654 .6888 .6532
HCR-1 .7167 .6833 .7058 .7167 .6934
HCR-2 .7189 .6900 .7100 .7200 .7000
HCR-5 .7201 .6999 .7210 .7201 .7031
his/her published patents and a company is represented byits held patents. Our heterogeneous co-ranking method isreferred to as HCR. We tried different settings for the propa-gation. For example, HCR-2 indicates the heterogeneous co-ranking method with 2-step propagation. We preprocessedeach patent by (a) removing stopwords and numbers; (b)removing words that appear less than three times in the cor-pus; and (c) downcasing the obtained words. Moreover, weperformed company name disambiguation (e.g., IBM versusIBM Corp.) based on a company name dictionary.Results and Analysis Table 2 shows the ranking perfor-mances for patents and companies. We see that the pro-posed HCR method (with all settings) clearly outperformsthe baseline method using language model. In terms ofMAP, the improvement of HCR-5 over language model is5.3% for patent ranking and 5.01% for company ranking.Our method benefits from the topic modeling results whichconsider the companies, patents, and inventors in a unifiedmodel while the language model can only use the content in-formation. In addition, the propagation process in our HCRmethods further leverages the network information.
We studied how the number of propagation step influencesthe ranking performance. Figure 2 shows the ranking per-formance in terms of MAP with varied propagation step.Increasing the propagation step from 1 to 5 results in im-proved performance. This is because that our methods in-tegrated more network information with more propagationsteps. However, continuing to increase the propagation stepmake the network information an dominate factor for thefinal ranking results, thus hurts the relevance performance.
LM HCR−1 HCR−2 HCR−5 HCR−10 HCR−200.66
0.68
0.7
0.72
0.74
0.76
MA
P
PatentCompany
Figure 2: Ranking performance with varied propa-gation step.
7.2 Results on Competitive AnalysisData Sets, Evaluation Measures, and Baselines Itis difficult to obtain the ground truth for evaluating ourmethod’s performance on competitor analysis. For a rel-atively fair comparison, we have obtained the competitorinformation from Yahoo! Finance8, which hosts informa-tion of all companies listed on NASDAQ, Dow Jones, S&P,etc. For each company, it gives a list of competitors basedon their business performance (such as revenue and employ-ees). It also provides the topic information for the competi-tors, such as Microsoft and Oracle compete on “Software”.In summary, we obtained 543 companies and their globalcompetitor information, and 326 companies and their com-petitors on 18 different topics.
We used P@N, MAP, and NDCG to evaluate our method(referred to as TopCom), and to compare with two baselinemethods. The first one (referred to as WBS) is to representeach company as a bag of words (by the vector space model[20]), and for a given company, its competitors are rankedaccording to the Cosine similarity of the company with eachcandidate. This method can identify the global but nottopic-level competitors. The second baseline method is fortopic-level competitors analysis. We used Latent DirichletAllocation (LDA) [1] to generate the topic-word distributionand then combine the language model (LM) for competitordiscovery. Specifically, we used all the patent titles to learnthe LDA model, and used language model to find the rel-evant patents for a given query. Finally we obtained thecompetitive companies by summing all the scores of theircorresponding patents. Hereafter we will use “LM+LDA” todenote this method. In addition, we compared our methodswith different scoring measures (Cf. Section 5):
• TopCom+TBD: It ranks the competitors by KL-divergence of topic distribution between each candi-date and the given company.
• TopCom+PBC: It ranks the competitors by theprobability-based correlation between each candidateand the given company.
• TopCom+DBC: It ranks the competitors by the dif-ference of its distribution in a specific topic betweeneach candidate and the given company.
• TopCom+HBC: It ranks the competitors by thehybrid-based score between each candidate and thegiven company.
Results and Analysis Table 3 shows the performance ofdifferent methods on global and topic-level competitor anal-ysis. We see that our method (TopCom) outperforms thebaseline methods. For the global competitor analysis, ourmethod with the probability-based correlation (PBC) scor-ing measure achieves the best performance in terms of P@1,P@5, N@1, and N@5. For the topic-level competitor analy-sis, the best performance is obtained by our method with thehybrid-based (HBC) scoring measure, which indicates that ascoring measure leveraging both topic and patent content in-formation can achieve a better performance than using onlyone of the information. The advantage of our method lies inthat in the proposed DICT model, we simultaneously modelthe topic distribution of companies, inventors, and patents;
8http://finance.yahoo.com/
1372
Table 3: Performance of competitor analysis. N@1
and N@5 indicate NDCG for the top 1 or 5 returned
competitors.
Methods P@1 P@5 MAP N@1 N@5
Global
WBS .2009 .1087 .2904 .2009 .2841
TopCom+TBD .1731 .0846 .3078 .1731 .2871
TopCom+PBC .2098 .1161 .2920 .2098 .3085
Topic
LM+LDA .1536 .1221 .2643 .1536 .2524
TopCom+DBC .1369 .1270 .2388 .1469 .2446
TopCom+HBC .1620 .1366 .2781 .1620 .2874
Table 4: Examples for topic-level competitor evolu-tion.
Cisco (Network Device) AT&T Corp. (Communication)
1996-2000 2006-2010 1996-2000 2001-2005 2006-2010IBM 3Com Lucent Lucent Lucent
Microsoft Juniper IBM NEC NECLucent Broadcom NEC Motorola IBM
AT&T Corp. Nortel Verizon IBM BellIntel Intel Microsoft Broadcom FujitsuSun Canon Samsung Intel Samsung3Com IBM Motorola Microsoft MotorolaDEC Fujitsu Ericsson Cisco VerizonHP Sony Alcatel Samsung AOL
while the two baseline methods only model the content in-formation (WBS) or only model the topic distribution ofpatents (LM+LDA).Case Study Table 4 shows example results of the topic-level competitor evolution analysis by our TopCom+PBCmethod. From the example, we have observed some inter-esting patterns. On topic “Network Devices”, Cisco’s early(1996-2000) competitors include IBM, Microsoft, Lucence,etc., but now it turns out to be 3Com, Juniper, and Broad-com, etc. Juniper seems to be a rising star in the “NetworkDevice” field; while a few other companies (such as AT&Tand Lucent) have passed their bloom on “Network Device”since 2001. Instead, AT&T and Lucent have become morefocused in the “Communication” field.
7.3 Results on Patent SummarizationData Sets, Evaluation Metrics, and Baselines Thepatent summarization method was first tested on bench-mark data sets TAC 2008 and 2009 9 before being appliedto patent data. TAC 2008 and 2009 datasets respectivelycontain 48 and 44 topics (queries). The document collectionfor each topic is given. The task is, similar to our problem,to generate a 100-word summary from the document collec-tion for each topic (query). Four human-written summariesare used as gold standard for each topic.
We compared our approach with two baseline methodsthat reported the state-of-the-art performance on this task.The baselines are: Maximal Marginal Relevance (MMR) [3]and Diversity Penalty (DP) [21]. We use ROUGE-1 andROUGE-2, two commonly used evaluation metrics in docu-ment summarization, as the evaluation measures.Results and Analysis Table 5 lists the results of differentsummarization methods on the TAC datasets. Our approachclearly outperforms the baseline systems on both datasets.The relative improvements over the baselines are about 6%and 10% in terms of ROUGE-1 and ROUGE-2. We also
9http://www.nist.gov/tac/
Table 5: Summarization performance on the TACdatasets. ILP is our approach.
Data MetricsMethods Gold
DP MMR ILP Standard
TAC2008ROUGE-1 0.349 0.348 0.371 0.414ROUGE-2 0.097 0.096 0.103 0.116
TAC2009ROUGE-1 0.334 0.343 0.372 0.444ROUGE-2 0.091 0.096 0.105 0.126
Figure 3: Demonstration system of PatentMiner.
compared our generated results with the human-written re-sults. The last column in Table 5 is the average results ofthe four human-written summaries for all topics. Therefore,these scores in the last column can be considered as the up-per bound of the summarization task. As shown in Table 5,the average score of our approach (0.371/0.372 by ROUGE-1) reaches 86.63% of that of gold standard (0.414/0.444),86.01% in terms of ROUGE-2, which further confirms theeffectiveness of the proposed approach.
7.4 Online SystemWe have developed a patent analysis and mining system:
PatentMiner, and implemented the proposed methods in thesystem. Figure 3 shows the screenshot of the system. Thetop-left of is the profile page for “Microsoft Corporation”.There are three plots respectively showing the patent appli-cation trend, topic trend, and major inventor trend. Theright side is the results of competitor evolution analysis ontopic “mobile phone”.
8. RELATED WORKThere are a few systems for patent search and analy-
sis such as Google Patent, WikiPatent, FreePatentsOnline,Patents, PatentLens, and PriorArtSearch. However, mostof these systems focus on search and provide limited macro-level analytic functions. Few systems provide the user withinsights into the micro-level analysis of the patent network.No existing system studies the problem of dynamic topicmodeling and the topic-level competitor analysis. Tsenget al. [19] introduce a series of text mining techniques forpatent analysis, including text segmentation and summaryextraction. However, they merely employ existing text min-ing techniques for patent analysis.
1373
Arnetminer [18] is a “sister” system of PatentMiner. Ituses a combination approach to build profiles for academicresearchers [16] and provides topic-level expertise search overacademic social networks [17]. Compared with these priorworks, the PatentMiner system distinguishes itself in thefollowing aspects: dynamic topic-driven modeling, heteroge-neous network co-ranking, competitive analysis, and patentsummarization, where we propose new approaches to over-come the drawbacks that exist in the traditional methods.
Considerable work has been conducted for extracting top-ics from text. For example, Hofmann [9] proposes the prob-abilistic latent semantic indexing (pLSI) and applies it toinformation retrieval. Blei et al. [1] propose Latent DirichletAllocation (LDA) by introducing a conjugate Dirichlet priorfor all documents. Several extensions of the LDA modelhave been proposed, for example, the Author model [10], theAuthor-Topic model [13], and the Author-Conference-Topicmodel [18]. The major difference of our ICT and DICT mod-els from existing models is that we simultaneously modeldynamic topics of different objects in the patent network.
9. CONCLUSIONWe introduce the architecture, algorithms, and main fea-
tures of the PatentMiner system. We design and imple-ment a general topic-driven framework for analyzing andmining the heterogeneous patent network. Specifically, wefirst propose a dynamic probabilistic model to model thetopical evolution of different objects in the heterogeneousnetwork. We then present a heterogeneous co-ranking ap-proach to rank the multiple objects. We further propose anapproach for topic-level competitor analysis. To help usersdigest the search result, we introduce a method to summa-rize the patent search results. We evaluate the proposedmethods on a real-world patent database and the experi-mental results validate the effectiveness of our methods.
There are many directions of this work. It would be in-teresting to further study influence between companies [15]:how a company’s technology innovation influences anothercompany’s marketing/R&D strategy? Another challenge ishow to integrate domain knowledge into the mining process.
Acknolwedgements The work is supported by a researchfund from ExxonMobil Corporation. Jie Tang has been sup-ported in part by the Natural Science Foundation of China(No. 61073073, No. 61170061), Chinese National Key Foun-dation Research (No. 60933013, No. 61035004).
10. REFERENCES[1] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent
dirichlet allocation. JMLR, 3:993–1022, 2003.
[2] C. Buckley and E. M. Voorhees. Retrieval evaluationwith incomplete information. In SIGIR’2004, pages25–32, 2004.
[3] J. Carbonell and J. Goldstein. The use of mmr,diversity-based reranking for reordering documentsand producing summaries. In SIGIR’98, pages335–336, 1998.
[4] D. Cohn and H. Chang. Learning to probabilisticallyidentify authoritative documents. In ICML’00, pages167–174, 2000.
[5] N. Craswell, A. P. de Vries, and I. Soboroff. Overviewof the trec-2005 enterprise track. In TREC 2005Conference Notebook, pages 199–205, 2005.
[6] J. Dean and S. Ghemawat. Mapreduce: Simplifieddata processing on large clusters. In OSDI’04, pages10–10, 2004.
[7] T. L. Griffiths and M. Steyvers. Finding scientifictopics. In PNAS’04, pages 5228–5235, 2004.
[8] M. Hertzum and A. M. Pejtersen. Theinformation-seeking practices of engineers: Searchingfor documents as well as for people. InformationProcessing & Management, 36(5):761–778, 2000.
[9] T. Hofmann. Probabilistic latent semantic indexing. InSIGIR’99, pages 50–57, 1999.
[10] A. McCallum. Multi-label text classification with amixture model trained by em. In Proceedings ofAAAI’99 Workshop on Text Learning, 1999.
[11] Q. Mei, X. Ling, M. Wondra, H. Su, and C. Zhai.Topic sentiment mixture: modeling facets and opinionsin weblogs. In WWW’07, pages 171–180, 2007.
[12] L. Page, S. Brin, R. Motwani, and T. Winograd. Thepagerank citation ranking: Bringing order to the web.Technical Report SIDL-WP-1999-0120, StanfordUniversity, 1999.
[13] M. Steyvers, P. Smyth, and T. Griffiths. Probabilisticauthor-topic models for information discovery. InKDD’04, pages 306–315, 2004.
[14] J. Tang, R. Jin, and J. Zhang. A topic modelingapproach and its integration into the random walkframework for academic search. In ICDM’08, pages1055–1060, 2008.
[15] J. Tang, J. Sun, C. Wang, and Z. Yang. Socialinfluence analysis in large-scale networks. In KDD’09,pages 807–816, 2009.
[16] J. Tang, L. Yao, D. Zhang, and J. Zhang. Acombination approach to web user profiling. ACMTKDD, 5(1):1–44, 2010.
[17] J. Tang, J. Zhang, R. Jin, Z. Yang, K. Cai, L. Zhang,and Z. Su. Topic level expertise search overheterogeneous networks. Machine Learning Journal,82(2):211–237, 2011.
[18] J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, and Z. Su.Arnetminer: Extraction and mining of academic socialnetworks. In KDD’08, pages 990–998, 2008.
[19] Y.-H. Tseng, C.-J. Lin, and Y.-I. Lin. Text miningtechniques for patent analysis. Inf. Process. Manage.,43:1216–1247, September 2007.
[20] C. van Rijsbergen. Information Retrieval.But-terworths, London, 1979.
[21] X. Wan, J. Yang, and J. Xiao. Towards an iterativereinforcement approach for simultaneous documentsummarization and keyword extraction. In ACL’07,pages 552–559, 2007.
[22] C. Zhai and J. Lafferty. A study of smoothing methodsfor language models applied to ad hoc informationretrieval. In SIGIR’01, pages 334–342, 2001.
[23] J. Zhang, J. Tang, and J. Li. Expert finding in a socialnetwork. In DASFAA’07, pages 1066–1069, 2007.
[24] X. Zhu and J. Lafferty. Harmonic mixtures:Combining mixture models and graph-based methodsfor inductive and scalable semi-supervised learning. InICML’05, pages 1052–1059, 2005.
1374