n eighborhood f ormation and a nomaly d etection in b ipartite g raphs jimeng sun, huiming qu,...
Post on 19-Dec-2015
217 views
TRANSCRIPT
NEIGHBORHOOD FORMATION AND ANOMALY DETECTION IN BIPARTITE GRAPHS
Jimeng Sun, Huiming Qu,
Deepayan Chakrabarti & Christos Faloutsos
Presented ByBhavana Dalvi
OUTLINE
Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
Author Paper graph
Authors Papers
a
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
Author Paper graph
Authors Papers
aWhich authors are most related to ‘a’ ?
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
Author Paper graph
Authors Papers
aWhich authors are most related to ‘a’ ?
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
Author Paper graph
Authors Papers
aWhich authors are most related to ‘a’ ?
0.8b
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
Author Paper graph
Authors Papers
aWhich authors are most related to ‘a’ ?
0.8
0.6
0.2
0.4
b
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
Author Paper graph
Authors Papers
a
Which is the uncommon paper written by ‘a’ ?
0.8
0.6
0.2
0.4
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
Author Paper graph
Authors Papers
a
Which is the uncommon paper written by ‘a’ ?
0.8
0.6
0.2
0.4
BIPARTITE GRAPHS AND INTERESTING QUESTIONS
P2P Network
10
users
files
Which users have similar preferences as a particular user?
Jimeng Sun’s presentation at ICDM 2005
Which files are downloaded by users with very different preferences?
OUTLINE
Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work
Neighborhood formation (NF)Input : query node q in V1
Output : relevance scores of all the nodes in V1 to q
Anomaly detection (AD)Input : query node q in V1, Output : normality scores for nodes in V2 that link to q
PROBLEM DEFINITION
V1 V2
q
E
OUTLINE
Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work
NEIGHBORHOOD FORMATION
Relevance (b, q) (# short length paths from q to b)
b
q
The connection that links only b and q brings more relevance than the connection which links b, q and other nodes.
b
q
EXACT NF ALGORITHM : RANDOM WALK WITH RESTART
Input : a graph G and a query node q
Output : relevance scores to q Construct the transition matrix where
every node in the graph becomes a state every state has a restart probability c to jump back to the query node q. transition probability
Find the steady-state probability u which is the relevance score of all the nodes to q
q
cc c
c
c
Jimeng Sun’s presentation at ICDM 2005
FINDING STEADY STATE PROBABILITIES
|V1| = k , |V2| = n M : k*n matrix representing weighted graph G Adjacency matrix : PA = col_norm(MA) qA : transform query node ‘a’ to (k+n)*1 vector
where only ath column has 1 and rest are 0. uA : steady state probability vector with restart
probability c
Bipartite structure :
k << n then savings are significant
EXTENSIONS TO NF ALGORITHM
Parallel NF If multiple queries, computation can be done in
parallel.
Approximate NF Cluster the nodes in to k partitions
(preprocessing) Given query node q, find partition Gi it belongs to Run Exact NF algorithm only on Gi
Set relevance = 0 for nodes not in Gi
OUTLINE
Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work
ANOMALY DETECTION
A node x in V2 is normal if
Nodes in V1 that links to x
are in same neighbourhood. e.g. V1
V2
V1 V2
low normalityhigh normality
xx
ANOMALY DETECTION ALGORITHM
Input : node t in V2, Bipartite transition matrix P,
Output : Normality score(t)
1. Set St = neighbours of t in V1
2. RSt : Pairwise relevance scores for nodes in St
3. Normality score ns(t) = function (RSt) e.g. mean over non-diagonal elements in
RSt
OUTLINE
Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work
DATASETS
datasets |V1| |V2| |E| Avgdeg (V1)
Avgdeg (V2)
Conference-Author (CA)
2687 288K 662K 510 5
Author-Paper (AP)
316K 472K 1M 3 2
IMDB 553K 204K 2.2M 4 11
DO THE NEIGHBORHOODS MAKE SENSE?
rele
van
ce s
core
rele
vanc
e sc
ore
rele
vanc
e sc
ore
most relevant neighbors most relevant neighbors
The nodes (x-axis) with the highest relevance scores (y-axis) are indeed very relevant to the query node.
HOW ACCURATE IS THE APPROXIMATE NF?
neighborhood size = 20 num of partitions = 10
Precision = fraction of overlaps between ApprNF and NF among top k neighbors
The precision drops slowly while increasing the number of partition The precision remain high for a wide range of neighborhood size
DO THE ANOMALIES MAKE SENSE?
avg
. nor
mal
ity
scor
e
Injection : • Inject 100 nodes in V2 connecting k nodes each in V1
where k = avg. degree of nodes in V2
• Nodes in V1 are randomly picked such that degree = 10 * avg. degree of nodes in V1
• Assumption : will induce connections across neighbourhoods
WHAT ABOUT THE COMPUTATIONAL COST?
Computational cost drops significantly even with small increment in number of partitions
OUTLINE
Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work
RELATEDWORK
Random walk on Graphs Page-Rank [ISDN 1998], Topic Sensitive Page-Rank [WWW 2002]
Outlier detection Outlier detection in high dimensional data : Aggarwal
and Yu [SIGMOD 2001] Outlier Detection Using Random Walks [ICTAI 2006]
Find outlier clusters
Graph partitioning : METIS package Spectral clustering methods Neighbourhoods can become personalized clusters
OUTLINE
Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work
CONCLUSIONS AND FUTURE WORK
Solution to two problems for Bipartite Graphs Neighborhood Formation (NF) Anomaly Detection (AD)
Random walk with restart along with graph partitioning can be used to solve NF efficiently.
AD can be done based on relevance scores generated by NF
Experiments on real datasets show good results.
Proximity Tracking on Time-Evolving Graphs (SIAM 2008 paper) Defines proximity scores in dynamic setting. Efficient incremental updates
THANK YOU