![Page 1: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/1.jpg)
Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution
Malik Magdon-IsmailRensselaer Polytechnic
Institute
![Page 2: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/2.jpg)
2
Our Group
M. Goldberg M-I B. Szymanski A. Wallace
Students: Mykola Hayvanovich Apirak Hoonlor Stephen Kelley Konstantin Mertsalov
![Page 3: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/3.jpg)
3
Motivation
Communications supporting IED planning have patterns and are correlated….
Analysis of the patterns can reveal the groups as well as their internal group structure.
![Page 4: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/4.jpg)
4
CommunicationsTime: January 12, 2005, 09:35
From: [email protected]
Subject: Hello
Message: Where have you been?
16:06:31] <FreeTrade> Republicans were the worst pacifists before ww1 and ww2[16:06:43] <SweetLeaf> France Fries[16:06:50] <FreeTrade> As a generality, of course their were Republican Hawks.[16:07:13] <FreeTrade> Sweet, good pun but bad story![16:07:18] <SweetLeaf> yup[16:07:23] <Lupine> anyways, he's perpetually tormented by presidential actions[16:07:25] <SweetLeaf> it aint good for no one[16:07:47] <SweetLeaf> I think they knew it was commiing[16:07:51] <FreeTrade> Rossevelt met monthly in New York with mostly trusted Republicans to talk about how to get america into the war.[16:08:10] <FreeTrade> and he spent 2 year with Churchill meeting him sometimes secretly in the ocean to discuss the same topic.[16:08:22] <FreeTrade> Exchanging a lot of letters.[16:08:25] <FreeTrade> telegrams[16:08:28] <Lupine> There really is nothing like a shorn scrotum. It's breathtaking, I suggest you try it.[16:08:55] <FreeTrade> Well they didnt literally meet in the ocean, they were on ships.
![Page 5: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/5.jpg)
5
Streaming ExampleTime From To Message10:00 Alice Charlie Golf tomorrow? Tell everyone.10:05 Charlie Felix Alice mentioned golf tomorrow.10:06 Alice Bob Hey, golf tomorrow. Spread the word.10:12 Alice Bob Tee off: 8am at Pinehurst.10:13 Felix Grace Hey guys, golf tomorrow.10:13 Felix Harry Hey guys, golf tomorrow.10:15 Alice Charlie Pinehurst Tee time: 8am.10:20 Bob Elizabeth We’re playing golf tomorrow.10:20 Bob Dave We’re playing golf tomorrow.10:22 Charlie Felix Tee time 8am at Pinehurst10:25 Bob Elizabeth We tee off 8am at Pinehurst.10:25 Bob Dave We tee off 8am at Pinehurst.10:31 Felix Grace Tee time 8am, Pinehurst.10:31 Felix Harry Tee time 8am, Pinehurst.
![Page 6: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/6.jpg)
6
Streaming ExampleTime From To Message10:00 Alice Charlie Golf tomorrow? Tell everyone.10:05 Charlie Felix Alice mentioned golf tomorrow.10:06 Alice Bob Hey, golf tomorrow. Spread the word.10:12 Alice Bob Tee off: 8am at Pinehurst.10:13 Felix Grace Hey guys, golf tomorrow.10:13 Felix Harry Hey guys, golf tomorrow.10:15 Alice Charlie Pinehurst Tee time: 8am.10:20 Bob Elizabeth We’re playing golf tomorrow.10:20 Bob Dave We’re playing golf tomorrow.10:22 Charlie Felix Tee time 8am at Pinehurst10:25 Bob Elizabeth We tee off 8am at Pinehurst.10:25 Bob Dave We tee off 8am at Pinehurst.10:31 Felix Grace Tee time 8am, Pinehurst.10:31 Felix Harry Tee time 8am, Pinehurst.
![Page 7: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/7.jpg)
7
Streaming ExampleTime From To10:00 Alice Charlie10:05 Charlie Felix10:06 Alice Bob 10:12 Alice Bob 10:13 Felix Grace10:13 Felix Harry10:15 Alice Charlie 10:20 Bob Elizabeth10:20 Bob Dave10:22 Charlie Felix10:25 Bob Elizabeth 10:25 Bob Dave10:31 Felix Grace10:31 Felix Harry
![Page 8: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/8.jpg)
8
Overview: SIGHTS & RDM
b u y , t r a d e . . . b u y
2 t r a d e . . . 2 t r a d e
3 , s e l l . . . 3 , h e l l
Pattern id = 2Pattern = “buy,”
Pattern id = 3Pattern = “2trade”bb
Level 0
Level 1
Level 2
Higher ranked leaders
Group leader
Subgroup leaders
Members
![Page 9: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/9.jpg)
9
Communications
Email, Telephone, Newsgroup, Weblog, Chatrooms, …
Time: January 12, 2005, 09:35
From: [email protected]
Subject: Hello
Message:
Where have you been lately?
Time: January 12, 2005, 09:35
From: [email protected]
Subject: Hello
Message:
Where have you been lately?
![Page 11: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/11.jpg)
11
Communication Graph
What are the social groups/coalitions?
![Page 12: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/12.jpg)
12
Social Groups are Clusters
Clusters may overlap.
![Page 13: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/13.jpg)
13
Social Groups are Clusters
Clusters may overlap.A cluster is a locally defined object.
![Page 14: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/14.jpg)
14
Social Groups are Clusters
Clusters may overlap.A cluster is a locally defined object.
Group members are more introverted than extroverted.
YES NO
![Page 15: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/15.jpg)
15
Social Groups are Clusters
Clusters may overlap.A cluster is a locally defined object.
Group members are more introverted than extroverted.
Social groups (clusters) persist
![Page 16: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/16.jpg)
16
SIGHTSStatistical Identification of Groups Hidden in Time and Space
- System for statistical analysis of social coalitions in communication networks
Data SourcesBlogsEmails (Enron)ChatroomSynthetic data
Coalition DiscoveryOverlapping ClusteringStreaming groupsPersistent groups.
Coalition AnalysisLeadersOpposing groupsTopic matching
VisualizationsSize-Density plotsStatic coalitionsDynamic coalitions
Groups matching analyst topic in red
Size vs. Density Plot
Visualization options
Choose time window
Groupmembers
Different analyses on dataset
Leader index
![Page 17: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/17.jpg)
17
Examples
Two clusters: Electric circuit design; Optimization of Neural Networks:
Intersection: “Sensitivity analysis in degenerate quadratic programming”
Citeseer
ENRON
GROUND TRUTH Group A
Dog Vulture Camel Yassir Hussein Bird (6 others)
Group B Ahmet Saleh Sarwuk Shaid Pavlammed Pavlah Osan Domenik
SIGHTS Group A
Dog Vulture Camel Gopher
Group B Ahmet Saleh Sarwuk Shaid Dajik
Ali Baba Data Set (DoD)
![Page 18: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/18.jpg)
18
Build a classifier to identify the relationship between sender and receiver of a message
EXAMPLE:“Do you have time to meet some time this week?”
“Lets meet 2pm today, ok?”
Which is advisor, which is student?
Recursive Data Mining (RDM)
![Page 19: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/19.jpg)
19
Hierarchical Pattern Construction (recursive definition)Captures patterns; patterns of patterns; patterns of patterns of
patterns… (can even capture long-range patterns)
Pattern Definition
Larger patterns
b u y , t r a d e . . . b u y
2 t r a d e . . . 2 t r a d e
3 , s e l l . . . 3 , h e l l
Pattern id = 2Pattern = “buy,”
Pattern id = 3Pattern = “2trade”
Pattern id = 4Pattern = “3,_ell”
Level 0
Level 1
Level 2
![Page 20: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/20.jpg)
20
Ensemble of classifiersClassifier for each level in the hierarchical approachFeatures gathered from the training messagesGlobal features include average length and number of sentencesApproximate matching allows treatment of noise
A Classifier – Joining the Pieces
![Page 21: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/21.jpg)
21
Binary classification: for a given message m,
is m sent by a person with role r?
r є {CEO, Manager, Trader, Vice-President}
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
CEO Manager Trader Vice-pres.Roles
1 -
F_
sco
re
NB RDM_NB SVM RDM_SVM CPAR
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
CEO Manager Trader Vice-pres.
Roles
1 -
F_
Sco
re
NB RDM_NB SVM RDM_SVM CPAR
Multi-classification: for a given message m,
which role r is the most likely for the sender?
r є {CEO, Manager, Trader, Vice-President}
The bars show the error of classification. Universally RDM_SVM outperforms other classifiers
Results on Enron
![Page 22: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/22.jpg)
22
Summing Up
SIGHTS:Structural; non-semantic; language
independentFinds groups, their dynamics and
structure; visual analytic capabilities.RDM
Uses statistical semantics; language independent
Identifies roles within the group
![Page 23: Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649dcd5503460f94ac1656/html5/thumbnails/23.jpg)
23
Thank You
http://www.cs.rpi.edu/~magdon