lecture 4: how do we mine, analyse & visualise the social web? (vu amsterdam social web course)

Social Web��2015

Lecture 4: How do we MINE, ANALYSE & VISUALISE the Social Web?

Anca Dumitrache & Lora AroyoThe Network Institute

VU University Amsterdam

•  25 billion tweets on Twitter in 2010, by 175 million users

•  360 billion pieces of contents on Facebook in 2010, by 600 million different users

•  35 hours of videos uploaded to YouTube every minute

•  130 million photos uploaded to flickr per month

The Age of BIG Data

Social Web 2015, Lora Aroyo

Science with BIG Data


BIG Data Challenges


enormous wealth of data = lots of insights•  insights in users’ daily lives and activities•  insights in history•  insights in politics•  insights in communities•  insights in trends•  insights in businesses & brands

Why?


enormous wealth of data = lots of insights•  who uploads/talks? (age, gender, nationality,

community, etc.)•  what are the trending topics? when?•  what else do these users like? on which platform?•  who are the most/least active users?•  ..…

Why?


Image: http://www.co.olmsted.mn.us/prl/

propertyrecords/RecordingDocuments/PublishingImages/forms.jpg

This doesn’t work


How about this?


Who uses it?


Politicians!Governmental

institutions!


Whole society!


Whole society!

repurposing data

danger of second order effect


Whole society!

repurposing data

discoveries & correlations

Web-Scale Pharmacovigilance: Listening to Signals from the Crowd, R.W. White et al (2013)


Scientists!

Bibliometrics


Culture !History!


Culture

Bill Howe, University of Washington


Entertainment !


You?!


Companies!


Who does it?


The Rise of the Data Scientist

Data Geeks Skills: !Statistics!

Data munging !Visualisation!


http://radar.oreilly.com/2010/06/what-is-data-science.html

The Rise of the Data Scientist


•  Data Science enables the creation of data products

•  Data products are applications that acquire their value from the data, and create more data as a result.

•  Users are in a feedback loop: they constantly provide information about the products they use, which gets used in the data product.

Data Science


Data Science Venn Diagram

Drew Conway


Popular Data Products

Data Science is about building products

not just answering questionsSocial Web 2015, Lora Aroyo

Popular Data Products

empower the others to use the data

empower the others to their own analysis


(Inspired by George Tziralis’ FOSS Conf’09, John Elder IV’s Salford Systems Data Mining Conf. and Toon Calders’ slides)

Data mining is the exploration & analysis of large quantities of data

in order to discover valid, novel, potentially useful, & ultimately understandable patterns in data

http://www.freefoto.com/images/33/12/33_12_7---Pebbles_web.jpg

Data Mining 101


Databases Statistics

Artificial Intelligence

Data Mining 101

• Data input & exploration

• Preprocessing• Data mining algorithms

• Evaluation & Interpretation


•  What data do I need to answer question X?

•  What variables are in the data?

•  Basic stats of my data?

Data Input & Exploration

“LikeMiner” Social Web 2015, Lora Aroyo

•  Cleanup!

•  Choose a suitable data model

• What happens if you integrate data from multiple sources?

•  Reformat your data

Preprocessing

“LikeMiner”


•  Classification: Generalising a known structure & apply to new data

•  Association: Finding relationships between variables

•  Clustering: Discovering groups and structures in data

Data Mining Algorithms


•  Filter users by interests

•  Construct user graphs

•  PageRank on graphs to mine representativeness

•  Result: set of influential users

•  Compare page topics to user interests to find pages most representative for topics

Mining in “LikeMiner”


Evaluation & Interpretation What does the pattern I found mean?!•  Pitfalls: • Meaningless Discoveries•  Implication ≠ Causality (Intensive care -> death)•  Simpson’s paradox•  Data Dredging•  Redundancy• No New Information

• Overfitting•  Bad Experimental Setup


Data Mining is not easy


Data Journalism


source: http://kunau.us/wp-content/uploads/2011/02/Screen-shot-2011-02-09-

at-9.03.46-PM-w600-h900.png

Mining Social Web Data


Source: http://infosthetics.com/archives/2011/12/all_the_information_facebook_knows_about_you.htmlSee also: http://www.youtube.com/watch?feature=player_embedded&v=kJvAUqs3Ofg

Single Person


http://www.brandrants.com/brandrants/obama/

Populations


Brand Sentiment via Twitter

http://flowingdata.com/2011/07/25/brand-sentiment-showdown/


Sentiment Analysis as Service


http://text-processing.com/demo/sentiment/


http://www.cs.cornell.edu/home/kleinber/networks-book/networks-book.pdf

Recommended Reading


http://www.actmedia.eu/media/img/text_zones/English/small_38421.jpg

Assignment 2: Semantic Markup •  Part I: enrich/create a Web page with semantic markup!

•  Step 1: Mark up two different Web pages with the appropriate markup describing properties of at least people, relationships to other people, locations, some temporally related data and some multimedia. You can also try out tools such as Google Markup Helper

•  Step 2: Validate your semantic markup. Use existing validator.•  Step 3: Explain why you chose particular markups. Compare the advantages and disadvantages of

the different markups. Include screenshots from validators.

•  Part II: analyse other team’s Web page markup - as a consumer & as a publisher!•  Step 1: Perform evaluation and report your findings (consider findability or content extraction)•  Step 2: Support your critique with examples of how the semantic markup could be improved.•  In introductory section explain what semantic markup is, what it is for, what it looks like etc. •  Support your choices and explanations with appropriate literature references. •  5 pages (excluding screen shots). •  Other group’s evaluation details in appendix.

•  Deadline: 3 March 23:59!

Image Source: http://blog.compete.com/wp-content/uploads/2012/03/Like.jpg

Final Assignment: Your SocWeb App

•  Create your own Social Web app (in a group)•  Use structured data, entity relations, data analysis, visualisation•  Write individual report on one of the main aspects of your app•  Pitch your app idea before finalising: 12 Mar, during Hands-on•  Submit final assignment : 27 March 23:59


image source: http://www.flickr.com/photos/bionicteaching/1375254387/

Hands-on Teaser

•  Build your own recommender system 101•  Recommend pages on del.icio.us •  Recommend pages to your Facebook friends


lecture 4: how do we mine, analyse & visualise the social web? (vu amsterdam social web course)

Education

lora aroyosocial web

data science social

data scientist social

lora aroyoscience

lora aroyoinspired

lora aroyopoliticians

lora aroyoscientists

lora aroyoentertainment