using clojure for sentiment analysis of the twittersphere (euroclojure 2014)

Sentiment Analysis of the Twittersphere

Leiningen Versus the Ants!

Clojure Versus Java!

FP Versus OOP

Leiningen Versus the Ants!


FP Versus OOP

Apache Ant

Leiningen Versus the Ants


FP Versus OOP

Apache Ant

Leiningen Versus the Ants

Clojure Versus Java

Functional Versus Object Oriented

Apache Ant

Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma!

1 FAVORITE


1 FAVORITE

Scores HIGH for flu symptoms


1 FAVORITE

Who is Catalina Rubottom?


1 FAVORITE

HDFS/Hadoop

Mongo/Aggregation

Mongo/MapReduce

Postgres

30 million geo-tagged tweets sent from UK

How can we do fast, real time analytics of

social media?

Store this!

Not this!

ptaoussanis/carmine

(:require [taoensso.carmine :as car]) !!(defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) !!(defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))

(:require [taoensso.carmine :as car]) !!(defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) !!(defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))

find value for keyfind bit of index

set bit to 1

(wcar* (car/bitcount "SCOTLAND") (car/bitcount "JOVIALITY") (car/bitcount "26062014114532"))

ENG

SCT

WAL

NIR

valuekey

ENG 1

SCT 0

WAL 0

NIR 0

(car/setbit “ENG” 0 1)

tweet-idkey

valuekey

ENG 0 1

SCT 1 0

WAL 0 0

NIR 0 0

(car/setbit “SCT” 1 1)

tweet-idkey

valuekey

ENG 1 0 1

SCT 0 1 0

WAL 0 0 0

NIR 0 0 0

(car/setbit “ENG” 2 1)

tweet-idkey

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/setbit “SCT” 15 1)

tweet-idkey

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "ENG")

8

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "SCT")

84

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "WAL")

843

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "NIR")

8431

valuekey

JOVIAL

SHY

HOSTILE

FATIGUE

valuekey

JOVIAL 1

SHY 0

HOSTILE 0

FATIGUE 0

(car/setbit “JOVIAL” 0 1)

tweet-idkey

valuekey

JOVIAL 0 1

SHY 0 0

HOSTILE 1 0

FATIGUE 0 0

(car/setbit “HOSTILE” 1 1)

tweet-idkey

valuekey

JOVIAL 0 0 1

SHY 1 0 0

HOSTILE 0 1 0

FATIGUE 0 0 0

(car/setbit “SHY” 2 1)

tweet-idkey

valuekey

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0

FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

(car/setbit “JOVIAL” 15 1)

tweet-idkey

valuekey

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0

FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

(car/bitcount [*])

8134

valuekey

(wcar*

(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")

(car/expire “ENGLAND&JOVIALITY" 10)

(car/bitcount “ENGLAND&JOVIALITY"))

How many people in England are Happy?

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1


ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1


ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

1

AND



ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

0 1

AND



ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

0 0 1

AND



ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1

AND



ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1


(car/bitcount “ENGLAND&JOVIALITY")

4

(wcar*

(car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY")

(car/expire "FATIGUE|HOSTILITY" 10)

(car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)"

"SCOTLAND" "FATIGUE|HOSTILITY")

(car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10)

(car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))

How many people in Scotland are Tired and Grumpy?

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0


HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0


HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH

OR


HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0

OR


HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 1 0

OR


HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0


HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

OR


SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0


SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

AND


SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 00 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0


2

RINGSyncing

Journaling

Business logic

What

Where

When

Mem

ory

Web

API

Incoming Data

RINGSyncing

Journaling

Business logic

What

Where

When

Mem

ory

Web

API

Incoming Data

adamwynne/twitter-api

RINGSyncing

Journaling

Business logic

What

Where

When

Mem

ory

Web

API

Incoming Data

clojurewerkz/meltdown

LMAX-Exchange/disruptor

RINGSyncing

Journaling

Business logic

What

Where

When

Mem

ory

Web

API

Incoming Data

dakrone/clojure-opennlp

“I’m loving #EuroClojure! ”

Positive Affect: enthusiastic, active, alert !

Negative Affect: subjective distress !

Emerged as distinctive, orthogonal dimensions

PANAS

1

23

4

51

3153

2335345221

PANAS

1

23

4

51

3153

2335345221

1 3 5 9 10 12 14 16 17 19

2 4 6 7 8 11 13 15 18 20

PANAS

1

2345

1

31

53

233534

5 22 1

1 3 5 9 10 12 14 16 17 19

2 4 6 7 8 11 13 15 18 20

39 19

positive affect! negative affect

PANAS-xGeneral Dimension Scales!Negative Affect (10) Positive Affect (10)

Basic Positive Emotions Scales!Joviality (8) Self-assurance (6) Attentiveness (4)

Basic Negative Emotions Scales!Fear (6) Hostility (6) Guilt (6) Sadness (5)

Other Affective States!Shyness (4) Fatigue (4) Serenity (3) Surprise (3)

PANAS-t

Accounts for bias on social media!

Outlines sanitisation

Validate against 10 real events

Sanitisation

• Exclude media/advertising/spam

• Account for text speak

• Account for smileys & emoj’s

• Word stemming (or lemmatisation)

• Part of Speech Tagging (POS)

LMAO

GetRichQuick.com

dakrone/clojure-opennlp

http://GetRichQuick.com



























SHYNESS FATIGUE

SERENITY SURPRISE

FEAR HOSTILITY

GUILT SADNESS

JOVIALITY SELF ASSURANCE ATTENTIVENESS

We have sentiment key

s!

RINGSyncing

Journaling

Business logic

What

Where

When

Mem

ory

Web

API

Incoming Data

mbostock/d3

[-75.14310264, 40.05701649]

Reverse Geocoding Issues• Don’t want external services

• Don’t want heavy IO

• Don’t want round trips to the database

• Accuracy not too much of a concern

Equirectangular Projection

Courtesy of Eric Gaba (Wikimedia Commons User: Sting)

x = λcosφ1 y = φ

Tissot's indicatrix of deformation

Courtesy of Eric Gaba (Wikimedia Commons User: Sting)

x (px)

y (px)

x (px)

y (px)

[lon, lat] = [55.86140, -4.26477]

[x, y] = [329, 601]

[x, y] = = Scotland!

Server side! (JavaFX)

Mike Bostock & Nick Rabinowitz

(:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) !(def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) !(defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) !(defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) !(defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])"))))

JavaFX

(:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) !(def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) !(defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) !(defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) !(defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])"))))

Stuart Sierras

Component lib!!

JavaFX

RINGSyncing

Journaling

Business logic

What

Where

When

Mem

ory

Web

API

Incoming Data

KirinDave/clj-time

“Thu Jun 26 14:35:57 +0000 2014”

over 86 thousand seconds in a dayover 31 million seconds in a year

mm:260620141435

HH:260620141400

Monday Tuesday Wednesday Thursday Friday Saturday Sunday

1

2 3 4 5 6 7 8

9 10 11 12 13 14 15

16 17 18 19 20 21 22

23 24 25 27 28 29

30 31

dd:260620140000

JAN FEB MAR APR

MAY JUL AUG

SEP OCT NOV DEC

MM:010620140000

2013

2015

yy:010120140000

(:require [clj-time.format :as time]) !

!

(def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) !

!

(defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) !

!

(defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))

RINGSyncing

Journaling

Business logic

What

Where

When

Mem

ory

Web

API

Incoming Data

ptaoussanis/carmine

SENTIMENT !

LOCATION !

mm:date-time HH:date-time dd:date-time MM:date-time yy:date-time

We have 11 keys in total!

What

Where

When

(:require [taoensso.carmine :as redis]) !

!

(defn create-tweet-id [] (redis/incr "global:tweet.id")) !

!

(defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) !

!

(defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))

{ 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }

weavejester/compojure

JSON query



JSON query

/api?_query={json}

(defmulti execute :query-type) !!(defmethod execute “count" [] … logic here) !!(defmethod execute “p-affect“ [] … logic here) !!(defmethod execute “n-affect“ [] … logic here)



JSON query

/api?_query={json}



JSON query

/api?_query={json}




JSON query

/api?_query={json}

{ 'query-type': ‘p-affect', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }

The code for this project is about to be open sourced

!

We would love to have more contributors! !

To find out more, follow: #APassionForCode

using clojure for sentiment analysis of the twittersphere (euroclojure 2014)

Data & Analytics