![Page 1: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/1.jpg)
@MargrietGr
Margriet Groenendijk, PhDDeveloper Advocate for IBM Cloud Data Services
Open Data Science Conference UK9 October 2016, London
How To Analyse Weather Data and Twitter Sentiment with Spark and Watson
![Page 2: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/2.jpg)
@MargrietGr
![Page 3: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/3.jpg)
@MargrietGr
![Page 4: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/4.jpg)
@MargrietGr
![Page 5: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/5.jpg)
@MargrietGr
Analyse Weather Data and Twitter Sentiment using Spark and Watson
People love to talk about the weather on Twitter
What insights can you find when combining the data?
What is the weather sentiment related to?
![Page 6: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/6.jpg)
@MargrietGr
Bluemix
Where to find the data?
Insights for Twitter
Weather Company Data
API services from IBM Bluemix
https://console.ng.bluemix.net/
![Page 7: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/7.jpg)
@MargrietGr
Bluemix
Where to store the data?
Availablefrom IBM Bluemix
Cloudant NoSQL DB
![Page 9: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/9.jpg)
@MargrietGr
Weather Company Data
Watson Tone Analyser
TweetsWeatherSentiment
Exploring the options
![Page 10: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/10.jpg)
Bluemix
![Page 11: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/11.jpg)
@MargrietGr
IBM Bluemix
▪ Free trial etc▪ lots of services etc
Free 30-day trial“Big Blue Box” containing all IBMs services
https://console.ng.bluemix.net/
![Page 12: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/12.jpg)
@MargrietGr
Add a service in Bluemix
Add a service
Search for weather
Account + spaces
![Page 13: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/13.jpg)
Weather Company Data for IBM Bluemix
![Page 14: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/14.jpg)
@MargrietGr
Weather Company Data for IBM Bluemix
2 Credentials3 Ready to
use the REST APIs
Add service 1
![Page 15: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/15.jpg)
@MargrietGr
Your own weather forecast in a Python notebook
![Page 16: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/16.jpg)
@MargrietGr
Weather Company Data API
Show json weather file
![Page 17: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/17.jpg)
@MargrietGr
Your own weather forecast in a Python notebook
London
![Page 18: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/18.jpg)
@MargrietGr
https://developer.ibm.com/clouddataservices/2016/10/06/your-own-weather-forecast-in-a-python-notebook/
Weather in UK on Friday evening 7 October
![Page 19: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/19.jpg)
@MargrietGr
Store weather data in Cloudant
![Page 20: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/20.jpg)
@MargrietGr
Python script run daily on a Bluemix VM service
https://python-cloudant.readthedocs.io
![Page 21: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/21.jpg)
@MargrietGr
Python script run daily on a Bluemix VM service
Add crontab job to run daily
![Page 22: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/22.jpg)
Cloudant
![Page 23: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/23.jpg)
@MargrietGr
▪ cloudant
▪ etc
![Page 24: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/24.jpg)
@MargrietGr
▪ geospatial index▪ show map with 100 cities :-)
![Page 25: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/25.jpg)
@MargrietGr
▪ geospatial index▪ show map with 100 cities :-)
![Page 26: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/26.jpg)
@MargrietGr
Weather Company Data
Watson Tone Analyser
TweetsWeatherSentiment
![Page 27: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/27.jpg)
Insights for Twitter
![Page 28: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/28.jpg)
@MargrietGr
Insights for Twitter
![Page 29: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/29.jpg)
@MargrietGr
Insights for Twitter
Only a 100…
![Page 30: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/30.jpg)
dashDB
![Page 31: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/31.jpg)
@MargrietGr
Add the dashDB service in Bluemix
Add a service
Search for dashDB
![Page 32: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/32.jpg)
@MargrietGr
![Page 33: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/33.jpg)
@MargrietGr
Use an existing service3
1
2
posted:2016-08-01,2016-10-01 followers_count:3000 friends_count: 3000 (weather OR sun OR sunny OR rain OR hail OR storm OR rainy OR drought OR flood OR hurricane OR tornado OR cold OR snow OR drizzle OR cloudy OR thunder OR lightning OR wind OR windy OR heatwave)
REST API docs:https://new-console.ng.bluemix.net/docs/services/Twitter/twitter_rest_apis.html#rest_apis
Search for tweets
4 Select table
![Page 34: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/34.jpg)
@MargrietGr
![Page 35: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/35.jpg)
@MargrietGr
Weather Company Data
Watson Tone Analyser
TweetsWeatherSentiment
![Page 36: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/36.jpg)
Explore the data
![Page 37: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/37.jpg)
IBM Data Science Experience
![Page 38: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/38.jpg)
@MargrietGr
Nested data…
![Page 39: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/39.jpg)
@MargrietGr
![Page 40: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/40.jpg)
@MargrietGr
Load tweets from dashDB with Spark SQL
![Page 41: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/41.jpg)
@MargrietGr
Clean data, summarise and load into pandas DataFrame
![Page 42: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/42.jpg)
@MargrietGr
Add weather to tweets
Weather data is nested, pyspark.sql struggles with thatThere is no location data of tweets
Only 10% of all tweets available in the free plan through the Decahose streamWeather API only has 24 hours of data available
![Page 43: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/43.jpg)
@MargrietGr
Weather Company Data
Watson Tone Analyser
TweetsWeatherSentimentX
![Page 44: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/44.jpg)
@MargrietGr
Weather Company Data
crontab -e
0 23 * * * /path/to/file/do_something.sh
python do_something.py
TweetsWeatherSentiment
![Page 45: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/45.jpg)
Watson Tone Analyser
![Page 46: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/46.jpg)
@MargrietGr
Add sentiment - example
![Page 47: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/47.jpg)
@MargrietGr
![Page 48: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/48.jpg)
@MargrietGr
![Page 49: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/49.jpg)
#Matthew
![Page 50: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/50.jpg)
@MargrietGr
Use an existing service3
1
2
posted:2016-08-26,2016-10-06 followers_count:1000 friends_count:1000 (matthew OR hurricane matthew OR hurricane)
REST API docs:https://new-console.ng.bluemix.net/docs/services/Twitter/twitter_rest_apis.html#rest_apis
#matthew tweets
4 Select table
![Page 51: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/51.jpg)
@MargrietGr
![Page 52: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/52.jpg)
@MargrietGr
![Page 53: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/53.jpg)
@MargrietGr
![Page 54: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/54.jpg)
@MargrietGr
Some lessons learnedAPIs are great!Can extend and build on this, as all data is in the Cloud
Weather data only available for 24 hrs, great for weather apps, but harder to combine weather with historical tweets, need a daily script
Now ready to build a more efficient workflow that will be easily able to handle millions of tweetsStart a more in depth analysis in the Data Science Experience
![Page 55: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/55.jpg)
@MargrietGr
▪ analyse data!▪ pretty plots
https://github.com/ibm-cds-labs/pixiedust
![Page 56: ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson](https://reader031.vdocuments.mx/reader031/viewer/2022030305/58724ebd1a28ab852f8b67c3/html5/thumbnails/56.jpg)
@MargrietGr
Margriet Groenendijk, PhDDeveloper Advocate for IBM Cloud Data Services
https://developer.ibm.com/clouddataservices/author/mgroenen/
Thank you!
Slides will be available onhttp://www.slideshare.net/MargrietGroenendijk