dundee university hacku 2013 - yql
DESCRIPTION
TRANSCRIPT
HackU 2013 University of Dundee
February 2013
HackU 2013 – University of Dundee
The web is full of juicy and
long lasting data
HackU 2013 – University of Dundee
And there is a lot of it around
HackU 2013 – University of Dundee
However, our attempts to get it can sometimes
be clumsy
HackU 2013 – University of Dundee
What we need is an easy way to get to that data
HackU 2013 – University of Dundee
What about using APIs?
Like RSS, JSON, XML, CSV, SOAP, REST, …
HackU 2013 – University of Dundee
We could use APIs. But there are thousands of APIs...
HackU 2013 – University of Dundee
and it’s painful to get your head around them!
HackU 2013 – University of Dundee
What if there was one API…
One API for everything on the web?
HackU 2013 – University of Dundee
One World One Web One API ?
Can it ever exist?
Sounds like
politics!
HackU 2013 – University of Dundee
YQL:
The API for the
web
Yes it does exist!
HackU 2013 – University of Dundee
The Yahoo Query Language, or short YQL is a unified interface language
to the web.
What are you waiting for? http://developer.yahoo.com/yql
HackU 2013 – University of Dundee
Accessing the web and its services now becomes easy as ABC SQL
select * from web
select {what} from {service} where {condition}
HackU 2013 – University of Dundee
Let’s try a real YQL example
select * from flickr.photos.search where
text="kitten" limit 5
I love kittens. YQL get me 5 kitten photos
HackU 2013 – University of Dundee
Let’s try a real YQL example select * from
flickr.photos.search where text="kitten" limit 5
Output can be XML or JSON
HackU 2013 – University of Dundee
I want to use YQL and I want it now. That’s the easy part…
http://query.yahooapis.com/v1/public/yql?q= <SQL_QUERY>&format=<FORMAT> e.g. http://query.yahooapis.com/v1/public/yql?q= select%20*%20from%20rss%20where%20url%3D%22http%3A%2F%2Frss.news.yahoo.com%2Frss%2Ftopstories%22&format=json
There’s also a private endpoint which uses oauth authentication http://developer.yahoo.com/yql/guide/authorization-access.html
HackU 2013 – University of Dundee
That’s not all… You can mix and match several web services using
the in() command
HackU 2013 – University of Dundee
Mix and match…
select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text=’dundee,uk') and license=4)
Find photos in Dundee, UK with a Creative Commons “By” license
HackU 2013 – University of Dundee
Now use your imagination… and build away!
http://isithackday.com/hacks/cantine/index.php?loc=covent+garden
HackU 2013 – University of Dundee
Yahoo! offers lots of data for free!
flickr.photos.exif flickr.photos.info flickr.photos.interes1ngness flickr.photos.recent flickr.photos.search flickr.photos.sizes flickr.places flickr.places.info geo.places geo.places.ancestors geo.places.belongtos geo.places.children geo.places.neighbors geo.places.parent geo.places.siblings geo.placetypes gnip.ac1vity
local.search music.ar1st.id music.ar1st.popular music.ar1st.search music.ar1st.similar music.release.ar1st music.release.id music.release.popular music.release.search music.track.id music.track.popular music.track.search music.video.category music.video.id music.video.popular music.video.search music.video.similar
mybloglog.community.find mybloglog.member mybloglog.member.contacts mybloglog.member.newwithcontacts mybloglog.member.newwithme mybloglog.members.find mybloglog.stats.find upcoming.category upcoming.country upcoming.events upcoming.events.bes1nplace upcoming.groups upcoming.metro upcoming.state upcoming.user upcoming.venue
HackU 2013 – University of Dundee
… and more of it. ALL FREE!
search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connec1ons social.contacts social.presence social.profile social.updates weather.forecast yahoo.iden1ty yap.setsmallview
search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connec1ons social.contacts social.presence social.profile social.updates weather.forecast yahoo.iden1ty yap.setsmallview
search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connec1ons social.contacts social.presence social.profile social.updates weather.forecast yahoo.iden1ty yap.setsmallview
HackU 2013 – University of Dundee
You want more data? amazon bitly delicious dopplr etsy friendfeed github Greader guardian imdb iploca1on las?m nestoria ne?lix
nmm nyt opensocial search shopping social twi@er update.groovy weather wesabe whitepages yahoo yelp zillow
http://github.com/spullara/yql-tables/tree/master
HackU 2013 – University of Dundee
Why don’t you create your own data source for YQL…
Using YQL’s Open Tables
All we need to do is create a schema that explains the data structure.
http://developer.yahoo.com/yql/guide/yql-opentables-
chapter.html
HackU 2013 – University of Dundee
What else can you do with YQL?
Get data of a website select * from html where url='http://www.telegraph.co.uk/' and xpath='//h3/a'
Get data of several websites in one query select * from rss where url in (<url1>,<url2>);
HackU 2013 – University of Dundee
Anything else? Yes...
Create a View (Filtered data set) use 'http://yqlblog.net/samples/searchrank.xml' as searchrank; select * from searchrank where query='pizza' and dispurl like '%pizzahut%'
More about YQL Execute: http://developer.yahoo.com/yql/guide/yql-execute-chapter.html
YQL, where to start…
User guide: http://developer.yahoo.com/yql YQL Console (Test your queries…) http://developer.yahoo.com/yql/console/
HackU 2013 – University of Dundee