Playing with the web

Christian Heilmann | http://wait-till-i.com | http://twitter.com/codepo8

Geek Meet Stockholm, December 2008

or “The geek shall inherit the earth”

The web is an awesome opportunity and media.

I learnt that pretty early and left an old media to work for

the new one.

One thing that keeps amazing me about it is how simple the

technologies driving it are.

You don’t need to be a rocket scientist to wreak havoc on the web.

I am not talking about malicious intent here.

I am talking about ethical hacking.

So today I am going to show some tools I love to use to

play and mess with the web.

Disrupting the process to foster and drive innovation.

Mr. Buzzwordwill see you now!

Let’s start with the first one – a true Swedish product!

Their product?

cURL

cURL allows you to get raw text data from any server.

You can create the full range of HTTP requests from a script

or the shell.

Including POST requests, simulating cookies and all the

other goodies :)

Which gives you an amazing amount of power

Say you want to build an API that doesn’t exist.

At Mashed08 I got bored.

http://www.flickr.com/photos/37996583811@N01/2600265124/

Someone came to me and asked if I knew of a currency

conversion API.

And I didn’t as there is none.(everybody pays good money for that data)

So I went to Yahoo Finance.

http://finance.yahoo.com/currency/convert?amt=1&from=USD&to=JPY&submit=Convert

Simple, predictable URL :)

I found the location of the result by viewing the source

of the page.

http://www.wait-till-i.com/2008/06/21/currency-conversion-api/

function convert($from,$to){ $url= 'http://finance.yahoo.com/currency/convert?amt=1&from='.$from.'&to='.$to.'&submit=Convert'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $feed = curl_exec($ch); curl_close($ch); preg_match_all("/tabledata1\">([^<]+)<\/td>/", $feed,$cells); return $cells[1][1];}echo convert(’USD’,'GBP’);

Turning this into an API was easy:

header('Content-type:text/javascript');$from = $_GET['from'];$to = $_GET['to'];$callback = $_GET['callback'];if(preg_match("/[A-Z|a-z]{3}/",$to) && preg_match("/[A-Z|a-z]{3}/",$from)){ $to = strToUpper($to); $from = strToUpper($from); $url= ‘http://finance.yahoo.com/currency/convert?’ . ‘amt=1&from=’.$from.’&to=’.$to.’&submit=Convert’; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $feed = curl_exec($ch); curl_close($ch);

preg_match_all("/tabledata1\">([^<]+)<\/td>/",$feed,$cells); if(is_numeric($cells[1][1])){ $out = ‘{"from":"’.$from.’","to":"’.$to.’","factor":"’.$cells[1][1].’"}’; } else { $out = ‘{"error":"Could not convert currencies, are you sure about the names?"}’; }} else { $out = ‘{"error":"Invalid Currency format, must be three letters"}’;}if(isset($callback)){ if(preg_match("/[a-z|A-Z|_|-|\$|0-9|\.]/",$callback)){ $out = $callback.’(’.$out.’)';

} else { $out = ‘{"error":"Invalid callback method name"}’; }}echo $out;

Using the API is as easy...

<script type="text/javascript"> function converted(obj){ if(obj.error){ alert(obj.error); } else { alert('one ' + obj.from + ' is ' + obj.factor + ' ' + obj.to); } }</script><script type="text/javascript" src="convert.php?from=gbp&to=usd&callback=converted"></script>

View Source is a good start.

However, much more win is Firebug.

It has never been easier to find things to get with cURL.

Say twitter information...

How about showing this as a cool chart?

We all like to show off what we do on the web.

Charts can be tricky...

Good thing that there’s Google Charts.

(yeah and YUI charts)

http://www.wait-till-i.com/2008/11/23/show-the-world-your-twitter-type-using-php-and-google-charts/

<?php$user = $_GET['user'];$isjs = "/^[a-z|A-Z|_|-|\$|0-9|\.]+$/";if(preg_match($isjs,$user)){ $info = array(); $cont = get('http://twitter.com/'.$user); preg_match_all('/<span id="following_count" class="stats_count numeric">([^>]+)<\/span>/msi',$cont,$follow); $info['follower'] = convert($follow[1][0]); preg_match_all('/<span id="follower_count" class="stats_count numeric">([^>]+)<\/span>/msi',$cont,$follower); $info['followed'] = convert($follower[1][0]);

preg_match_all('/<span id="update_count" class="stats_count numeric">([^>]+)<\/span>/msi',$cont,$updates); $info['updater'] = convert($updates[1][0]); $max = max($info); $convert = 100 / $max ; foreach($info as $k=>$f){ if($f === $max){ $type = $k; } $disp[$k] = $f * $convert; } if($type === 'updater'){ $t = ' is an '; } if($type === 'follower'){ $t = ' is a ';

} if($type === 'followed'){ $t = ' is being '; } $title = $user . $t . $type; $out = array(); foreach($info as $k=>$i){ $out[] = $k.'+('.$i.')'; } $labels = join($out,'|'); $values = join($disp,','); header('location:http://chart.apis.google.com/chart?cht=p3&chco=336699&'. 'chtt='.urlencode($title).'&chd=t:'.$values. '&chs=350x100&chl='.$labels);}

function get($url){ $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $feed = curl_exec($ch); curl_close($ch); return $feed;}function convert($x){ $x = str_replace(',','',$x); $x = (int)$x; return $x;}?>

What if you need to mix and match and filter data?

http://www.wait-till-i.com/2008/09/28/useful-tweets-with-pipe/

https://pipes.yahoo.com/

https://pipes.yahoo.com/

And *dum de dum de*...

http://developer.yahoo.com/yql/console/

/* Useful tweets badge by Christian Heilmann*/var tweets = function(){ var x = document.getElementById('mytweet'); if(x){ var twitterUserId = x.className.replace('user-',''); var s = document.createElement('script'); s.type = 'text/javascript'; s.src = 'http://pipes.yahoo.com/pipes/pipe.run?' + '_id=f7229d01b79e508d543fb84e8a0abb0d&_render=json' + '&id=' + twitterUserId + '&_callback=tweets.tweet'; document.getElementsByTagName('head')[0].appendChild(s); }; function tweet(data){ if(data && data.value && data.value.items){ if(typeof data.value.items.length !== 'undefined'){ var ul = document.createElement('ul'); var all = data.value.items.length; var end = all > 5 ? 5 : all; for(var i=0;i < end;i++){

var now = data.value.items[i]; var li = document.createElement('li'); var a = document.createElement('a'); a.href = now.link; a.appendChild( document.createTextNode(now.title) ); li.appendChild(a); ul.appendChild(li); } x.appendChild(ul); } } }; return{ tweet:tweet }}();

One of my favourite toys:

Using GreaseMonkey you can change any web page out

there with DOM and JavaScript.

You can for example prototype an enhancement and show it to people with a

single link.

By playing with the web you can do jobs that until now

cost a lot of money.

Say you want to help your clients find good keywords to promote their product online.

You can do some research, surf all the competitors’ sites

and note down the descriptions, keywords and

titles.

Or you can be a man and use cURL to write a script to do

that.

Or you can be a clever man and keep your eyes open and

check if there is an API for that.

http://developer.yahoo.com/search/boss/

http://boss.yahooapis.com/ysearch/web/v1/donkeys?format=xml&appid=...

http://boss.yahooapis.com/ysearch/web/v1/donkeys?format=xml&view=keyterms&appid=...

All you need to do is getting the top 20, analyzing the keyword frequency and

create a top 20.

http://developer.yahoo.net/blog/archives/2008/11/boss_keywords.html

Then you take YUI CSS grids, and spend 30 minutes playing

with colours and fonts.

And you have a product:

http://keywordfinder.org

This is all cool, but does it bring us anywhere?

Yes, if you get excited about the web and its opportunities

you can move things.

It takes determination!

And the outcome can be rewarding beyond anything

you ever imagined.

http://www.youtube.com/watch?v=CwsDKaalgq8&

http://www.youtube.com/watch?v=QiuT0y0KR6I

So by all means, put all your wonderful products on the

web.

Especially when they cater a very specific need.

The prime number shitting bear...

http://alpha61.com/primenumbershittingbear/

And never stop to fiddle, tweak and poke at the things people offer you on the web.

So thank you for having me here and listening.

And I hope you will have an awesome Christmas!

Christian Heilmann http://scriptingenabled.org | http://wait-till-i.com

twitter/flickr: codepo8