please hold: your call is in a queue. an introduction to queuing in drupal
DESCRIPTION
Please hold: your call is in a queue. An introduction to queuing in Drupal. My talk from Drupalcamp London 1st-3rd March 2013. A version of this talk with minor modifications was also given at the Drupal Learning April 2013 meetup in London (http://www.meetup.com/Learning-Drupal-Meetup/events/111107222/). It's not just the Brits that love queues, Drupal loves them too. Maybe your website is making requests to another internal or third party system. Maybe a third party is sending messages to you. Either way, as you begin to grow your services are increasingly likely to run into problems completing these requests in a predictable amount of time and with guaranteed success. Queueing is one solution to this, balancing load, ensuring jobs are processed successfully - but the uses don't end there. This session will provide an overview of queueing patterns, discuss some scenarios where you may want to implement queues, and look at how Drupal implements queueing - as well as some of the gotchas in implementing queue processing at scale.TRANSCRIPT
![Page 1: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/1.jpg)
PLEASE HOLD: YOUR CALL IS IN A QUEUE
An introduction to queueing in Drupalby Tom Phethean
Sunday, 3 March 13
![Page 2: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/2.jpg)
QUEUING IN 30 MINUTES...
• I hate waiting in line...
•Drupal Queue API
• Beyond Core
•Queue jumpers and other gotchas
•Questions
Sunday, 3 March 13
![Page 3: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/3.jpg)
I HATE WAITING IN LINE...
Sunday, 3 March 13
![Page 4: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/4.jpg)
Sunday, 3 March 13
![Page 5: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/5.jpg)
Sunday, 3 March 13
![Page 6: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/6.jpg)
Sunday, 3 March 13
![Page 7: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/7.jpg)
Sunday, 3 March 13
![Page 8: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/8.jpg)
QUEUES WILL FORM WHEN PROCESSES WITH VARIABILITY
ARE LOADED TO HIGH LEVELS OF UTILISATION
WITHOUT CONSTRAINT
Sunday, 3 March 13
![Page 9: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/9.jpg)
SOMETHING WILL QUEUE SOMEWHERE
Sunday, 3 March 13
![Page 10: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/10.jpg)
SOMETHING WILL QUEUE SOMEWHERE
Whether you like it or not
Sunday, 3 March 13
![Page 11: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/11.jpg)
DO YOU WANT TO BE IN CONTROL?
Sunday, 3 March 13
![Page 12: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/12.jpg)
DO YOU WANT TO BE IN CONTROL?
</silly_question>
<silly_question>
Sunday, 3 March 13
![Page 13: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/13.jpg)
DRUPAL TO THE RESCUE!
Sunday, 3 March 13
![Page 14: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/14.jpg)
CORE QUEUE API
• Two kinds of queue: reliable and non-reliable
• Provides interface to common queue functions
•MemoryQueue implements QueueInterface
• SystemQueue implements ReliableQueueInterface
• In Drupal 7 (and 8), Batch API extends SystemQueue
Sunday, 3 March 13
![Page 15: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/15.jpg)
WHERE CAN I FIND QUEUE API?
•Drupal 7:
• ./modules/system/system.queue.inc
•Drupal 8:
• ./core/lib/Drupal/Core/Queue
Sunday, 3 March 13
![Page 16: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/16.jpg)
QUEUE API METHODS
• createQueue()
• createItem()
• claimItem()
• releaseItem()
• numberOfItems()
• deleteItem()
• deleteQueue()
Sunday, 3 March 13
![Page 17: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/17.jpg)
Code time.
Sunday, 3 March 13
![Page 18: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/18.jpg)
/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}
/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');
$queue_data = array ( 'stuff' => 'test', );
return $queue->createItem($queue_data);}
/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');
return $queue->numberOfItems();}
/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');
$item = $queue->claimItem(30);
if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}
/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}
/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}
Sunday, 3 March 13
![Page 19: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/19.jpg)
/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}
/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');
$queue_data = array ( 'stuff' => 'test', );
return $queue->createItem($queue_data);}
/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');
return $queue->numberOfItems();}
/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');
$item = $queue->claimItem(30);
if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}
/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}
/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}
Sunday, 3 March 13
![Page 20: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/20.jpg)
/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}
/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');
$queue_data = array ( 'stuff' => 'test', );
return $queue->createItem($queue_data);}
/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');
return $queue->numberOfItems();}
/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');
$item = $queue->claimItem(30);
if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}
/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}
/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}
Sunday, 3 March 13
![Page 21: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/21.jpg)
/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}
/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');
$queue_data = array ( 'stuff' => 'test', );
return $queue->createItem($queue_data);}
/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');
return $queue->numberOfItems();}
/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');
$item = $queue->claimItem(30);
if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}
/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}
/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}
Sunday, 3 March 13
![Page 22: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/22.jpg)
/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}
/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');
$queue_data = array ( 'stuff' => 'test', );
return $queue->createItem($queue_data);}
/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');
return $queue->numberOfItems();}
/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');
$item = $queue->claimItem(30);
if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}
/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}
/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}
Sunday, 3 March 13
![Page 23: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/23.jpg)
That’s it!
Sunday, 3 March 13
![Page 24: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/24.jpg)
BEYOND CORE
• Drupal Queue (backport to D6) - http://drupal.org/project/drupal_queue
• Queue UI - http://drupal.org/project/queue_ui
• Examples - http://drupal.org/project/examples
• Queue backends:
• Beanstalkd - http://drupal.org/project/beanstalkd
• STOMP - http://drupal.org/project/stomp
• Redis - http://drupal.org/project/redis_queue
Sunday, 3 March 13
![Page 25: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/25.jpg)
CUSTOM BACKENDS
•Default queue class:
• $conf['queue_default_class'] = 'SystemQueue';
• $conf[‘queue_default_reliable_class’] = ‘SystemQueue’;
•Override queue classes:
• $conf['queue_class_{queue name}'] = 'StompQueue';
Sunday, 3 March 13
![Page 26: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/26.jpg)
CUSTOM BACKENDS
•Default queue class:
• $conf['queue_default_class'] = 'SystemQueue';
• $conf[‘queue_default_reliable_class’] = ‘SystemQueue’;
•Override queue classes:
• $conf['queue_class_{queue name}'] = 'RedisQueue';
Sunday, 3 March 13
![Page 27: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/27.jpg)
QUEUES AND DRUSH
• drush queue-list
• drush queue-run [queue name]
• ...use with caution: queue-run will always delete your queue item
Sunday, 3 March 13
![Page 28: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/28.jpg)
WHEN TO QUEUE
• Remote service call triggered by user interface
• Complex business rules around data processing
•Need guaranteed success
Sunday, 3 March 13
![Page 29: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/29.jpg)
SOME EXAMPLES
• Publishing user information to a remote CRM
• Backend order fulfilment
• Email sending (see notifications module)
Sunday, 3 March 13
![Page 30: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/30.jpg)
GOTCHAS
Sunday, 3 March 13
![Page 31: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/31.jpg)
MULTIPLE CONSUMERS
Sunday, 3 March 13
![Page 32: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/32.jpg)
MULTIPLE CONSUMERS
• Identify what it is you’re processing
• Use Lock API to secure a lock on it:
• lock_acquire(‘my_id’)
• lock_release(‘my_id’)
• Check the “thing” is in a state you expect it to be
• Release or Delete from queue as appropriate
Sunday, 3 March 13
![Page 33: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/33.jpg)
DEBUGGING YOUR QUEUE
Sunday, 3 March 13
![Page 34: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/34.jpg)
DEBUGGING YOUR QUEUE
• watchdog() is your friend
• Log EVERYTHING
• Abstract it so you can toggle verbose logging
• Use Graylog, LogStash etc to save your database from death by logs
Sunday, 3 March 13
![Page 35: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/35.jpg)
THINGS CAN GO WRONG...
Sunday, 3 March 13
![Page 36: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/36.jpg)
THINGS CAN GO WRONG...
•Monitor your queue depths with Munin or Nagios
• Alert if they get too deep
• Know what the alert means (and listen to it)!
Sunday, 3 March 13
![Page 37: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/37.jpg)
THINGS BREAK. BUT WHEN THEY DO, KNOW THAT EVERY ITEM IN YOUR QUEUE
IS A REQUEST WHICH WOULD OTHERWISE HAVE BEEN LOST.
Sunday, 3 March 13
![Page 38: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/38.jpg)
IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,
Sunday, 3 March 13
![Page 39: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/39.jpg)
IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,
MAKE THROUGHPUT PREDICTABLE,
Sunday, 3 March 13
![Page 40: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/40.jpg)
IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,
MAKE THROUGHPUT PREDICTABLE, REMOVES PROCESSING TIME FROM
THE USER EXPERIENCE
Sunday, 3 March 13
![Page 41: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/41.jpg)
IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,
MAKE THROUGHPUT PREDICTABLE, REMOVES PROCESSING TIME FROM
THE USER EXPERIENCE AND ENSURES THAT PROBLEMS ARE
HANDLED GRACEFULLY
Sunday, 3 March 13
![Page 42: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/42.jpg)
IN OTHER WORDS...
Sunday, 3 March 13
![Page 43: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/43.jpg)
Sunday, 3 March 13
![Page 44: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/44.jpg)
QUESTIONS?
Tweet: @tsphethean Drupal.org: tsphethean
IRC: tsphetheanSunday, 3 March 13
![Page 45: Please hold: your call is in a queue. An introduction to queuing in Drupal](https://reader033.vdocuments.mx/reader033/viewer/2022060109/5552f4aeb4c90587048b4ce9/html5/thumbnails/45.jpg)
IMAGE ATTRIBUTIONS
• Image of supermarket queue - http://petoneponderings.blogspot.co.uk/2012_12_01_archive.html
• Post Office queue - http://www.flickr.com/photos/welshkaren/5367517662/
• Apple Store queueing http://www.telegraph.co.uk/technology/apple/7854982/Apple-iPhone-4-more-than-a-quarter-of-users-think-video-calling-is-a-gimmick.html
• Supermarket crush - http://www.dailymail.co.uk/news/article-2078597/Boxing-Day-sales-Record-numbers-shops-open-80-push-grab-shoppers.html
• Drupal Superhero - http://www.origineight.net/blog-post/web-sites-you-manage
• Lots of Shoppers - http://www.dailymail.co.uk/news/article-2253523/United-Nations-Bargain-Hunters-Britons-the-queue-China-Middle-East-lead-rush-designer-brands.html
• Confused baby - http://thementalpausechronicles.blogspot.co.uk/2008/08/head-scratching.html
• Mark Sonnabaum - http://www.whatwouldmarksonnabaumdo.com
Sunday, 3 March 13