queue your work
DESCRIPTION
Uncon talk from PHPBenelux 2014.TRANSCRIPT
![Page 1: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/1.jpg)
Queue your work
Jurian Sluiman - uncon session PHPBenelux 2014
![Page 2: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/2.jpg)
About me
• Jurian Sluiman• Founder @ soflomo (Delft, The Netherlands)• Web development (eCommerce, health care)• Zend Framework 2 contributor• Blogger: http://juriansluiman.nl• Twitter: @juriansluiman
![Page 3: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/3.jpg)
Queue systems
• Execute tasks in background• Return your response fast!
Server
request
response
![Page 4: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/4.jpg)
Queue systems
• Execute tasks in background• Return your response fast!
• Producer: piece of code pushing new jobs• Worker: piece of code consuming jobs
Server(Producer)
request
response WorkerQueue
![Page 5: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/5.jpg)
Queue systems
Server(Producer)
request
response
Worker
Server(Producer)
request
response
Server(Producer)
request
response
Worker
Queue
Queue
Worker
Worker
![Page 6: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/6.jpg)
So why?
Advantages• Asynchronous• Resilience• Scalable• Atomic
Disadvantages• Asynchronous• Complexity (is it?)
![Page 7: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/7.jpg)
Types
1.Dedicated job queues• Gearman, beanstalkd, Celery
2.Message queues• RabbitMQ, ZeroMQ, ActiveMQ
3.SaaS queues• Amazon SQS, IronMQ
4.Databases• Redis, (My)SQL
A list of most of the queues: http://queues.io
![Page 8: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/8.jpg)
Job queues
1.Gearman• Widely known • PECL extension available (http://php.net/gearman)
2.Beanstalkd• Small footprint• Very easy setup
![Page 9: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/9.jpg)
Gearman
1. Run task in background// producer$client = new GearmanClient();$client->addServer(); // 127.0.0.1:4730$client->doBackground('email', $workload);
// worker$worker = new GearmanWorker();$worker->addServer();$worker->addFunction('email', function($job) { $params = $job->getWorkLoad(); mail($params['to'], $params['subject'], $params['msg']);};
while($worker->work());
![Page 10: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/10.jpg)
Gearman
2. Normal tasks$client = new GearmanClient();$client->addServer();$result = $client->doNormal('generate-key');// use $result
![Page 11: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/11.jpg)
Gearman
2. Normal tasks$client = new GearmanClient();$client->addServer();$result = $client->doNormal('generate-key');// use $result
3. Multiple servers$servers = array('192.168.1.200', '192.168.1.201');shuffle($servers);
$client = new GearmanClient();$client->addServers(implode(',', $servers));
![Page 12: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/12.jpg)
Gearman
Advantages• Multi-tenant• Job status• Priorities• Persistent
Disadvantages• Requires compilation!• Pre-defined functions• Opt-in persistency
![Page 13: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/13.jpg)
Beanstalkd
1. Run task in background// producer$client = new Pheanstalk_Pheanstalk('127.0.0.1');$client->put($data);
// worker$worker = new Pheanstalk_Pheanstalk('127.0.0.1');while($job = $worker->reserve()) { $data = $job->getData(); $function = $data['function']; $args = $data['args'];
call_user_func_array($function, $args);}
![Page 14: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/14.jpg)
Beanstalkd
2. Priority & delayed task$client = new Pheanstalk_Pheanstalk('127.0.0.1');$prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY;$delay = 5 * 60;$client->put($data, $prio, $delay);
![Page 15: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/15.jpg)
Beanstalkd
2. Priority & delayed task$client = new Pheanstalk_Pheanstalk('127.0.0.1');$prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY;$delay = 5 * 60;$client->put($data, $prio, $delay);
3. Using tubes// producer$client = new Pheanstalk_Pheanstalk('127.0.0.1');$client->putInTube('image', $data);
// worker$client = new Pheanstalk_Pheanstalk('127.0.0.1');$job = $client->reserveFromTube('image');
![Page 16: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/16.jpg)
Beanstalkd
Advantages• Fast!• Priorities & delays• Job life cycle• Time-to-run• Persistent• Flexible payload
Disadvantages• Single-tenant• Fairly unknown
![Page 17: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/17.jpg)
How fast?
Gearman• 1mln jobs• Push:~205 seconds
~4900 ops/sec• Pull: ~ 180 seconds
~ 5500 ops /sec
Beanstalkd• 1mln jobs• Push:~120 seconds
~ 8300 ops/sec• Pull: ~ 150 seconds
~ 6600 ops/sec
Tested on a 2.5GHz VPS, 1 core, 1GB RAM – http://gist.github.com/juriansluiman/8593421
![Page 18: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/18.jpg)
Message queues
1.RabbitMQ• AMQP implementation (Message-Oriented-Middleware)• High availability, multi-tenancy, persistency
2.ØMQ “ZeroMQ”• Higher throughput than TCP• Flexible socket library, create your own patterns
![Page 19: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/19.jpg)
ØMQ
Example// producer$context = new ZMQContext();$producer = new ZMQSocket($context, ZMQ::SOCKET_REQ);$producer->bind(“tcp://localhost:5555”);
$producer->send($payload);
// worker$context = new ZMQContext();$worker = new ZMQSocket($context, ZMQ::SOCKET_REP);$worker->connect(“tcp://*:5555”);
while(true) { $payload = $worker->recv();}
![Page 20: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/20.jpg)
Message queues
Advantages• Extremely flexible• AMQP• Message broker• Extreme & advanced
Disadvantages• Extremely flexible• DIY• No “best way”• Routing, pub/sub,
ventilators, channels
![Page 21: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/21.jpg)
Software-as-a-Service
1.Amazon Simple Queue Service (SQS)• Useful within EC2 instances• http://aws.amazon.com/sqs
2.IronMQ• They say they're better than SQS• http://iron.io/mq
![Page 22: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/22.jpg)
Advantages• No maintenance• Easy to set-up• Easy to scale
Disadvantages• HTTP latency• Distributed (SQS)• Cost
Software-as-a-Service
![Page 23: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/23.jpg)
1.Redis• Extremely light-weight key/value store• BRPOPLPUSH• Watch back Ross Tuck @ PHPBenelux 2013
2.(My)SQL• Only if you have to
Databases
![Page 24: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/24.jpg)
Databases
Advantages• Well known set-up• OK for shared hosting
Disadvantages• Not meant for jobs• DIY
![Page 25: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/25.jpg)
Queue abstraction layers
1.SlmQueue (April 2012)
2.php-queue (September 2012)
3.BBQ (June 2013)
4.Laravel Queue component
![Page 26: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/26.jpg)
SlmQueue
• Supports Beanstalkd, SQS and Doctrine• Redis and Gearman support coming• Integrated with ZF2, but not required• Packagist: slm/queue + slm/queue-beanstalkd• GitHub: http://github.com/juriansluiman/SlmQueue
![Page 27: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/27.jpg)
SlmQueue
SlmQueue\Job\JobInterfaceinterface JobInterface{ public function execute();}
![Page 28: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/28.jpg)
SlmQueue
SlmQueue\Job\JobInterfaceinterface JobInterface{ public function execute();}
SlmQueue\Queue\QueueInterfaceinterface QueueInterface{ public function push(JobInterface $job, $options); public function pop();}
![Page 29: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/29.jpg)
SlmQueue
SlmQueue\Worker\WorkerInterfaceinterface WorkerInterface{ public function processQueue($name, $options); public function processJob(JobInterface $job);}
![Page 30: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/30.jpg)
Workers
Run via ZF2 appphp public/index.php queue beanstalkd default
Configuration• Time-out for blocking calls• Maximum number of cycles• Maximum memory usage• Signal handlers for SIGTERM and SIGINT
![Page 31: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/31.jpg)
Dependency injection
MyModule\Job\Emailclass Email extends AbstractJob{ protected $transport;
public function __construct(Transport $transport) { $this->transport = $transport; }
public function execute() { $payload = $this->getContent(); $message = new Message; $message->setTo($payload['to']); $message->setMessage($payload['message']);
$this->transport->send($message); }}
![Page 32: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/32.jpg)
Dependency injection
MyModule\Factory\EmailJobFactoryuse MyModule\Job\Email as EmailJob;class EmailJobFactory implements FactoryInterface{ public function createService(ServiceLocator $sl) { $transport = $sl->get('MyEmailTransport'); return new EmailJob($transport); }}
module.config.php'slm_queue' => [ 'job_manager' => [ 'MyEmailJob' => 'MyModule\Factory\EmailJobFactory' ]]
![Page 33: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/33.jpg)
Dependency injection
Example: ZF2 Controllerpublic function fooAction(){ $payload = array( 'to' => '[email protected]', 'message' => 'Hi there!', );
$this->queue('default') ->push('MyEmailJob', $payload);}
![Page 34: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/34.jpg)
Queue aware jobs
MyModule\Job\Fooclass Foo extends AbstractJob implements QueueAwareInterface{ use QueueAwareTrait;
public function execute() { // work here
$job = new BarJob(); $this->getQueue()->push($job); }}
![Page 35: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/35.jpg)
Pro-tips
1.Start experimenting now!
2.Atomic jobs
3.Log everything
4.Use a control system like supervisord
![Page 36: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/36.jpg)
Questions?
Jurian Sluiman - uncon session PHPBenelux 2014
![Page 37: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/37.jpg)
Jobs in services
MyModule\Service\Fooclass Foo{ protected $queue;
public function __construct(QueueInterface $queue) { $this->queue = $queue; }
public function doSomething() { // work here
$job = new BarJob; $this->queue->push($job); }}
![Page 38: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/38.jpg)
Lazy servicesclass Buzzer{ public function __construct() { sleep(5); }
public function buzz() { // do something }}
Lazy services with a Proxy pattern by Marco Pivetta (Ocramius)
![Page 39: Queue your work](https://reader033.vdocuments.mx/reader033/viewer/2022050801/5557920bd8b42ad4278b5183/html5/thumbnails/39.jpg)
Lazy servicesclass BuzzerProxy extends Buzzer{ private $sl; private $instance;
public function __construct(ServiceLocator $sl) { $this->sl = $sl; }
private function initialize() { $this->initialized = true; $this->original = $this->sl->get('Buzzer'); } public function buzz() { if (!$this->initialized) { $this->initialize(); } return $this->instance->buzz(); }}