low latency logging with rabbitmq (php london - 4th sep 2014)
DESCRIPTION
Logging is an absolute must for any API or web application, but when starting out, questions such as "how can we do it without disrupting everything else" and "what is the easiest way to log" often come up. We’re going to examine a tried and tested method to carry out high-performance, low-latency logging using the power of RabbitMQ to ensure minimal impact to the performance of your runtime application. The talk will show you that a really great logging architecture is a low-cost investment in your application that will definitely pay off in the long run.TRANSCRIPT
![Page 1: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/1.jpg)
Low Latency Loggingwith RabbitMQ
PHP London4th September 2014
![Page 2: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/2.jpg)
James Titcumbwww.jamestitcumb.comwww.protected.co.ukwww.phphants.co.uk@asgrim
Who is this guy?
![Page 3: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/3.jpg)
Let’s go back to basics...
![Page 4: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/4.jpg)
Errors
![Page 5: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/5.jpg)
source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
![Page 6: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/6.jpg)
![Page 7: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/7.jpg)
![Page 8: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/8.jpg)
![Page 9: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/9.jpg)
![Page 10: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/10.jpg)
error_reporting(0);
![Page 11: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/11.jpg)
They look rubbish!
![Page 12: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/12.jpg)
@
![Page 13: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/13.jpg)
Log to file only
![Page 14: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/14.jpg)
Ways Around// Will raise E_WARNING
fopen($somefile, 'r');
![Page 15: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/15.jpg)
Ways Around// No error! :)
if (file_exists($somefile)) {
fopen($somefile, 'r');
} else {
// nice handling...
// maybe throw exception...
}
![Page 16: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/16.jpg)
Exceptions
![Page 17: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/17.jpg)
Jargon Buster
● throwTriggering
● tryTry to run
● catchHandle it
● finallyRun code after try/catch
![Page 18: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/18.jpg)
source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
![Page 19: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/19.jpg)
C♯/C++
![Page 20: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/20.jpg)
Catchable
![Page 21: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/21.jpg)
Turn into fatal errorsif not caught
![Page 22: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/22.jpg)
Classy
![Page 23: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/23.jpg)
Descriptive exceptions
![Page 24: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/24.jpg)
Example (exception class)class DivisionByZeroException
extends LogicException
{
}
![Page 25: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/25.jpg)
Example (throw)class Mathematics {
public function divide($a, $b) {
if ($b == 0) {
$msg = sprintf(“Divide %s by zero”, $a);
throw new DivisionByZeroException($msg);
}
return ($a / $b);
}
}
![Page 26: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/26.jpg)
Example (catch)$maths = new Mathematics();
try
{
$result = $maths->divide(5, 0);
}
catch (DivisionByZeroException $exception)
{
$logger->warning($exception->getMessage());
}
![Page 27: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/27.jpg)
Logging
![Page 28: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/28.jpg)
![Page 29: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/29.jpg)
What is “logging”?
![Page 30: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/30.jpg)
What is “logging”?Keeping a record of all events...
![Page 31: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/31.jpg)
What is “logging”?Keeping a record of all events...
exceptions, errors, warnings, info, debug
![Page 32: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/32.jpg)
Logging is a paper trail
![Page 33: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/33.jpg)
Log like you just don’t care
![Page 34: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/34.jpg)
![Page 35: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/35.jpg)
Log wherever you like
![Page 36: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/36.jpg)
What about Apache’s error_log?
source: http://up-ship.com/blog/?p=20903
![Page 37: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/37.jpg)
Doin’ it rightwrong… // Magic file that makes your entire project work perfectly
@ini_set('display_errors', 0);
@error_reporting(0);
function __globalErrorHandler() {
return true;
}
@set_error_handler('__globalErrorHandler');
@set_exception_handler('__globalErrorHandler');
@register_shutdown_function(function() {
if(error_get_last())
echo "Script executed successfully!";
}); https://github.com/webarto/boostrap.php
![Page 38: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/38.jpg)
Requirements (for everyone)
● Fire & forget● Minimum or zero latency● Highly available● Log everything● PSR-3 compatible
![Page 39: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/39.jpg)
PSR-3
![Page 40: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/40.jpg)
● monolog (PSR-3)● Drupal - PSR-3 Watchdog● phpconsole● log4php● RavenPHP + Sentry● FirePHP (dev environment)● Roll your own
Logging Options
![Page 41: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/41.jpg)
source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html
![Page 42: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/42.jpg)
Capturing Logging
Use “capture methods”, send to $logger
● set_exception_handler()○ Handles all uncaught exceptions
● set_error_handler()○ Handles most errors
● register_shutdown_function()○ Handles fatal errors
![Page 43: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/43.jpg)
Sending Log Messages
● Handler/Adapter translates● However you want…● Monolog has loads:
○ syslog-compatible / error_log○ Email, HipChat○ AMQP, Sentry, Zend Monitor, Graylog2○ Redis, MongoDB, CouchDB
![Page 44: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/44.jpg)
Capture Method
Logger (PSR-3)
Handler / Adapter
Data Storage
Typical PSR-3 Compatible Design
![Page 45: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/45.jpg)
Monolog\ErrorHandler->handleException()
Monolog\Logger->log()
Monolog\Handler->handle()
Monolog
![Page 46: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/46.jpg)
Low LatencyHigh Performance
![Page 47: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/47.jpg)
Slow Logging
ApplicationBrowser Log Server
HTTP request
Send log message to database
Error!
Acknowledge message
HTTP response to client
![Page 48: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/48.jpg)
Zero Latency Logging (ideal)
ApplicationBrowser Log Server
HTTP request
Send log message to database
Error!
HTTP response to client
![Page 49: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/49.jpg)
UDP?
![Page 50: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/50.jpg)
Low Latency Logging (balance)
ApplicationBrowser Log Server
HTTP request
Send log message to database
Error!
HTTP response to client
![Page 51: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/51.jpg)
…so how?
![Page 52: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/52.jpg)
Say hello to RabbitMQ
![Page 53: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/53.jpg)
RabbitMQ - Basic
source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html
Producer Consumer
test_queue
1 2 3 4 5
![Page 54: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/54.jpg)
RabbitMQ - Exchanges
source: http://www.rabbitmq.com/tutorials/tutorial-three-php.html
Exchange
Consumertest_queue
1 2 3 4 5
Consumer
Consumer
Consumer
Consumer
![Page 55: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/55.jpg)
Using Queues === Fast!Add RabbitMQ to logging architecture...
![Page 56: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/56.jpg)
Ed\Log\Handler\ErrorHandler->handleException()
Ed\Log\Logger->log()
Ed\Log\Publisher\AmqpPublisher->publish()
Logging Server
Low Latency (using AMQP)
RabbitMQ JSON payload
![Page 57: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/57.jpg)
Fetch message
Low Latency Logging (with AMQP)
ApplicationBrowser Log Server
HTTP request
JSON via AMQP
Error!
HTTP response
RabbitMQ
![Page 58: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/58.jpg)
Why bother?● Scalability
RabbitMQ
Application A
Application B
Application C
Log Worker
Log Worker
Log Worker
Log Worker
![Page 59: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/59.jpg)
Single Point of Failure...
● RabbitMQ can do HA or clustering
RabbitMQNode 1
RabbitMQNode 3
RabbitMQNode 2
RabbitMQNode 4
RabbitMQNode 5
RabbitMQNode 6
![Page 60: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/60.jpg)
Questions?
![Page 61: Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)](https://reader034.vdocuments.mx/reader034/viewer/2022042607/557b853cd8b42a62418b47f1/html5/thumbnails/61.jpg)
James Titcumbwww.jamestitcumb.comwww.protected.co.ukwww.phphants.co.uk@asgrim
Thanks for watching!