Stream processing in Mercari
Developers Summit 2015 Autumn LT@kazeburo Masahiro Nagano
Me
•長野雅広(Masahiro Nagano)
• @kazeburo
• Mercari, Inc.
• Principal Engineer, Site Reliability
Your Friendly Mobile Marketplace
Stream processing for Monitoring
Monitoring
collect metrics value• Just get current value/point
• e.g. load avg, memory usage
• Get counter/incremental value and calculate changes
• e.g. traffic, cpu usage
• Parse log and count value by your self
• e.g. access_log, error_log, custom_log
Log monitoringPast and present
While ago• Make a script for parsing and count logs
• Execute periodically from monitoring tool
• Too many tail (io) and grep (cpu) (;_;)
#!/bin/shset -eLOG_WATCH=20000TARGET=$(date -d '61 second ago' +%H:%M:[0-9][0-9])tail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" | wc -ltail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" | grep '" 500 ' wc -l
fluentd + datacounterWeb
access_log
in_tail
Web
access_log
Web
access_log
aggregate
datacounter
<match mercari.access_log> type datacounter count_interval 1m count_key status pattern1 2xx ^2\d\d$ pattern2 3xx ^3\d\d$ pattern4 4xx ^4\d\d$ pattern5 5xx ^5\d\d$ tag status.datacount.web</match>
<match status.datacount.web> type zabbix zabbix_server 10.x.x.x</match>
Cons of fluentd+datacounter
• Gigantic fluend.conf
• need a configuration file generator?
• Write fluend plugin for responding various needs?
• restart fluentd is required for adding a new tabulation
Norikra
• Norikra is an awesome open source product by Mr. tagomori, provides “Stream processing” with SQL
• We LOVE SQL!!
219.109.xx.xx - - [09/Nov/2014:06:50:07 +0900] "GET /mt.js HTTP/1.1" 200 6339 "http://blog.nomadscafe.jp/2013/06/webdb-pressvol172.html" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"50.116.xx.xx - - [09/Nov/2014:06:54:41 +0900] "GET /2014/09/line-isucon4-51192.html HTTP/1.1" 200 30194 "http://www.google.co.uk/url?sa=t&source=web&cd=1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.92 Safari/535.2"76.164.xx.xx - - [09/Nov/2014:07:09:21 +0900] "GET / HTTP/1.0" 200 75606 "http://blog.nomadscafe.jp/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
SELECT COUNT(1, status like “5%”) AS count_5x, COUNT(1, status like “2%”) AS count_2x FROM access_log.win:time_batch(1 min)
{ “count_2x”: 765, “count_5x”: 2 }
Norikra
After NorikaWeb
access_log
in_tail
Web
access_log
Web
access_log
aggregate
SQL投入
Norikra
Pros of Norikra• Do not need a gigantic configuration file
• Do not required restarting a daemon
• Schema less
• analysis any log data like error_log, application specific log
• SQL
• every engineer/producer can write SQL
Graph/Alert• zabbix
• nagios
• GrowthForecast (graph only)
• Data Dog
• mackerel
Worker/Batch
WebWeb
Stream Processing in MercariWeb/App
access_log
Worker/Batch
create SQL
Norikra
error_log
app_log
error_log
app_logdefine Alert
Mackerel
Generates Graphautomatically!!
Alternatives and Competitors• PipelineDB
• www.pipelinedb.com
• Azure Stream Analytics
• azure.microsoft.com/ja-jp/services/stream-analytics/
• Amazon Kinesis Analytics(coming soon)
• aws.amazon.com/kinesis/analytics/
end