Stream processing in Mercari

Developers Summit 2015 Autumn LT@kazeburo Masahiro Nagano

Me

•長野雅広(Masahiro Nagano)

• @kazeburo

• Mercari, Inc.

• Principal Engineer, Site Reliability

Your Friendly Mobile Marketplace

Stream processing for Monitoring

Monitoring

collect metrics value• Just get current value/point

• e.g. load avg, memory usage

• Get counter/incremental value and calculate changes

• e.g. traffic, cpu usage

• Parse log and count value by your self

• e.g. access_log, error_log, custom_log

Log monitoringPast and present

While ago• Make a script for parsing and count logs

• Execute periodically from monitoring tool

• Too many tail (io) and grep (cpu) (;_;)

#!/bin/shset -eLOG_WATCH=20000TARGET=$(date -d '61 second ago' +%H:%M:[0-9][0-9])tail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" | wc -ltail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" | grep '" 500 ' wc -l

fluentd + datacounterWeb

access_log

in_tail

Web

access_log

Web

access_log

aggregate

datacounter

<match mercari.access_log> type datacounter count_interval 1m count_key status pattern1 2xx ^2\d\d$ pattern2 3xx ^3\d\d$ pattern4 4xx ^4\d\d$ pattern5 5xx ^5\d\d$ tag status.datacount.web</match>

<match status.datacount.web> type zabbix zabbix_server 10.x.x.x</match>

Cons of fluentd+datacounter

• Gigantic fluend.conf

• need a configuration file generator?

• Write fluend plugin for responding various needs?

• restart fluentd is required for adding a new tabulation

Norikra

• Norikra is an awesome open source product by Mr. tagomori, provides “Stream processing” with SQL

• We LOVE SQL!!

219.109.xx.xx - - [09/Nov/2014:06:50:07 +0900] "GET /mt.js HTTP/1.1" 200 6339 "http://blog.nomadscafe.jp/2013/06/webdb-pressvol172.html" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"50.116.xx.xx - - [09/Nov/2014:06:54:41 +0900] "GET /2014/09/line-isucon4-51192.html HTTP/1.1" 200 30194 "http://www.google.co.uk/url?sa=t&source=web&cd=1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.92 Safari/535.2"76.164.xx.xx - - [09/Nov/2014:07:09:21 +0900] "GET / HTTP/1.0" 200 75606 "http://blog.nomadscafe.jp/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"

SELECT COUNT(1, status like “5%”) AS count_5x, COUNT(1, status like “2%”) AS count_2x FROM access_log.win:time_batch(1 min)

{ “count_2x”: 765, “count_5x”: 2 }

Norikra

After NorikaWeb

access_log

in_tail

Web

access_log

Web

access_log

aggregate

SQL投入

Norikra

Pros of Norikra• Do not need a gigantic configuration file

• Do not required restarting a daemon

• Schema less

• analysis any log data like error_log, application specific log

• SQL

• every engineer/producer can write SQL

Graph/Alert• zabbix

• nagios

• GrowthForecast (graph only)

• Data Dog

• mackerel

Worker/Batch

WebWeb

Stream Processing in MercariWeb/App

access_log

Worker/Batch

create SQL

Norikra

error_log

app_log

error_log

app_logdefine Alert

Mackerel

Generates Graphautomatically!!

Alternatives and Competitors• PipelineDB

• www.pipelinedb.com

• Azure Stream Analytics

• azure.microsoft.com/ja-jp/services/stream-analytics/

• Amazon Kinesis Analytics(coming soon)

• aws.amazon.com/kinesis/analytics/

end