batch processing and stream processing by sql

Batch processing andStream processing by

SQL@tagomoris (TAGOMORI Satoshi)2014/07/08Hadoop Conference Japan 2014 #hcj2014

14年7月8日火曜日

TAGOMORI Satoshi (@tagomoris)LINE Corporation

Analytics Platform Team


SQL


BATCHand/or

STREAM14年7月8日火曜日

Analytics data flow overviewservers Fluentd

Cluster

archive

visualization

notifications

Hadoop / HivePresto

Fluentd

Norikra

applicationmetrics

“Log analysis systems and its designs in LINE corp. 2014 early”http://www.slideshare.net/tagomoris/log-analysis-system-and-its-designs-in-line-corp-2014-early14年7月8日火曜日

http://www.slideshare.net/tagomoris/log-analysis-system-and-its-designs-in-line-corp-2014-early

http://www.slideshare.net/tagomoris/log-analysis-system-and-its-designs-in-line-corp-2014-early

servers FluentdCluster

archive

visualization

notifications

Hadoop / HivePresto

Fluentd

Norikra

applicationmetrics

STREAM

BATCH


servers FluentdCluster

archive

visualization

notifications

Hadoop / HivePresto

Fluentd

Norikra

applicationmetrics

STREAM

BATCHSQL


SQL is NOT the best.

But,SQL is better than NONE.


What supports SQL:

RDBMSApache Hive (on MR/Spark/Tez)

Facebook Presto, Cloudera Impala, Apache DrillGoogle BigQuery, ......


SQL

SQLSQL

SQL (2/6)SQL

SQL

SQL SQL


DB Batch ShortBatch

non-SQL NoSQL HadoopMRPig ----

SQL RDBMS HivePrestoImpala

Drill


Batch processing.

ORStream processing?


Batch processing

Hadoop/Hive

Target window: hours - weeks (or more)

Total throuput: HIGHEST

Query Latency: LARGEST (20sec - mins - hours)


Short Batch processing

Presto, Impala, Drill

Target window: seconds - hours (- days)

Total throughput: Normal

Query latency: Small (seconds - mins)


Stream processing

Storm, Kafka, Esper, Norikra, Fluentd, ....

Spark streaming(?)

Target window: seconds - hours

Total throughput: Normal

Query latency: SMALLEST (milliseconds)

Queries must be written BEFORE DATA

Once registered, runs forever


Data flow and latencydata windowquery execution

BatchShortBatch Stream

incrementalquery exection


Data windowTarget time (or size) range of queries

Batch (or short-batch)

FROM-TO: WHERE dt >= ‘2014-07-07 00:00:00‘

AND dt <= ‘2014-07-08 23:59:59’

Stream

“Calculate this query for every 3 minutes”

Extended SQL required


Stream processing with SQLEsper: Java library to process StreamWith schema


Stream processing with SQLEsper: Java library to process StreamEsper EPL

SELECT param1, param2FROM tblWHERE age > 30


Stream processing with SQL

SELECT param, COUNT(*) AS cFROM tblWHERE age > 30GROUP BY param

Esper: Java library to process StreamEsper EPL


Stream processing with SQL

SELECT param, COUNT(*) AS cFROM tbl.win:time_batch(1 hour)WHERE age > 30GROUP BY param

Esper: Java library to process StreamEsper EPL


Norikra:Schema-less Stream Processing with SQL

OSS, based on Esper EPL, GPLv2

Without pre-defined schema

Complex event processing (w/ nested hash/array) w/ SQL

HTTP RPC w/ JSON or MessagePack (fluentd plugin available!)

Dynamic query registration/removing

Ultra fast bootstrap (in 3 minutes!)

UDF plugins by Java/Rubyhttp://norikra.github.io/


http://norikra.github.io/

http://norikra.github.io/

Distributed processing OR NOT?

Norikra is NOT a distributed processing platform.

Of course, SCALE OUT IS FANTASTIC.

Is non-distributed software useless?

MySQL

MySQL Cluster

Norikra can handle 10k events/sec

on 2CPU (8core) server


DB Batch ShortBatch Stream

non-SQL NoSQL HadoopMRPig ----

StormKafka

Dataflow(G)

SQL RDBMS HivePrestoImpala

DrillNorikra


Lambda architecture

Just same 2 process on:Stream processingBatch processing

http://lambda-architecture.net/




Replayable processing

Stream processingMUST NOT be replayable

Queries on stream processingSHOULD be replayable


Hybrid processing: for fault-torelance

Stream processing: executes queries in normalBatch processing: executes recovery queries


Hybrid processing: for latency-reduction + accuracy

Stream processing: for prompt reports (速報値)

Batch processing: for fixed reports (確定値)


Hybrid stream processing: against complexity

Non-SQL stream processing: for simple, fixed, high-traffic eventsSQL stream processing: for complex, fragile events


Case study in LINE

Prompt-report & fixed-report

Norikra + Hive Hybrid

Error detection from application and access logs

Norikra + Fluentd Hybrid

Realtime aggregation for complex and simple(fixed) objects

Norikra + Fluentd Hybrid


Hive: fixed-reportsSELECT yyyymmdd, hh, campaign_id, region, lang, COUNT(*) AS click, COUNT(DISTINCT member_id) AS uuFROM ( SELECT yyyymmdd, hh, get_json_object(log, '$.campaign.id') AS campaign_id, get_json_object(log, '$.member.region') AS region, get_json_object(log, '$.member.lang') AS lang, get_json_object(log, '$.member.id') AS member_id FROM applog WHERE service='myservice' AND yyyymmdd='20140708' AND hh='00' AND get_json_object(log, '$.type')='click') xGROUP BY yyyymmdd, hh, campaign_id, region, lang


Norikra: prompt-reports

SELECT campaign.id AS campaign_id, member.region AS region, member.lang AS lang, COUNT(*) AS click, COUNT(DISTINCT member.id) AS uuFROM myservice.win:time_batch(1 hours)WHERE type="click"GROUP BY campaign.id, member.region, member.lang


More queries, more simplicityand less latency.

Thanks!


batch processing and stream processing by sql

Technology