QMapper for Smart Grid: Migrating SQL-based Applicationto HiveYue Wang, Yingzhong Xu, Yue Liu, Jian Chen and Songlin HuSIGMOD’15, May 31–June 4, 2015

Content

•Introduction•System Overview•Query Rewriting•Cost Model•Implementation•Experiments

Introduction• High-level query languages such as Hive, Pig based

on MapReduce have been widely used

• Performance bottlenecks of current RDBMS-based infrastructure appear in traditional enterprises

• Hive can not fully support the SQL syntax at the moment

• Even if some SQL queries used in RDBMS can be directly accepted by Hive, their performance might be very low in the Hadoop

contribution

•Translate SQL to optimized HiveQL

•A cost model is proposed to reflect the execution time of MapReduce jobs

•An algorithm is designed to reorganize the join structure so as to construct the near-optimal query

SECICS System

•The total amount of data is 20TB and there is about 30GB new data added into the database every day

•three kinds of data in SECICS:▫Meter data:collected by smart meters▫Archive data: records the detailed

archived information of meter data▫Statistic data: the result of offline batch

analysis

Background

Background• Low data write throughput

▫RDBMS with complex indexes can not provide enough write throughput

• Unsatisfied statistics analyzing capability▫The average processing time even reaches 3 to

4 hours• Weak scalability

▫scaling out RDBMS mostly leads to redesign of the sharding strategies as well as a lot of application logic.

• Uncontrollable resource competition

The migration of Stored Procedures

Overview

Four Components•SQL Interpreter:

▫resolves the SQL query provided by a user and parses that query into an Abstracted Syntax Tree

•Query Rewriter:▫a Rule-Based Rewriter (RBR) checks if a

query matches a series of static rules, new equivalent queries will be generated

▫Cost-Based Optimizer (CBO) is used to further optimize the join structure for each query

Four Components

•Statistics Collector▫collecting statistics of related tables and

their columns•Plan Evaluator

▫The queries with equivalent join cost generated by RBR will be sent to it

QUERY REWRITING

•Rule-based Rewriter▫detect the SQL clauses that are not

supported well by Hive and transform them into HiveQL

▫initial rules are first invoked to check if the query can be rewritten

▫the RBR will traverse the subqueries of each query and apply rules to them recursively

▫all rewritten queries are generated and sent to the CBO

Example

•lvRate(uid,deviceid,isMissing,date,type)•dataProfile(dataid,uid,isActive)•dataRecord(dataid,date,consumption)•powerCut(uid,date)•gprsUsage(deviceid,dataid,date,gprs)•deviceInfo(deviceid,region,type)

Basic UPDATE Rule• This rule translates UPDATE into SELECT

statement by putting the simpleCondition to selectList

• UPDATE lvRate a SET a.isMissing=trueLEFT OUTER JOIN dataProfile b ON a.uid=b.uidLEFT OUTER JOIN dataRecord c on b.dataid=c.dataidAND a.date=c.date WHERE c.dataid IS NULL

• INSERT OVERWRITE TABLE lvRate SELECT a.uid,a.deviceid,IF(c.dataid IS NULL,true,false) as isMissing,a.date,a.type FROM lvRate LEFT OUTER JOIN dataProfile b ON a.uid=b.uid LEFT OUTER JOIN dataRecord c ON b.dataid=c.dataid AND a.date=c.date

(NOT) EXISTS Rule• transforms that subquery into a LEFT OUTER

JOIN and replaces that (NOT) EXISTS condition with join Column IS (NOT) NULL

• DELETE FROM lvRate a WHERE NOT EXISTS (SELECT 1 FROM powerCut b WHERE a.uid=b.uid AND a.date=b.date )

• INSERT OVERWRITE TABLE lvRate SELECT a.uid,a.deviceid,a.isMissing,a.date,a.type FROM lvRate a LEFT OUTER JOIN ( SELECT uid,date FROM powerCut) b ON a.uid=b.uid AND a.date=b.date WHERE b.uid IS NULL

Cost-based Optimizer• SELECT sum(gprs), type FROM gprsUsage A JOIN deviceInfo

B ON A.deviceid = B.deviceid JOIN dataRecord C ON A.dataid = C.dataid AND A.date = C.date JOIN dataProfile D ON C.dataid = D.dataid LEFT OUTER JOIN powerCut E ON D.uid = E.uid AND A.date = E.date WHERE E.uid IS NULL AND A.date=’2014-01-01’ GROUP BY B.type

• SELECT sum(gprs), type FROM( SELECT T1.gprs, T1.date, T1.type, T2.uid FROM (SELECT A.gprs, A.dataid, A.date, B.type FROM gprsUsage A JOIN deviceInfo B ON A.deviceid = B.deviceid WHERE A.date=’2014-01-01’ )T1 JOIN (SELECT C.dataid, C.date, D.uid FROM dataRecord C JOIN dataProfile D ON C.dataid = D.dataid)T2 ON T1.dataid = T2.dataid

Cost-based Optimizer

Cost-based Optimizer

•Different from traditional databases, MapReduce- based query processing will write join intermediate results back to HDFS and the next join operation will read it from HDFS too, causing big I/O costs

• the main difference in intermediate results is that the left-deep plan generates A B C and the bushy plan generates C D

•B may has worse performance as jobs will compete for computing resources

COST MODEL

•Cost of MapReduce▫Map phase can be divided into three

subphases, which are Map, Spill and Merge.

▫Reduce phase also includes three parts, Shuffle, Merge and Reduce

•Map▫For each Mapper:

Mapper Cost Model

•Spill

•Merge

•Different from normal MapReduce jobs, in Hive, the internal logic of mappers may vary depending on the specific table to be processed.

Reduce

•In the reduce phase, shuffle is responsible for fetching mappers outputs to their corresponding reducers

•Merge

•Reduce

•Total Cost

Cost of Operators in Map and Reduce

•In order to calculate the costs, a few sample queries based on TPC-H are designed as probes to collect the execution time of operators

•given a chain with n operators,the cost is evaluated as:

Cost of Workflow

•A HiveQL query is finally compiled to MapReduce workflows (a directed acyclic graph) where each node is a single MapReduce job and the edge represents the dataflow

Experiments•evaluate the correctness and efficiency of

Qmapper•the efficiency of translating SQL into HiveQL

and the efficiency of HiveQL execution comparing QMapper with manually translated work

•TPC-H will demonstrate the execution efficiency of HiveQL generated by Qmapper

•Smart Grid application will show the correctness and translation efficiency of QMapper

Join Performance

Scalability

Accuracy of Cost Model

Thanks