sql on accumulo

SQL-on-AccumuloA talk about how we may be able to do it one day

DONALD MINER

1

May 5th, 2014

2

A brief history of Hadoop and SQL

time1980 2000

SQL

3

A brief history of Hadoop and SQL

time1980 2000

Hadoo

p HDFS

& M

apRed

uce

4

BIG DATASQL

5

A brief history of Hadoop and SQL

time1980 2000

SQL-

on-H

adoo

p

SQL-on-Hadoop

6

SQL-on-Accumulo would be nice

Problem: Accumulo is just a data store We’ll have to do query somewhere else

7

WWHBD?(What Would HBase Do?)

8

WWHBD? - Hive

• Hive Runs in MapReduce Map col family and col qualifiers to columns Maintained by Hive community

• Impala and Shark inherit functionality from Hive

9

WWHBD? - next level

Problem: Hive, Impala, and Shark don’t know how HBase works … and don’t care

• Apache Phoenix Specifically SQL-on-HBase Currently Apache incubator project Client-embedded JDBC driver Uses series of scans and coprocessors

• Pivotal’s HAWQ and PXF PXF is external table functionality in HAWQ Native support for HAWQ: uses push down filters,

range scans, etc. to efficiently slurp data into HAWQ

10

ACCUMULO-143

people. technology. integrity.

11

SQL-on-Accumulo Status

Hive (and somewhat Impala and Shark)• Github project by Brian Femiano [1]

Doesn’t work on new versions Hasn’t been touched in 9 months Wasn’t committed into trunk

• Some rumors that some orgs have done it themselves (but no public information)

people. technology. integrity.

[1] https://github.com/bfemiano/accumulo-hive-storage-manager (google for “accumulo hive”)

12

SQL-on-Accumulo Status

Phoenix• Discussion on mailing list last week• Some differences between iterators and

coprocessors makes this interesting

Pivotal’s HAWQ and PXF• In development• Will support visibility labels • Pushdown and optimizations with iterators

people. technology. integrity.

13

Visibility Design Problems

These problems are unique to Accumulo

• SELECT and visibility labels Assume two cells, only uniqueness is visibility…

Which do I pick in a SELECT? Timestamps have this problem, but have a logical

assumption (most recent)

• Authorizations in SQL How do you tell the execution engine which

authorizations to use? Table definition? (hard to change) SQL statement? (extend SQL language?) Based on login? (how do you downgrade?)

14

What are the next steps?

I guess that’s up to the community

15

QUIZ: What is this definition trying to say?

Big Data:• Volume• Variety• Velocity• Veracity

A warning about SQL-on-Accumulo

16

QUIZ: What is this definition trying to say?

Big Data:• Volume• Variety• Velocity• Veracity

Answer: RDBMS/SQL suck at all these things

A warning about SQL-on-Accumulo

17

QUIZ: What is this definition trying to say?

Big Data:• Volume• Variety• Velocity• Veracity

Answer: RDBMS/SQL suck at all these things

A warning about SQL-on-Accumulo

What does SQL-on-Accumulo still suck at?*Added context for my internet viewers since this could be controversial if taken literally and I’m not talking to my slides: I’m trying to say that SQL-on-X can’t solve all of the worlds problems, but it can solve a good number of them very well. It also tees up the conversation that SQL is not the end-all-be-all… there are ways that it could be made better to adapt to “the big data use case”. Don’t take this the wrong way, SQL-on-Hadoop and SQL-on-Accumulo would be incredible useful, but it doesn’t solve 100% of the problems.

SQL-on-Accumulo

DONALD MINER

18

dminer@clearedgeit.com @donaldpminer

Questions?