secondary indexing in phoenix jesse yates hbase committer software engineer sf hbase user group –...

Secondary Indexing in Phoenix

Jesse YatesHBase CommitterSoftware Engineer

SF HBase User Group – September 26, 2013

James TaylorPhoenix LeadSoftware Engineer

2

Agenda

• About

• Indexes In Phoenix

• Immutable Indexes

• Mutable Indexes

• Demo!

• Roadmap

SF HUG – Sept 2013

https://www.madison.k12.wi.us/calendars

3

Phoenix

• Open Source– https://github.com/forcedotcom/phoenix

• “SQL-skin” on HBase– Everyone knows SQL!

• JDBC Driver– Plug-and-play

• Faster than HBase– in some cases


4

Secondary Indexes

• Sort on ‘orthogonal’ axis

• Save full-table scan

• Expected database feature

• Hard in HBase b/c of ACID considerations


5

• About



• Mutable Indexes

• Demo!

• Roadmap

Agenda


6

Indexes In Phoenix

• Creating an index– DDL statement– Creates another HBase table behind the scenes

• Deciding when an index is used– Transparent to the user– (but user can override through hint)– No stats yet

• Knowing which table was used– EXPLAIN <query>


7

Creating Indexes In Phoenix

• CREATE INDEX <index_name>ON <table_name>(<columns_to_index>…)INCLUDE (<columns_to_cover>…);

• Optionally add IMMUTABLE_ROWS=true property to CREATE TABLE statement


8

Creating Indexes In Phoenix

CREATE TABLE baby_names ( name VARCHAR PRIMARY KEY, occurrences BIGINT);

CREATE INDEX baby_names_idx ON baby_names(occurrences DESC,

name);


9

Deciding When To Use

• Transparent to the user• Query optimizer does the following:– Compiles query against data and index tables– Chooses “best” one (not yet stats driven)• Can index even be used?

– Active, Using columns contained in index (no join back to data table)

• Can ORDER BY be removed?• Which plan forms the longest start/stop scan key?


10


SELECT name, occurrences FROM baby_names ORDER BY occurrences DESC LIMIT 10;

SELECT name, occurrences FROM baby_names_idxLIMIT 10


ORDER BY not necessary since rows in index table are already ordered this way

11


SELECT name, occurrences FROM baby_names WHERE occurrences > 100;

SELECT name, occurrences FROM baby_names_idxWHERE occurrences > 100;


Uses index, since we can form start row for scan based on filter of occurrences

12


SELECT /* NO_INDEX */ nameFROM baby_names WHERE occurrences > 100;

SELECT /*+ INDEX (baby_names baby_names_idx other_baby_names_idx) */name,occurrences FROM baby_namesWHERE occurrences > 100;


Override optimizer by telling it not to use any indexes

Tell optimizer priority in which it should consider using indexes`

13

Knowing which table was used

EXPLAIN SELECT name, occurrences FROM baby_names ORDER BY occurrences DESC LIMIT 10;

CLIENT PARALLEL 1-WAY FULL SCAN OVER BABY_NAMES_IDX SERVER FILTER BY PageFilter 10CLIENT 10 ROW LIMIT


14

• About



• Mutable Indexes

• Demo!

• Roadmap

Agenda


15

Immutable Indexes

• Immutable Rows

• Much easier to implement

• Client-managed

• Bulk-loadable


16

• About



• Mutable Indexes

• Demo!

• Roadmap

Agenda


17

Mutable Indexes

• Global Index

• Change row state– Common use-case– “expected” implementation

• Covered Columns/Join Index


18

1.5 years*


19

Internals

• Index Management– Build index updates– Ensures index is ‘cleaned up’

• Recovery Mechanism– Ensures index updates are “ACID”


20

“There is no magic”

- Every programming hipster (chipster)


21

Mutable Indexing: Standard Write Path

Client HRegion

RegionCoprocessorHost

WAL


MemStore


22

Mutable Indexing: Standard Write Path

Client HRegion


WAL


MemStore


23

Mutable Indexing

RegionCoprocessor

Host

WAL

RegionCoprocessor

Host

Indexer Builder

WAL Updater

Durable!

IndexerIndex Table

Index TableIndex Table

Codec


24

Index Management

• Lives within a RegionCoprocesorObserver• Access to the local HRegion• Specifies the mutations to apply to the index

tables

public interface IndexBuilder {public void setup(RegionCoprocessorEnvironment env);public Map<Mutation, String> getIndexUpdate(Put put);public Map<Mutation, String> getIndexUpdate(Delete delete);

}


25

Why not write my own?

• Managing Cleanup – Efficient point-in-time correctness– Performance tricks

• Abstract access to HRegion– Minimal network hops

• Sorting correctness– Phoenix typing ensures correct index sorting


26

Example: Managing Cleanup

• Updates can arrive out of order– Client-managed timestamps


ROW FAMILY QUALIFIER TS VALUE

Row1 Fam Qual 10 val1

Row1 Fam2 Qual2 12 val2


27


Index Table


ROW FAMILY QUALIFIER TS

Val1|Row1 Index Fam:Qual 10

Val1|Val2|Row1 Index Fam:QualFam2:Qual2

12


13

28








29








30




Va1|Row1 Index Fam:Qual 10



12

Va1l|Val2|Row1 Index Fam:QualFam2:Qual2

12


13

31




Va1|Row1 Index Fam:Qual 10



12

Va1l|Val2|Row1 Index Fam:QualFam2:Qual2

12


13

32

Managing Cleanup

• History “roll up”• Out-of-order Updates• Point-in-time correctness• Multiple Timestamps per Mutation• Delete vs. DeleteColumn vs. DeleteFamily

Surprisingly hard!SF HUG – Sept 2013

33

Phoenix Index Builder

• Much simpler than full index management• Hides cleanup considerations• Abstracted access to local state


public interface IndexCodec{public void initialize(RegionCoprocessorEnvironment env);public Iterable<IndexUpdate> getIndexDeletes(TableState state);public Iterable<IndexUpdate> getIndexUpserts(TableState state);

}

34

Phoenix Index Codec


35

Dude, where’s my data?


Ensuring Correctness

36

HBase ACID

• Does NOT give you:– Cross-row consistency– Cross-table consistency

• Does give you:– Durable data on success– Visibility on success without partial rows


Key Observation

“Secondary indexing is inherently an easier problem than full transactions… secondary index updates are idempotent.”

- Lars Hofhansl

37 SF HUG – Sept 2013

38

Idempotent Index Updates

• Doesn’t need full transactions

• Replay as many times as needed

• Can tolerate a little lag– As long as we get the order right


39

Failure Recovery• Custom WALEditCodec– Encodes index updates– Supports compressed WAL

• Custom WAL Reader– Replay index updates from WAL


<property><name>hbase.regionserver.wal.codec</name> <value>o.a.h.hbase.regionserver.wal.IndexedWALEditCodec</value>

</property><property>

<name>hbase.regionserver.hlog.reader.impl</name> <value>o.a.h.hbase.regionserver.wal.IndexedHLogReader</value>

</property>

40

Failure Situations

• Any time before WAL, client replay

• Any time after WAL, HBase replay

• All-or-nothing


41

Failure #1: Before WAL

Client HRegion


WAL


MemStore


42

Failure #1: Before WAL

Client HRegion


WAL


MemStore

No problem! No data is stored in the WAL, client just retries entire update.


43

Failure #2: After WAL

Client HRegion


WAL


MemStore


44

Failure #2: After WAL

Client HRegion


WAL


MemStore

WAL replayed via usual replay mechanisms


45

“Magic”

• Server-short circuit• Lazy load columns• Skip-scan for cache• Parallel Writing• Custom MemStore in Indexer• Caching HTables• Pluggable Index Writing/Failure Policy• Minimize byte[] copy (ImmutableBytesPtr)


46

• About



• Mutable Indexes

• Demo!

• Roadmap

Agenda


47

Demo


48

• About



• Mutable Indexes

• Demo!

• Roadmap

Agenda


49

Roadmap

• Next release of Phoenix

• Performance improvements

• Functional Indexes

• Other indexing approaches (Huawei, SEP)


50

Open Source!

• Main: https://github.com/forcedotcom/phoenix

• Indexing:https://github.com/forcedotcom/phoenix/tree/mutable-si


(obligatory hiring slide)

We’re Hiring!

Questions? Comments?

[email protected]@jamesplusplus

[email protected]@jesse_yates

53

Appendix

• AsyncHBaseWriter– github.com/jyates/phoenix/tree/async-hbase– 2x+ slower*

* Written in 2hrs, not 100% correct either


secondary indexing in phoenix jesse yates hbase committer software engineer sf hbase user group –...

Documents