phoenix secondary indexing - la hug sept 9th, 2013

Secondary Indexing in Phoenix

Jesse YatesHBase CommitterSoftware Engineer

LA HBase User Group – September 4, 2013

2

Agenda

• About

• Other Indexing Frameworks

• Immutable Indexes

• Mutable Indexes in Phoenix

• Mutable Indexing Internals

• Roadmap

LA HUG – Sept 2013

https://www.madison.k12.wi.us/calendars

3

About me

• Developer at Salesforce– System of Record, Phoenix

• Open Source– Phoenix– HBase– Accumulo


4

Phoenix

• Open Source– https://github.com/forcedotcom/phoenix

• “SQL-skin” on HBase– Everyone knows SQL!

• JDBC Driver– Plug-and-play

• Faster than HBase– in some cases


5

Why Index?

• HBase is only sorted on 1 “axis”

• Great for search via a single pattern

Example!LA HUG – Sept 2013

6

Example

name:type:

subtype:date:

major:minor:

quantity:


7

Secondary Indexes

• Sort on ‘orthogonal’ axis

• Save full-table scan

• Expected database feature

• Hard in HBase b/c of ACID considerations


8

Agenda

• About





• Roadmap


9 LA HUG – Sept 2013

http://www.wired.com/wiredenterprise/2011/10/microsoft-and-hadoop/

10

Other (Major) Indexing Frameworks

• HBase SEP– Side-Effects Processor– Replication-based– https://github.com/NGDATA/hbase-sep

• Huawei – Server-local indexes– Buddy regions– https://github.com/Huawei-Hadoop/hindex


11

Agenda

• About





• Roadmap


12

Immutable Indexes

• Immutable Rows

• Much easier to implement

• Client-managed

• Bulk-loadable


13

Bulk Loading

phoenix-hbase.blogspot.com


14

Index Bulk Loading

Identity Mapper

Custom Phoenix Reducer


HFile Output Format

15

Index Bulk LoadingPreparedStatement statement = conn.prepareStatement(dmlStatement); statement.execute();

String upsertStmt = "upsert into core.entity_history(organization_id,key_prefix,entity_history_id, created_by, created_date)\n" + "values(?,?,?,?,?)";

statement = conn.prepareStatement(upsertStmt);… //set values

Iterator<Pair<byte[],List<KeyValue>>> dataIterator = PhoenixRuntime.getUncommittedDataIterator(conn);


16

Agenda

• About





• Roadmap


17

The “fun” stuff…


18

1.5 years


19

Mutable Indexes

• Global Index

• Change row state– Common use-case– “expected” implementation

• Covered Columns


20

Usage

• Just SQL!

• Baby name popularity

• Mock demo


21

Usage• Selects the most popular name for a given yearSELECT name,occurrences FROM baby_names WHERE year=2012 LIMIT 1;

• Selects the total occurrences of a given name across all years SELECT /*+ NO_INDEX */ name,sum(occurrences) FROM baby_names

WHERE name='Jesse' GROUP BY name;

• Selects the total occurrences of a given name across all years allowing an index to be used

SELECT name,sum(occurrences) FROM baby_names WHERE name='Jesse' GROUP BY NAME;


22

Usage• Update rows due to census inaccuracy

– Will only work if the mutable indexing is working

UPSERT INTO baby_names SELECT year,occurrences+3000,sex,name FROM baby_names WHERE name='Jesse';

• Selects the now updated data (from the index table)

SELECT name,sum(occurrences) FROM baby_names WHERE name='Jesse' GROUP BY NAME;

• Index table still used in scans

EXPLAIN SELECT name,sum(occurrences) FROM baby_names WHERE name='Jesse' GROUP BY NAME;


23

Agenda

• About





• Roadmap


24

Internals

• Index Management– Build index updates– Ensures index is ‘cleaned up’

• Recovery Mechanism– Ensures index updates are “ACID”


25

“There is no magic”

- Every programming hipster (chipster)


26

Mutable Indexing: Standard Write Path

Client HRegion

RegionCoprocessorHost

WAL


MemStore


27

Mutable Indexing: Standard Write Path

Client HRegion


WAL


MemStore


28

Mutable Indexing

RegionCoprocessor

Host

WAL

RegionCoprocessor

Host

Indexer Builder

WAL Updater

Durable!

IndexerIndex Table

Index TableIndex Table

Codec


29

Index Management

• Lives within a RegionCoprocesorObserver• Access to the local HRegion• Specifies the mutations to apply to the index

tables

public interface IndexBuilder {public void setup(RegionCoprocessorEnvironment env);public Map<Mutation, String> getIndexUpdate(Put put);public Map<Mutation, String> getIndexUpdate(Delete delete);

}


30

Why not write my own?

• Managing Cleanup – Efficient point-in-time correctness– Performance tricks

• Abstract access to HRegion– Minimal network hops

• Sorting correctness– Phoenix typing ensures correct index sorting


31

Example: Managing Cleanup

• Updates can arrive out of order– Client-managed timestamps


ROW FAMILY QUALIFIER TS VALUE

Row1 Fam Qual 10 val1

Row1 Fam2 Qual2 12 val2


32


Index Table


ROW FAMILY QUALIFIER TS

Val1|Row1 Index Fam:Qual 10

Val1|Val2|Row1 Index Fam:QualFam2:Qual2

12


13

33








34








35




Va1|Row1 Index Fam:Qual 10



12

Va1l|Val2|Row1 Index Fam:QualFam2:Qual2

12


13

36




Va1|Row1 Index Fam:Qual 10



12

Va1l|Val2|Row1 Index Fam:QualFam2:Qual2

12


13

37

Managing Cleanup

• History “roll up”• Out-of-order Updates• Point-in-time correctness• Multiple Timestamps per Mutation• Delete vs. DeleteColumn vs. DeleteFamily

Surprisingly hard!LA HUG – Sept 2013

38

Phoenix Index Builder

• Much simpler than full index management• Hides cleanup considerations• Abstracted access to local state


public interface IndexCodec{public void initialize(RegionCoprocessorEnvironment env);public Iterable<IndexUpdate> getIndexDeletes(TableState state;public Iterable<IndexUpdate> getIndexUpserts(TableState state);

}

39

Phoenix Index Codec


40

Dude, where’s my data?


Ensuring Correctness

41

HBase ACID

• Does NOT give you:– Cross-row consistency– Cross-table consistency

• Does give you:– Durable data on success– Visibility on success without partial rows


Key Observation

“Secondary indexing is inherently an easier problem than full transactions… secondary index updates are idempotent.”

- Lars Hofhansl

42 LA HUG – Sept 2013

43

Idempotent Index Updates

• Doesn’t need full transactions

• Replay as many times as needed

• Can tolerate a little lag– As long as we get the order right


44

Failure Recovery• Custom WALEditCodec– Encodes index updates– Supports compressed WAL

• Custom WAL Reader– Replay index updates from WAL


<property><name>hbase.regionserver.wal.codec</name> <value>o.a.h.hbase.regionserver.wal.IndexedWALEditCodec</value>

</property><property>

<name>hbase.regionserver.hlog.reader.impl</name> <value>o.a.h.hbase.regionserver.wal.IndexedHLogReader</value>

</property>

45

Failure Situations

• Any time before WAL, client replay

• Any time after WAL, HBase replay

• All-or-nothing


46

Failure #1: Before WAL

Client HRegion


WAL


MemStore


47

Failure #1: Before WAL

Client HRegion


WAL


MemStore

No problem! No data is stored in the WAL, client just retries entire update.


48

Failure #2: After WAL

Client HRegion


WAL


MemStore


49

Failure #2: After WAL

Client HRegion


WAL


MemStore

WAL replayed via usual replay mechanisms


50

Agenda

• About



• Mutable Indexes

• RoadmapLA HUG – Sept 2013

51

Roadmap

• Next release of Phoenix

• Performance testing

• Increased adoption

• Adding to HBase (?)


52

Open Source!

• Main: https://github.com/forcedotcom/phoenix

• Indexing:https://github.com/forcedotcom/phoenix/tree/mutable-si


(obligatory hiring slide)

We’re Hiring!

Questions? Comments?

[email protected]@jesse_yates

phoenix secondary indexing - la hug sept 9th, 2013

Technology

cleanup index table

correct index

secondary indexing

secondary indexes sort

jesse group

phoenix open source

local hregion

cleanup updates