HBASE @ FLIPBOARD

What is Flipboard?

Publications

People

Topics

Scale

100+ million users

250,000+ new users/day

15+ million magazines

Why HBase?• Transferrable Hadoop operational expertise

• MapReduce!

• Write throughput

• Better elasticity than MySQL, our other primary data store

• Strong consistency

• Column-oriented, as opposed to simple K/V

What do we use it for?

• User-generated magazines, likes, comments

• Vanity metrics for those magazines- daily + all-time counters/HLLs

• Follow graph

• RSS feeds

• More and more every day…

Magazine Storage• Stored in a single HBase table

• Magazines live in one column family (“magazine”)

• Articles in temporal order in another CF (“article”)

• Logically, everything shared is tagged with magazine ID (prefix compression helps here)

• Makes the calculation of everything a user shared efficient

User Magazines

magazine:<magazineid> magazine:<magazineid> magazine:<magazineid>

sha1(userid) MagazineData (serialized JSON) MagazineData (serialized JSON) MagazineData (serialized JSON)

Magazine CF of Collection Table Listing magazines a user has created is a single read

Data is stored in serialized JSON for language interoperability but is parsed and serialized by plain old java objects

User Magazines

[Reverse Unix TS]:[magazineid]

[Reverse Unix TS]:[magazineid]

sha1(userid) Article Data(Serialized JSON)

Article Data(Serialized JSON)

Articles CF of Collection Table Kept in temporal order so that most recently shared articles are first

Access patterns are usually newest first. HBase Filters are used to slice wide rows.

User Magazines

like:[userid] reflip:[new article id] comment:[timestamp][userid]

Article ID JSON (who/when) JSON (where it was

reflipped) JSON (comment/person)

Social Activity

One cell per like, since you can only do it once per user Can be many comment and reflip cells by one user per article

Alternative orderings can be computed from Elasticsearch indexes

User Magazines

magazine:<magazineid> contributor:<magazine>:<userid>

sha1(userid) JSON metadata JSON metadata

Multiple Contributors Magazine CF contains magazines that user can share into.

Contributor CF contains user’s magazines that others are allowed to share into.

User Magazines

<metric>:<day> <metric>_count

magID long count for day alltime count

Per magazine metrics live in Stats CF

Atomic increments for counters, both a per day count and a total count: Total Articles Contributors

etc.

User Magazines

<unique>:<day> <unique>_count

magID HLL for individual day Premerged HLL

Unique readers kept in each magazine’s row as a serialized HyperLogLog

Allows for merging unique data over day ranges or displaying all time

Social Graph

follow:userid follower:userid stats:<counter>

sha1(userid) JSON person that I follow

JSON person that follows me

long count of followers/following

Stored in friends table, follower/followers/stats CFs; metadata in MySQL

Alternative indexes in Elasticsearch

HBase Table Access Patterns• Tables optimized for application access patterns (“design for the questions, not the

answers”)

• Fetching an individual magazine- collection table, magazine CF, [magazine ID] -> cell

• Fetching an individual article - article table, article:[article ID] cell

• Fetch an article’s stats - article table, article:stats cells

• Fetching a magazine’s articles: collection table, article CF, with cell limit and column qualifier starts with magazine id

• Fetching a user’s magazines- collection table, magazine CF, [magazine ID] in the CQ

Client Stats

• Articles: sum(magazine stats:article_count for each magazine)

• Magazines: count(collection:magazine) cells • Followers: friends:stats:follower_count +

sum(magazine stats:subscriber_count for each magazine)

More Client Stats

• Summary stats use counters from the article table, detailed stats (who liked the article?) read cells

• We can cache the feed of items, but the stats/like state is calculated per user

• likes: article:stats:like_count • reflips: article:stats:reflip_count • comments: article:stats:comment_count

Even More Client Stats

AsyncHBase Usage• Our fork adds column filters on wide rows- we’d like to get these upstream

• Stats requests require scatter/gather reads for several tables, sometimes over multiple HBase clusters

• HBaseAsync requests are grouped into a single Deferred

• Most requests are a get on a single row, no multi row scans

• Most requests wait once until the results are returned or a deadline expires

• If data is returned late or HBase regions are not available, partial calculations are allowed (we just display the stats we’ve got)

Handling HBase Failures• Most patterns are read before write which causes early failure

• We can tolerate some data loss (atomic increments, vanity stats)

• Individual servers track inflight requests to HBase, slow puts and gets, and report to Graphite

• Various levels of caching allow HBase recovery/region reassignment without end users noticing

• Read Only mode - writes are stopped at the application layer

• Ability to switch to replica under duress

Current HBase Fleet

• 15 clusters

• ~100 tables

• ~250TB in HDFS

• ~250 RegionServers

• Busiest clusters: 100,000+ qps, 1000 regions

HBase Fleet, continued

• All in EC2 😳

• Nothing in VPC, yet

• Each cluster lives within an AZ

• 1 durable cluster doing cross-AZ HBase-level replication

• 1 cluster running Stargate (it works, but we’re not in love with it)

HBase History at FlipboardOldest current production instances launched in 2011

(This cluster is going away soon 😀)

HBase Version Distribution• 0.90: dwindling, thankfully

• 0.94: Moved to Snappy; Stargate cluster for RSS storage; Python writers, Go readers

• 0.96: First CDH5/Java 7/Ubuntu Precise clusters; magazines live in one of these

• 0.98: pre-calculated user homefeeds, more

• 1.0, 1.1: Soon…

Which instance types?• Started off with m1.xlarges for the 1.6TB of ephemeral (spinning)

disk; when we started using HBase, AWS didn’t have SSDs

• Moved to hi1.4xlarges (16 cores, 60GB RAM, 2x1TB local SSD)

• Moved to i2s (next-gen SSD instances, made for databases) as soon as AWS let us launch them!

• ❤ i2s; some 2x, some 4x

AWS tips• Use instance storage, not EBS

• Rely on HDFS to keep your bits replicated instead of using EBS

• Cross-AZ latency is minimal, but traffic is expensive!

• Push HDFS snapshots to S3 (we trigger this from Jenkins)

• If you jack up your network-related timeouts to handle AWS’ network flakiness, your MTTR rises, so be careful…

• Upgrade often, you’ll get more sleep!

Clients

• Java, Scala- AsyncHBase, which we love. Added column filtering for wide rows.

• Python + Go: protobufs over HTTP via Stargate, which works

• We use HAProxy everywhere, so we use that to load balance requests to Stargate servers

What’s next for HBase at Flipboard?• Moar HBase

• 1.0, now that CDH has it in 5.4.0; 1.1 when CDH gets it, hopefully soon…

• Region replicas (HBASE-10070) will help with use cases that can tolerate timeline consistency; 1.1 will have many improvements here

• Compaction throttling! (HBASE-8329)

• Java 8 + G1, Ubuntu Trusty, 3.13 kernel

• EC2 placement groups + VPC enhanced networking, once we’re in (no charge for these)

• HTrace (we use a little Zipkin, would love to get more HBase visibility)

• Multitenancy improvements in Apache HBase will help us put more customers on a cluster

Wish List $

HydraBase! (HBASE-12259)

Async client in Apache HBase so it keeps pace (HBASE-12684 is a start!)

Native Go client for 0.96+

Thanks!

• Matt Blair (@mb on Flipboard, @mattyblair on Twitter)

• Jason Culverhouse (@jsonculverhouse on both)

• Sang Chi (@sangchi on Flipboard, @sandbreaker on Twitter)