hbasecon 2012 | building mobile infrastructure with hbase
DESCRIPTION
In this session you will learn the common mistakes made when deploying a high write environment when building an analytics database in HBase, as well as tips on how to diagnose and debug performance bottlenecks, and an overview of an open source monitoring utility developed at Urban Airship for finding HBase hotspots. This session will also present a case study on how Urban Airship replaced a tag system running on a highly sharded PostgreSQL cluster to HBase, the options explored to create a high throughput Boolean tag system and how it was ultimately built on HBase.TRANSCRIPT
Building Mobile Infrastructure with HBaseMay 2012Nate Putnam
About Me
• Core Data and Analytics (1 year)
• Previously Engineer at Jive Software (4 years)
• Contributor to HBase/Zookeeper
In this Talk
• About Urban Airship
• What is mobile infrastructure?
• Common mistakes
• Performance tuning and monitoring
• Tag system use case
• Questions
What is an Urban Airship?• Hosting for mobile services that developers should not build themselves
• Unified API for services across platforms
• SLAs for throughput, latency
By The Numbers
• Hundreds of millions devices
• Front end API sustains thousands of requests per second
• Millions of Android devices online all the time
• 6 months for the company to deliver 1M messages, hundred million plus a day now.
Mobile Infrastructure ?
• Similar services as any large website.
• Identity
• Messaging
• Reporting
• Segmentation
• Waste time, waste battery, lose customers
Mobile Infrastructure ?
Mistakes
• Didn’t have any Hadoop experience, mistakes were made.
• Get READY, XMAS IS COMING!
Mistakes
XMAS :)
Mistakes
• Not tuning OS
• XFS
• Disable swap
• Large page support
Mistakes
Mistakes
Using RAID or virtualized disks
Mistakes
Not using Cloudera
Mistakes
• Not tuning Hadoop
• dfs.datanode.max.xcievers
• Replication factor
• Number of mappers
Mistakes
• Not tuning HBase
• Handler count
• Region Size
• Block Cache
• HFile block size
Mistakes
• Coding against HBase
• Use lightweight access (checkAndPut, exists)
• HBaseTestingUtility is great
• Batching *or* concurrency
Mistakes
Mistakes
• Schema
• Non distributed keys
• Too many column families
• Compression
• Versions
Mistakes
• How do you tell if your region sizes are good?
• How do you tell if your splits are good?
• http://bobcopeland.com/blog/2012/04/graphing-hbase-splits/
Mistakes
Mistakes
Mistakes
Performance Monitoring
• Exposes Yammer metrics around HBase client calls.
• Written by @dave_revell
• Simple integration with your existing Java code
Performance Monitoring
Performance Monitoring
Performance Monitoring
https://github.com/urbanairship/statshtable
Tags System
• Give an object one or more aliases
• Ability to query based on those aliases
Tag System
• Hard on relation databases
• Large datasets are spread out on disk
• Query plan can fall back to full table scan
• Long reads hurt other parts of the system
Tag System
Tag System
• Legacy system running on Postgres
• Sharded dataset, adding new shards is tedious
• Needed something that was low latency high throughput
• Support for grouping, and, or, not
Tag System
• Bake off between HBase and Lucene
• HBase won because of operational ease and known scalability characteristics
Tag System
• Match can return full row or key
• Only touch disk when necessary
• Brute force, fast, simple
Tag System
Tag System
• Good
• Fast, 3 node ec2 ~200k/sec, ~.5 seconds to first result
• 3.5 million send time decreased 9x
• Scales easily
• Bad
• Key partitions can’t be changed
• Inefficient in the scaled down case
Thanks!
• HBase
• Urban Airship http://www.urbanairship.com
• We’re hiring! http://urbanairship.com/company/jobs/
• @nateputnam
Questions?