solving multi-tenancy and g1gc in apache hbase
TRANSCRIPT
![Page 1: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/1.jpg)
HBase @ HubSpot
![Page 2: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/2.jpg)
Multitenancy
![Page 3: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/3.jpg)
●
●
●
What does Multitenancy Mean?
![Page 4: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/4.jpg)
●
●
●
Benefits of Multitenancy
![Page 5: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/5.jpg)
Identifying Bad Actors
![Page 6: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/6.jpg)
●
●
●
Our Tool: HBasetracing
![Page 7: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/7.jpg)
●
●
●
●
Ad-hoc Querying with HBasetracing
![Page 8: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/8.jpg)
●
●
●
HBasetracing Roll-up
![Page 9: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/9.jpg)
Example Incident
![Page 10: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/10.jpg)
Dealing with Bad Actors
![Page 11: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/11.jpg)
●
●
●
●
Quotas
![Page 12: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/12.jpg)
●
●
●
HADOOP_USER_NAME
Deploying Quotas
![Page 13: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/13.jpg)
●
○
○
●
When Quotas Help
![Page 14: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/14.jpg)
Remember this?
![Page 15: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/15.jpg)
●
●
●
When Quotas Fall Flat
![Page 16: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/16.jpg)
●
●
Detention Queues
![Page 17: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/17.jpg)
●
The Dream
![Page 18: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/18.jpg)
Handling Failure & Managing Risk
![Page 19: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/19.jpg)
●
●
●
Read Replicas
![Page 20: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/20.jpg)
●
● Consistency.TIMELINE
● isStale()
Read Replica Usage
![Page 21: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/21.jpg)
●
●
Read Replica Timeout Settings
![Page 22: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/22.jpg)
●
●
●
Read Replica Limitations
![Page 23: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/23.jpg)
●
●
●
Cluster Replication
![Page 24: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/24.jpg)
●
●
●
Failover Client
![Page 25: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/25.jpg)
●
●
●
@StaleReadOnly Annotation
![Page 26: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/26.jpg)
Monitoring
![Page 27: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/27.jpg)
●
●
●
●
![Page 28: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/28.jpg)
●
● next()
●
●
●
![Page 29: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/29.jpg)
●
●
●
●
![Page 30: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/30.jpg)
●
●
![Page 31: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/31.jpg)
G1GC:Making it work with HBase
![Page 32: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/32.jpg)
Why G1GC?● Designed for large heaps.
○ Divides heap into many smaller G1 regions.○ G1 regions scanned and collected independently.
● Instead of occasional very long pauses, G1GC has more frequent, shorter pauses.
If tuned properly, G1GC can provide performant GC that scales well for large RegionServer heaps.
![Page 33: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/33.jpg)
The Need for TuningOut of the box, G1GC hurt our HBase
clusters’ performance:
● Too much time spent in GC pauses.● Occasional very long GC pauses.● “To-space Exhaustion”, leading to Full GCs,
which led to slow RegionServer deaths.
![Page 34: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/34.jpg)
![Page 35: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/35.jpg)
●
●
●
●
Recommended Defaults
![Page 36: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/36.jpg)
Important Metrics for Tuning● G1GC Eden & Tenured size.
○ GC logs: “[Eden: … Survivors: … Heap: …]”
● HBase memory used by Memstore.○ RegionServer JMX: “memStoreSize”
● HBase memory used by Block Cache.○ RegionServer JMX: “blockCacheSize”
● HBase memory used by “static index”.○ RegionServer JMX: “staticIndexSize”
![Page 37: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/37.jpg)
Necessary Tuning Params● JVM args:
-Xms, -Xmx-XX:G1NewSizePercent -XX:InitiatingHeapOccupancyPercent (aka “IHOP”)
● HBase configs (hbase-site.xml):hfile.block.cache.sizehbase.regionserver.global.memstore.size
![Page 38: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/38.jpg)
Necessary Tuning: MethodA. Find max block cache size, memstore size,
and static index size from the past month.B. Sum 110% of (A) maxes, add heap waste.C. Set IHOP and heap size such that Initiating
Heap Occupancy > (B) by at least 10% heap.D. Ensure IHOP + G1NewSizePercent < 90%.
– 90% = 100% - G1ReservePercent (default 10)
![Page 39: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/39.jpg)
Necessary Tuning: cont.In hbase-site.xml:● Set hfile.block.cache.size ratio value to 110%
max block cache size from the past month.● Set hbase.regionserver.global.memstore.size
ratio value to 110% max Memstore size from the past month.
![Page 40: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/40.jpg)
![Page 41: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/41.jpg)
![Page 42: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/42.jpg)
Further Tuning & Considerations
● -XX:G1ReservePercent○ Accommodating for burst-y usage.
● -XX:G1HeapRegionSize○ Reducing occurrence of humongous objects.○ Reducing long tail of slow GCs in some cases.
● -XX:G1NewSizePercent○ Tuning individual pause time vs. % time in GC.
![Page 43: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/43.jpg)
HBase Usage & Tuning Limits
A Full GC isn’t necessarily G1GC’s fault. There’s a level of “bad usage” that’s unreasonable to tune around:● Unexpected, excessively burst-y traffic.● Too many/enormous Humongous objects.
In either of these cases, the real solution is tofix the client code.
![Page 44: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/44.jpg)
Usage Note: Caching isn’t Free!
Yellow: % time spent in Mixed GC (left axis) | Blue: block cache churn, MB/sec (right axis)
![Page 45: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/45.jpg)
...to Summarize:● Tune heap size, IHOP, & HBase memory
caps based on HBase memory usage.● Tune Eden size based on % time in GC &
average Young GC pause times.● Make adjustments as needed, based on
cluster usage.● Look for suboptimal usage in your HBase
clients to further improve HBase GC.
![Page 46: Solving Multi-tenancy and G1GC in Apache HBase](https://reader034.vdocuments.mx/reader034/viewer/2022042510/587c19201a28abb5068b4d25/html5/thumbnails/46.jpg)
Links & Reference
Blog Post — http://bit.ly/hbasegc
G1GC CollectD Plugin — http://bit.ly/collectdgc
G1GC Log Visualizer — http://bit.ly/gclogviz