memory kinetics presentation
DESCRIPTION
A definitive approach to Java heap optimizationTRANSCRIPT
Memory Kineticsfor Scale-up and Optimization
Kenny Yu
Session ID: CON5144
a.
A large heap (-Xmx) is not good enough --Excessive GC activitiesPremature tenuringAllocation failuresOutOfMemoryError
Problems
High throughputLow pause timeLow cost of GC
Objectives
Objectives explained: Good GC
Approaches
Generational distributionGC intervals ⇦ Eden + SRMinimal major GC ⇦ OC
Approaches
Capacity = ArrivalRate * HoldTime
Applicable to subsystemsEdenSurvivorOldGen
Regardless of distribution of retention time
Fundamentals: Little's Law
1. Optimize GC interval and memory region sizes
Approaches
a. The application is re-designed to polarize the generational distribution:Lighter sessionsLong-living caches
Generational hypothesis: enforced
Pop
ulat
ion
Age
1. 180335.787: [Full GC [PSYoungGen: 57985K->0K(1850304K)] [PSOldGen: 1945111K->1483937K(2097152K)] 2003097K->1483937K(3947456K) [PSPermGen: 263211K->263211K(329728K)], 10.0093850 secs] 180352.182: [GC [PSYoungGen: 1603456K->42582K(1873344K)] 3087393K->1526520K(3970496K), 0.3015250 secs]!un-optimized
Measure fixed OldGen Utilization size
1.OUfixed = OU on full GC = 1483937KSurvivor copy rate = 42582/0.3 = 142 M/sFull GC interval = (OC - OUF) / (ΔOU / Δt)We want to make this 24 hrs.
1. Too brief: excessive overflow to OldGen
Too long: large footprint + long pause
Determine young GC interval
1. To measure: gc-free response time (RT0) at low stress
To set: young gc interval (Tygc) > 4 * RT0
To measure: memory allocation rate under load
Determine Eden Capacity
13714.538: [GC 13714.538: [ParNew: 4399678K->33530K(4806016K), 0.0518020 secs] 5692328K->1326181K(6903168K), 0.0519700 secs] [Times: user=0.15 sys=0.00, real=0.05 secs]
13768.579: [GC 13768.579: [ParNew: 4402682K->27479K(4806016K), 0.0508480 secs] 5695333K->1320130K(6903168K), 0.0510040 secs] [Times: user=0.17 sys=0.00, real=0.05 secs]
Δm/Δt = (4 402 682 - 33 530) / (68 - 14) = 80 M/s
EC = Δm/Δt * Tygc = 80M *30 = 2400M
Determine Survivor Capacity
1. set: SC > 2 * max(SU)
1. In our example, SC = 200 M to handle 2x surges.
To measure: Survivor utilization on oversized SC
EU ∝ responses/s
SU ∝ responses/s
Fix Survivor Ratio
1. SurvivorRatio is independent of workload,It is a property of the application.
1. In our example, 2400M/200M = 12
To measure: overflow rate under load-- How fast OldGen utilization grows
Determine Promotion Rate
jstat -gc pid 10000 60S0C S1C S0U S1U EC EU OC OU YGC FGC436864.0 436864.0 41019.9 0.0 4369152.0 3511510.8 2097152.0 1319641.3 127 4436864.0 436864.0 0.0 33325.8 4369152.0 197903.8 2097152.0 1319645.8 128 4436864.0 436864.0 0.0 33325.8 4369152.0 971835.5 2097152.0 1319645.8 128 4 436864.0 436864.0 0.0 33325.8 4369152.0 2003800.7 2097152.0 1319645.8 128 4436864.0 436864.0 0.0 33325.8 4369152.0 3422374.2 2097152.0 1319645.8 128 4436864.0 436864.0 48272.6 0.0 4369152.0 1306482.6 2097152.0 1319646.9 129 4
PromoRate = ΔOU/Δt = (6.9-1.3)/50 = 0.11 k/s
With adequate full GC interval (1 day)
Determine Old Gen Capacity
1. SpareOC = PromoRate * 3600 * 24= 10 M
OC = OUfixed + SpareOC
Summary of approach
Condition Measure Determine
Upon full GC fixed OldGen util OldGen capacity
Low load response time YGC interval
Load to 50% CPU memory alloc rate Eden size
Oversized survivor
survivor utilization SurvivorRatio
Load, EC, SC promotion rate FGC interval
1. -XX:SurvivorRatio=10 -XX:-UsePSAdaptiveSurvivorSizePolicy -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSScavengeBeforeRemark -XX:CMSInitiatingOccupancyFraction=95 -XX:+UseCMSInitiatingOccupancyOnly -Xms7g -Xmx7g -Xmn5g
JVM Memory Configuration
1.Keep constants:Fixed OldGen utilizationSurvivor ratio
CPU speed-dependentYoung GC interval
Proportional to CPU and Throughput:Eden sizeSpare OC
Memory scale-up to faster and more CPUs
1. Full GC once at nightMemory problems resolvedGC pause time 50 ms Application became CPU-boundResponse time outliers eliminated -- stability
Performance Results
• DIRECTV dev team, led by Bob Hays, Ankush Walhekar and Harsha Anna, for the application and configuration, and for redesigning the application for object lifetime management. Jamal Haskins for timely assistance on build and deploy.
Owen Li, for helping with LoadRunner scripts
Louis Liao for tuning Linux OS
Rao Uppalapati, Prem Chaitanya and Abhiram Madadi, for setting up environments
DIRECTV for support and providing the equipment and funding the trip
Acknowledgments
• Thank you.
Questions?