typical issues with middleware · pre-java 8 memory layout heap perm gen code cache threads, libs,...
TRANSCRIPT
![Page 1: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/1.jpg)
Typical Issues with Middleware
HrOUG 2016
Timur Akhmadeev
October 2016
![Page 2: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/2.jpg)
About Me
Database Consultant at Pythian
10+ years with Database and Java
Systems Performance and Architecture
OakTable member
3rd conference as a speaker
timurakhmadeev.wordpress.com
pythian.com/blog/author/akhmadeev
twitter.com/tmmdv
![Page 3: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/3.jpg)
About Pythian
10K+Systems
400+People in 200 cities in 35
countries
Founded in
1997
Global Leader In IT Transformation And Operational Excellence
Unparalleled Expertise
• Top 5% in Databases, Applications, Infrastructure, Big Data, Cloud, Data
Science, and DevOps
Unmatched Certifications
• 9 Oracle ACEs, 4 Oracle ACE Directors, 1 Oracle ACE Associate
• 6 Microsoft MVPs, 1 Microsoft Certified Master
• 5 Google Platform Qualified Developers
• 1 Cloudera Champion of Big Data
• 1 Mongo DB Certified DBA Associate Level
• 1 DataStax Certified Partner, 1 MVP
Broad Technical Experience
• Oracle, Microsoft, MySQL, Oracle EBS, Hadoop, Cassandra, MongoDB,
virtualization, configuration management, monitoring, trending, and more
![Page 4: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/4.jpg)
Agenda
• Background
• Architecture
• Typical Issues
• Approach to Troubleshooting
![Page 5: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/5.jpg)
Architecture
Net
Dev
DBA
NOC
Storage
![Page 6: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/6.jpg)
Typical MW Issues
• Failures
– Out of Memory, Crashes
• Stability
– Hangs, changes in response times
![Page 7: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/7.jpg)
Typical Reasons of Issues
![Page 8: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/8.jpg)
Inefficient Memory Usage
8
![Page 9: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/9.jpg)
OS Memory Usage – Database
• Still very common to miss HugePages
• HugePages are a must
– Lock SGA in memory
– Reduce OS page tables footprint
– Reduce sys% CPU time
• THP have to be disabled
![Page 10: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/10.jpg)
OS Memory Usage – Middleware
• Possible to use HugePages with Java
• Recommended by Oracle
– Oracle Commerce MAA Configuration Best Practices,
July 2015
• Recommended by VMware
– Large Pages Performance – case study
• Recommended to disable THP
![Page 11: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/11.jpg)
Pre-Java 8 Memory Layout
HeapPerm
Gen
Code
Cache
Threads, Libs,
mmap-ed files,
internal structs
-Xms10G
-Xmx10G
-XX:InitialCodeCacheSize=128M
-XX:ReservedCodeCacheSize=128M
-XX:PermSize=256M
-XX:MaxPermSize=256M
Dynamic, app specific
![Page 12: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/12.jpg)
Pre-Java 8 Memory Layout
Threads, Libs,
mmap-ed files,
internal structs
-Xms10G
-Xmx10G
-XX:InitialCodeCacheSize=128M
-XX:ReservedCodeCacheSize=128M
-XX:PermSize=256M
-XX:MaxPermSize=256M
Dynamic, app specific
Can be allocated in HugePages with
-XX:+UseLargePages
![Page 13: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/13.jpg)
Java 8 Memory Layout
HeapPerm
Gen
Code
Cache
Threads, Libs,
mmap-ed files,
internal structs
-Xms10G
-Xmx10G
-XX:InitialCodeCacheSize=128M
-XX:ReservedCodeCacheSize=128M
Dynamic, app specific
![Page 14: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/14.jpg)
Java 8 Memory Layout
HeapMeta
space
Code
Cache
Threads, Libs,
mmap-ed files,
internal structs
-Xms10G
-Xmx10G
-XX:InitialCodeCacheSize=128M
-XX:ReservedCodeCacheSize=128M
Dynamic, app specific
-XX:MetaspaceSize=256M
-XX:MaxMetaspaceSize=256M
![Page 15: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/15.jpg)
Java 8 Memory Layout
Threads, Libs,
mmap-ed files,
internal structs
-Xms10G
-Xmx10G
-XX:InitialCodeCacheSize=128M
-XX:ReservedCodeCacheSize=128M
Dynamic, app specific
-XX:MetaspaceSize=256M
-XX:MaxMetaspaceSize=256M
Can be allocated in HugePages with-XX:+UseLargePages
-XX:+UseLargePagesInMetaspace
-XX:-UseCompressedClassPointers
![Page 16: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/16.jpg)
Java 8 Memory and HugePages
-XX:+UseLargePages
• If not enough pages, default pages are used
• For Metaspace in HugePages as well:
-XX:+UseLargePagesInMetaspace
-XX:-UseCompressedClassPointers
![Page 17: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/17.jpg)
Memory Usage – Java Heap
• Application creates objects in heap
• Heap is cleaned up automatically
• Cleaning is called Garbage Collection
• Major cause of pause time with Java based apps
![Page 18: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/18.jpg)
![Page 19: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/19.jpg)
Memory Usage – Java Heap
• Long GC pause is a result of
– poor sizing and configuration
– insufficient heap
– memory leak
– application deficiencies
• May lead to java.lang.OutOfMemoryError
• More OOM https://plumbr.eu/outofmemoryerror
![Page 20: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/20.jpg)
Memory Usage – Java Heap
Young Generation
Survivor0
(S0)
Survivor1
(S1)
Old GenerationEden
![Page 21: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/21.jpg)
Memory Usage – Java Heap
![Page 22: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/22.jpg)
Memory Usage – Java Heap
![Page 23: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/23.jpg)
Memory Usage – Java Heap
![Page 24: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/24.jpg)
Memory Usage – Java Heap
• High allocation rates in general is not an issue
– As long as objects become garbage quick enough
• Short requests are usually easier to handle
• Long running requests are challenging for GC
– Those that keep large active data set
• Large live set is an issue
– GC takes proportionally more time
![Page 25: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/25.jpg)
High Level Comparison of Collectors
Feature ParallelOld mCMS G1
Live Data Set Small to Medium Medium to Large Medium to XXL
Major GC pauses Up to few secs 50..500ms+ Up to few secs
Memory Usage Minimal Medium Large
Target Throughput Latency Throughput or
Latency
Downsides High pause times
with large live sets
* Fragmentation
* Serial Full GC if
promotion failure
* Complicated
* Often it is slower
than CMS (yet)
![Page 26: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/26.jpg)
![Page 27: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/27.jpg)
![Page 28: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/28.jpg)
Tools for GC Monitoring
• GC log + GCViewer / http://gceasy.io
• jstat: command line, tabular output
jstat -gcutil PID 5s 10
• jconsole/jvisualvm/jmc
![Page 29: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/29.jpg)
Resources Allocation
41
![Page 30: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/30.jpg)
Memory Allocation
![Page 31: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/31.jpg)
CPU Allocation
![Page 32: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/32.jpg)
Resources Allocation Advice
• Follow generic sizing rules
• Do not allocate less than 4 vCPUs per JVM
• Run 1 App Server per VM
![Page 33: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/33.jpg)
Resources Allocation Advice
• Do not allocate more than 2 node MW cluster
– If you don’t know how many you really need
• Allocate dedicated instances for critical services
• Split short & batch tasks between nodes
![Page 34: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/34.jpg)
Resources Allocation Advice
Know the limits of an App Server instance– concurrent users
– requests/second
– traffic/second and /request
– queries/second and /request
– garbage/second
– how big live set could be
– bottlenecks
![Page 35: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/35.jpg)
Resources Allocation – DB Connections
• Large dynamic connection pools do not work
http://www.youtube.com/watch?v=Oo-tBpVewP4
http://www.youtube.com/watch?v=XzN8Rp6glEo
• The problem is easy to appear with
– large MW clusters
– multiple connection pools to same DB
![Page 36: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/36.jpg)
DB Connections Advice
DBCPUs*10/N
![Page 37: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/37.jpg)
Optimistic Use of Oracle RAC
49
![Page 38: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/38.jpg)
Optimistic Use of RAC
• Clients want RAC because “HA & scalable”
– Especially those clients that never had it in-house
• Expectations are
– all apps scale well in RAC
– RAC provides protection from node failures
• Often licenses are acquired in advance
![Page 39: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/39.jpg)
Optimistic Use of RAC
• SQL spending time in gc waits
• App behaves worse than with single instance DB
• Sometimes clients think it’s not enough HW and
try to add more nodes to RAC
![Page 40: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/40.jpg)
RAC Advice
• Treat it as a consolidation platform
• Use services. Even without RAC!
• Service Affinity to single node
![Page 41: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/41.jpg)
Optimistic Use of RAC
![Page 42: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/42.jpg)
RAC Advice
• Using 5y+ HW makes no sense
• Follow OraCHK recommendations (carefully)
• Active GridLink with WebLogic
– https://docs.oracle.com/middleware/1212/wls/JDBCA/
gridlink_datasources.htm
![Page 43: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/43.jpg)
Unreliable Statistics Management
57
![Page 44: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/44.jpg)
Typical Statistics Management
• Default task makes changes in production
– silently with no change control
– same effect as testing code in production right away
• Usually runs way more often than needed
– Some clients run it manually even more often
• Histograms by default METHOD_OPT
![Page 45: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/45.jpg)
Histograms
• By default Oracle creates a histogram when
– Column is used in SQL condition
– Skew in the column data distribution
![Page 46: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/46.jpg)
Histograms
• When App really needs a histogram
– Column is used in SQL condition
– Skew in the column data distribution
– App uses literals in SQL condition
– Histogram helps SQL to run optimally
![Page 47: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/47.jpg)
All Columns Columns with automatically
created histograms
Histogram
required
![Page 48: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/48.jpg)
Consequences of Histograms
• As a result of unnecessary histograms
– Increased number of plans in memory
– Unnecessary CPU, memory and disk overheads
– Unexpected plan changes with bind peeking
• Adaptive features suppose to help sometimes
• People “fix” it with different plan stability options
– Without even trying to analyze the cause
![Page 49: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/49.jpg)
Statistics Management Advice
• Statistics as a code
– Do not let Oracle to change your code at random
– Set statistics as part of the code delivery
– Fix Min/Max, bad histograms, partition stats, temp
tables, new tables, etc.
• Create histograms manually
![Page 50: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/50.jpg)
Inadequate Monitoring and
Troubleshooting
64
![Page 51: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/51.jpg)
![Page 52: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/52.jpg)
![Page 53: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/53.jpg)
![Page 54: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/54.jpg)
Troubleshooting
![Page 55: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/55.jpg)
Minimal Diagnostics
• OS level metrics
• GC activity: log and/or jstat
• Thread Dumps: top + poor man’s profiler– http://www.pythian.com/blog/a-simple-way-to-monitor-
java-in-linux/
– jvmtop https://github.com/patric-r/jvmtop
– SJK https://github.com/aragozin/jvm-tools
• Heap Dump + Memory Analyzer
![Page 56: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/56.jpg)
Troubleshooting
• Recommended things to have
– Application Performance Management software
• AppDynamics
• NewRelic
– Java Flight Recorder
– JVisualVM
![Page 57: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/57.jpg)
APM
![Page 58: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/58.jpg)
Summary
• Memory efficiency
• Start small. Scale up first
• RAC as a consolidation platform
• Think and plan stats management
• Don’t just restart. Gather diagnostics.
![Page 59: Typical Issues with Middleware · Pre-Java 8 Memory Layout Heap Perm Gen Code Cache Threads, Libs, mmap-ed files, internal structs-Xms10G-Xmx10G-XX:InitialCodeCacheSize=128M-XX:ReservedCodeCacheSize=128M-XX:PermSize=256M-XX:MaxPermSize=256M](https://reader034.vdocuments.mx/reader034/viewer/2022051823/5fed5a45906cbe2f4c44a970/html5/thumbnails/59.jpg)
Thank You!
Q & A
73