Table of Contents1. WildFly Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

1.1. The Author of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

1.2. What this book covers: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

1.2.1. Who this book is for . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  2

1.2.2. How to Contact Us . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  2

1.2.3. Piracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  2

1.2.4. Book Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3

1.2.5. Conventions used in this book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3

2. Strategies for tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  4

2.1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  4

2.2. The tuning process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5

2.2.1. Data gathering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  6

2.2.2. Data analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  7

2.2.3. Verify the hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  7

3. Tools for monitoring performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  9

3.1. Using JConsole to monitor WildFly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  9

3.1.1. What JConsole can do for you . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  10

3.1.2. Connecting to a remote WildFly Server in Standalone Mode . . . . . . . . . . . . . . . . . . . . . . . . .  16

3.2. Connecting to a remote WildFly Server in Domain Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  18

3.3. Using Java VisualVM to monitor WildFly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  18

3.3.1. Connecting to a Local WildFly JVM Using VisualVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  19

3.3.2. Profiler vs Sampler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  20

3.3.3. Collecting snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  23

3.3.4. Connecting to a Remote WildFly JVM Using VisualVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  24

3.4. Using hawtio to monitor the application server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  25

3.5. Gathering statistics using the native Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . .  26

3.6. Using the ELK stack to get actionable insights from WildFly . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  28

3.6.1. Installing the ELK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  28

3.6.2. WildFly Data mining with ELK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  28

3.6.3. Configuring lostash to poll metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  30

3.6.4. Managing metrics from kibana. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  34

4. JVM Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  39

4.1. Finding the optimal JVM settings for your environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  39

4.1.1. Finding the maximum memory to assign to your application . . . . . . . . . . . . . . . . . . . . . . . .  39

4.1.2. Setting the minimum memory to assign to your application . . . . . . . . . . . . . . . . . . . . . . . . .  41

4.1.3. Setting the Java Heap sub-spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  41

4.1.4. Choosing the right garbage Collector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  43

4.1.4.1. Serial Garbage Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  44

4.1.4.2. Parallel Garbage Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  44

4.1.4.3. CMS Garbage Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  45

4.1.4.4. G1 Garbage Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  46

4.1.5. Choosing the right Garbage collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  47

4.1.6. Debugging the garbage collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  48

4.1.7. Using -verbosegc to debug Garbage collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  49

4.2. Other JVM parameters optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  50

4.2.1. String optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  50

4.2.2. Capturing Heap dump when running out of memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  51

4.2.3. Enabling Large Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  51

4.2.4. Enabling Aggressive Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  52

5. Tuning Data Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  53

5.1. Datasource Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  53

5.1.1. Monitoring Datasources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  55

5.2. Using Agroal Datasource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  56

5.2.1. Configuring the Agroal Datasource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  57

5.2.2. Configuring an Agroal XA Datasource. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  58

5.3. Tuning Resource Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  59

5.4. JDBC Performance tuning guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  61

5.4.1. Statement Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  61

5.4.2. JDBC batching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  63

5.4.3. Use correct getXXX() method to retrieve Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  64

5.5. JPA/Hibernate Performance tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  65

5.5.1. Choosing the best fetching strategy for performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  65

5.5.1.1. Option 1: Use a Fetch Join in your JPQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  67

5.5.1.2. Option 2: Use the Criteria API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  68

5.5.1.3. Option 3: Use Entity Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  69

5.5.1.4. Using Dynamic Entity Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  70

5.5.2. Using Batch inserts/updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  71

5.5.3. Using named queries to improve performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  72

5.5.4. Choosing the right Unique Generator for your Tables in Hibernate/JPA . . . . . . . . . . . . . . .  73

5.5.5. Choosing the right column types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  74

5.5.6. Limiting the amount of data fetched with pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  74

5.5.7. Configuring Second Level cache to improve the performance . . . . . . . . . . . . . . . . . . . . . . . .  75

5.5.7.1. Setting a timeout for your Query cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  77

5.5.7.2. Monitoring the Second Level cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  78

5.5.8. Debugging Hibernate/JPA Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  79

6. EJB Subsystem Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  81

6.1. Tuning SLSBs and MDBs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  81

6.1.1. Creating a new EJB Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  83

6.2. Message Driven Bean Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  83

6.3. Finding the optimal Pool size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  84

6.3.1. Do you need an EJB Pool ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  86

6.3.1.1. Turning off the EJB pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  86

6.4. Configuring the EJB Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  86

6.4.1. Enabling Passivation for Stateful Session Beans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  87

6.4.2. Monitoring the EJB cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  88

6.5. Configuring EJB Thread Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  89

6.5.1. How to create a custom EJB Thread Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  91

7. Tuning the Messaging subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  93

7.1. Guidelines to improve the performance of JMS applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  93

7.2. Tuning ArtemisMQ performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  95

7.2.1. JMS and transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  96

7.2.2. Using asynchronous send acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  96

7.2.3. Configuring Thread Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  98

7.2.4. Monitoring Messaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  99

8. Transactions Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  102

8.1. Tuning XA Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  103

8.2. Monitoring Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  104

9. Optimizing the Logging Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  105

9.1. Choose wisely what to log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  105

9.2. Disable logging to the console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  106

9.3. Configure the Loggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  106

9.4. Disable auto-flush . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  106

9.5. Choose an appropriate format for your logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  106

9.6. Evaluate if you need an Asynchronous Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  107

10. Undertow Subsystem Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  109

10.1. Configuring the Web Server Thread pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  109

10.1.1. Monitoring Undertow workers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  110

10.2. Configuring a custom Worker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  112

10.3. Monitoring HTTP sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  112

10.4. Configuring Buffer Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  112

10.5. Configuring Undertow’s Buffer Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  114

10.6. Undertow JSP Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  115

10.6.1. Enabling JSP hot deploy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  115

10.6.2. Features not available any more in the Servlet container . . . . . . . . . . . . . . . . . . . . . . . . . .  116

10.6.3. Configuring Undertow’s Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  116

10.7. Servlets tuning best practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  117

10.7.1. Use the servlet init() method to cache static data, and release them in the destroy()

method.

 117

10.7.2. Use preferably server side redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  118

10.7.3. Avoid reverse DNS lookups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  118

10.7.4. Be careful with Servlet’s flush! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  119

11. JGroups Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  121

11.1. Choosing the optimal transport protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  122

11.1.1. Limitations in UDP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  122

11.2. Configuring Buffer sizes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  122

11.2.1. Changing WildFly default buffer sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  123

11.3. Enabling Non-blocking JGroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  124

11.4. Configuring Message Bundling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  124

11.4.1. Disadvantages of Message bundling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  125

11.5. Configuring Jumbo Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  126

11.6. JGroups Thread Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  127

11.6.1. Thread Pool Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  127

11.7. Monitoring JGroups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  128

11.8. Using Probe script to monitor JGroups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  131

11.9. Testing the performance of your network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  132

11.9.1. Measuring unicast performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  133

12. Tuning the application environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  134

12.1. File Descriptor tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  134

12.2. TCP/IP Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  136

12.3. Use Large Memory Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  138

1. WildFly Performance TuningAuthor : Francesco Marchioni

© ItBuzzPress 2018

Red Hat, Red Hat Enterprise Linux, JBoss, are trademarks of Red Hat, Inc., registered in theUnited States and other countries.

Linux ® is the registered trademark of Linus Torvalds in the United States and othercountries.

Java ® is a registered trademark of Oracle and/or its affiliates.

1.1. The Author of the BookFrancesco Marchioni is a Quality Engineer working at Red Hat since 2014. He has joined the JBosscommunity in early 2000 when the application server was a mere EJB container, running release2.x.

In 2008 he started an IT portal focused on JBoss products

(http://www.mastertheboss.com) which is pleased to serve an average of 8000 daily visits.

He has authored the following titles:

JBossAS 5 Development, Packt Publishing (December 2009)

JBoss �AS 5 Performance Tuning, Packt Publishing (December �2010)

JBoss �AS 7 Configuration, Deployment, and Administration, Packt Publishing �(December 2011)

Infinispan �Data Grid Platform, Packt Publishing (June 2012) �co-authored with Manik Surtani(Infinispan Project lead)

JBoss �AS 7 Development, Packt Publishing (June 2013)

Enterprise �Application Server CookBook, ItBuzzPress (September �2013)

WildFly Administration Guide (June 2014 – Updated on �March 2018 )

Practical �Java EE Development on WildFly (June 2014 – Updated on �April 2018 )

In March 2018 Francesco published his first sci-fi novel named Chronicles from a Simulated Worldwhich covers in an earthy and rational style the discussion about the Simulation Hypothesis.

1.2. What this book covers:Strategies for tuning : Introduces to the basic strategies for tuning, describing how data should beinitially gathered, analyzed and verified.

1

Tools for monitoring performance: Discusses the tools that can be used to monitor theperformance of the application server,from the standard JMX toolings to ELK stacks and advancedCommand Line functions.

JVM Tuning: Covers JVM tuning, guiding you to an optimal configuration of the Virtual MachineEnvironment and its Garbage Collection policies.

Tuning Data Access: Tuning Data access: spanning from JDBC to JPA/Hibernate, it covers every bestpractice to improve the most common bottleneck of every application.

EJB Subsystem Tuning: Discusses the best tactics to tune the EJB Container and the objects runningin it.

Tuning the Messaging subsystem: Contains guidelines to achieve the best performance formessaging applications and the WildFly broker (ArtemisMQ).

Transactions Tuning: Discusses some options to control Transaction storage to improve itsperformance.

Optimizing the Logging Performance : Tuning how data is logged and some common pitfalls toavoid.

Undertow Subsystem Tuning: Contains best practices to tune the Web Server (Undertow) and therelated io subsystem.

JGroups Tuning: Discusses JGroups cluster tools and the options available to measure itsperformance.

Tuning the application environment: Environment tuning is the last aspect discussed in the book,covering some best practices for File Descriptors, TCP-IP configuration and Storage tuning.

1.2.1. Who this book is for

This book is well-suited for Java Enterprise architects who design and configure Enterpriseapplications. It is also great for Java developers and System Administrators who want to get in-depth details of the application server and of the correct tuning methodology. Finally, also Qualityassurance teams will also find this book useful as they will learn methodologies and instruments totest the performance of the application server.

1.2.2. How to Contact Us

Please address comments and questions concerning this book to the publisher:info@itbuzzpress.com.

For more information about our books, and future projects see our website at:http://www.itbuzzpress.com

1.2.3. Piracy

The uploading/downloading of copyrighted material without the express written consent of the

2

content copyright holder is strictly forbidden. Piracy is an illegal act that you may aspire toreconsider. Besides this, piracy is not a victimless crime! It is financially damaging and personallyhurtful to company employees and their families. Legitimate users suffer as well. We appreciateyour help in protecting the valuable content of this book.

1.2.4. Book Version

This is the version 1.0 of the book – Updated 1st December 2018

1.2.5. Conventions used in this book

This book contains lots of script files and commands to be executed on your machine. Much efforthas been put to make the code as much as readable as possible.

The following script snippet (in a blizzard blue) identifies a command to be executed on youroperating system’s shell:

$ ./jboss-cli.sh

As you can see from the prompt, we have assumed that you are executing on a Linux/Unixmachine. At the beginning of the book, we have also provided the equivalent Windows syntax ofsome core commands:

C:\Users\jboss\wildfly-14.0.0.Final\bin jboss-cli.bat

To avoid being repetitive, we have however used the Linux shell syntax for the rest of the book.

Within the book, you will find also some gray-filled block of code like the following one:

[disconnected /] patch apply /tmp/wildfly-9.0.1.Final.patch

{

"outcome" :"success",

"result" : {}

}

This piece of code identifies a command to be executed within the application server’s CommandLine Interface ([Using the CLI]). Therefore, executing this command in the operating system’s shellwill obviously return an error.

3

2. Strategies for tuningWildFly is the new name for the application server formerly known as JBoss AS. Since its firstrelease (WildFly 8) the application server has evolved with new functionalities and featuring anembedded intelligence to self-tune some of the key configuration parameters. Although a simpleapplication can rely on the default settings (calculated usually on the amount of memory or CPUavailable), any mission-critical application requires a performance-trained eye. The reason for it isthat the amount of variables involved in a complex application is so high that it’s impossible toprovide an efficient configuration without a performance tuning cycle.

In this book we will cover the most common strategies to tune a complex software like anapplication server, then we will gradually approach the core components that make up WildFlyinfrastructure. As the application server is running within a JVM, we have to discuss optimizationswhich can be applied so that the JVM performs at its best.

2.1. IntroductionPerformance tuning, in a nutshell, is a process driven by identifying the most evident bottlenecksand making the appropriate changes to diminish or eliminate the effect of the bottleneck.Generally, performance tuning is performed reactively, either while the system is not yet inproduction or after it is live.

The most efficient way to tune an application is to have a well-established performance baselinethat you can use as a comparison, should a performance issue arises. As a performance expert,your first task is to identify the peak periods of your environment and install a monitoring tool thatgathers performance data for those high-load times. Ideally, you should be gathering data sincewhen the application is in its initial stage, during the QA cycle. Data gathering should continue alsowhen the system is first put in production.

In general terms, your baseline data should include the following information:

• Application statistics (number of transactions, response time, etc.)

• Middleware statistics (amount of memory used, number of total connections opened, numberof threads running, etc)

• External systems statistics ( database or message broker statistics, etc)

• Operating System statistics (amount of RAM available, file descriptors opened, disk I/Ostatistics, network statistics)

In this book, we will go in deep into the application server’s indicators which are the purpose ofthis book. However, focusing only on the application server is a common pitfall which can mistakethe symptoms of a problem for the actual problem itself. It is important to recognize thatperformance issues can propagate from the application server to external systems and vice-versa,passing through complex network environments. For this reason, we will include in our missionalso common symptoms which indicate the issue could be elsewhere in your environment.

4

2.2. The tuning processIt is a common belief that the tuning process begins when customers complain about poor responsetime. This is unfortunately too late in the software process to use some of the most effective tuningstrategies. At that time, you are usually unable or unwilling to completely redesign the application.Therefore, you may only improve performance marginally by tweaking memory settings andtuning your I/O.

On the other hand, a solid tuning strategy is an iterative process which is part of the softwaredevelopment cycle and it is composed of three distinct steps, as highlighted by the followingpicture:

• Data gathering: that is collecting statistics about your application under load.

• Data analysis: that is making the hypothesis on the cause of any possible distortion of datagathering from the expected values.

• Verify hypothesis: by applying the required changes, you will be able to verify if the bottleneckis solved. Otherwise, the process continues.

5

Let’s break done more in detail the process in the next sections.

2.2.1. Data gathering

So, the first obvious step in order to begin your performance tuning is that you are able to collectsome data about your application. This can be in turn broken down in the following list ofactivities:

• Create an environment with the same settings (hardware/software) where the application canbe tested

• Isolate the testing environment so that that the data collected is not influenced by externalapplications

• Install the appropriate tools that will start the load test and the application. The amount of loadshould be as much as possible equivalent to a production load of the system

Usually, in this first phase, no configuration changes are applied to the application, unless themonitoring exposes a critical performance threat. In some cases, experienced performanceengineers can identify potential problems just by looking at data gathering alone, although it ishighly recommended to ascertain a performance degradation before applying any change.

Without a well-measured performance issue, you shouldn’t optimize just becauseyou think you will get a performance improvement. There are obvious Javaoptimizations (like not doing string concatenation inside a tight loop) butanything that isn’t a trivial change should be avoided until it can be measured.The biggest problems with "unconsidered optimization" is that it makes code un-maintainable and is often the cause of performance problems.

Not all early optimizations are however to be regarded as evil. Rather it depends on what you aretrying to optimize. For example, here are some good early optimizations you should apply in thevery beginning of your project:

• Architectural optimizations: that is, the way the application is structured and how the singlelayers are composed.

• Data flow optimizations: that is how data transits from inside and outside of an application.

• If you are developing unit test as part of your coding, you might have early indications ofproblems. The catch is that you need to connect to external systems from your tests which is notalways doable.

On the other hand, some optimizations which can be applied in the mid development cycleinclude:

• Data structures optimization: that introduces new data structures that have better performanceor less overhead

• Algorithms optimization: for example choosing between a quicksort3 and heapsort

Finally, some end development cycle optimizations include:

6

• Finding code hotpots: such as loops or functions that should be optimized

• Profile-based optimizations: using a tool which is able to capture metrics from yourapplication’s resources.

2.2.2. Data analysis

When you have collected a sufficient amount of data, you will start to analyze them and have thefirst evidence of which areas show a performance penalty.

Finding which component takes most of the time in your application might be just the surface of aproblem which is in a totally different area. To make sure you are able to solve the problem at theroot you should follow a well-defined strategy:

1. Identify the locations of any bottleneck.

2. Make some hypothesis about the possible cause of the bottleneck.

3. Consider any factors that may prove/disprove your hypothesis.

Based on this process, you should be able to adapt your test environment to be executed with newsettings. For example, if you believe your application code is flawless and you require a largeramount of Connections from the pool, then executing the test with a new max-pool-size mayconfirm your hypothesis In the end, by carefully examining performance monitors, you can isolatethe problem and thus identify how to solve the bottleneck. The re-execution of the test will be thenext step.

2.2.3. Verify the hypothesis

Performance tuning is an iterative process, therefore you need to execute again your tests until thebottleneck has been solved. Typically, the data analysis has provided one or more hypothesis aboutthe bottleneck of your application. You should be able to establish a priority list so that you applychanges individually. This is a key factor of performance tests: you cannot verify your hypothesisapplying multiple changes at the same time as it’s difficult to identify which one actually varied theresult. So apply the next change in your priority lists and continue the performance tuning process,until the bottleneck is solved.

As a side note consider that optimizing code can introduce new bugs so theapplication should be tested during the optimization phase. A particularoptimization should not be considered valid until the application using thatoptimization’s code path has passed the quality assessment.

7

Chapter recap

• Performance tuning is an iterative process which should be part of every softwaredevelopment cycle.

• Design your project’s blueprint using the best architectural practices, which include alsoperformance considerations.

• Premature optimization takes up a lot of time and makes the code hard to read andmaintain with no guarantee that it will be effective.

• Define how fast your application needs to be. If you develop unit test as you code, you canhotspot some bottlenecks in the early phases of your project, provided that you canconnect or emulate the connection to external systems.

8

3. Tools for monitoring performanceIn order to gather statistics about WildFly and the applications running on it, we need some tools.First of all, the application server itself is able to produce metrics which can be gathered just usingits Command Line Interface.

Besides it, there are several free tools which can be used to specifically monitor the applicationserver’s JVM and its MBeans, such as JConsole and JVisualVM. Both these tools enable basicmonitoring of JVM processes, thread and memory usage, loaded classes, and other available JVMmetrics.

Much the same way, you can use other tools which are able to connect to the application server’sengine through the JMX Protocol, such as Hawtio which features a modular Web console with lotsof valuable plugins.

Finally, if you want to aggregate, your data and have some visual and actionable insights werecommend having a look at the ELK Stack which is discussed at the end of this chapter.

3.1. Using JConsole to monitor WildFlyWhile JConsole is not a very sophisticated tool, it does give some valuable information aboutresource consumption and access to the MBeans running in the JVM. Furthermore, the JConsoleplug-in API provides a mechanism by which you can, for example, add a tab to access your ownapplication’s MBeans.

JConsole ships with the JDK (but not the JRE) in the $JAVA_HOME/bin directory, however, werecommend using the preconfigured jconsole.sh wrapper script is bundled with WildFly. Using thiswrapper script ensures that all the required libraries are added to the classpath, and also providesaccess to the WildFly management CLI from within JConsole.

To launch JConsole, open a terminal or command window, change to the $JBOSS_HOME/bin directory,and execute:

$ ./jconsole.sh

Mind to include the current path ("./") when executing JConsole otherwise youmight launch the jconsole script which is in the $JAVA_HOME/bin path (in case youhave put $JAVA_HOME/bin in your System’s PATH).

When JConsole starts, it shows a window listing the managed Java VMs on the machine. The ProcessId (PID) and command line arguments for each Java VM are also displayed. Select the applicationserver JVM and connect to it. You’ll see a window similar to the following picture:

9

How to find out the Process used by WildFly? Apart from using the 'ps' built-inLinux command, having the JDK Installed you can easily find it with:

jps -lv | grep -i jboss | cut -d ' ' -f 1

3.1.1. What JConsole can do for you

The UI of JConsole is made up of several tabs:

The Overview tab displays graphical monitoring information about CPU usage, memory usage,thread counts, and the classes loaded in the Java VM, all in a single screen. This can be used tomonitor the application for a period of time.

10

The Memory tab provides information about memory consumption and memory pools. If thememory curve touches the maximum memory allowed, then we can suspect OutOfMemory or highmemory usage. A well-behaved application should exibit a rhythmic graph curve like in thefollowing picture:

11

You can use Perform GC option to manually execute the Garbage Collection. This ishelpful to confirm OutOfMemory Error/high memory usage in case the garbagecollection produced no effect.

The Threads tab provides information about thread use. It gives the number of threads running inthe application. We can get the Stack Trace of a particular thread by clicking the thread name onthe left side panel. If the number of threads increasing is continuous, then we can suspect thethread leak.

12

The Threads also includes a Detect Deadlock option: By clicking on it, any detected deadlockedthread will be displayed in a new tab that appears next to the Threads tab.

The Classes tab displays information about class loading. This gives the number of classes loaded inthe JVM.

13

The VM Summary tab provides information about the Java VM.

Accessing the VM Summary Tab is useful for a quick look into the remote JVMparameters such as Garbage collector’s policy:

14

The MBeans tab displays information about all the MBeans registered with the platform MBeanserver in a generic way. This is a handy shortcut to reach the application server’s MBeans and canbe an alternative way to manage the application server.

Some application server parameters might now be still available in the standardmanagement interfaces, therefore, it’s needed to access this information throughthe MBeans.

15

Finally, it’s worth mentioning that JConsole includes also a Tab named WildFly CLI which allowsexecuting WildFly Command Line in graphical mode.

3.1.2. Connecting to a remote WildFly Server in Standalone Mode

In order to connect to a remote WildFly Server using JConsole, you will need at first a managementuser to authorize access to the server. You can create one using the add-user.sh script available inthe $JBOSS_HOME/bin folder. Here’s an example:

./add-user.sh -u admin1234 -p Password1!

As an example, we will connect to a WildFly server running in a Docker container at the IP Address172.17.0.2. In order to connect from JConsole we need to use the protocol JMX over remote with thefollowing URI:

  service:jmx:remote+http://172.17.0.2:9990

Additionally, you must include in the Username and Password the credentials for your User asindicated by the following picture:

16

In a few seconds, you will be able to reach JConsole’s main UI:

17

3.2. Connecting to a remote WildFly Server in DomainModeWhen running WildFly in Domain mode, the main difference is that individual servers do not havea management port. Only the Host Controller exposes a management port. Therefore, we need toset the remoting-connector in the jmx subsystem to not use the management endpoint as follows:

/profile=full/subsystem=jmx/remoting-connector=jmx:add(use-management-endpoint=false)

In terms of configuration, this will result in the following change in domain.xml:

<subsystem xmlns="urn:jboss:domain:jmx:1.3">  <expose-resolved-model/>  <expose-expression-model/>  <remoting-connector use-management-endpoint="false"/></subsystem>

Next, define an ApplicationRealm user -with the add-user script- to be used for connecting to yourremote WildFly servers:

./add-user.sh -a -u user1234 -p Password1!

Finally, execute the following commands from the CLI to add a remoting port to the socket bindinggroup, and add remoting to the ApplicationRealm.

/socket-binding-group=full-sockets/socket-binding=remoting:add(port=4447)

/profile=full/subsystem=remoting/connector=remoting-connector:add(socket-binding=remoting,security-realm=ApplicationRealm)

The configuration is complete. In order to connect to the single WildFly servers of your Domain,you will use the jmx over remoting protocol, which is available on port 4447, as we configured itearlier:

service:jmx:remote://192.168.1.109:4447

As for standalone mode, you will need to add the username and password fromyour Application Realm in order to connect to the Server.

3.3. Using Java VisualVM to monitor WildFlyJava VisualVM provides a more advanced set of options to monitor Java processes. It is located in

18

the $JAVA_HOME/bin folder of your Oracle JDK.

The following sections provide instructions for using VisualVM to connect to a local or remoteWildFly JVM. As for JConsole, we will at first check out how to connect to a local JVM process andthen to a remote server.

3.3.1. Connecting to a Local WildFly JVM Using VisualVM

To connect to a WildFly JVM running on the same machine as VisualVM:

1. Open VisualVM, and find the Applications panel on the left side of the VisualVM window.

2. Under Local, double-click the WildFly JVM process that you want to monitor.

For a standalone WildFly server, there is one WildFly JVM process.

A WildFly managed domain host has multiple JVM processes you can connect to: a Host controllerJVM process, a Process controller JVM process, and a JVM process for each WildFly Server on thehost. You can find out the active WildFly server instance with this command:

$ ps -ef | grep "org.jboss.as.server"

Or, if you prefer to capture just the Process ID (PID):

$ pgrep -f "org.jboss.as.server"

Once connected, your UI will look like this:

19