12 Best Practices for Optimizing the JBoss EAP Platform

Over the past several years, the Web Architecture team at NBC Universal Studios has dedicated a significant amout of their time and resources to improving the performance of their highly distributed enterprise architecture. In this multi-part series, Richir Choudhry, Alpesh Vaghela and Vaidyanathan Kothandaraman will be sharing their insights, tips and the best practices they employed during their efforts to enhance application performance without actually editing or refactoring any of the production code.

Optimization Techniques for JBoss EAP 4.3 Production Environments

The environment was built from an Apache, Red Hat, JBoss and Sun JDK core, though many of the performance improvement tips can be universally applied to other production environments.

Best Practices

The following are the tried and true best practices used at NBC Universal Studios for all JBoss EAP 4.3 production environments. Some of the best practices are common sense, while others require some fine tuning and testing.

1) Use the most recent version of JDK /JRE
For each major Java™ release "train" (e.g.J2SE 1.4.2, J2SE 5.0, J2SE 6.0) Sun/Oracle publishes update releases on a regular basis. For example the most recent update release of Java SE 1.6.0_21. Update releases often include bug fixes and performance improvements. Deploying the most recent update release for Java™ is an easy way tol benefit from the latest and greatest performance improvements from Sun/Oracle.

2) Insure OS patches are up-to-date
Even though Java is cross platform it does rely on the underlying operating system and therefore it is important that the OS basis for the Java™ Platform is as up-to-date as possible. In our case we used Red Hat Enterprise 5.3 on VM.

3) Use trimmed version of Web Server (Apache2.2.8, 2.2.15), by compiling it specific for the needs of the applications or env.
Apache comes with various configuration parameters to give more and more flexibility to the application. It’s very much required to pick what is good for us and what is the need of the application. Based on our specific needs, we pruned out various Apache modules, and recompiled for our specific requirements.

4) Place the code on the local env not on the storage; content can reside on the storage.
By placing the code on the local (a carved out space on virtual machine (VM Ware) from the storage), there is no traversing to the storage, rather it utilizes the higher capacity bandwidth, which is dedicated for the VM Ware env.

5) Trim the application server (Jboss EAP 4.3) by exactly using what you need from the application server
By trimming the JBoss 4.3 EAP  we removed a number of services which comes with a J2EE container by default like JMS, session based clustering, large size of object pooling, connection poking , XA Transaction, Modjk tomcat connectivity and much more.

6) Use info logging
Use only info type of logging in production; this reduces I/O to the file system.

7) Test various JVM configuration settings and make sure to use what is best for the env
We used this to find out what is the right size for our env and how it fluctuates based on the different application needs. Based on our test, we found that 2 VCPU with 4 GB of RAM is best suited for our applications/env. Details of the most successful JVM configurations will be detailed in a follow-up article.

8) Test various CPU or VCPUS, RAM options to find out what’s best suited for the env

The right size of the CPU & RAM varies due to type of hardware used; design an type of application, how the multithreaded env is used/programmed, type of platform and so on and so fourth. We used VM Ware RED HAT 5.3 Enterprise edition on ESX farms. We did various rounds of test, as detailed in appendix 4, and found that 2VCPU along with 4 GB of RAM per Jboss EAP4.3 is needed.

9) Use hardware based load balancing to provide a HA env; Avoid clustering if it's not the business need

Its tempting to use software based clustering and it can be done at various levels and in various ways. But in all possible scenarios it creates an overhead.

Example:  HAPartition is a general-purpose service used for a variety of tasks in AS clustering. At its core, it is an abstraction built on top of a JGroups Channel that provides support for making/receiving RPC invocations on/from one or more cluster members. HAPartition allows services that use it to share a single Channel and multiplex RPC invocations over it, eliminating the configuration complexity and runtime overhead of having each service create its own Channel.

10) Disable hot deployment if its not needed in production

Looking at the hot deployment features supported by various vendors everybody thinks it’s “cool idea” but not robust as it appears. None of the vendors recommend wholeheartedly these in production environment, so you have to be careful when you decide to use this in your production server running your website or mission critical applications. In addition to the above during hot deployment Container spends significant CPU cycles polling for changes in applications. This is the sole reason we decided not to use hot deployment.

11) Start JBoss EAP 4.3 always in production mode if you can

During our series of test we did find the best startup mode among the various modes of startup (default, all, minimal and production) is production. This gave us the best possible performance.

12) Eliminating Variables

Be aware that various system activities and the operations of other applications running on your system can introduce significant variance into the measurements of any application's performance, including Java applications. The activities of the OS and other applications may introduce CPU, Memory, disk or network resource contention that may interfere with your measurements.