Problem Description

Out Of Memory (OOM) - An application displays Out of Memory errors due to memory exhaustion, either in java heap or native memory.
Memory Leak - Constant memory growth in either java heap or native memory, which will eventually end up in out of memory situation.
The techniques to debug the memory leak situations are the same as the out of memory situations.

Problem Troubleshooting
Please note that not all of the following items would need to be done.  Some issues can be solved by only following a few of the items.

Quick Links:

Java heap, Native memory, Process size
Difference between process address space and physical memory
Why does the OOM problem occur and What does the JVM do in this situation?
Steps to debug the problem
HP JVM specific tools/tips
Jrockit specific features
Known Jrockit issues
Sample initial response e-mail from a Frontline DRE to a customer
Bibliography

Java heap, Native memory and Process size

Java heap – This is the memory that the JVM uses to allocate java objects. The maximum value of java heap memory is specified using –Xmx flag in the java command line. If the maximum heap size is not specified, then the limit is decided by the JVM considering factors like the amount of physical memory in the machine and the amount of free memory available at that moment. It is always recommended to specify the max java heap value.

Process size – Process size will be the sum of the java heap, native memory and the memory occupied by the loaded executables and libraries. On 32 bit operating systems, the virtual address space of a process can go up to 4 GB. Out of this 4 GB, the OS kernel reserves some part for itself (typically 1 – 2 GB). The remaining is available for the application.

Windows – By default, 2 GB is available for the application and 2 GB is reserved for Kernel’s use. However, on some variants of Windows, there is a /3GB switch which can be used to change this ratio such that the application gets 3 GB. More details on the /3GB switch can be found here –
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ddtools/hh/ddtools/bootini_1fcj.asp

RH Linux AS 2.1 – 3 GB is available for the application.

For other operating systems, please refer to the OS document.

Difference between process address space and physical memory

Each process gets its own address space. In 32 bit operating systems, this address space will range from 0 to 4 GB. This is independent of the available RAM or swap space in the machine.
The total physical memory available on the machine is the sum of RAM and the swap space available on that machine. All the running processes share this physical memory.
The memory address within a process is virtual. The kernel maps this virtual address to the physical address. The physical address points to a location somewhere in the physical memory.
At any given time, the sum of all the virtual memory used by the running processes in a machine cannot exceed the total physical memory available on that machine.

Why does the OOM problem occur and What does the JVM do in this situation?

Out of memory in java heap

JVM throws java out of memory (java OOM) error if it is not able get more memory in java heap to allocate more java objects. The JVM cannot allocate more java objects if the java heap is full of live objects and it is not able to expand the java heap anymore.
In this situation, the JVM let the application decide on what to do after throwing the java.lang.OutOfMemoryError. For example, the application may handle this error and decide to shut down itself in a safe way or decide to run ignoring this error. If the application doesn’t handle this error, then the thread that throws this error will exit (you will not see this thread if you take a java thread dump).
In case of weblogic server, this error is handled if it is thrown by an execute thread and the error is logged. If this error is being thrown continuously, then the core health monitor thread shuts down the weblogic server.

Out of memory in native heap

JVM throws native out of memory (native OOM) if it is not able to get any more native memory. This usually happens when the process reaches the process size limitation on that OS or the machine runs out of RAM and swap space.
When this happens, the JVM handles the native OOM condition, logs a message saying that it ran out of native memory or unable to acquire memory and exits. If the JVM or any other loaded module (like libc or a third party module) doesn’t handle this native OOM situation, then the OS will send a sigabort signal to the JVM which will make the JVM exit. Usually, the JVMs will generate a core file when it gets a sigabort signal.

Steps to debug the problem

1. Find out whether it is a java OOM or native OOM

a) If the stdout/stderr message says that this is a java.lang.OutOfMemoryError, then this is java OOM
b) If the stdout/stderr message says that it failed to acquire memory, then this is a native OOM

Please note that the above messages goes to stdout or stderr and not to the application specific log files like weblogic.log

2. Steps to follow in case of java OOM
2.1) Collect and analyze the verbose gc output

a) Add ‘-verbosegc’ flag in the java command line - This will print GC activity info to stdout/stderr. Redirect the stdout/stderr to a file. Run the application until the problem gets reproduced.
b) Make sure that the JVM does the following before throwing java OOM

Full GC run: Does a full GC and all the un-reachable, phantomly, weakly and softly reachable objects are removed and those spaces are reclaimed. More details on different levels of object reachability can be found here :
http://java.sun.com/developer/technicalArticles/ALT/RefObj

We can check whether full GC was done before the OOM message. A message like this is printed when a full GC is done (format varies depending on the JVM – Check JVM help message to understand the format)

[memory ] 7.160: GC 131072K->130052K (131072K) in 1057.359 ms

The format of the above output is (I will stick to the same format throughout this document) :

[memory ] <start>: GC <before>K-><after>K (<heap>K), <total> ms
[memory ] <start> - start time of collection (seconds since jvm start)
[memory ] <before> - memory used by objects before collection (KB)
[memory ] <after> - memory used by objects after collection (KB)
[memory ] <heap> - size of heap after collection (KB)
[memory ] <total> - total time of collection (milliseconds)

However, there is no way to conclude whether the soft/weak/phantomly reachable objects are removed using the verbose messages. If we suspect that these objects are still around when OOM was thrown, we need to take this up with the JVM vendor.

If the garbage collection algorithm is a generational algorithm (gencopy or gencon in case of Jrockit and the default algorithm in case of other JDKs), you will also see verbose output something like this :
[memory ] 2.414: Nursery GC 31000K->20760K (75776K), 0.469 ms

The above is the nursery GC (or young GC) cycle which will promote live objects from nursery (or young space) to old space. This cycle is not important for our analysis. More details on generational algorithms can be found in JVM documentation.

If the GC cycle doesn’t happen before java OOM, then it is a JVM bug.

Full compaction: Make sure that the JVM does proper compaction work and the memory is not fragmented which could prevent large objects being allocated and so java OOM error.
Java objects need the memory to be contiguous. If the available free memory is fragmented, then the JVM will not be able to allocate a large object, as it may not fit in any of the available free chunks. In this case, the JVM should do a full compaction so that more contiguous free memory can be formed to accommodate large objects.
Compaction work involves moving of objects (data) from one place to another in the java heap memory and updating the references to those objects to point to the new location. JVMs may not compact all the objects unless if there is a need. This is to reduce the pause time of GC cycle.

We can check whether the java OOM is due to fragmentation by analyzing the verbose gc messages. If you see something like this where the OOM is being thrown even whether there is free java heap available, then it is due to fragmentation.

[memory ] 8.162: GC 73043K->72989K (131072K) in 12.938 ms
[memory ] 8.172: GC 72989K->72905K (131072K) in 12.000 ms
[memory ] 8.182: GC 72905K->72580K (131072K) in 13.509 ms
java.lang.OutOfMemoryError

In the above case you can see that the max heap specified was 128MB and the JVM threw OOM when the actual memory usage is only 72580K. The heap usage is only 55%. Therefore, you can say that the effect of fragmentation in this case is to throw OOM even when there is 45% of free heap. This is a JVM bug or limitation. We need to take this up with the JVM vendor.

2.2) If the JVM does its work properly (all the things mentioned in the above step), then the java OOM could be an application issue. The application might be leaking some java memory constantly, which may cause this problem. Or, the application uses more live objects and it needs more java heap memory. The following things can be checked in the application:

a) Caching in the application - If the application caches java objects in memory, then we should make sure that this cache is not growing constantly. There should be a limit for the number of objects in the cache. We can try reducing this limit to see if it reduces the java heap usage.
Java soft references can also be used for data caching as softly reachable objects are guaranteed to be removed when the JVM runs out of java heap.

b) Long living objects - If there are long living objects in the application, then we can try reducing the life of the objects if possible. For example, tuning HTTP session timeout will help in reclaiming the idle session objects faster.

c) Memory leaks – One example of memory leak is when using database connection pools in application server. When using connection pools, the JDBC statement and resultset objects must be explicitly closed in a finally block. This is due to the fact that calling close() on the connection objects from pool will simply return the connection back to the pool for re-use and it doesn’t actually close the connection and the associated statement/resultset objects.

It is recommended to follow the coding practices suggested in the following URLs to avoid memory leaks in your application.

JDBC - http://e-docs.bea.com/wls/docs81/jdbc/troubleshooting.html#1026696

JNDI - http://e-docs.bea.com/wls/docs81/jndi/jndi.html#472853

JMS - http://e-docs.bea.com/wls/docs81/jms/implement.html#1194127

d) Increase the java heap - We can also try increasing the java heap if possible to see whether that solves the problem.

e) Work around - As a temporary work around, the application may be gracefully re-started when the java heap usage goes about 90%. When following this work around, the java max heap can be set to as high as possible so that the application will take more time to fill all the java heap. The java heap usage can be monitored by adding '-verbosegc' flag in the java command line which will send the GC/ heap usage info to stdout or stderr.

2.3) If none of the above suggestion is applicable to the application, then we need to use a JVMPI (JVM Profiler Interface) based profiler like Jprobe or OptimizeIt to find out which objects are occupying the java heap. The profilers also give details on the place in the java code from where these objects are being created. This document doesn’t cover the details on each profiler. The profiler documentation can be referred to understand how to set and start the application with this profilers. In general, JVMPI based profilers have high overhead and drastically reduce the performance of the application. Therefore, it is not advisable to use these profilers in production environments.
http://www.borland.com/optimizeit
http://www.quest.com/jprobe

3. Steps to follow in case of native OOM

3.1) Collect the following information

a) –verbosegc output to monitor the java heap usage. This will help to understand the java memory requirement for this application.
It should be noted that independent of the actual java heap usage by the application, the amount of max heap specified (using –Xmx flag in the java command line) is reserved at the JVM startup and this reserved memory is not available for any other purpose.
In case of Jrockit, use ‘-verbose’ instead of ‘-verbosegc’ as this gives codegen information in addition to GC information.

b) Record the process virtual memory size periodically from the time the application was started until the JVM runs out of native memory. This will help to understand whether the process really hits the size limitation on that OS.
In case of Windows, use the following procedure to monitor the virtual process size:
1) In the Start -> Run… dialog, enter “perfmon” and click OK.
2) In the “Performance” window that pops up, click on the ‘+’ button (above the graph).
3) Select the following options in the resulting dialog:

• Performance object: Process (not the default Processor)
• Select counter from list: Virtual Bytes
• Select instances from list: Select the JVM (java) instance
• Click “Add”, then “Close”

In case of Unix or Linux, for a given PID, the virtual memory size can be found using this command – ‘ps –p <PID> -o vsz’
In case of Linux, each java thread within a single JVM instance is shown as a separate process. It is enough if we take the PID of the root java process. The root java process can be found using the ‘—forest’ option for ‘ps’ command. For example, ‘ps –lU <user> --forest’ will give a ASCII tree art for all the processes started by the specified user. You can find the root java from the tree art.

3.2) Memory availability in the machine
If the machine doesn’t have enough RAM and swap space, then the OS will not be able to give more memory to this process that could also result in out of memory. Make sure that the sum of RAM and swap space in the disk is sufficient to cater to all the running processes in that machine.

3.3) Tuning java heap
If the java heap usage is well within the max heap, then reducing the java max heap will give more native memory to the JVM. This is not a solution but a workaround that can be tried. Since the OS limits the process size, we need to strike a balance between the java heap and the native heap.

3.4) Native memory usage by the JVM
The amount of native memory usage by the JVM is expected to flatten out after all the classes are loaded and the methods have been called (code generation is over). This usually happens within the first few hours for most of the applications. After that, the JVM uses only little native memory that may be due to run time class loading, code generation due to optimization etc.
In order to narrow down the problem, run time optimizations can be disabled and check whether that makes any difference. In case of Jrockit, -Xnoopt flag can be used to disable run time optimizations. In case of SUN hotspot JVM, -Xint flag will make the JVM to run in interpreted mode (no code generation). If the native memory usage continues to grow constantly throughout the run, then this could be a memory leak in the native code.

3.5) Third party native modules or JNI code in the application
Check whether they are using any third party native module like database drivers. These native modules could also allocate native memory and the leak may be from these modules. In order to narrow down the problem, we should attempt to reproduce the problem without these third party modules. For example, we can use pure java drivers instead of native database drivers.
Check whether the customer application uses some JNI code. This could also be causing native memory leak and we can try to run the application without the JNI code if possible.

3.6) If the source of native memory cannot be found after the above steps, then we need to work with the JVM vendor to get a special build which can trace the native memory allocation calls and give more information about the leak.

HP JVM specific tools/tips

The following URL gives some tools and tips specific to OOM situations with HP JVM HP JVM tools/tips

Jrockit specific features

1. Jrockit 81SP1 and above supports JRA recording (Java Runtime Analyzer). This is useful to gather information at JVM run time which will give information about the application like, number of GCs running, number of soft/weak/phantom references, hot methods etc. It is useful to make a recording for few minutes and analyze the data if the JVM has performance problems or hang problems. More details on this can be found in Jrockit docs.
http://e-docs.bea.com/wljrockit/docs142/userguide/jra.html

2. (BEA internal use only) Jrockit 81SP3 has a in-build java OOM diagnostic tool which is very useful to debug java OOM problems. This is only for internal support use and so it is not documented in our public documentation. This tool has very light overhead compared to JVMPI based profilers. More details on this tool can be found in the internal documentation page
http://wiki.jrpg.bea.com/Wiki.jsp?page=MemoryLeakTool

3. (BEA internal use only) Jrockit engineering has developed a library which can be used to detect native memory leaks. More details on this can be found in the internal docs
http://wiki.jrpg.bea.com/Wiki.jsp?page=Libmemoryprofile

Known Jrockit issues (BEA internal use only)

1. Java heap fragmentation issues – CR130147 – Some type of objects are pinned objects, meaning, the JVM won’t move those objects throughout their lifetime. This could lead to OOM even when there is lot of free java heap. This issue is fixed in 81SP3.
2. OOM thrown even when softly referenced objects exist - CR100810, CR129265. These issues will be fixed in 81SP3/70SP5
3. Gencopy algorithm has a limitation that makes the JVM to use only about 50% of the specified max heap. This is because of the fact that the old generation space in this case is split into 2 portions (“from” and “to”) and if one of the portions become full, then the JVM has to throw OOM. Gencopy will be removed in 81SP3.
4. Gencon/gencopy - JVM might throw OOM even if the nursery is empty (when old generation space if full). This is a product limitation.
5. Native memory leak in Windows - CR128798 – This issue is fixed in 81SP2RP1 release
6. JVM process size cannot grow beyond 2.1 GB in linux which limits the max java heap – This will be fixed in 81SP3 (CR133516).

Sample initial response e-mail from a Frontline DRE to customer

This is a sample e-mail that a Frontline DRE can send a customer when handling this type of case. Please note that case particulars vary widely, so each e-mail should be tailored to the specifics of the customer.

Bibliography

• Sun Java Web Site
• Sun Java Newsgroups
• JRockit Documentation
• dev2dev (JRockit)
• JRockit newsgroup