We recently worked with a customer who saw Native Heap Exhaustion on their Weblogic server. Weblogic9 running on Solaris with LiveCycle 821.x.
While majority of customers have upgraded to LiveCycle ES2.x or LiveCycle ES3 which require 64bit OS + JVM combinations, some customers are still on LiveCycle 821.x with 32bit OS/JVM combinations.
Out Of Memory exceptions in JVM logs could indicate either JVM heap exhaustion or Native Heap exhaustion. Closer look to the stacktrace could help determine the root cause.
On 32 bit systems, since max memory space that can be accessed by a process is 4GB; choice of max JVM heap setting could be critical.
If JVM heap is too high, it leaves less room for native heap, and you could see Out Of Memory exceptions in JVM logs.
In this particular scenario, exceptions we saw in WebLogic server logs were:
<<anonymous>> <BEA1-53BF593214445136C2FE> <> <1318337670542> <000000> <stalling action-instance: 742271 with message: /u02/app/adobe/bea/jdk150_10/jre/lib/sparc/libawt.so: Can’t load Sparc 32-bit .so on a Sparc 32-bit platform: java.lang.UnsatisfiedLinkError: /u02/app/adobe/bea/jdk150_10/jre/lib/sparc/libawt.so: Can’t load Sparc 32-bit .so on a Sparc 32-bit platform
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
Closer look at the stacktrace indicates that JVM was not able to load the native .SO file on Solaris due to native heap exhaustion.
Their JVM was set to use 3GB of Xmx (max heap), leaving very limited room for the native heap.
In this case we resolved the problem by reducing the Max JVM heap to 2GB.
It might sound counter intuitive, but increasing JVM heap to deal with Out Of Memory scenarios might not be the right thing to do all the time. It depends…