Posts tagged "WebSphere"

LiveCycle ES2: Unexpected exception. Failed to create directory: \\<cluster>\GDS

Issue

If you have configured the WebSphere node agents to start as Windows services instead of using the batch files in your cluster you may experience the following behaviour:

  • in AdminUI > Services > Applications and Services > Application Management, you may receive an ALC-DSC-033-000 error
  • in AdminUI > Services > Applications and Services > Service Management, you will see many services marked Inactive, and you cannot stop/start them

You will also notice the following errors in the SystemOut.log on the individual cluster nodes when starting the servers:

E com.adobe.idp.dsc.registry.service.impl.ServiceStoreImpl createServiceConfigurationFromBOI
    COULD NOT PROCESS SERVICE: Process Management (system)/Queue Sharing VERSION: 1.0 MARKING AS INACTIVE
E com.adobe.idp.dsc.registry.service.impl.ServiceStoreImpl createServiceConfigurationFromBOI TRAS0014I:
    The following exception was logged java.lang.ClassNotFoundException: server.startup : 1Class name
    com.adobe.idp.workflow.dsc.descriptor.WorkflowDSCDescriptor from package com.adobe.idp.workflow.dsc.descriptor not found.
....
W com.adobe.idp.dsc.registry.service.impl.ServiceRegistryImpl getHeadActiveConfiguration
    Head active ServiceConfiguration for service: Process Management (system)/Queue Sharing version: 1.0 is not active as expected.
    The service was probably Marked INACTIVE due to an error loading the service.  Please look at the logs to see if this service was Marked INACTIVE.
...

The SystemOut.log will also contain the following error:

E com.adobe.idp.DocumentFileBackend getTimeSkewDelta DOCS001: Unexpected exception. Failed to create directory: \\es2cluster\GDS.
E com.adobe.idp.Document passivate DOCS001: Unexpected exception. While re-passivating a document for inline data resizing..
E com.adobe.idp.Document passivate TRAS0014I: The following exception was logged com.adobe.idp.DocumentError:
Failed to create directory: \\es2cluster\GDS
    at com.adobe.idp.DocumentFileBackend.getTimeSkewDelta(DocumentFileBackend.java:113)
    at com.adobe.idp.DocumentFileBackend.<init>(DocumentFileBackend.java:86)
    at com.adobe.idp.Document.getGlobalBackend(Document.java:3153)
    at com.adobe.idp.Document.writeToGlobalBackend(Document.java:2977)
    at com.adobe.idp.Document.doInputStream(Document.java:1698)
    at com.adobe.idp.Document.passivateInitData(Document.java:1457)
    at com.adobe.idp.Document.passivate(Document.java:1235)
    at com.adobe.idp.Document.passivateGlobally(Document.java:1205)
    at com.adobe.idp.dsc.management.impl.ArchiveStoreImpl.createNewArchive(ArchiveStoreImpl.java:636)
    at com.adobe.idp.dsc.management.impl.ArchiveStoreImpl._getArchive(ArchiveStoreImpl.java:438)

Reason

The Windows services that you have created do not have the correct privileges for accessing the LiveCycle GDS directory which is required for the proper operation of the LiveCycle server.

Solution

Open the Services view from the Control Panel in Windows.
Select the WAS NodeAgent service and goto properties > LogOn > Log on as > This account.
Browse for an Administrator account with sufficient privileges to access the GDS share.
Restart the NodeAgent.
Then restart the cluster from the WAS ND admin console.

Additional Information

To create the nodeagent as a Windows service instead of using the batch files:

1. Change to the \bin directory where WebSphere is installed, for example:

C:\Program Files\IBM\WebSphere\AppServer\bin

2. Run the following command:

wasservice -add ctgNode01_nodeagent
          -servername nodeagent
          -profilePath “C:\Program Files\IBM\WebSphere\AppServer\profiles\ctgAppSrv01″
          -wasHome “C:\Program Files\IBM\WebSphere\AppServer”
          -logFile “C:\Program Files\IBM\WebSphere\AppServer\profiles\ctgAppSrv01\logs\nodeagent\startNode.log”
          -logRoot “C:\Program Files\IBM\WebSphere\AppServer\ctgAppSrv01\logs\nodeagent”
          -restart true
          -startType automatic

You can do this for each cluster node.  Now when you restart Windows, the nodeagent on each cluster node will start automatically.

reference: (1911394)

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 9.0/10 (1 vote cast)

LiveCycle ES2: SECJ0305I error in WebSphere log when using an email endpoint

Issue

When using an email endpoint in LiveCycle ES2 on WebSphere, you may notice the following error in the server log each time the email endpoint is invoked, or scans for new emails:

00000020 RoleBasedAuth A   SECJ0305I: The role-based authorization check failed for naming-authz operation NameServer:bind_java_object. 
The user UNAUTHENTICATED (unique ID: UNAUTHENTICATED) was not granted any of the following required roles: CosNamingWrite, CosNamingDelete, CosNamingCreate.
00000020 EmailReaderIm E com.adobe.idp.dsc.provider.service.email.impl.EmailReaderImpl getEmailSourceLock NO_PERMISSION exception caught

Reason

This error is repeatedly sent to the server log due to missing permissions for the CORBA naming service groups in WebSphere.

Here is the configuration with the missing privileges:

Solution

By making a small change in the WAS admin console you can resolve these errors.  You need to add the privileges for “Cos Naming Write”, “Cos Naming Delete” and “Cos Naming Create” to the CORBA naming service groups.

Open the WebSphere administration console

Goto Environment > Naming > CORBA Naming service groups

Add the following privileges “Cos Naming Write”, “Cos Naming Delete” and “Cos Naming Create”

Restart the WebSphere application server for the changes to take affect

Here is the correct configuration:

reference: (182724546/2998051)

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 7.0/10 (1 vote cast)

LiveCycle ES: OutOfMemoryError: SystemFailure Proctor : memory has remained chronically below 1.048.576 bytes

Issue

 The following exception appears repeatedly in the LiveCycle server log files after the server has been running for some time:

17.04.10 12:04:22:174 CEST 000000bc WorkPolicyEva W com.adobe.idp.dsc.workmanager.workload.policy.WorkPolicyEvaluationBackground
Task run Policies violated for statistics on ‘adobews__1730804381:wm_default’. 
Cause: com.adobe.idp.dsc.workmanager.workload.policy.WorkPolicyViolationException: Policy Violation. Will attempt to recover in 60000 ms

17.04.10 12:04:49:402 CEST 000002bc ExceptionUtil E CNTR0020E: EJB encountered an undefined exception calling the method “getObjectType” 
for Bean “BeanId(dms_lc_was#adobe-pof.jar#adobe_POFDataDictionaryLocalEJB, null)”. 
Exception details: java.lang.OutOfMemoryError: SystemFailure Proctor : memory has remained chronically below 1.048.576 bytes 
(out of a maximum of 1.073.741.824 ) for 15 sec.
 At com.gemstone.gemfire.SystemFailure.runProctor(SystemFailure.java:573)
 At com.gemstone.gemfire.SystemFailure$4.run(SystemFailure.java:536)
 At java.lang.Thread.run(Thread.java:811)

When these errors have been repeated in the log for some time the LiveCycle server usually stops processing new work items and goes into a kind of standby mode, waiting on the garbage collector to clean up the Java heap.  This problem has been reported so far on WebSphere only.

Explanation

LiveCycle has a specific threshold for memory usage which, when reached, results in a call to the garbage collector to clean up the heap.  LiveCycle allows a certain amount of time for the garbage collector to react and clean the heap. If this does not happen in the specified time window, then LiveCycle goes into standby mode to prevent data loss due to a memory deficit.

There are two areas of concern to the out-of-memory condition: Adobe Work Manager and the LiveCycle cache subsystem.  Both of these subsystems use a similar approach to identifying a potential or imminent memory deficit.

1. Adobe Work Manager

Adobe Work manager periodically checks the apparent available memory as a percent of the maximum allowable size of the heap. It uses this check as a metric of overall system load, which it then uses to plan the execution of background tasks. When encountering what it perceives as a memory deficit, the Work Manager generates the exception above and refrains from scheduling background tasks. Upon resolution of the memory deficit by a garbage collection event, the Work Manager resumes normal operation.

You can ignore the memory-availability checking of the Work Manager and LiveCycle will continue work as expected. You can configure Work Manager to refrain from using memory availability as a factor in its scheduling. To ignore memory checking in the Work Manager, use the following property:

-Dadobe.workmanager.memory-control.enabled=false

2. LiveCycle cache subsystem

The LiveCycle cache subsystem also periodically checks the apparent available memory in the same way as Work Manager. When the LiveCycle cache subsystem perceives a critical memory deficit, it enters an emergency shutdown mode and no longer operates properly. Consequently, LiveCycle functions that use the cache subsystem no longer operate properly. There is no means to recover other than restarting the applications.

The critical logic within the cache subsystem senses the memory using runtime.freeMemory(). The logic depends on the ability to trigger a garbage collection via System.gc() within the configured wait period defined for assessing the memory deficit. Reviewing the circumstances when System.gc() can be called, this subsystem falls within the best practices described by IBM. That is, a GC is requested only when memory is indeed reaching capacity and the JVM would, of its own accord, be scheduling a GC.

When System.gc() is disabled via -Xdisableexplicitgc, the GC trigger is ignored and the cache subsystem waits for a GC to occur due to the JVMs actions. When the system is lightly loaded or idle, the rate of memory growth is slow. The time between memory exceeding the cache subsystem’s tolerance, and when memory is exhausted sometimes exceeds the period defined for waiting for deficit resolution. This interval is sometimes detected as a critical memory deficit, triggering an emergency shutdown.

The default configuration of the cache subsystem is:

• chronic_memory_threshold — 1 MB

• MEMORY_MAX_WAIT —  15 seconds

Solution

Adobe recommends avoiding these emergency shutdown activities being triggered by enabling System.gc(), and allowing the cache subsystem to operate as designed.

Adobe recognizes that this solution isn’t feasible for all customers, and has identified the following work-around configuration.  If you cannot enable the system Garbage Collector by removing the –Xdisableexplicitgc parameter then you will need to apply the 3 parameters as mentioned below.

-Dgemfire.SystemFailure.chronic_memory_threshold=0 -- zero bytes

With this parameter set to 0, the cache subsystem doesn’t attempt to react to any memory deficit under any circumstances. This setting alleviates the problem reported above. But, it also disables the cache subsystem’s ability to react to actual severe memory deficits. The likelihood of an actual system OOM is low. The impact of such a situation extends well beyond the cache subsystem into other LiveCycle and customer applications. So the recover activities of this one subsystem are unlikely to be critical to the overall state of system health.  Adobe feels that it is reasonable to disable it.

-Dgemfire.SystemFailure.MEMORY_POLL_INTERVAL=360 -- six minutes or 1/10th hour
-Dgemfire.SystemFailure.MEMORY_MAX_WAIT=7200 -- two hours

With the memory deficit and associated potential recovery activities disabled, it is not necessary or useful to set these timer values to monitor memory aggressively. Instead, set the MEMORY_MAX_WAIT to a time period approximating the longest likely duration between garbage collections on a lightly loaded system — about two hours. The poll interval is set to ten times per hour.  The basic polling logic cannot be entirely disabled, but these long poll intervals minimize the already light impact of the thread activity.

Additional information

 LiveCycle Cache Subsystem Configurable JVM Properties

For reference, the controlling parameters for the LiveCycle cache subsystem mentio1ned above are defined as follows:

gemfire.SystemFailure.chronic_memory_threshold = <bytes>

default=1048576 bytes

This setting is the minimum amount of memory that the cache subsystem tolerates before attempting recover activities.  The initial recovery activity is to attempt a System.gc().  If the apparent memory deficit persists for the MEMORY_MAX_WAIT, the cache subsystem enters a system failure mode.  If this value is 0 bytes, the cache subsystem never senses a memory deficit or attempts to take corrective action.

gemfire.SystemFailure.MEMORY_POLL_INTERVAL = <seconds>

default is 1 sec

This setting is the interval, in seconds, that the cache subsystem thread awakens and assesses system free memory.

gemfire.SystemFailure.MEMORY_MAX_WAIT = <seconds>

default is 15 sec

This setting is the maximum amount of time, in seconds, that the cache subsystem thread tolerates seeing free memory stay below the minimum threshold. After exceeding the specified time interval it declares a system failure.

reference: (181544183/2607146)

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 7.0/10 (2 votes cast)

LiveCycle ES2: errors after updating to WebSphere 6.1 Fixpack 35 (6.1.0.35)

Issue

 If you are running LiveCycle ES2 on WebSphere and you update to WAS FP35 (6.1.0.35) you may notice some random errors in the server logs such as:

com.adobe.livecycle.docconverter.client.ValidationException: ALC-CVT-S00-003: Error while validating PDF/A conformance of a document: docConverter.pdf
 at com.adobe.livecycle.docconverter.DocConverterServiceImpl.isPDFA(DocConverterServiceImpl.java:463)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:618)
...
Caused by: com.adobe.internal.pdfm.pdfa.PDFAServiceException: PDFM_I27002: PDF/A Validation was unable to release document 
"{SharedPDFMDocHandle name=docConverter.pdf ID=270667810 getStore={SharedStore name=docConverter.pdf ID=637281788 (faux clone of 1810918384) 
idp=<document state="passive" senderVersion="3" persistent="false" senderPersistent="false" passivated="true" senderPassivated="false" deserialized="true" 
senderHostId="127.0.0.1/4.241.117.208/4.241.117.207/4.241.117.214/4.241.117.234/4.241.117.233/4.241.117.232/4.241.117.236/4.241.117.227/4.241.117.226/
4.241.117.210/4.241.117.216/4.241.117.209/4.241.117.215/4.241.117.218/172.16.252.148/4.239.195.28/4.100.183.28" callbackId="0" senderCallbackId="16" 
callbackRef="com.adobe.idp._IDocumentCallbackStub:IOR:00bdbdbd0000002849444c3a636f6d2f61646f62652f6964702f49446f63756d656e7443616c6c6261636b3a312e30000000000
100000000000000c4000102bd000000194431303041455539313535382e443130302e696e7465726e00bd24550000002c4a4d42490000001071960b49000000000000000000000000000000000000
002400000008000000120000000000000007000000010000001400bdbdbd0501000100000000000101000000000049424d0a0000000800bd00011500000200000026000000020002bdbd49424d040
00000050005020102bdbdbd0000001f0000000400bd0003000000200000000400bd0001000000250000000400bd0003" isLocalizable="true" isTransactionBound="false" 
defaultDisposalTimeout="600" disposalTimeout="600" maxInlineSize="65536" defaultMaxInlineSize="65536" inlineSize="0" contentType="null" length="95543">
<cacheId/><localBackendId><DocumentFileID fileName="/usd/as91558a/work/was-v61/apps/jvmdms.a.1/temp/AdobeDocumentStorage/local/docm1302791768337/1925ec208e7deba285534660215ae481"/>
<localBackendId/><globalBackendId><DocumentFileID fileName="/usd/as91558a/work/was-v61/apps/jvmdms.a.1/native/adobe/jvmdms.a.1.as91558.1/DocumentStorage/docm1302792793931/d570dc3cf31fc6f368d4ee043ede67d5"/>
<globalBackendId/><senderLocalBackendId/><senderGlobalBackendId/><inline/><senderPullServantJndiName>adobe/idp/DocumentPullServant/adobews__957269328</senderPullServantJndiName>
<attributes/></document> pdfDoc=null sharedData=null} getPdfDoc=null getCounted=null refCount=0}".
    at com.adobe.internal.pdfm.pdfa.PDFAService.isPDFA(PDFAService.java:273)
    at com.adobe.internal.pdfm.pdfa.PDFAService.isPDFA(PDFAService.java:193)
    at com.adobe.livecycle.docconverter.DocConverterServiceImpl.isPDFA(DocConverterServiceImpl.java:461)
... 71 more
Caused by: com.adobe.internal.pdfm.NotAPDFException: PDFM_S00025: Document docConverter.pdf of type application/octet-stream is not a PDF.
    at com.adobe.livecycle.assembler.SharedPDFMDocHandle.open(SharedPDFMDocHandle.java:232)
    at com.adobe.internal.pdfm.PDFMDocHandle.acquire(PDFMDocHandle.java:608)
    at com.adobe.internal.pdfm.PDFMDocHandle.acquirePDF(PDFMDocHandle.java:662)
    at com.adobe.internal.pdfm.pdfa.PDFAService.isPDFA(PDFAService.java:217)
... 73 more

The errors that occur depend on what kind of processes and activities you are running in the LiveCycle server, so they could be very different to the above.  The cause of these errors is the WebSphere version and changes that were made to the JDK used by WebSphere.  With WAS FP35 (and probably FP33 and 37) the default behaviour of IgnoreMalformedInput has been set to false, resulting in this change in behaviour for the String constructor in the JDK, causing these issues in LiveCycle.

Solution

 Contact IBM to ensure you have the latest fixes which should contain the solution to this problem as outlined below.  The latest fixes should reset the default behaviour of IgnoreMalformedInput=true.

As an immediate workaround you can manually set the JVM parameter “-Dcom.ibm.IgnoreMalformedInput=true” in your WebSphere administration console.

Additional information

 Information from IBM related to the changes that cause this behaviour and the relevant workaround and fixes:

String constructor does not operate correctly with UTF-16 in Java SR12 FP1

This is a known issue and we had to make changes because of the behavior difference between IBM and OracleJVMs.  The change was to enable conformity with the Oracle JDK for malformed URLs.  To revert back to the original behavior you must add the following  parameter to the “Generic JVM Args” section in the WAS Admin console (instructions in the link below):

-Dcom.ibm.IgnoreMalformedInput=true

Where to set generic JVM arguments in WebSphere Application Server: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21417365

For more details  on the issue please refer to the APAR- IZ80870  (link below).

IZ80870: REPLACING THE ILLEGAL UTF8 BYTE SEQUENCES WITH \UFFFD: http://www-01.ibm.com/support/docview.wss?uid=swg1IZ80870

Please note, this change went into SR 12 FP1, which is why the same behavior does not occur in the previous SDK releases.  Here is the SDK fix list confirming the above APAR went into SR 12 FP1:

http://www.ibm.com/developerworks/java/jdk/aix/j532/fixes.html#SR12FP1

reference: (182181050/2838202)

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 9.0/10 (1 vote cast)

LiveCycle ES: “DocumentBuilder returned is not DOM3 compliant, using Xerces’s DOM parser” warning

Issue

After installing LiveCycle ES Update 1 SP2 with WebSphere 6.1 as the application server, you may observe the following warnings in the SystemOut.log:

[06.03.09 11:21:40:015 CET] 0000002d SAMLToken W com.adobe.idp.common.errors.Logger$LogConsumer run UserM:GENERIC_WARNING:
[Thread Hashcode: 1753770120] DocumentBuilder returned is not DOM3 compliant, using Xerces's DOM parser

[06.03.09 11:21:45:140 CET] 000000be Reference I org.apache.xml.security.signature.Reference verify
Verification successful for URI "#d143154d8548fe37c847df7490b054cc"

Reason

This warning message is logged if the XML parser available to LiveCycle ES is not DOM 3 compliant. This warning has been reported on some Websphere installations, but seems to have been resolved in the latest fixpacks from IBM.

Solution

Upgrade to the latest WebSphere fixpack.  You can suppress the warning in the log by setting the log level for com.adobe.idp.um.businesslogic.authentication.SAMLToken to “ERROR”.

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 0.0/10 (0 votes cast)

LiveCycle 7: disable “An active transaction should be present…” message from the WebSphere log output

Issue

 If you are running LC7 products in the WebSphere application server you may notice the following message appearing repeatedly in the log file:

[21/09/07 15:40:24:562 IST] 72d82e0c ConnectionMan W J2CA0075W: 
An active transaction should be present while processing method allocateMCWrapper.
[21/09/07 15:40:24:578 IST] 72d82e0c ConnectionMan W J2CA0075W: 
An active transaction should be present while processing method initializeForUOW.

You may want to disable this message, to prevent it appearing in the log.

Reason

This message can be ignored as it is merely an informational message.  This message can appear so much, that it starts to fill up the log and makes it very difficult to see any actual problems in the log.

Solution

To simply suppress the messages from being logged to the SystemOut.log, do the following:

Open the <WEBSPHERE_HOME>\properties\j2c.properties file in a text editor.  Find the following block:

<!-- The cm-properties are in a comment block. Uncomment to use -->
 <!--
 <cm-properties>
     <manageCachedHandles>false</manageCachedHandles>
     <logMissingTranContext>true</logMissingTranContext>
 </cm-properties>
 -->

Remove the comments and change the true to false.  It should look like the following once complete:

<!-- The cm-properties are in a comment block. Uncomment to use -->
 <cm-properties>
 <manageCachedHandles>false</manageCachedHandles>
     <logMissingTranContext>false</logMissingTranContext>
 </cm-properties>

___________________________________
reference: (1-25647507)

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 0.0/10 (0 votes cast)