Posts tagged "cluster"

LiveCycle ES3 SP1: Gemfire errors in server log after installing patches on top of 10.0.2

Issue

If you are using TCP locators in a LiveCycle cluster with ES3 (10.0.2) and have installed a service pack or patches on top of this version, you may notice some errors in the server log similar to the following:

com.adobe.livecycle.cache.CacheRuntimeException: Error loading cache configuration
at com.adobe.livecycle.cache.adapter.GemfireCacheAdapter.getCacheContainer(GemfireCacheAdapter.java:164)
at com.adobe.livecycle.cache.adapter.GemfireCacheAdapter.createRootRegion(GemfireCacheAdapter.java:853)
at com.adobe.livecycle.cache.adapter.GemfireCacheAdapter.init(GemfireCacheAdapter.java:218)
at com.adobe.livecycle.cache.adapter.GemfireCacheAdapter.<init>(GemfireCacheAdapter.java:207)
at com.adobe.livecycle.cache.adapter.CacheAdapterFactory.getCache(CacheAdapterFactory.java:103)

Caused by: com.gemstone.gemfire.SystemConnectException: Attempt to connect to distributed system timed out

If you have a UDP-based cluster caching mechanism, you are not affected by this issue.

Reason

This issue occurs when the Gemfire version on the server is different from the Gemfire version used by the TCP locators.  Gemfire.jar and GemFireVersion.properties are delivered in two different places in the LC installation – 1. Core EAR 2. /lib/caching directory.

Gemfire version 6.5.1.17 was used for LC ES3 (10.0.2).  To address some known issues a new Gemfire version (6.5.1.35) was used in later versions and patches.  This newer Gemfire version was used for the Core EAR, but unfortunately not included for updating the caching directory.

To verify the issue, compare the Gemfire version in the GFLocator.log file for your TCP Locator with the Gemfire version in the Gemfire logs for your application server at the following location:

  • JBoss: <adobe_temp_dir>/adobejb_<server_name>/caching/Gemfire.log
  • Weblogic: <adobe_temp_dir>/adobewl_<hostname>/caching/Gemfire.log
  • WebSphere: <adobe_temp_dir>/adobews_*/caching/Gemfire.log

You will find that Gemfire on the LiveCycle server has a higher version.

Solution

This issue has been addressed in SP2 and later patch versions.

Note: the patch installer will only update the default caching directory (<lc_install_dir>/lib/caching) on the machine where LC is installed.  If you are running LC on various machines, and/or use custom caching directories, you will need to update the 2 Gemfire files for your TCP locators manually.  All TCP locators need to be restarted before starting LC servers.

Workaround

For existing patches you can use the following workaround to fix the issue on your LiveCycle installation.

  1. Stop the LiveCycle server(s).
  2. Stop the TCP Locator(s).
  3. Go to <LC_INSTALL_DIR>/deploy folder.
  4. Open adobe-core-<appserver_name>.ear.
  5. Go to TCP locator installation directory. Typically the default location is <LC_INSTALL_DIR>/lib/caching.
  6. gemfire.jar and GemFireVersion.properties can be found here.
  7. Replace the two files above in the TCP locator with the same files from the core ear.
  8. Repeat the replacement procedure (steps 5 – 7) for each TCP locator.
  9. Start TCP Locator(s).
  10. Start LiveCycle server(s) in the Cluster.

Verify the Gemfire versions in GFLocator.log and Gemfire.log files. You should see the same Gemfire version in both files.

reference: (183615151/3327651)

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 7.7/10 (3 votes cast)

LiveCycle ES2: Unexpected exception. Failed to create directory: \\<cluster>\GDS

Issue

If you have configured the WebSphere node agents to start as Windows services instead of using the batch files in your cluster you may experience the following behaviour:

  • in AdminUI > Services > Applications and Services > Application Management, you may receive an ALC-DSC-033-000 error
  • in AdminUI > Services > Applications and Services > Service Management, you will see many services marked Inactive, and you cannot stop/start them

You will also notice the following errors in the SystemOut.log on the individual cluster nodes when starting the servers:

E com.adobe.idp.dsc.registry.service.impl.ServiceStoreImpl createServiceConfigurationFromBOI
    COULD NOT PROCESS SERVICE: Process Management (system)/Queue Sharing VERSION: 1.0 MARKING AS INACTIVE
E com.adobe.idp.dsc.registry.service.impl.ServiceStoreImpl createServiceConfigurationFromBOI TRAS0014I:
    The following exception was logged java.lang.ClassNotFoundException: server.startup : 1Class name
    com.adobe.idp.workflow.dsc.descriptor.WorkflowDSCDescriptor from package com.adobe.idp.workflow.dsc.descriptor not found.
....
W com.adobe.idp.dsc.registry.service.impl.ServiceRegistryImpl getHeadActiveConfiguration
    Head active ServiceConfiguration for service: Process Management (system)/Queue Sharing version: 1.0 is not active as expected.
    The service was probably Marked INACTIVE due to an error loading the service.  Please look at the logs to see if this service was Marked INACTIVE.
...

The SystemOut.log will also contain the following error:

E com.adobe.idp.DocumentFileBackend getTimeSkewDelta DOCS001: Unexpected exception. Failed to create directory: \\es2cluster\GDS.
E com.adobe.idp.Document passivate DOCS001: Unexpected exception. While re-passivating a document for inline data resizing..
E com.adobe.idp.Document passivate TRAS0014I: The following exception was logged com.adobe.idp.DocumentError:
Failed to create directory: \\es2cluster\GDS
    at com.adobe.idp.DocumentFileBackend.getTimeSkewDelta(DocumentFileBackend.java:113)
    at com.adobe.idp.DocumentFileBackend.<init>(DocumentFileBackend.java:86)
    at com.adobe.idp.Document.getGlobalBackend(Document.java:3153)
    at com.adobe.idp.Document.writeToGlobalBackend(Document.java:2977)
    at com.adobe.idp.Document.doInputStream(Document.java:1698)
    at com.adobe.idp.Document.passivateInitData(Document.java:1457)
    at com.adobe.idp.Document.passivate(Document.java:1235)
    at com.adobe.idp.Document.passivateGlobally(Document.java:1205)
    at com.adobe.idp.dsc.management.impl.ArchiveStoreImpl.createNewArchive(ArchiveStoreImpl.java:636)
    at com.adobe.idp.dsc.management.impl.ArchiveStoreImpl._getArchive(ArchiveStoreImpl.java:438)

Reason

The Windows services that you have created do not have the correct privileges for accessing the LiveCycle GDS directory which is required for the proper operation of the LiveCycle server.

Solution

Open the Services view from the Control Panel in Windows.
Select the WAS NodeAgent service and goto properties > LogOn > Log on as > This account.
Browse for an Administrator account with sufficient privileges to access the GDS share.
Restart the NodeAgent.
Then restart the cluster from the WAS ND admin console.

Additional Information

To create the nodeagent as a Windows service instead of using the batch files:

1. Change to the \bin directory where WebSphere is installed, for example:

C:\Program Files\IBM\WebSphere\AppServer\bin

2. Run the following command:

wasservice -add ctgNode01_nodeagent
          -servername nodeagent
          -profilePath “C:\Program Files\IBM\WebSphere\AppServer\profiles\ctgAppSrv01″
          -wasHome “C:\Program Files\IBM\WebSphere\AppServer”
          -logFile “C:\Program Files\IBM\WebSphere\AppServer\profiles\ctgAppSrv01\logs\nodeagent\startNode.log”
          -logRoot “C:\Program Files\IBM\WebSphere\AppServer\ctgAppSrv01\logs\nodeagent”
          -restart true
          -startType automatic

You can do this for each cluster node.  Now when you restart Windows, the nodeagent on each cluster node will start automatically.

reference: (1911394)

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 9.0/10 (1 vote cast)