If you are using the SOAP port for any of your LiveCycle ES2 services, you may encounter an issue that causes repetitive “Failed job” error messages, originating from the Quartz scheduler.
The message looks like below:
12:30:03,375 INFO DSCJobStoreTX:3271 – ClusterManager: Scanning for instance “XXXXX”‘s failed in-progress jobs.
12:30:10,891 INFO DSCJobStoreTX:3420 – ClusterManager: detected 1 failed or restarted instances.
These error messages continue to appear even after the node servicing the request has been shut down and another node has completed the pending job.
The reason to create this blog is for consolidating resolution steps for all Application Servers in one place.
JBoss: The steps to resolve this on JBoss are documented in the 8.2 Clustering JBoss Guide : Under Troubleshooting Section – Page#92 & 93 – Under “SOAP port test causes Quartz scheduler to create infinite loop of “Failed job” messages in log file”. But similar steps do not exist for Websphere and Weblogic in 8.2 Documentation.
Websphere: The steps to resolve this on WebSphere are documented in the 9.0 Troubleshooting Guide: Page#30 – Under “To prevent repetitive “Failed job” error messages originating from the Quartz scheduler”. But similar steps do not exist for JBoss and Weblogic in the ES2/ES2.5 Documentation.
Weblogic: Steps to resolve this on Weblogic do not exist in any documentation.
The below resolution steps apply to all versions of LiveCycle (8.2/ES2/ES2.5)
To avoid this issue, locate the log4j.xml file on each JBoss
server in the cluster, and add the following two code snippets under Limit Categories:
1. <category name=”org.quartz.impl.jdbcjobstore”>
2. <category name =“com.adobe.idp.scheduler.jobstore.DSCJobStoreTX”>
To avoid this issue, use the admin console to change the log configuration for all nodes in the WebSphere
cluster. Set the log level for the following packages to severe:
To avoid this issue, you have to first construct a Log Filter and then apply that to the different logging on Weblogic.
To construct a log filter expression:
1. If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit
2. In the left pane of the Console, under Domain Structure, select the domain name.
3. Select Configuration > Log Filters.
4. Under Log Filters, click the name of the log filter for which you want to define a log filter expression.
5. On the Settings for [Log Filter Name], click Edit.
6. On the Settings for [Log Filter Name] > Configuration page, type or paste the following directly into the Log Filter Expressions field:
(MESSAGE LIKE ‘%DSCJobStoreTX:%’)
7. Click Finish. click Save.
8. In the left pane of the Console, expand Environment and select Servers.
9. Under Servers, click the name of the server instance whose logging you want to configure.
10. On the Logging: General page, click Advanced.
11. Under Message Destinations:
a. In the Filter list box, select the filter just configured in above steps.
The Filter should be selected for categories -
- Log File and
- Std Out
12. Click Save.
13. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.