How to Prevent Repetitive “Failed job” Error Messages from the Quartz Scheduler

If you are using the SOAP port for any of your LiveCycle ES2 services, you may encounter an issue that causes repetitive “Failed job” error messages, originating from the Quartz scheduler.

The message looks like below:
12:30:03,375  INFO DSCJobStoreTX:3271 – ClusterManager: Scanning for instance “XXXXX”‘s failed in-progress jobs.
12:30:10,891  INFO DSCJobStoreTX:3420 – ClusterManager: detected 1 failed or restarted instances.

These error messages continue to appear even after the node servicing the request has been shut down and another node has completed the pending job.

Consolidating Information:
The reason to create this blog is for consolidating resolution steps for all Application Servers in one place.
JBoss: The steps to resolve this on JBoss are documented in the 8.2 Clustering JBoss Guide : Under Troubleshooting Section – Page#92 & 93 – Under “SOAP port test causes Quartz scheduler to create infinite loop of “Failed job” messages in log file”. But similar steps do not exist for Websphere and Weblogic in 8.2 Documentation.
Websphere: The steps to resolve this on WebSphere are documented in the 9.0 Troubleshooting Guide: Page#30 – Under “To prevent repetitive “Failed job” error messages originating from the Quartz scheduler”. But similar steps do not exist for JBoss and Weblogic in the ES2/ES2.5 Documentation.
Weblogic: Steps to resolve this on Weblogic do not exist in any documentation.

Resolution Steps:

The below resolution steps apply to all versions of LiveCycle (8.2/ES2/ES2.5)
To avoid this issue, locate the log4j.xml file on each JBoss
server in the cluster, and add the following two code snippets under Limit Categories:
1. <category name=”org.quartz.impl.jdbcjobstore”>
<priority value=”ERROR”/>
2. <category name =“com.adobe.idp.scheduler.jobstore.DSCJobStoreTX”>
<priority value=”ERROR”/>

To avoid this issue, use the admin console to change the log configuration for all nodes in the WebSphere
cluster. Set the log level for the following packages to severe:
● org.quartz.impl.jdbcjobstore
● com.adobe.idp.scheduler.jobstore.DSCJobStoreTX

To avoid this issue, you have to first construct a Log Filter and then apply that to the different logging on Weblogic.

To construct a log filter expression:
1.    If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit
2.    In the left pane of the Console, under Domain Structure, select the domain name.
3.    Select Configuration > Log Filters.
4.    Under Log Filters, click the name of the log filter for which you want to define a log filter expression.
5.    On the Settings for [Log Filter Name], click Edit.
6.    On the Settings for [Log Filter Name] > Configuration page, type or paste the following directly into the Log     Filter Expressions field:
7.    Click Finish. click Save.
8.    In the left pane of the Console, expand Environment and select Servers.
9.    Under Servers, click the name of the server instance whose logging you want to configure.
10.    On the Logging: General page, click Advanced.
11.    Under Message Destinations:
a.    In the Filter list box, select the filter just configured in above steps.
The Filter should be selected for categories –
– Log File and
– Std Out
12.    Click Save.
13.    To activate these changes, in the Change Center of the Administration Console, click Activate Changes.

About Ameeth Palla

Ameeth Palla is a Technical Account Manager for the Adobe Digital Enterprise Platform team. Previously he was in the role of a Technical Expert for LiveCycle for the the Technical Response Team (what is now the Sr.Support Architect role). In both roles, Ameeth handled several technical issues for various customers and worked closely to aid with Sales POC's, Development/Implementation and Manitenance of LiveCycle/ADEP Projects. In the current role he handles several high profile customer accounts and provides guidance in all aspects of LiveCycle/ADEP. Also he is a Certified LiveCycle ES2.5 Process Management Expert. He was nominated for the 'Adobe Founders Award' and the 'Excellence Matters' award for LiveCycle/ADEP BU.
This entry was posted in Adobe LiveCycle ES2 (9.0.x), Adobe LiveCycle ES, General Interest and tagged , , , , . Bookmark the permalink.

Comments are closed.