Determining if the LiveCycle Scheduler has Started in Cluster Mode

When running Adobe LiveCycle in a cluster it is important that the LiveCycle scheduler service is running in cluster mode.  Internally the scheduler service uses Quartz and if it does not run in cluster mode, then Quartz instances on each server in the cluster will run independently of each other thinking they are alone.  This can result in serious issues including deadlocks in the Quartz tables (the tables in the database with names like qrtz_*).

Under normal circumstances the scheduler will automatically realize it is in a cluster and start Quartz in cluster mode.  In the last while I’ve been involved in a few issues where something went wrong and Quartz was not started in cluster mode, so I thought it would be a good idea to write down how to determine whether or not the scheduler was starting in cluster mode, as it may not be apparent if you don’t know what you are looking for.

The key to making this determination is to look in the application server log files during start up at the section of logs starting with the following line:

“[com.adobe.idp.scheduler.SchedulerServiceImpl] IDPSchedulerService onLoad”

This is the indication that the scheduler service is starting up and the lines that follow it will tell you whether or not it is starting in clustered mode properly.  It is important to locate this first line in the logs because some application servers (and other third party applications) use Quartz as well, and their Quartz instances should not be confused with the instance being used by the scheduler service.  The following is an example of the output from a LiveCycle server I have running here:

INFO  [com.adobe.idp.scheduler.SchedulerServiceImpl] IDPSchedulerService onLoad
INFO  [com.adobe.idp.scheduler.SchedulerServiceImpl] Start scheduler service
INFO  [com.adobe.idp.scheduler.SchedulerServiceImpl] Context classloader is org.jboss.util.loading.DelegatingClassLoader
INFO  [com.adobe.idp.scheduler.SchedulerServiceImpl] Initializing SCHEDULER FACTORY from properties …
INFO  [com.adobe.idp.scheduler.SchedulerServiceImpl] SCHEDULER FACTORY initialized.
INFO  [com.adobe.idp.scheduler.SchedulerServiceImpl] Creating new scheduler. isInUpgrade:false
INFO  [com.adobe.idp.scheduler.DSCSchedulerFactory] Using custom data access locking (synchronization): org.quartz.impl.jdbcjobstore.UpdateLockRowSemaphore
INFO  [org.quartz.core.QuartzScheduler] Quartz Scheduler v.1.6.0 created.
INFO  [com.adobe.idp.scheduler.jobstore.DSCJobStoreTX] JobStoreTX initialized.
INFO  [com.adobe.idp.scheduler.DSCSchedulerFactory] Quartz scheduler ‘IDPSchedulerService’ initialized from an externally provided properties instance.
INFO  [com.adobe.idp.scheduler.DSCSchedulerFactory] Quartz scheduler version: 1.6.0
INFO  [com.adobe.idp.scheduler.SchedulerServiceImpl] Scheduler loaded with name IDPSchedulerService
INFO  [org.quartz.core.QuartzScheduler] Scheduler IDPSchedulerService_$_bl1297894319250 started.

Notice the last line with the name of the Quartz instance listed: “IDPSchedulerService_$_bl1297894319250”.  In this case these are taken from the logs of a server in a cluster.  The name of the scheduler’s Quartz instance will always begin with the string “IDPSchedulerService_$_”, what is appended to the end of that tells you whether or not Quartz is running in clustered mode.  In this case I know it is running in clustered mode because the name has been appended with a complex unique identifier.  If it was not running in clustered mode the name would have been appended with a simple number, (ie: 20).

So, to summarize; in cluster mode the name of the Quartz instance started by the scheduler will appear in a log line looking like this:

INFO  [org.quartz.core.QuartzScheduler] Scheduler IDPSchedulerService_$_bl1297894319250 started.

In a non-cluster, the log line would like more like this:

INFO  [org.quartz.core.QuartzScheduler] Scheduler IDPSchedulerService_$_20 started.

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 6.3/10 (3 votes cast)
Determining if the LiveCycle Scheduler has Started in Cluster Mode, 6.3 out of 10 based on 3 ratings

About Chris Trubiani

Computer Scientist at Adobe Systems Canada
This entry was posted in Adobe LiveCycle ES2 (9.0.x) and tagged , . Bookmark the permalink.

7 Responses to Determining if the LiveCycle Scheduler has Started in Cluster Mode

  1. Pingback: Parth on Livecycle » Determining if the LiveCycle Scheduler has Started in Cluster Mode

  2. JB says:

    If the LiveCycle Scheduler is starting in stand alone mode in a clustered environment, how do I get it to run in clustered mode?

    • Chris Trubiani says:

      Well, the short answer is you could force it to start in cluster mode by modifying a properties file at the root of the scheduler DSC jar file and redeploying the service. But the longer, and real, answer is that if it is not starting in clustered mode within a cluster then something is going wrong. It would be better to investigate and determine what the real cause of the problem is. There have been some problems fixed in this area recently, so it may just be that you need to install a patch to fix a problem. I’d recommend contacting enterprise support and they should be able to help you quickly.

  3. Ameeth Palla says:

    Note – If TCP Locators are used for Cluster Caching then the Scheduler does not run in a cluster mode. This is a bug and has been fixed via QF’s for LC versions,,, Please contact Adobe Support to obtain the respective QF’s.

  4. Tom Migot says:

    Note – When using Websphere application server (minimum requirement for LCES 8.2.1) or, there is no trace of org.quartz.core.QuartzScheduler in the logs. Applying the Websphere FixPack 19 solve this.
    The method described above seems to be the only mean to determine the good functioning of the Scheduler in cluster mode. Therefore i would highly recommend to upgrade the WAS to at least

  5. Pingback: JVM arguments required by LiveCycle Cluster are not getting picked up by JBoss | LiveCycle Product Blog

  6. Julien FERRE says:

    Another way of checking the current running instance of the scheduler is to go into the database and check the content of the table QRTZ_SCHEDULER_STATE. If it is empty then you run in standalone mode otherwise there is one row per instance of the clustered Scheduler.