How to Fix com.adobe.idp.scheduler.SchedulerException: Failure Occurred During Job Recovery

Problem Scenario:

LiveCycle uses Quartz scheduler. Typical use case is when “watched folder” endpoint related schedules get translated as quartz triggers. Quartz takes care of executing the trigger based on frequency and updates a table with the latest execution count.

We ran into a production scenario where customer was running LiveCycle 8213 against Oracle 10g db and one fine day their weblogic servers wouldn’t start.

LiveCycle Scheduler component was failing during startup with a message “Failure occurred during job recovery.”

——–
####<Dec 16, 2011 1:07:31 PM EST> <Error> <com.adobe.idp.scheduler.SchedulerServiceImpl> <auslyncas72> <LCServer1> <[ACTIVE] ExecuteThread: ’0′ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<anonymous>> <BEA1-014B3386796FD9C837B9> <> <1324058851904> <000000> <Scheduler loading caused exception Invocation error.
ALC-DSC-003-000: com.adobe.idp.dsc.DSCInvocationException: Invocation error.    at com.adobe.idp.dsc.component.impl.DefaultPOJOInvokerImpl.invoke(DefaultPOJOInvokerImpl.java:152)
    at com.adobe.idp.dsc.interceptor.impl.InvocationInterceptor.intercept(InvocationInterceptor.java:140)
com.adobe.idp.scheduler.SchedulerException: Failure occured during job recovery.
    at com.adobe.idp.scheduler.SchedulerServiceImpl.createScheduler(SchedulerServiceImpl.java:517)
    at com.adobe.idp.scheduler.SchedulerServiceImpl.startSchedulerServiceInternal(SchedulerServiceImpl.java:776)
    at com.adobe.idp.scheduler.SchedulerServiceImpl.startSchedulerService(SchedulerServiceImpl.java:722)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
——–

Solution:

After investigating further we found that one of the quartz tables “QRTZ_SIMPLE_TRIGGERS” had a record which reached upper limit for a particular column (TIMES_TRIGGERED).

This column is defined as Number(7), so largest value it can accomodate is 9999999.

In order to resolve this we manually reset the column value to “0″, with following query.

update adobe.qrtz_simple_triggers q set times_triggered=0 where trigger_name=”WatchedFolder:181″;

This problem is fixed with LC 8.2.1 SP4 (value is reset programmatically).