Archive for July, 2012

LifeCycle of LiveCycle Domain Synchronization

A LiveCycle Enterprise Domain is synchronized with users and groups from an external entity.
The external entity can be either of the following,

  • Custom SPI – Custom Service Provider Interface allows one to connect to a external system other than LDAP for fetching users to LiveCycle.
    One can create a Custom SPI by implementing DirectoryUserProvider and DirectoryGroupProvider Interfaces.
    The following document details step by step procedure for creating a Directory Provider,
  • LDAP – Integrating an LDAP with an Enterprise Domain is very clearly detailed in this blog,
    As of LiveCycle ES3, following LDAP servers are supported,

    • Sun ONE 5.2
    • Sun ONE 6.3
    • Microsoft Active Directory 2003
    • Microsoft Active Directory 2008
    • IBM Tivoli Directory Server 6.0
    • Novell eDirectory 8.7.3
    • Lotus Domino v8.x
    • ADAM 1.1.3790.2075
    • OpenLDAP 2.3.43-12.el5_5.3.i386

I’ll talk about the following related to Domain Synchronization,

Working of Domain Synchronization with an LDAP Server

An Enterprise Domain registered with an LDAP server works in 3 steps during a sync,

  • Syncrhonize Users from LDAP
  • Syncrhonize Groups from LDAP
  • Syncrhonize Group Members from LDAP

Let’s take an example where an Enterprise Domain is registered with a Microsoft Active Directory with following specifications,

  • UniqueId for both Users and Groups is set to ObjectGUID.
  • Batch size is by default set to 200, which means that the Users, Groups, Group Members will be fetched from an LDAP in a batch of 200.
  • Let’s consider some test Users from the Active Directory LDAP,
    User 1
    dn: CN=foo,CN=users,DC=example,DC=com
    sAMAccountName: foo
    memberOf: CN=baz,CN=Users,DC=example,DC=com
    objectGUID: 1
    User 2
    dn: CN=bar,CN=users,DC=example,DC=com
    sAMAccountName: bar
    memberOf: CN=baz,CN=Users,DC=example,DC=com
    objectGUID: 2

In an Active Directory LDAP server, the ObjectGUID field is a binary field.
For simplicity sake I’m using it as numeric field.

The user synchronization phase in this example, involves the following steps:

  • Fetch 200 users (configurable batch size via editing config.xml) from LDAP.
  • Determine the value for the unique identifier for each user.
  • Look for a record in LiveCycle user table where the canonicalName matches with the user’s unique identifier.
    • Case A – If the record exists then update the user properties.
    • Case B – If the record does not exist then create a new user record.
    • Case C – If the record exists in LiveCycle Db but marked Obsolete, then mangle the userID of the previous record and continue creating the new record.
  • Once all the users have been fetched, the users present in LiveCycle Db but not modified in the current synch cycle are marked OBSOLETE.

A similar logic is used in the group synchronization phase.
Group membership phase comes into picture only if both the User and Groups synchronization for a Domain are enabled.
Group Membership links the synched User and Group on behalf of the Membership at the LDAP end between both the principals.

UniqueId used in LDAP Synchronization

In the synchronization process, the unique identifier attribute for users and groups plays a very important role.
It serves as the key attribute for helping migrate Principals registered with one LDAP to another.
This LDAP attribute used as UniqueId should fulfill the following requirements in the LiveCycle database,

  • Unique – The identifier should be unique across the whole user/group repository.
  • Immutable – It does not change for a given user/group.
  • Not recycled – The identifier once assigned to a user/group is never reused.

The following LDAP attributes are not a good candidate for a unique identifier:

  • distinguishedName
    For example, consider that a user’s distinguished name (dn) CN=foo, ou=finance, DC=bar,DC=com is used as the unique identifier.
    In this case, if foo moves to different department, then the dn will change.
  • loginId
    A user’s login ID (samAccountName in Active Directory and uid in SunOne) are sometime recycled when an old user leaves and the new user with the same name arrives.
    In this case, the new user is given the same userId.
  • email
    Email is fairly a good candidate. However, it might cause some issues when the emails are recycled or modified.

Therefore, the synchronization logic by default, uses the following LDAP attributes for different LDAP server,

  • objectGUID – for Microsoft Active Directory
  • objectGUID – for ADAM
  • nsuniqueId – for SunOne
  • guid – Novell eDirectory
  • dominoUNID – Lotus Domino
  • ibm-entryuuid – IBM Tivoli

However, these attributes can be replaced with any other attribute of LDAP which fulfills the above mentioned requirements.

User Indentity in LiveCycle

The user identity of a user in LiveCycle is governed by the following rules.
In a LiveCycle domain,

  • A user’s loginid (edcprincipaluserentity.uidstring) is unique but with respect to it’s own domain, i.e. same userId can co-exist in another domain.
  • A user’s canonicalName (edcprincipalentity.canonicalname) is unique  but with respect to it’s own domain, i.e. same canonicalName can co-exist in another domain.
  • Each user and group is assigned an oid (, which is used to refer the user by other systems in LiveCycle.
    It’s unique across all domains.
    Therefore, a process refers to a user using its Oid.
  • A user can be uniquely referred using:
    • User’s LiveCycle DomainName and LoginId
    • User’s LiveCycle DomainName and CanonicalName
    • User’s Oid

Unique Identifier Migration

There are times when an Enterprise needs to migrate it’s users to another LDAP, or let say an Enterprise may have multiple LDAP servers but now wants to consolidate the principals in a single domain.
In broader sense the migration works as follows,

  • Modify the unique identifiers from old one to new ones according to the new LDAP server.
  • The next Synchronization detects that the unique identifier has changed. Therefore, the Synchronization logic is modified.
  • Fetch 200 users from LDAP.
  • For each user, determine the value for the old unique identifier.
  • Look for a record in LiveCycle user table, where the record’s canonicalName matches with the old unique identifier.
    • Case A – If the record exists, update the user properties and the canonicalName.
    • Case B – If the record does not exist, create a new user record with the new canonicalName.
  • Once all the users are fetched, find all the users who have not been modified in the current cycle, and mark them OBSOLETE.
  • In the end, the canonicalName (unique identifier) for all users is changed to the new Unique Id.
  • Ensure that during this Synchronization process, the value for old unique identifier is not changed at the Old LDAP’s end.

Scenario 1

An Enterprise has configured an LDAP DirectoryProvider in LiveCycle, which uses email as the unique identifier.
Due to various reasons mentioned above, an Enterprise may want to move to objectGUID, whereas other details, such as the LDAP server, domain, and so on, remain unchanged.
In this case, after the unique identifier has been changed, the canonical name for all active Users will be migrated to the new one.

Scenario 2

An Enterprise has configured a SunOne LDAP DirectoryProvider, which uses nsuniqueId as the unique identifier.
As a part of an IT exercise all the users at the LDAP end have migrated from SunOne to Active Directory.
This means that all the LiveCycle Users that are a part of the Enterprise Domain registered with Sunone will have to migrate to Active Directory.
Strict care has to be taken to preserve the User’s Identity in such a way that the said User’s work doesn’t get affected.

  • LDAP1 – Old LDAP server, SunOne
  • LDAP2 – New LDAP server, Active Directory

The user accounts will be migrated from LDAP1 to LDAP2.
While doing this, a user account will be active only in one of the LDAP servers.
After the migration, it will be marked inactive in the other one.
As UniqueId will be different between LDAP1 and LDAP2, one needs to change the unique identifier in a way that the user identity is maintained.
i.e. the user identity should be federated between both the LDAPs based on a particular attribute which can also serve as the uniqueId during migration.

The following steps need to be performed to conclude a successful migration,

  • In the Directory Provider configured with LDAP1, change the unique identifier from nsuniqueId to uid.
    It assumes that the value for uid(from SunOne) and samAccountName(from Active Directory) remains same between LDAP1 and LDAP2.
  • Run the Synchronization. It will change the unique identifier from nsuniqueId to uid, and thereby update the canonicalName for the Users in LiveCycle database.
  • During the migration, LDAP1 will have users disabled while the same users will be in active in LDAP2.
    Now, one needs to configure LDAP2 as the new Directory Provider in the same LiveCycle domain with unique identifier set to samAcccountName.
    For example, the user Bob is disabled in LDAP1 and enabled in LDAP2.
    In this case, when the Synchronization runs,

    • It fetches the user Bob from LDAP1, which is inactive, and therefore will be disabled.
    • It fetches the users Bob from LDAP2, which is active. Its unique identifier will be used to look up the user Bob in LiveCycle database.The record will be found (from the Synchronization by LDAP1) and updated.
  • After migration is complete, the new Directory Provider can switch to the commonly used uniuqeId attribute and synchronize again, i.e. objectGuid in this case.

Bookmark and Share

Capturing LDAP Traffic between LiveCycle server and LDAP server

In an Enterprise Domain, principals(users/groups) are synched to LiveCycle from an LDAP server via Directory Providers.
In an Enterprise or Hybrid Domain, the LiveCycle users might be authenticated using an LDAP server.
In both the above cases, it becomes necessary to sniff the traffic if one of the principal hasn’t reached LiveCycle or the sync has broken in between for some unexplained reason.
Or let say authenticating a user through an LDAP server is failing.
The server logs of an Application server detail only the issue occurring at it’s own end,
but can’t trace the issue occurred during the transfer of data or the issue occurred at LDAP server end.

In such cases capturing the log details right from the start of fetching principal to transporting it to LiveCycle server, gives a clear insight into the working of LDAP protocol and the issue occurred.

There are many ways of capturing LDAP traffic

  • Wireshark is a great tool for capturing the LDAP traffic between an LDAP server and LiveCycle server.
  • Snoop is a utility used for capturing traffic on a Solaris machine.
  • TCPDump can be used to capture the LDAP traffic on a Linux machine.
  • LDAPDecoder can be used to capture the traffic by acting as a proxy between the LiveCycle server and the LDAP server.

I prefer using LDAPDecoder because of the following reasons,

      • It’s not always possible to install a Wireshark on customer’s production system.
        Also using a Wireshark needs root access else the LDAP traffic can’t be captured.
        Wireshark doesn’t work on headless unix/linux servers.
      • Snoop and TCPDump help in capturing traffic on headless unix/linux servers but they capture all of the traffic while one may be interested in only the LDAP traffic.
      • LDAPDecoder works well on both UI and non-UI systems.
        On Non-UI systems the LDAPDecoder can work independently as well as in tandem with Snoop and TCPDump.
        The data captured from Snoop and TCPDump can be given the LDAPDecoder in order to interpret LDAP information in specific.
      • LDAPDecoder can be used to capture LDAP traffic between a remote LiveCycle server and remote LDAP server.

LDAPDecoder acts as a proxy between both the LDAP Server and LiveCycle server.
So, the client forwards the request to LDAPDecoder which decodes the request and then forwards it to the server.
Once it gets a response from the LDAP server it decodes it and passes on the LiveCycle server thereby completing the whole flow of LDAP communication.

Okay, enough of theory, let’s get to some practical now,
1. Extract the LDAPDecoder.jar from LDAPDecoder.
2. Start the LDAPDecoder as follows on command line,
java -jar LDAPDecoder.jar -h ldapserver -p 389 -L 390 -f output.log

-h ldapserver -> the host name or ip address of the LDAP server
-p 389 -> the port of the LDAP server
-L 390 -> the port of the LDAPDecoder server which the LiveCycle server should send requests to.
-f output.log -> the file in which the LDAP traffic will be captured, i.e. the request from the LiveCycle server and response from the LDAP server.

Additionally one can specify -s parameter to make communication between LDAP Server and LiveCycle server in SSL mode.
For more such options just type, java -jar LDAPDecoder.jar

3. Open the Enterprise or Hybrid domain in Edit mode.
4. In the server field, mention the hostname or the IPaddress of the machine on which the LDAPDecoder server is running.
5. In the port field, mention the port to which the LDAPDecoder server is listening, i.e. the port specified with -L parameter, e.g. 390.

6. Save the domain.
7. Sync the Domain in case of Enterprise or make authentication calls in case of either Enterprise or Hybrid domain.
8. Check the details of LDAP request response flowing between LiveCycle server and LDAP server in the log file, i.e. the file specified with the -f parameter, e.g. output.log

For more information on LDAPDecoder and how to interpret Snoop and TCMPDump data, refer the following doc,

Bookmark and Share

Using JMX-Console for configuring and debugging LiveCycle applications

 JMX-Console ??

JMX(Java Managed Extensions) is a Java Technology which provides a way to manage running applications via various utilities and tools.
The running services are registered as mbeans and can be accessed/controlled remotely via JMX-Console.

LiveCycle currently supports 3 App Servers namely, JBoss, Weblogic and Webspehere.
While Weblogic and Websphere don’t provide a UI to connect to JMX managed bean instances,
Jboss provides a management console called JMX-Console to do the same.
In order to manage beans for Weblogic and Websphere,
Java provides a generic Utility called JConsole which can be connected to any running Application Server and relevant tasks can be performed.

Why JMX-Console

  • There are times when server gets stuck at some error, e.g. Server goes into hang state. In such cases, needs arises to get an insight into the running system so that the issue can be narrowed down.
  • At times one would like to get some information about the registered LiveCycle service, e.g. some metadata, some server information, some kernel information.
  • Sometimes one needs to change some settings in the server at run time.
  • One may need to redeploy a web application.
  • One might need to change the logging level for a particular service or decrease the verbosity of the logs.
  • Most of these changes mandate restarting of the server which can be very tardy with a LiveCycle bundled with too many components.
  • There are times when the host machine is not accessible directly to debug the issue.

Some benefactions of JMX-Console for debugging LiveCycle issues

I’ll be discussing about the following topics that can be helpful while debugging a LiveCycle application/service.
a. Login into JMX-Console
b. Redeploying a LiveCycle war
c. Configuring the log levels for a specific package
d. Generating Thread Dump
e. Stopping LiveCycle Server Instance
f. Starting/Stopping the CRX server
g. Command line JMX management

NOTE: the changes made on JMX-Console remain active till the server is running and vanish once the server is restarted.

Login into JMX-Console

Redeploying a LiveCycle web service

Whenever one needs to change some settings in a war, Jboss doesn’t allow to do so until the server is stopped.
Of course, one can always copy the war/ear outside and edit and then hotswap with the existing running war.
But the following seems like a safe and graceful way to modify a war without shutting down the whole running LiveCycle server.

  • Navigate to http://LCServer:LCPort/jmx-console
  • Under the section “Object Name Filter”, click on the link named jboss.web.deployment
  • To the right hand side under the section jboss.web.deployment, there will be a listing of context-roots to which various wars are associated.
  • Click on the context-root which that is associated with the concerned war.
  • This will open the JMX Mbean view of the associated war.
  • The mbean names are of format “war=/context_root_associated_with_war”.
  • Now one can simply start/stop/redeploy the war by clicking on the related “Invoke” button.

Configuring log levels for package(s)

The log levels for a package are changed in Jboss via log4jService located at, \\Jboss\server\server_instance\conf\jboss-log4j.xml.
Or by changing the located in a particular war.
JMX-Console provides a remote way of doing the same.

  • Navigate to http://LCServer:LCPort/jmx-console
  • Under the section “Object Name Filter”, click on the link named jboss.system
  • Then click on link service=Logging,type=Log4jService under the section “jboss.system” on the right hand side.
  • There are various settings one can alter for logging as per need.
    • Custom Logging File – As pointed out before, the log4J settings are controlled via the file named jboss-log4j.xml.
      One can even change the settings of the server to point to a custom made logging.xml file.
      Replace the value of the attribute named “ConfigurationURL” from “resource:jboss-log4j.xml” to the custom logging.xml file.
      The custom file however needs to be placed at the same location alongside the jboss-log4j.xml.
    • Get specific Log Level details – Have a look at the jboss-log4j.xml.
      It consists of tags named “categories” with different priority values.
      In Log4j world, the category is called “Logger” and the priority value is called the “Log Level”.
      If one wants to know the log level for a particular category specified in the jboss-log4j.xml,
      then enter the name of the Logger for the Operation named “getLoggerLevel” and click on the Invoke button.
    • Set/Edit new Logger and Log level – One can create new Categories/Logger on the fly and associate a particular package with a Log Level.
      Enter the name of the Logger and the Log Level for the Operation named “setLoggerLevel” and click Invoke.
      Till the server instance is up and running, the log level for the particular package will remain active.
    • Set/Edit multiple Loggers and Log level – One can also set multiple Loggers with a particular Log Level under the Operation named “setLoggerLevels”.
    • Reconfigure Logging – If the logging is not getting reflected for some unknown reason,
      then click on Invoke button for the Operation named “reconfigure”.
      This is analogous to editing of a jboss-log4j.xml where the saving the changes the logging is reconfigured.
    • Edit root threshold Log level – When no Categories are specified then the Log level of Root takes control.
      This is specified under as the value for the attribute named “DefaultJBossServerLogThreshold”.
      Change the value to a required level and click on “Apply Changes”.
      This will activate the Log Level for all those packages which don’t have a category explicitly specified.

Generating Thread Dump

A Java thread dump is a way of finding out what every JVM thread is doing at a particular point in time.
This is very helpful in cases where the server has become unresponsive or went into an unexplained hang state.
Thread Dump allows one to get an insight into the failing JVM process.
As an analysis of the dump, it can be known where exactly the threads are stuck.

  • In order to get the Thread Dump via JMX-Console,Navigate to http://LCServer:LCPort/jmx-console
  • Under the section “Object Name Filter”, click on the link named jboss.system
  • Then click on link type=ServerInfo under the section “jboss.system” on the right hand side.
  • Click on Invoke button of the “listThreadDump” operation to generate the ThreadDump.
  • The Thread Dump can then be saved to the file system for analysis.

Stopping LiveCycle Server Instance

The JBoss server instance can be stopped online via JMX-Console.
Especially useful when one doesn’t have direct access to the host machine.
i. Navigate to http://LCServer:LCPort/jmx-console
ii. Under the section “Object Name Filter”, click on the link named jboss.system
iii. Then click on the link type=Server under the section jboss.system on the right hand side.
iv. Click on Invoke button of the “shutdown” operation to shutdown the server instance.

Starting/Stopping the CRX server

Have a look at second section of the blog entry,

While JMX-Console provides UI way of debugging and changing the settings,
the same can be achieved through command line by another utility named twiddle that comes bundled with Jboss and is located in the bin folder of the same.
It can perform every task that a JMX-Console can do through UI.
I found a nice article depicting the usage of Twiddle with examples. Have a look at,

Note: The JMXConosle.war is not shipped with the turnkey installations previous to LC ES3.
In such cases, it needs to be downloaded and deployed to the deploy folder of the related Server_Instance.

Bookmark and Share