Avoiding LiveCycle Kerberos based SSO problems for Active Directory users with large group memberships

Scenario:

Many LiveCycle customers implement kerberos based SSO in their deployments.

We recently came across an interesting customer situation where SSO did not work for some domain users.
To recap working scenario – If LiveCycle SSO is configured correctly, when domain user opens workspace URL from  domain machine, workspace should directly open without asking for user credentials again.

In this case, it wasn’t working for some domain users; but worked for others.

Investigating further we found that non-working users were part of 100+ AD groups resulting in a large kerberos token size.

Customer was using JBoss 4.2.1 with LiveCycle ES2.5.

HTTP traces for working versus non-working scenario revealed that authenticate header was getting truncated in case of non-working scenario. Example below:

Working Http header
header=WWW-Authenticate=Negotiate
oYGqMIGnoAMKAQChCwYJKoZIhvcSAQIComoEaGBmBgkqhkiG9xIBAgICAG9XMFWgAwIBBaEDAgEPokkwR6ADAgEXokAEPtWpZjlA1z6oofQdcYB583
MLxlmoHwCFOtQ3VaGkG38A6zyKf9JogE/KrLPTScZnVCjbEMIeh7cCgUaJTNjqoycEJWAjBgkqhkiG9xIBAgIBAREA/////+sKU3YfjaMD +LWYtwumSTY=

Non Working Http header
header=WWW-Authenticate=Negotiate

Analysis and Solution:

LiveCycle does not have any hard limit on kerberos token size per se. Also Livecycle does not use group information associated with kerberos token for authentication.

However, we found that application servers (in this case JBoss) typically set http header limit due to which packet containing large tokens was getting dropped.

By default Jboss sets maxHttpHeaderSize to 8192 (8KB).

There are couple of options to solve this.   In this case customer chose option 1) below; however I wanted to lay out both options for completeness.

Option 1)

Change Jboss config file to increase http header limit.
Modify maxHttpHeaderSize to say =”16384″ in location below and restart Jboss.

${jboss.home}/server/${jboss.server}/deploy/jboss-web.deployer/server.xml
—
<Connector port=”8080″ address=”${jboss.bind.address}”
maxThreads=”250″ maxHttpHeaderSize=”16384″
—

Option 2)

Microsoft AD allows a flag to be set on user object in AD, so that extra PAC information (containing  groups) won’t be sent across by AD. One could set this flag for “service” user id in AD which is used for LiveCycle SSO.

This flag is called “NO_AUTH_REQUIRED” and was introduced in a hot fix on Windows 2003 AD.

Here is the link to download the hot fix, and also it explains where the flag should be set.
http://support.microsoft.com/kb/832572