October 2008 Archives

The default settings for LiveCycle are not optimal for the processing of large documents. Large documents can be defined as anything that is larger than 50 MB.

Java Heap Size
You might need to set heap size bigger than 1.2 GB If that is the case, you cannot use 32-bit JDKs on Windows or Linux. You need to move to a complete 64-bit environment with a 64-bit OS, a 64-bit JDK along with a 64-bit appserver.

A common mis-conception is that LiveCycle will be able to use all of the RAM on a server. But LiveCycle only has access to what the JVM has access to. And that is limited by its configuration which is usually 256 MB unless changed. The JVM argument that controls the minimum memory is -Xms and the same for the maximum is -Xmx. If -Xmx is set to 1 GB, LiveCycle will only get to use 1 GB even if your server has 8 GB of RAM. The JBoss and WebLogic Turnkey Installations of LiveCytcle set this to 1 GB.

Appserver Transaction Timeout
Most application servers have a transaction timeout that is configurable. You need to change this to a higher value. For example, we recently had to set this to 30 minutes for a 500 MB Microsoft Word document that was being converted to PDF by PDF Generator ES.

For JBoss, this configuration is in the %JBOSS_HOME%\server\all\conf\jboss-service.xml Using a reliable text editor, (avoid Notepad), increase the value of the TransactionManager's TransactionTimeout attribute from 300 seconds to a higher value. Re-start the appserver instance.

For WebSphere, using the WebSphere Admin Console, navigate to the appserver instance hosting LiveCycle. In 'Container Services'->'Transaction Service', set the values for 'Total transaction lifetime timeout' and the 'Maximum transaction timeout', and re-start the appserver instance.

ORB Service Request Timeout
This needs to be configured to a high enough value. The LiveCycle installation and administration guides provide details on how to do this for your J2EE appserver.

SOAP Request Timeout
If using SOAP, this needs to be configured to a high enough value.

LiveCycle Orchestration Timeout
This needs to be configured to a high enough value. Using Workbench, navigate to the active version of the orchestration, right-click the mouse, and choose 'Properties' to get the appropriate dialog.

LiveCycle Service Timeout
Many LiveCycle services have timeout settings of their own. For example, for PDFGenerator, the PDFGConfigService lets you configure this. Navigate to Services->Applications and Services->Service Management. Locate PDFGConfigService and click it. Change the value of 'Server Conversion Timeout' from the default 270 seconds. Save and then re-start the appserver instance.

For Output ES, navigate to Services->Applications and Services->Service Management. Filter the list by choosing the category 'Output' from the dropdown list. Click on 'OutputService: 1.1'. In the 'Configuration' tab, change 'Transaction timeout' from the default 180 seconds to a higher number as required. Save and then re-start the service.

Default Document Max Inline Size
Set this value to a number higher than your largest document. More details are available here. Please note that setting this to 50 MB or higher will blow your Java heap and crash the appserver if you don't set the JVM heap size correspondingly higher.

Disk Space
Large documents require additional diskspace for temporary File I/O. The filesystem that hosts the 'LiveCycle Temporary Folder' requires several GB of disk space if processing a large number of BIG documents. If some of those transactions timeout or throw an exception, this filesystem runs the risk of running out of diskspace because of orphaned files. Periodic cleanup might be required. The folders of interest are %LC_TEMP_FOLDER%\AdobeDocumentStorage\local\ and %LC_TEMP_FOLDER%\pdfg-livecycle\.

CPU Cycles
Large document conversion is CPU-intensive. Conversion of Microsoft Office documents to PDF is single-threaded. As a result, concurrent mulitple coversion requests will queue up unless the solution is load-balanced across multiple servers. The 500 MB Word document conversion consistently used 3 GHz of CPU cycles for over 15 minutes.

The presence of hyphens in the names of tablespaces or users in Oracle will cause exceptions in WebSphere when it is being configured for LiveCycle ES. The SystemOut.log will not report this but the WebSphere FFDC (First Failure Data Capture) folder will contain entries like as follows:

Exception = java.sql.SQLException
Source = com.ibm.ws.rsadapter.jdbc.WSJdbcStatement.pmiExecuteUpdate
Stack Dump = java.sql.SQLException: ORA-00922: missing or invalid option
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)

The 'Default document max inline size' is an in-memory buffer that holds documents that are being processed by a LiveCycle orchestration. Its default setting is 65536 bytes (64 KB) which is too low for many orchestrations. If you proces a document larger than this value, temporary file I/O occurs to the filesystem that hosts the 'LiveCycle Temporary Folder'. If some of these transactions timeout or throw an exception, this filesystem runs the risk of running out of diskspace because of orphaned files. Periodic cleanup might be required. The folder of interest is %LC_TEMP_FOLDER%\AdobeDocumentStorage\local\.

Our tests show a 15% improvement in transaction throughput if the 'Default document max inline size' is larger than the largest document used anywhere in the orchestration. Therefore, as a performance tuning exercise, you will be better off figuring out the largest document that could possibly be used and then setting this value to a number that is higher.

You can configure this using the LiveCycle Admin UI. For 8.2.1, navigate to Settings->Core System Settings->Configurations. After making the change, the appserver instance will have to be re-started.

Values can be 1048576 (1 MB), 2097152 (2 MB) 5242880 (5 MB) or higher. However, please note that a very high value such as 300 MB will cause distress to your appserver instance's Java heap setting. In 32-bit JDK environments, such high values will increase the risk of hitting OutOfMemoryExceptions.

Depending on your use case, you will also have to increase the cache size for Forms ES as well as Output ES. In Admin UI, navigate to Services and then choose either 'LiveCycle Forms ES' or 'LiveCycle Output ES'. Change the value for Max Cache Document Size (KB) under the 'Global Cache Settings' section. Otherwise, you might get warnings such as "Ignoring cache write as Object size is greater than Max Document size in cache setting". Save the settings and then re-start the application server for these changes to take effect.

The LiveCycle ES JBoss and WebLogic Turnkeys in Windows both use MySQL Community Edition as their database. By default, MySQL does not allow remote or local access to it using tools such as MySQL Administrator, Oracle SQL Developer, Aquafold Aqua Data Studio or Navicat.

MySQL Administrator is a very useful tool. It is part of what MySQL calls 'MySQL GUI Tools' which can be downloaded here. One of its best features is the ability to optimize the tables which keeps the database running efficiently.

To enable remote access, the following steps are required:

Change directory to MySQL's /bin folder:
cd C:\Adobe\LiveCycle8.2\mysql\bin

Login as user 'root' with password 'password'
mysql -u root -ppassword
or without a password
mysql -u root

If MySQL was installed separately (in the case of non-turnkey install on Linux), you need to set the password for the 'root' user first. Here is an example for SUSE Enterprise Linux 10:
cd /usr/bin
mysqladmin -u root password 'your_new_password'

Verify that you can run commands. Tell MySQL to list its databases
show databases;
You should see multiple databases listed, including adobe (used by LiveCycle) and adobe_meta (used by BAM)

Tell MySQL to allow your user in. Here the user is 'root' with password 'password' logging in from a machine with the DNS name 'laptop.company.com'
GRANT ALL PRIVILEGES ON *.* TO 'root'@'laptop.company.com' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;

You should see a response like this :
Query OK, 0 rows affected (0.03 sec)

Logout
quit;
You should see a response like this :
Bye

You can now login as 'root' from the machine 'laptop.company.com' using MySQL Administrator. To optimize the Adobe LIveCycle tables, login and lcik on the schemata called 'adobe'. This will display the list of tables on the right in the 'Schema Tables' tab. On the right pane, at the bottom, click on 'Maintenance'. Check the radiobuton for 'Optimize Tables' and proceed.

During the development phase, LiveCycle developers sometimes require an e-mail server to test mail functionality. It might be too much paperwork to get corporate IT to provision additional e-mail accounts for test purposes on the corporate mail server. In such cases, Apache's JAMES mail server comes in very handy. It is a light-weight mail server built using Java that you can run locally on your laptop/desktop without any external dependencies. It requires about 30 MB of RAM. JAMES is an acronym for Java Apache Mail Enterprise Server.

To get it to work, the following steps are required:

- Download it from Apache's JAMES website
- Extract the .zip file to a suitable location. No install is required
- Make sure that the environment variable JAVA_HOME is poroperly set to the right JDK
- To start JAMES, navigate to the /bin folder and run run.bat from a command prompt

- By default, JAMES listens on port 4555 for its Remote Manager Service. This service lets you configure JAMES. It uses the IANA-assigned default ports for POP3 (110) and SMTP (25).

- To add users, telnet to JAMES from a command prompt using the following command:
telnet localhost 4555
The default administrator user is 'root' with password 'root'

- Use the folowing command to list all available commands:
help

- Add a user (livecycle) with password (password) using the following command:
adduser livecycle password

- Add an additional user (user1/password) for verification purposes

- Verify using the following command:
listusers

- Logout using the command quit.

- Download Mozilla's Thunderbird mail client and install it. You can also use Microsoft Outlook.
- Configure the accounts livecycle and user1. The respective e-mail addresses would be livecycle@localhost and user1@localhost
- Send mail from one to the other and vice versa and verify that both work.

During its installation on AIX, Solaris and Linux, LiveCycle extracts 2.5 GB of its bits into the /tmp file system. Many Unix system administrators usually provision only about 500 MB for the /tmp filesystem.

In such cases, you can launch the installer with an argument re-directing the temporary file i/o to another filesystem with at least 3 GB of disk space. The following command re-directs the file i/o to /idp/temp/ on AIX:
./aix_livecycle8_setup.bin -is:tempdir /idp/temp

If you are running LiveCycle in a complete 64-bit environment (64-bit OS, 64-bit JDK), please note that you'd still need to install a 32-bit version of the same JDK if you plan to use PDF Generator. If your appserver is WebSphere or WebLogic, this 32-bit additional JDK can be Sun HotSpot.

The high-level steps are as follows:
- Download and install the Sun HotSpot 32-bit JDK (version 1.5, version 6.0 is NOT supported)
- Create a new system environment variable called JAVA_HOME_32
- Set its value to the folder where you installed the 32-bit JDK

Additional details are available in the 'Preparing to Install LiveCycle ES' guide for your respective J2EE appserver available here.

If a user is to be authorized to invoke the Reader Extensions service via EJB or SOAP, that user has to be assigned a custom-made role with the requisite permissions. Here are instructions to do that.

CREATE CUSTOM ROLE
- In the LiveCycle Admin UI, navigate to Settings->User Management->Role Management
- Click 'New Role'
- Provide a name, for example 'Reader Extensions Invoke' and click 'Next'. The description is optional
- Click 'Find Permissions'
- Check the boxes for 'Credential Read', 'ReaderExtensions Invoke', 'Service Invoke' and 'Service Read' and click OK.
- Click Next, and then Finish.

CREATE USER
- In the LiveCycle Admin UI, navigate to Settings->User Management->Users and Groups
- Click 'New User' and create a new user. First and Last Names as well as userID and password are required.
- There is no requirement to assign the user to any group, it is entirely optional
- Click 'Find Roles' and check the box for the custom role you previously created and click OK
- Click Finish.

LiveCycle ES Update 1 (8.2.1) is supported on 64-bit JDKs. 64-bit JDKs liberate us from the 1.2 GB maximum heap size limitation experienced on 32-bit JDKs on Windows. With all LiveCycle components deployed (including Content Services ES), LiveCycle's heap size requirements are now close to the limits afforded by 32-bit JDKs. However, the installation of the JBoss Turnkey will fail under a 64-bit JDK at the point where it tries to create a Windows service.

It is assumed that you are running on 64-bit CPU(s) and a 64-bit version of Windows. To confirm, download CPU-Z and run it. On an Intel CPU, the instruction sets supported will list EM64T, if the CPU is 64-bit. For an AMD CPU, it should say AMD64.

If you don't care whether JBoss is configured as a Windows service to start automatically on server re-start, you can easily bypass this step and continue. All you have to do are the following:

- Download and install the 64-bit version of Sun's HotSpot JDK from http://java.sun.com Please note the 6.0 (1.6) version of any JDK is not tested or supported by LiveCycle 8.2.1

- Configure the %JAVA_HOME% and %Path% system environment variables properly

- Run 'Group Policy Editor' (gpedit.msc)
- Assign the following user rights to the user that will be launching and running JBoss:
: Lock pages in Memory

- Start the Turnkey install and continue until it fails on the creation of the JBoss windows service
- Exit the LiveCycle Configuration Manager
- Edit JBoss' run.bat and add the following entries. These lines configure JBoss to start with a minimum and maximum heap size of 2 GB. This means that your system should have at least 4 GB of physical RAM installed. This also assumes a single dual-core CPU. If you've got a single quad-core CPU, change -XX:ParallelGCThreads from 2 to 4.

set JAVA_OPTS=%JAVA_OPTS% -server
set JAVA_OPTS=%JAVA_OPTS% -Dadobeidp.serverName=lc_server1
set JAVA_OPTS=%JAVA_OPTS% -Dfile.encoding=utf8
set JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true
set JAVA_OPTS=%JAVA_OPTS% -XX:PermSize=512m -XX:MaxPermSize=512m
set JAVA_OPTS=%JAVA_OPTS% -Xms2048m -Xmx2048m -XX:+UseLargePages
set JAVA_OPTS=%JAVA_OPTS% -Djboss.partition.name=partitionLIVECYCLE
set JAVA_OPTS=%JAVA_OPTS% -Djava.io.tmpdir=C:\Adobe\LiveCycle8.2\lc_java_io_tmpdir
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseParallelOldGC -XX:ParallelGCThreads=2
set JAVA_OPTS=%JAVA_OPTS% -DCASBOOTPROPS=bam.properties
set JAVA_OPTS=%JAVA_OPTS% -Dcom.celequest.bootstrap.envpropname=CASBOOTPROPS

- From a command prompt, start JBoss with the command run -c all. If this fails with an error such as this:
"Error occurred during initialization of VM
Could not reserve enough space for object heap",
chances are that your memory is fragmented. Reboot the server and continue.
- Once JBoss is fully initialized, start LiveCycle Configuration Manager again
- Skip all steps until "Initialize Database'
- Continue from there

Please note that with this configuration, although MySQL will be configured as a Windows service to startup automatically on server re-start, JBoss has to be manually started every time. You have the option of configuring it as a 'scheduled task' to start on server re-start. For instructions, see here.

Here's a blog entry that describes how to get a 64-bit JBoss instance working on Linux.

Testing with Adobe LiveCycle ES shows that performance on servers that carry Sun UltraSPARC T1 and T2 processors is not great. IBM's Processor Value Unit (PVU) calculation bears this out.

PVU is used by IBM to license its middleware and is based on the relative performance of processors. IBM assigns a PVU of only 30 to the T1, 50 to the T2 but 100 to the UltraSPARC IV. SPARC64 VI and VII processors also get a PVU of 100.

If you plan to deploy LiveCycle on Solaris on SPARC, you will get more value for your money by deploying it on servers with UltraSPARC IV, SPARC64 VI or VII processors. The new (Nov 2008) M3000 family of SPARC Enterprise servers carry SPARC64 VII processors.