Often times we run into situations where form rendering or flattening (via Forms Service or Output Service) fails and server log shows a high level error message like -
<Error> <com.adobe.formServer.PA.XMLFormAgentWrapper> <BEA-000000> <ALC-OUT-002-004: Unable to find service: XMLFormService, error: Connection to failed service.>
Stacktraces are not typically useful in that case because problem is typically related to something on the native side viz. Things related to XMLForms native process.
We recently encountered this error on LiveCycle ES3 (a.k.a. ADEP) deployed on Weblogic 10.3.4 and RHEL 5.5 64-bit server.
As discussed in various blog posts and also during Best Practices session at MAX (Slidedeck available here), FormServer and OutputServer both use XMLForms native (C++) code behind the scene for rendering.
FormService and OutputService Java implementations that run in the app server use CORBA/IIOP to communicate with XMLForm native code. During the first render operation after the appserver is started, separate XMLForm process instances gets launched.
On Unix/Linux platforms <XMLFormService/bin>/XMLForm.exe is a shell script that gets called. It prepares the environment and calls the XMLForm binary which is responsible for actual rendering.
As we can see there are a lot of moving parts here. The following tips could help collect lower level information to solve potential problems encountered by the native layer.
- On Unix/Linux deployments locate <XMLFormService/bin>/XMLForm.exe shell script (Yes, it has “.exe” extension but it’s an ASCII file) and edit using text editor.
- Navigate to end of the file and you should see an “exec” line.
- Modify “exec” line to include Corba debug parameters and add ldd/env commands before “exec” line as shown below.
ldd $myPath/bin/XMLForm > /tmp/XMLForm.ldd_dump.txt
env > /tmp/XMLForm.env_dump.txt
exec $myPath/bin/XMLForm $* ORBtraceLevel 40 -ORBtraceThreadId 1
- After that either restart your app server or kill XMLForms process instance(s) if running.
- Render a form using FormsService or OutputService and new instance of XMLForms process should get launched.
- Analyze ldd_dump.txt, env_dump.txt, StdOut and StdErr logs to see if there are any issues loading native shared libraries and/or Corba debug messages.
Analysis and Solution:
In this particular customer scenario we found couple of things by inspecting these logs.
- ldd capture showed that native library paths had “null” as a directory name.
We solved that by adding correct -Dadobeidp.RootDirectory=<path> JVM argument in managed server startup script. More about it in this blog.
- Stdout logs showed that “libgcc_s.so.1″ which was available in LD_LIBRARY_PATH was not corresponding to version GCC_4.0.0. Actual Message was:
/opt/bea/user_projects/domains/XPRESSION/adobe/wl1/XMLFormService/bin/XMLForm: /opt/xpression/Drivers/libgcc_s.so.1: version `GCC_4.0.0′ not found (required by /opt/bea/user_projects/domains/XPRESSION/adobe/wl1/XMLFormService/bin/libACE.so)
Typically libgcc_s.so.1 corresponding to GCC_4.0.0 is located in /lib directory.
You could verify the GCC version by executing following command:
strings /lib/libgcc_s.so.1 | grep GCC
If the version shows up correctly, as a quick test, you could add following entry in XMLForm.exe prior to “exec” line. (Of course, you need to kill any running instances of XMLForms process for it to be effective).
Once that works, as a permanent solution, you want to modify the LD_LIBRARY_PATH in the script that builds profile for the user that runs the application server. Typically “.profile” or “.bash_profile” in user home directory are potential candidates.
It might be a little investigative work to find actual script and you could start looking at the script files with filenames starting with a “.” (dot without quotes) located in user’s home directory.
In this scenario, we found that LD_LIBRARY_PATH was set in “.bash_profile”; so we modified it there to resolve the problem.
As an alternative, you could also modify LD_LIBRARY_PATH in application server startup script.