Debugging Form Rendering Issues Related to XMLForm Native Process

Scenario:

Often times we run into situations where form rendering or flattening (via Forms Service or Output Service) fails and server log shows a high level error message like –
——
<Error> <com.adobe.formServer.PA.XMLFormAgentWrapper> <BEA-000000> <ALC-OUT-002-004: Unable to find service: XMLFormService, error: Connection to failed service.>
——
Stacktraces are not typically useful in that case because problem is typically related to something on the native side viz. Things related to XMLForms native process.

We recently encountered this error on LiveCycle ES3 (a.k.a. ADEP) deployed on Weblogic 10.3.4 and RHEL 5.5 64-bit server.

Some Background:

As discussed in various blog posts and also during Best Practices session at MAX (Slidedeck available here), FormServer and OutputServer both use XMLForms native (C++) code behind the scene for rendering.

FormService and OutputService Java implementations that run in the app server use CORBA/IIOP to communicate with XMLForm native code. During the first render operation after the appserver is started, separate XMLForm process instances gets launched.

On Unix/Linux platforms <XMLFormService/bin>/XMLForm.exe is a shell script that gets called. It prepares the environment and calls the XMLForm binary which is responsible for actual rendering.

As we can see there are a lot of moving parts here. The following tips could help collect lower level information to solve potential problems encountered by the native layer.

  • On Unix/Linux deployments locate <XMLFormService/bin>/XMLForm.exe shell script (Yes, it has “.exe” extension but it’s an ASCII file) and edit using text editor.
  • Navigate to end of the file and you should see an “exec” line.
  • Modify “exec” line to include Corba debug parameters and add ldd/env commands before “exec” line as shown below.

—-
ldd $myPath/bin/XMLForm > /tmp/XMLForm.ldd_dump.txt
env > /tmp/XMLForm.env_dump.txt
exec $myPath/bin/XMLForm $* ORBtraceLevel 40  -ORBtraceThreadId 1
—-

  • After that either restart your app server or kill XMLForms process instance(s) if running.
  • Render a form using FormsService or OutputService and new instance of XMLForms process should get launched.
  • Analyze ldd_dump.txt, env_dump.txt, StdOut and StdErr logs to see if there are any issues loading native shared libraries and/or Corba debug messages.

Analysis and Solution:

In this particular customer scenario we found couple of things by inspecting these logs.

  1. ldd capture showed that native library paths had “null” as a directory name.

    We solved that by adding correct -Dadobeidp.RootDirectory=<path> JVM argument in managed server startup script.
    More about it in this blog.
  2. Stdout logs showed that “libgcc_s.so.1″ which was available in LD_LIBRARY_PATH was not corresponding to version GCC_4.0.0. Actual Message was:

/opt/bea/user_projects/domains/XPRESSION/adobe/wl1/XMLFormService/bin/XMLForm: /opt/xpression/Drivers/libgcc_s.so.1: version `GCC_4.0.0′ not found (required by /opt/bea/user_projects/domains/XPRESSION/adobe/wl1/XMLFormService/bin/libACE.so)

Typically libgcc_s.so.1 corresponding to GCC_4.0.0 is located in /lib directory.
You could verify the GCC version by executing following command:

strings /lib/libgcc_s.so.1 | grep GCC

If the version shows up correctly, as a quick test, you could add following entry in XMLForm.exe prior to “exec” line. (Of course, you need to kill any running instances of XMLForms process for it to be effective).

LD_LIBRARY_PATH=/lib:$LD_LIBRARY_PATH

Once that works, as a permanent solution, you want to modify the LD_LIBRARY_PATH in the script that builds profile for the user that runs the application server. Typically “.profile” or “.bash_profile” in user home directory are potential candidates.

It might be a little investigative work to find actual script and you could start looking at the script files with filenames starting with a “.” (dot without quotes) located in user’s home directory.

In this scenario, we found that LD_LIBRARY_PATH was set in “.bash_profile”; so we modified it there to resolve the problem.

As an alternative, you could also modify LD_LIBRARY_PATH in application server startup script.

VN:F [1.9.22_1171]
Was this helpful? Please rate the content.
Rating: 8.0/10 (1 vote cast)
Debugging Form Rendering Issues Related to XMLForm Native Process, 8.0 out of 10 based on 1 rating

About Santosh Tatke

Santosh Tatke is a Sr. Support Architect at Adobe Systems. He specializes in ADEP (formerly LiveCycle) technology stack and works closely with customers, solution consultants and partners to troubleshoot, analyze and resolve technical challenges during POC, implementation, deployment or upgrade. He is a Certified LiveCycle ES2.5 Process Management Expert.
This entry was posted in ADEP, Adobe LiveCycle ES2 (9.0.x), Document Services, General Interest and tagged , , , , , , . Bookmark the permalink.

5 Responses to Debugging Form Rendering Issues Related to XMLForm Native Process

  1. Pingback: LiveCycle XMLForms Native Process and WebSphere Global Security | Adobe LiveCycle Blog

  2. Ben Walsh says:

    Hi Santosh,

    Great blog. How do I do this on Windows?

    Regards,

    Ben

    • Santosh Tatke says:

      Hi Ben,

      In case of windows, xmlform.exe is a binary executable, so changes are not possible. However, we’ve not seen such problems on windows env. In case of Windows, if xmlforms process crashes, Dr Wtsn generated by logs should be useful to debug the problem.

      Thanks,
      Santosh

      • David Kacetl says:

        Hi, I have the same problem, we are using LC on windows. Have you already solved it (enable debug log on windows)?

  3. We are also having this problem on RHEL5. Is there some way this issue might be related to the JDK version?

    I haven’t had a look at the debug files, I’ll see what’s in them.