Recently in Data Binding Category

One of the formfeed blog commenters ("mo") asked about preserving data during an import operation.   I gave her (him?) a flippant reply with some hand-waving about saving/restoring data before/after import. Then I tried it myself and discovered it was not nearly as easy as I thought.

Here's the problem description:  Your data arrives in two separate data files.  You need to import them both into your form.  Problem is that importing new data replaces existing data.  Loading the second data file will discard the data from your first import. 

Let's set up a specific example -- Suppose my data looks like:

<multidata>
  <set1> ... </set1>
  <set2> ... </set2>
</multidata>

We want to be able to load set1 and set2 from different data files.

There are a couple of solutions to this problem.   But first some review on dataset handling within XFA/PDF forms.  Normally form data gets stored under $data -- which is a shortcut to: xfa.datasets.data.  If the root node of your form is "multidata", then your data appears under xfa.datasets.data.multidata.

The XML hierarchy looks like this:

<xfa:datasets xmlns:xfa="http://www.xfa.org/schema/xfa-data/1.0/">
  <xfa:data>
    <multidata>
      <set1> ... </set1>
      <set2> ... </set2>
    </multidata>
  </xfa:data>
</xfa:datasets>

When Acrobat performs a data import, it replaces the <xfa:data> element. But it *appends to* any other datasets. 

Solution 1: Preserve/Restore data before/after import

If during import you could arrange your data to look like:

<xfa:datasets xmlns:xfa="http://www.xfa.org/schema/xfa-data/1.0/">
  <xfa:data>
    <multidata>
      <set2> ... </set2>
    </multidata>
  </xfa:data>
  <set1> ... </set1>
</xfa:datasets>

In this case, <set1> would be preserved and only <set2> would be replaced.  Then after the import is complete,  you'd move <set1> back where it belonged.  The way to control the import is to use the Acrobat script function: importXFAData();  Here's the outline of the script to import set2:

1. Copy set1 data to be a child of xfa:datasets
2. Remove the set1 subform
3. Call importXFAData() to load set2
4. Move the set1 data back under <multidata>
5. Re-add the set1 subform

There are a number of tricky parts:

• When importXFAData() is successful, it causes a remerge.  When remerge happens, any script commands after the call to importXFAData() will not execute.  The workaround is to perform steps 4 and 5 in a separate form:ready script.
• importXFAData() does not return a status.  You have no way of knowing if the user cancelled.  If they did cancel, you need to restore set1 without depending on the form:ready script.
• If the user is running in Reader, then importXFAData() will throw an error.  We need to catch this error and restore set1 data.
• If the user imports data from the Acrobat menu (Forms/Manage Form Data/Import Data) then all your clever script won't run and set1 data will get cleared.  You need to figure out how to remove this option from the Acrobat menu.

Note that this specific example assumes you are loading data in the order set1 then set2.  The form could be coded more generally to load the data in any order.  It would just be a bit more complicated.  You'd need to move both set1 and set2 and then after the load you'd figure out which one(s) need to be moved back.

Here is a sample form.  Here is sample set2.xml data you can load.  Have a look at the button click and form:ready events for all the gory details.

Solution 2: Bind set1 outside of xfa:data

Instead of temporarily arranging your data so that set1 is under <xfa:datasets>, you could permanently arrange your data this way.  In the binding expression for the set1 subform, specify "!set1" -- which is a shortcut for xfa.datasets.set1.  Now whenever you import data for set2, it will leave set1 untouched.  However, this introduces a new problem.  Whenever you import new set1 data you will end up with multiple copies of set1.  You need a form:ready script that will delete all but the last copy.  This also means that the data file holding set1 needs to include the <xfa:datasets> element so that it can correctly specify the location for set1. 

Here is a sample file with set1 data and set2 data. The script to trim back the extra copies of set1 data is found in the multidata form:ready event.

My personal preference would be to use Solution 2.  The script is simpler.  The user can use the menu commands for loading the data.  But this approach might not be possible if your data is bound to a schema.

Jeff K asked for a sample form where we populate a listbox from the results of a web service.  He was even kind enough to point me at a set of public domain web services that could be referenced from a sample form.

The web services can be found at: http://www.webservicex.net

The sample I wrote is based on the USA Zip Code Information. Specifically the request to GetInfoByCity: http://www.webservicex.net/WCF/ServiceDetails.aspx?SID=35

The specific WSDL file can be found at: http://www.webservicex.net/uszip.asmx?wsdl

This query takes a city name as input, and returns xml data with all the zipcodes that match that city name -- in each state that has a city with that name.

Here are the steps I followed to create the sample:

Create a Data Service

Once I downloaded a local copy of the WSDL file, I used it to create a data connection named CityQuery.  When I expanded the data hierarchy to look at the input and output data, I found one data field in the request area (USCity).  Good so far.  But I was expecting more in the response area.  All I found was: "GetInfoByCityResult":

Turns out this web service uses <s:any> element -- which means Designer has no idea what data will be found under that node and cannot offer specific guidance for binding decisions.

If you run the sample at: http://www.webservicex.net/WCF/ServiceDetails.aspx?SID=35 , you will discover the data format by looking at the returned result:

<NewDataSet>
  <Table>
    <CITY>Mission</CITY>
    <STATE>KS</STATE>
    <ZIP>66202</ZIP>
    <AREA_CODE>913</AREA_CODE>
    <TIME_ZONE>C</TIME_ZONE>
  </Table>
  <Table>
    <CITY>Mission</CITY>
    <STATE>SD</STATE>
    <ZIP>57555</ZIP>
    <AREA_CODE>605</AREA_CODE>
    <TIME_ZONE>C</TIME_ZONE>
  </Table>
  <Table> ... </Table>
  <Table> ... </Table>
</NewDataSet>

Set up List Box Binding

I created a text field (City) and bound it to the request data: USCity.

Then I created a drop down list and set up bind to the returned data.

The binding wizard took me only as far as the GetInfoByCityResult element.  The rest I had to enter manually -- based on what the sample data looked like.  The bind expression for items is:

!connectionData.CityQuery.Body.GetInfoByCityResponse.GetInfoByCityResult.NewDataSet.Table[*]

Note that the expression starts with "!".  This character is used in SOM as a shortcut to the dataset elements i.e. the elements below xfa.datasets.  The data exchanged with this web service is gathered under xfa.datasets.connectionData.CityQuery. 

Execute the WSDL

The last step is to make sure the SOAP request happens at the right time.  On the exit event of the City field I added this script:

form1.#subform[0].City::exit - (JavaScript, client)

// Invoke the web service

xfa.connectionSet.CityQuery.execute(false);

ZipCodes.selectedIndex = 0;

The form now works.  Every time you exit the City field, the SOAP request is made and the zip code list box gets populated.

The Deep End

I wanted to populate another drop down list with more than just the ZIP codes. I wanted to include city name and state name.  The hard part about this is that once the WSDL request completes, the transaction data is removed. The moment in time where you can access the returned WSDL data is in the postExecute event (preExecute fires before a web service request, and postExecute fires afterward).  But here is the problem.  Designer does not provide an interface to specify a postExecute script.  I had to add one in the XML source view:

<event activity="postExecute" 
       ref="xfa.connectionSet.CityQuery"

       name="event__postExecute">

</event>

Once you've specified this event in XML source view, you can edit the script in Designer by selecting "Events with Scripts" in the script editor.

By specifying preExecute and postExecute events, you can access the SOAP data before and after the web service request. 

preExecute SOAP request:

<xfa:datasets xmlns:xfa="http://www.xfa.org/schema/xfa-data/1.0/">
   <!-- Form data (data bound to fields and subforms) -->

   <xfa:data>

   <dd:dataDescription xmlns:dd=http://ns.adobe.com/data-description/

                          dd:name="CityQueryGetInfoByCitySoapInDD">

      <CityQuery>

         <soap:Body xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">

            <tns:GetInfoByCity xmlns:tns="http://www.webserviceX.NET">

               <tns:USCity dd:minOccur="0" dd:nullType="exclude"/>

   <connectionData xmlns="http://www.xfa.org/schema/xfa-data/1.0/">

      <CityQuery xmlns="">

         <soap:Body xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">

            <tns:GetInfoByCity xmlns:tns="http://www.webserviceX.NET">

               <tns:USCity>Mission</tns:USCity>

</xfa:datasets>

Note that during preExecute, your script may modify the outgoing request data.

postExecute SOAP response:

<xfa:datasets xmlns:xfa="http://www.xfa.org/schema/xfa-data/1.0/">

   <!-- Form data (data bound to fields and subforms) -->

   <xfa:data> ... </xfa:data>

   <!-- The distilled WSDL schema used by XFA to construct the SOAP request -->
   <dd:dataDescription xmlns:dd=http://ns.adobe.com/data-description/ 
    ...

   </dd:dataDescription>

   <connectionData xmlns="http://www.xfa.org/schema/xfa-data/1.0/">

      <CityQuery xmlns="">

            <GetInfoByCityResponse xmlns="http://www.webserviceX.NET">

               <GetInfoByCityResult>

</xfa:datasets>

I was then able to write a script to populate the "Choice" drop down list with zipcode, city and state information:

var vTables = xfa.datasets.connectionData.CityQuery.Body.GetInfoByCityResponse.GetInfoByCityResult.NewDataSet.Table.all;

var aChoices = [];

Choice.selectedIndex = 0;

When I occasionally browse around some of the LiveCycle forums, I frequently see questions around how to populate a drop down list.  I have put together a sample form that illustrates several different options.

Data Source

There are two basic sources for updating a drop down list definition: data binding or script.  If the list contents are defined as part of your form data, and if they don't change during your form session, then use data binding.  If the definitions are more fluid, then use script.

The preOpen Event

The most important take-away from this blog entry concerns what event to use for updating lists.  I have seen customer forms that update list contents using change, exit, calculate, validate, mouseover, and enter events.  However, the proper place to do it is in the preOpen event.  The preOpen event fires when the user activates the control for the choice list.  Think of it as the "just-in-time" option.  If you try to maintain your list box definition from other events then often your script will update your list box contents too frequently. 

The only reason for updating a choice list sooner than the preOpen event is if you need to assign a value to a list field.  e.g. if your drop down list has a display value: "ONE" and a bound value: "1", then assigning
field.rawValue = 1; will cause the field to display "ONE".  Obviously this works only if the field has up to date list contents.  If you need your list contents updated more frequently, you should still put the list populating logic in the preOpen event, and use execEvent("preOpen") to populate the list from other contexts where it's needed.

The sample form has four choice lists that get populated from their preOpen event, using data found in form field values, JavaScript arrays and XML data.

Script Commands

There are two script methods for setting your list box contents:
field.addItem()
   and
field.setItems(). 

The API: field.addItem() works in all versions of Reader.  field.setItems() was introduced in Reader 9.0 and is much faster and more convenient.  The sample form has script that illustrates how to use each method.

Binding to Data

I constructed some sample data (dogs.xml) that looks like this:

<dogs>
  <category file="ugliest">
    <rank>1</rank><dog>Chinese Crested</dog>
    <rank>2</rank><dog>Pug</dog>
    <rank>3</rank><dog>Shih Tzu</dog>
    <rank>4</rank><dog>Standard Schnauzer</dog>
    <rank>5</rank><dog>Chinese Shar Pei</dog>
    <rank>6</rank><dog>Whippet</dog>
    <rank>7</rank><dog>Dandie Dinmont Terrier</dog>
    <rank>8</rank><dog>Japanese Chin</dog>
    <rank>9</rank><dog>French Bulldog</dog>
    <rank>10</rank><dog>Chihuahuas</dog>
  </category>
  <category file="dumbest">
   ...
  </category>
  <category file="comicBook">
   ...
  </category>
  <category file="smartest">
   ...
  </category>
</dogs>

On my form I want two drop down lists: one with the names of the categories and a second that gets populated with the contents of the category.  Since this XML is part of my data, I bound the category using these expressions:

The second list has a preOpen event that locates the category in the data, and then populates a second listbox from the category contents:

// Find the data group that corresponds to the category chosen
// in the Category field

var vDogs = xfa.datasets.data.dogs.category.all;
var vChoice = null;
var i;
for (i = 0; i < vDogs.length; i++) {
    if (vDogs.item(i).file.value === Category.rawValue) {
        vChoice = vDogs.item(i);
        break;
    }
}
if (vChoice !== null) {
    // vChoice.dog.all is the equivalent of the
    // SOM expression: "category.dog[*]"
    var vDisplayValues = vChoice.dog.all;
    var vBindValues = vChoice.rank.all;
    for (i = 0; i < vDisplayValues.length; i++) {
        this.addItem(vDisplayValues.item(i).value,
                     vBindValues.item(i).value);   
    }
}

Performance

Populating a list from data is very efficient.  But populating large lists or many lists using addItem() can be slow.

The performance gains of setItems() over addItem() is substantial.  If your form makes extensive use of choice lists or has choice lists with large contents, you will appreciate the improvements of setItems().  Of course, this option is available only in forms designed for Reader 9 or later.

Web Services

In some cases, the definition of the choice list may be held on a server.  In this scenario, the best strategy is to add a WSDL connection to your form that retrieves the list contents.  Have your list box bind its contents to the data retrieved via the SOAP call.

Away for a week

After I eat lots of turkey on the weekend (Canadian Thanksgiving), I'm spending next week trying to empty the job jar at home.

The XFA processor has a notion of data records.  A record is the data that populates one instance of your form.  If you are designing interactive forms, you have probably never had more than one data record in your form.  The notion of multiple records is primarily for processing large print jobs in LiveCycle Print.  Consider the case where you want to print customer statements.  You extract an XML file from your database with the data for all 2000 customers.  Then in a single call to LiveCycle Print you print all 2000 records.

I won't go into all the details of setting up a multi-record print.  My intention today is to show how you can leverage multiple record support in interactive forms.

Data Record

Suppose you have a form that binds to a root data node named: "form1".  When there is a single data record, your data looks like this:

<xfa:datasets xmlns:xfa="http://www.xfa.org/schema/xfa-data/1.0/">
   <xfa:data>
      <form1>...</form1>
   </xfa:data>
</xfa:datasets>

In the simple case, a form with three records will look like:

<xfa:datasets xmlns:xfa="http://www.xfa.org/schema/xfa-data/1.0/">
   <xfa:data>
      <form1>...</form1>
      <form1>...</form1>
      <form1>...</form1>
   </xfa:data>
</xfa:datasets>

But by default Acrobat/Reader will display only the first record.  But with a few simple script commands you can navigate to different records as well as add and remove records.

Script Expressions

The Current Record
The current data record is easy to find.  We have set up a convenience property: xfa.record.

Record Count
xfa.record.parent.nodes.length;

Current Record Number
xfa.record.index;

Record Count
xfa.record.parent.nodes.length;

Strictly speaking, this will not always return the record count... but that's in the deep end.

Add a Record
var newRecord = xfa.datasets.createNode("dataGroup", xfa.record.name);
xfa.record.parent.nodes.append(newRecord);

Remove a Record
var vRecordToRemove = xfa.record.parent.nodes.item(xfa.record.index);
xfa.record.parent.nodes.remove(vRecordToRemove);

Goto a Record
xfa.datasets.dataWindow.gotoRecord(nRecord);

Sample

I have attached a sample that exercises all these script commands.

Exporting XML data

One thing you will notice when you export or submit data as XML, the result will change depending on if there is one or more than one record included.  From the example above, if I export as XML and there's one record I'll get:

<form1> ... </form1>

If I export a data with 3 records, we need to add an aggregating element so that it remains valid XML:

<xfa:data xmlns:xfa="http://www.xfa.org/schema/xfa-data/1.0/">
    <form1>...<form1>
    <form1>...<form1>
    <form1>...<form1>
</xfa:data>

Changing the Current Record

You need to be aware that when you call dataWindow.gotoRecord(), the XFA processor will perform a re-merge with the new data.  Doing a remerge means that you lose any changes you've made to your form objects.  e.g. you will lose field highlighting, choice list definitions populated by script etc.

The Deep End

There are some advanced options for handling very large datasets on the server.  If you think about it, a large print job could contain many thousands of records, and the input could be a data file that is many gigabytes.  To handle that case there are options in configuration that allow you to process the file incrementally.  Part of that involves specifying a record element by level or by name.  e.g. you could say that the name of the record node is "customer".  In this sample XML:

<customerRecords>
  <jobDetails>...</jobDetails>
  <customer> ... </customer>
  <customer> ... </customer>
  <customer> ... </customer>
  <customer> ... </customer>
</customerRecords>

there are 4 records, but the <customer> parent element has 5 children.  That's why, strictly speaking, the expressions: xfa.record.index; and xfa.record.parent.nodes.length; are not always reliable.

But if you haven't set the config file options, then there's no problem.

Data Window

To round out the story of data records, we need to say a bit more about incremental processing -- commonly referred to as "lazy loading".  The idea is that the XFA processor holds only a small number of data records in memory.  As the XFA processor loads the data file it progressively loads data records into a window and when they've been processed they are discarded.  The window can specify how many records before and after the current record are kept in memory.  In this way we can process a multi-gigabyte data file without loading it all into memory.

As mentioned, there are configuration options for setting up the data window behaviour as well as a dataWindow scripting object.  But these details would need to be the topic of another blog entry.

It has been quiet for a while. That is because I took on a more ambitious task over the last couple weeks.  The result is a new XFA/PDF debugger tool.

In the past I've posted samples (tools) that help users debug their merge (form dom) and view their form layout. The new tool consolidates and extends those capabilities and implements the debugger in flash -- a SWF embedded in a PDF. 

This tool will be useful to anyone having problems designing dynamic forms:

• Subforms aren't being created from data where you expected they would
• Layout has a mysterious blank page
• Garbled or overlapping layout
• Layout did not appear in the order you expected
• leaders and trailers aren't being created
• leaders and trailers are created too often
• Content seems to be missing

If you're encountering any of these symptoms then this tool could help.

Usage

You need to be running Acrobat (not Reader) to use the tool.  And you need to be using version 9. 

To start a debug session, populate your PDF form with data and save it.  Then open the PDF from the XFADebugger.pdf tool.  You will see a snapshot of the form represented in three vertical boxes:

• Form DOM tree
• Data DOM tree
• Layout -- page display (with content areas rendered as grey boxes)

From there on it is hopefully self-explanatory.  You can expand/collapse the trees.  Nodes in the trees are colour coded:

• grey -- node is not bound
• black -- node is bound once
• blue -- data node is bound to more than one form node
• green -- represents a subform leader/trailer that has been added by the layout process

Selecting a node in the form tree will:

• if bound, highlight the corresponding node in the data tree
• if not hidden, highlight the area on the page display where that node is rendered.
• Display any interesting attributes/properties that impact merge and layout

Selecting a node in the data tree will (if bound)

• select all the form node(s) that are bound to this data
• display the attributes and highlight the corresponding form node(s)

Warnings

The tool reports on suspicious conditions on the form that could impact merge or layout.  Clicking on the "Find Warnings" button will cycle through any warnings found in the form.  For each warning, the corresponding nodes are highlighted and the the warning text appears in red.

These are the warning conditions detected:

Object extends outside its parent container
Just what it says.  When the offending form node is highlighted on the page layout in dark blue, it's parent node is highlighted in pale blue.

leader/trailer subforms must not be flowed
Leader and trailer subforms must always have positioned content.  As you might imagine having a variable-sized leader/trailer would make it pretty difficult for the layout algorithm to reserve space for the leader/trailer subform.

Subform is splittable but cannot split because the parent is not splittable
Designer also warns about this condition.

Object is growable, but not-splittable.  With enough data, it could grow too large for its content area
If an object can grow vertically without any upper limit, it could eventually grow too big to be rendered inside a content area.

Object is growable but its parent is not.  With enough data, it could grow too large for its parent
If you place a growable object inside a subform with a fixed layout size, you could end up with an object that's too big for its container.

Keep with previous' conflicts with 'break after' on previous element
Conflicting break/keep directives. (By the way, the keep will trump the break -- but this is condition is a leading cause of mysterious blank pages in your output)

'Break before' conflicts with 'keep with next' on previous element
Same as previous except the other way around.

Repeating subforms should not specify keep with next or keep with previous
Having a keep on a repeating subform will result in the entire group of subforms being un-splittable.

Subforms with repeating children should be splittable
If a subform has a repeating child, it is likely to require a split.

Multiple repeating form nodes are bound to the same data. This might cause a different merge result when the form is re-opened
This takes more explanation.  Consider this template definition:

<subform name="S0"><occur min=1 max=10/><bind ref="S[*]"/></subform>
<subform name="S1"><occur min=1 max=1/><bind match=once/></subform>

When this form is first opened without any data, we will create two instances of <S> in the data -- one for S0 and one for S1. Then when we save/close/reopen, subform S0 will bind to both instances of <S> and subform S1 will create a new instance of <S>. i.e. after save/close/reopen there is one more subform than there was before.  This is a form design issue that crops up occasionally and can be very confusing for novice form authors.

Rows with more than one multi-line field might have difficulty splitting

If you have a splittable table row with more than one multi-line field, you might find that it does not split.  The algorithm for splitting rows requires finding a common font baseline between rows on the sibling cells.  For current shipping product, the check for the baseline is very exact.  If there is any difference between the fields that can cause the lines to be offset slightly, then the split algorithm will not find a split point.  Some of the attributes that affect the position of the baselines include: top margin, paragraph space before/space after, line spacing, vertical justification, typeface, font size, vertical scale... and probably a couple more I haven't thought of.

As is the nature of warnings, not all warnings are problems that need to be fixed.  Your form might report warnings that are innocuous. 

Here is a sample of a very badly designed form that manages to have (at least) one instance of each warning.

How the Tool Works

The sample has two parts.  There is a base PDF with a document-level JavaScript defining:
function PDFLoader() that will:

• Select and open an XFA-based PDF
• Extract an XML snapshot representing the state of the form after it has opened

The form has a page-sized embedded SWF which holds the implementation of the debugger.  The SWF has a button that calls the document-level JavaScript using a call to ExternalInterface.call("PDFLoader").

Once the SWF has the XML snapshot of the form, it renders it and doesn't communicate with the base PDF anymore.

Other Uses

Educational

Loading up a form and seeing the form/data/layout graphically displayed can help to get insight on how the merge and layout processes work.

Quality Assurance

There are two ways that this tool can be used or adapted to maintain quality in your forms. 

1) loading and viewing your dynamic form in the debugger lets you verify that merge and layout are happening as designed.  Just because your form looks ok on screen doesn't necessarily mean that your data merged correctly or that your layout is behaving as planned.  You might be surprised by what you see.  You should make it a habit to check for warnings.

2) Adapt the XML snapshot to produce 'gold data' for your form.  When you are satisfied that your form is working correctly, produce a snapshot of the form that you can save as a baseline.  Then if your form gets modified -- perhaps some cosmetic changes -- you can compare the new snapshot to the baseline and confirm that any changes are as expected.

Futures

There are undoubtedly more form design problems that could be flagged by this tool. If you have suggestions for other conditions to detect, please let me know.

The form DOM could include more objects -- instance managers and draw elements.  For now I've left them out because they clutter the form tree too much.

Updates

June 1, 2009

• Fixed bug where field splittable status was reported incorrectly
• Increased the tolerance when checking for objects outside their extent
• Added a new warning: "Rows with more than one multi-line field might have difficulty splitting"