Archive for February, 2009

Form DOM Debugging Tool

One area where a novice form designer often needs help is in figuring out how to bind their form to data.  Ok, scratch that.  Advanced form designers often need help in this area as well.  There is lots of reading you can do in the XFA specification to learn how the template DOM gets merged with the data DOM to produce the form DOM.  Designer does a great job of generating binding SOM expressions for you.  But even still, when you are dealing with a complex schema, it can be hard to figure out where things went wrong.

A good way to debug this problem is to visualize the resulting DOMs.  Since we have full scripting access to the form DOM and to the data DOM, we can add a visual display of the DOMs to our form.  That’s the approach that today’s sample takes.  I took a work-in-progress purchase order form and added a "debugging subform" (domDump) at the end of the form in order to display the DOMs.  When you open the form and look at the last page you will see two side-by-side subforms.  The left side shows a tree view of the form DOM, the right side shows a tree view of the data DOM.  Some of the things you should note:

  • Entries in the tree are color-coded depending on their bind status
  • The display is cut off at one page.  To see more, use the scroll buttons at the top
  • Collapse and expand sections of the trees using the +/- buttons on the rows
  • Set focus in a field on either side of the tree and the display will highlight that entry and the corresponding entry(s) in the other DOM in red.
  • Set focus on a row in the form DOM, and some binding information is displayed in the top right of the display
  • Shift+click on a row in the form DOM will set focus to the corresponding field on the form.
  • By default we display the current state of the form at the docReady event.  If you want to refresh/rebuild the tree views, click the refresh button.

To try this out on your own forms, take the domDump subform and add it to your object library in Designer.  When you want to debug a form, drag the domDump subform onto your form.  (The subform needs to be displayed on a new master page with a content area that is at least 8×10.5 inches.) After debugging, when you are happy with your form, remove domDump.

There are some restrictions on the usage of this tool:

  • It works only on interactive forms
  • It works only on dynamic forms
  • Because of the extensive use of JavaScript objects, it does not work on forms with version 8.1 strict-scoping turned on

There is code in the script to check that these conditions have been met.

I won’t go into an in-depth description of the JavaScript that makes this debugger work.  It’s complicated :-) The intent is that form designers can use it without understanding the internals.  The one area where users might be tempted to tweak the script is to customize the information that gets displayed when a row in the form DOM is highlighted.  If that interests you, look for the script function: debugDisplay().

For interest sake, I’ve included one of my previous samples with the debugging subform added.  When I first added the dompDump subform I had to press the "refresh" button in order to see the completed DOMs.  That’s because the transpromo content and the debugging content are both populated from the docReady event — and the transpromo happens last.

Futures

  1. Admittedly, displaying tree views using dynamic subforms is non-trivial and a bit clunky.  One possible enhancement would be that rather than display the tree view on the form, we could export all the data for the tree views.  Then we could write a cool flash app to load the data give a proper rich user experience.  The only drawbacks with that approach is that a) it becomes a multi-step process and b) you lose the ability to "shift+click" to the corresponding form field.
  2. It would be great to have a similar debugging capability for WSDL connections.

March 19 Update

After using the form DOM debugger with several forms, I’m hooked.  I couldn’t resist making a couple improvements to it.  I’ve lifted most of the restrictions as to what flavour of forms it can be used in.  It now works with a broader range of template versions and strict scoping variations.  It also works for non-interactive documents.  For non-interactive, the tree display will spill over multiple pages and give a full dump — rather than windowing the content on a single page.  The only remaining restriction is that it works only with dynamic forms.

June Update

There is a follow-up debugger effort that supercedes the sample in this entry. Please have a look here.

Working with Data Fragments

Many of you will be familiar with the idea of constructing a form using template fragments. Template fragments are a powerful way to construct a form experience with modular parts. But there are some workflows where template fragments do not (yet) have all the functionality we might like. In cases where the content of the form is determined at runtime, template fragments might not have the flexibility you need. Take the transpromo examples from earlier posts(here and here). In those samples, the advertisement was baked into he template.  But in real life the actual ad that gets inserted will change. On starting a new marketing campaign, a company may want to issue a new set of ads to embed in their output. The actual ad chosen will vary depending on data in the form. Is the client single? Insert the sports car ad. Do they have a new baby? Insert the minivan ad. Some of our customers are building these kinds of applications using what we call “stitching solutions”. They have written Java libraries that assemble XFA templates on-demand. Writing a Java solution is fine for some, but eventually we have to make this easier.

One of the solutions available using today’s LiveCycle products leverages the notion of data fragments (instead of template fragments). I use the term “data fragments” to refer to the idea of embedding rich content in data. You might be surprised at how much you can customize the look of your document via XML instance data. You can add images, rich text, hyperlinks, positioned text and even floating fields.

A solution that uses data fragments to place ads in statements might look like this:

  1. An application for authoring data fragments representing the advertisements
  2. An application for adding metadata to the ads and storing them in a repository
  3. A rules engine for selecting ads from the repository based on correlating transaction data with ad metadata
  4. Print engine to render the statements with the ads

I can’t give you the whole solution, but can offer a sample that would help you get started with parts 1 and 4:

  • AdGenerator.pdf: This is a PDF for generating a data fragment and adding it to statement data. (Ideally we’d define something fancier for designing data fragments – maybe a slick flash app side-by-side with the PDF.)
  • statement.xdp: A sample credit card statement that includes a placeholder subform to render the ad data fragment.

Here is a copy of AdGenerator.pdf, populated and ready to export data. (Please give special notice to the image artwork that I worked so hard on.)

Here is a copy of the resulting statement generated with the ad.

How AdGenerator.pdf works

The advertisement is a growable subform that holds:

  • An image field
  • A repeating subform housing rich text

The form has various field and button controls to add the image and to add and position the rich text.  To understand how to use the form, read the instructions on the form itself. The scripts are also well documented and a good source for discovering the techniques used.

Dynamic properties

The (x,y) placement of rich text and the size of the ad subform are controlled by dynamic properties. Authoring these means going into XML Source view and adding the appropriate <setProperty> elements. E.g.:

<field name="Image" w="203.2mm" h="25.4mm">
   
<setProperty target="h" ref="height"/>
   
<setProperty target="w" ref="width"/>

Repeating, positioned text

Another case where we had to use XML source mode: Having added the TextData subform as a positioned child of advertisement, add an <occur/> element to allow it to repeat:

<occur min="0" max="-1"/>

The buttons and fields that position the text will update both the text coordinates as well as the data that the text coordinates are dynamically bound to.

Data References

You can personalize the text on the ad by injecting data values inline with the text.

On loading transaction data, we populate a listbox with all the data values found in the instance data. Adding the data reference uses the same mechanism as floating fields in Designer. We inject a special <span/> element into the xhtml with an embedded data reference. E.g.:

<span xfa:embed="$data.Statement[0].AccountNumber[0]" />

Styling Rich Text

In order to style your rich text in Acrobat, you need to bring up the properties editor (ctl + e). To add a hyperlink, select some text and choose “Hyperlink…” from the context menu. (By the way, there seems to be a bug here.  Rich text editing never works for me in Designer preview.  I use standalone Acrobat — with Designer shut down).

Calling FormCalc Functions From JavaScript

For form calculations, validations and general scripting problems, our form designers can choose between using FormCalc and JavaScript. FormCalc was designed as an expression grammar targeting forms functionality — and designed to be easy to use for a novice form author familiar with expressions in Excel.  JavaScript on the other hand, is a much more general purpose and powerful scripting language.  Over time, form authors have gravitated more toward JavaScript, mainly because of its familiarity.  However, there are functionality gaps — specifically a set of built in functions that are available in FormCalc but not in JavaScript. 

Many of the built-in FormCalc functions can be easily mimicked in JavaScript.  It doesn’t take a rocket scientist to write a JavaScript script object function to imitate the FormCalc sum() function. However, there are some functions that are not so easily mimicked.  The most notable of these are the Parse() and Format() functions available in FormCalc.  Parse() and Format() are the entry points into the very rich picture clause processing functionality.  When you consider the inherent locale handling, date/time functionality, different calendars etc. it’s plain to see that you don’t want to do this in JavaScript.

But now we have a problem.  Many users are committed to JavaScript because they built frameworks for controlling validations and other form behaviours.  I did the same in the series of blog posts on validations (the most recent version of the framework was in the sample from this post).  The problem is that you cannot directly call FormCalc from JavaScript.  So it would seem that you can’t enjoy both the power and familiarity of JavaScript as well as the built-in functions of FormCalc.  Well, … actually you can.

There are two design patterns I’d like to cover:

  1. Validating fields with picture clauses
  2. General mechanism for calling FormCalc functions

Validating Fields with Picture Formats

Picture clause validations are a simple, declarative mechanism for form authors to ensure that input data complies with a specific format.  If the field value can successfully be formatted with the validation picture clause, then the field is considered valid.  e.g. if your validation picture clause is "Date{YYYYMMDD}", then the field is considered valid only if its value can be formatted as a date.  If you were to express this validation as a script, you could write this FormCalc expression:

form1.#subform[0].DateTimeField1::validate – (FormCalc, client)
format("YYYYMMDD", $) <> ""

Now the question is how to tap into this functionality from JavaScript.  The short answer:

  1. Define a *display* picture format (do not a validation picture format)
  2. Write a JavaScript validation script that returns true when the field.rawValue is different from field.formattedValue

Most often when you use a validation picture clause you also use a display picture.  In fact, there’s really no reason why these picture clauses need to be different.  Combine that knowledge with an understanding of how the field.formattedValue property works:  when a field value can be formatted using the display picture, field.formattedValue will return the result of the format operation.  If the format operation fails, field.formattedValue returns the same as field.rawValue.  So to find out if a field was formatted correctly, use this JavaScript validation:

form1.#subform[0].DateTimeField1::validate – (JavaScript, client)
this.rawValue != this.formattedValue;

Mechanism for Calling FormCalc Functions from JavaScript

The solution is to use the execCalculate() method to indirectly cross the bridge between JavaScript and FormCalc.  When you open the attached sample, you will find a subform called "fc" that holds a script object called "func".  "func" has a series of embedded JavaScript functions that mimic the FormCalc functions with the same name.  Each function populates form variables with the function name and input parameters.  It then calls execCalculate() on the fc subform and returns the resulting value:

FCfromJS.fc.#variables[0].func – (JavaScript, client)
function Format(vPicture, vValue)
{
    F.value = "Format";
    P1.value = vPicture;
    P2.value = vValue;
    this.execCalculate();
    return ResultString.value;
}

The subform calculation script looks like this:

 FCfromJS.fc::calculate – (FormCalc, client)
; execute the requested function based on the input
; request parameters
if (F == "WordNum") then
    ResultString = WordNum(P1)

elseif (F == "Parse") then
    ResultString = Parse(P1, P2)

elseif (F == "Format") then
    ResultString = Format(P1, P2)

elseif (F == "Uuid") then
    ResultString = Uuid(P1)

elseif (F == "UnitValue") then
    ResultString = UnitValue(P1, P2)

else
    ResultString = ""
endif

The field that wants to use the format() functionality has a simple calculate script:

FCfromJS.#subform[1].format.Result::calculate – (JavaScript, client)
fc.func.Format(PictureClause.rawValue, Value.rawValue);

This fc subform can easily be incorporated as a custom library object in Designer that can be dragged onto any form.  It should be pretty easy to follow the design pattern if you want to extend the sample and add other FormCalc functions.

One usage note — if you want to call these functions from initialization scripts, then be sure to place the fc subform at the beginning of your template.  This is necessary because the fc subform has an initialization script that creates the necessary form variables.  By placing this subform at the top of the form hierarchy, we’ll be certain that the fc initialization event fires before other initialization events.

August 12, 2009 Update

I have updated the sample form:

  • Added access to the formcalc Get() function. 
  • Added a version checking mechanism (see this post for details).
  • The subform housing this functionality is now available as a downloadable fragment: scFormCalc.xdp
  • The functionality is accessed using the subform "enter" event rather than the calculate.  The calculate event was introducing unwanted calculation dependencies.

Form Compatibility

In order for customers to successfully distribute forms, they require predictability.  Predictability is more important than bug-free software (if there were such a thing).  Predictability means the form you originally developed continues to work the same when opened in a new version of Adobe Reader.  "the same" means it has the same set of features and the same set of bugs as in the previous version.

Same features, same bugs.  If you want the new features and bug fixes you need to bring your form into Designer and target it for a newer release of Reader.  But even so, for complex forms you probably prefer to get the new features without necessarily getting the new bug fixes.  Bug fixes can change form appearance and/or script behaviour.  What users most often want is to use that cool new feature, but not to tweak their layout due to the fixes in form rendering.

To accommodate these needs, there are two dials that control the form: the target version and the original version.

Target version

field.setItems() was an enhancement to the XFA object model for Reader 9.  Suppose Adobe Reader 9 opens a form Designed for Reader 8.  What should it do if this Reader 8 form happens to have a script calling field.setItems() ?  It should do the same thing that Reader 8 does: throw an error. 

Each form is stamped with a target version.  The stamp is expressed as the XFA version.  Reader 8 uses XFA version 2.6.  Reader 9 uses XFA version 2.8.  Whenever Reader is asked to execute a script function, it first checks whether that script function is valid for that version of XFA.

This applies to more than just script functions.  It applies to XFA markup as well.   e.g. Hyphenation was added in XFA 2.8.  If Reader encounters a 2.6 form that has the markup commands to turn on hyphenation, the hyphenation will not be enabled.

The XFA version is defined by the XML namespace of the template: e.g.

<template xmlns="http://www.xfa.org/schema/xfa-template/2.8/">

The target version is controlled in the Designer UI under form properties/defaults.  Note that by default, Designer will set the target version one version back from the current shipping version of Adobe Reader.

Original Version

Form behaviour is determined by more than just new features, it is also determined by bug fixes. In some cases fixing a bug in Reader can actually break existing forms that inadvertently relied on the buggy behaviour. Therefore by default we preserve old behaviours when migrating a form forward.

We preserve the old behaviour by storing the original version of the form in a processing instruction in the template. When you modify your form in Designer to take it from Reader 8.1 to Reader 9.0, we embed the original version inside the form so that you end up with the version 9 features, but the version 8.1 behaviours.  The processing instruction looks like this:

<?originalXFAVersion http://www.xfa.org/schema/xfa-template/2.6/?>

When Reader 9 opens a form with an original version of 2.6, it makes sure that the behaviours are consistent with Reader 8.1.

This is all well and good, but there are times where you really, really need the latest bug fixes.  Unfortunately there’s no UI to move the original version forward.  If you need the latest and greatest bug fixes, the go into the XML source tab, find the originalXFAVersion processing instruction and delete it.  Now your form will get all the behaviours and features of XFA 2.8 and Reader 9.

Sample

I have attached a sample form that demonstrates changing the target and original versions.  There are three interesting part to the form.

1. The form has both target version and original version set to 2.6 (Reader 8.1).  There is a button on the form that extracts these values from the template and populates a couple of fields with the result.

2. When you open the form in Designer, you will see a field displaying current date and time in ISO8601 format.  This field has a calculation that looks like this:

this.rawValue = "20090205T163000";
xfa.host.currentDateTime();

When you preview this form, the first line executes fine, but the second line encounters the currentDateTime() function that was added to XFA version 2.8.  As a result, the script fails on the second line, and the value of the field remains "20090205T163000".  If you open the script console in Acrobat (control+J) you’ll see the error.

3. The bottom part of the form illustrates a bug fix that was made in Reader 9.  Hidden fields are not supposed to affect layout.  The hidden property has always behaved properly in flowed subforms, but we discovered a bug in positioned containers.  Prior to Reader 9 a hidden field would expand the extent of a growable, positioned subform.  In this form, the subform is taller than it should be.

Update The Target Version

Change the target version (in the form properties/defaults tab) from "Acrobat and Adobe Reader 8.1 or later" to "Acrobat and Adobe Reader 9.0 and later".  Now when you preview the form you will see:

1. The target version is now 2.8, but the original version remains 2.6.

2. The calculation now works without error. The field displays the current date and time.

3. The growable subform renders at the same height as it did in Reader 8.1

Update the Original Version

Go into XML source mode and remove the originalXFAVersion processing instruction.  Now when you preview the form you will see:

1. Both target and original versions are 2.8

2. The calculation continues to work

3. The growable subform has collapsed to an extent that does not include the hidden field

The Deep End

Discovering Processing Instructions at Runtime

I was able to get at the processing instructions by loading an E4X XML object with the results of xfa.template.saveXML().  I don’t recommend doing this in a production form, since the result of calling saveXML() on the template can be a very, very large string — especially if there are images embedded in the template. 

If you wanted just the target version (the xml namespace), there is an easy way to get it with the ns property: xfa.template.ns. 

FormTargetVersion

I simplified a couple of details in order to make this easier to explain.  The full story is a little more complicated.

When Designer opens a form that has markup that is newer than the target version, it will automatically update the namespace of the XFA template.  So for example: Designer opens a 2.6 template with a target version of Reader 8.1.  It discovers syntax for turning on letter spacing, so it will automatically update the template to 2.8.  But the target version remains Reader 8.1.  This is because the UI for target version is actually driven by another processing instruction:

<?templateDesigner FormTargetVersion 26?>

Under most circumstances, Designer will keep the FormTargetVersion and the XFA template version synchronized.  If the template version is newer, then you will undoubtedly find a bunch of messages in your warnings tab.  This is Designer’s way of telling you that you’re using features that do not work in your target version.  Until you clean those up, your form will not work correctly in your target version.

Choosing Selected Changes

Changing the original version is pretty high level switch.  You either get all the bug fixes for a new release, or you get none.  In reality, there are qualifiers that allow you to preserve selected behaviours.  Suppose that in my sample form I wanted the bug fixes that came with Reader 9/XFA 2.8 except for the fix for hidden fields in positioned subforms.  In that case I can specify the processing instruction as:

<?originalXFAVersion http://www.xfa.org/schema/xfa-template/2.8/ v2.6-hiddenPositioned:1?>

The qualifier: v2.6-hiddenPositioned:1 tells Reader 9 to preserve the XFA 2.6 behaviour for hidden fields in positioned subforms.

Scripting bug fixes

Most of the time, I am happy with the default behaviour where original version behaviour is enforced by default.  However there is one exception.  My previous post described how the behaviour of strict scoping changed from 8.1 to 9.0.  The difference is that with strict scoping on in 8.1 we release JavaScript variables declared in script objects.  In 9.0 we preserve these variables.  If you are targeting 9.0 and make extensive use of script objects, make sure that you set your original version to 9.0 as well.  

Changing Default Behaviours

There have been a couple of times where we have changed behaviour and made the new behaviour the default without protecting it with the original version mechanism.  The most infamous was when we added "direct rendering" for dynamic forms.  In this case we had developed a way to dramatically improve the performance of dynamic forms.  We had to choose between turning the new behaviour (and the performance boost) on for all forms or just for those forms designed for 8.1 and later.  We chose to make the new behaviour the default.  If this caused problems, the form author could "opt out" by inserting an originalXFAVersion processing instruction.

This is described in the knowledge base article: http://kb.adobe.com/selfservice/viewContent.do?externalId=kb402409

Scope of JavaScript Objects

In a previous post, I went into the details of how XFA objects are integrated with the JavaScript interpreter.  One of the challenges in integrating the XFA object model with JavaScript has been determining how long to keep objects/variables in scope.  Keeping too many objects in scope leads to high memory usage and performance issues.  To get a sense of the size of the problem, bear in mind that it is not just fields and subforms that result in JavaScript objects, but their properties as well.  A script that accesses field.border.fill will create three JavaScript object references.  When you multiply the number of fields and subforms by the number of properties that can be accessed via script, you can see that there is potential to create an unwieldy number of XFA object references in JavaScript. 

JavaScript Variable Cleanup

We wanted to change the Reader to clean up JavaScript object references more aggressively, but we couldn’t change the default behaviour without breaking existing forms.  The solution was to expose strict scoping rules as an option in Designer:

image

This setting allows us to change form behaviours to clean up object references more frequently.  Strict scoping was first available for forms targeting Reader 8.1.  However, in 8.1, the cleanup was more aggressive than necessary.  Reader 8.1 released references to *all* JavaScript variables after each script execution.  In Reader 9 we eased off a bit and no longer release JavaScript variables declared in XFA Script Objects. 

As a result, when targeting forms for 8.1 with strict scoping on, you need to make all your XFA Script Objects re-entrant.  Do not rely on JavaScript variables sticking around.  You might not immediately realize that your variable references have been released, since they will continue to work until the JavaScript engine decides to perform its garbage collection.

Private Properties

I have seen customers defining script that looks like this:

MyField.myCustomProperty = "hello world";

What happens here is that the JavaScript interpreter does what is expected for any JavaScript object — it adds a new property to the object hash map.  This coding practise is pretty attractive when you consider the benefits of associating arbitrary JavaScript variables or functions with XFA objects. 

But there is a problem — associating new properties with JavaScript object references requires keeping those JavaScript objects in scope.  Since with strict scoping the JavaScript references are released, the private properties are also lost.  Rather than have people discover this the hard way by having variables mysteriously disappear at random intervals (during garbage collection), Reader 9 will issue an error when script attempts to assign an unknown property — when strict scoping is on.

Recommendation

The non-strict mode of variable management will continue to be supported, but I do not recommend relying on those behaviours.  I recommend strict scoping for a couple of reasons:

  1. Bug fixes:  We have branched our code internally to take one path for strict scoping (variable cleanup) and a separate path for non-strict scoping.  Going forward, only the strict scoping code path will get script bug fixes.
  2. Scalable Design Patterns:  When creating script-intensive large forms with non-strict scoping you are liable to eventually run into the performance issues of too many JavaScript objects.  My preference is (where possible) to use design patterns that work well with both small and large forms.

My preferred alternative is to use form variables on subforms and the desc or extras elements on fields as described in a previous post.