FormFeed

Knowing a bit more about how field values are represented in JavaScript could make a difference in how you write script.  Today I’ll give an overview of how field values are processed.   For starters, here is a sample form that will illustrate some of the points I make below.

But first, we need to be mindful that picture clauses (patterns) impact our field values.  A brief recap of the various picture clauses (patterns) we use:

• format pattern: Used to format field data for display
• edit pattern: Used to format field data when the user has set focus in a field and is about to edit a value.
• data pattern: Used to format field data when it is saved to XML.
• validation patten: Used for field validation — does not impact field value

With that background, let’s look at the properties available:

field.rawValue: the field value in its raw form. i.e. a format that has no patterns applied and a format that is consistent across locales and is suitable for calculations.

field.editValue: The field value when formatted with the edit pattern. If there is no edit pattern you will get a reasonable default — usually the rawValue, except in the case of a date field where you’ll get some locale-sensitive short date format.  If a field value cannot be formatted using the edit pattern, then field.editValue will revert to be the same as field.rawValue.

field.formattedValue: Same as field.editValue except using the display pattern.

Some miscellaneous facts about field values:

• field.editValue and field.formattedValue always return a string. If the field is null, these will return an empty string
  
  
• rawValue returns a JavaScript type corresponding to the kind of field.  e.g. A text field is a string, a numeric/decimal field is a number
  
  
• when a field is empty, rawValue will always be null
  
  
• rawValue is what will be stored in the XML by default (assuming no data pattern has been specified)
  
  
• There are two ways to check for a null value:
  • field.rawValue === null
  • field.isNull
• The rawValue of a date field does not have a type date.  We chose to represent dates as a string — in the form that they will be saved.  The format used for the string is YYYY-MM-DD.
  
  
• If you use JavaScript typeof to determine what kind of a value a field has, be aware that typeof(null) returns “object”

Knowing all this, there are some implications for the code you write:

• If you need a JavaScript Date from a date field, you can use this function:

  function ConvertDate(sDate) {
     if (sDate === null) {
        return null;
     }
     var parts = sDate.split("-");
     // Convert strings to numbers by putting them in math expressions
     // Convert month to a zero-based number
     return new Date(parts[0] * 1, parts[1] - 1, parts[2] * 1);
  };
              

• You should never code: field.rawValue.toString()
  This will result in a JavaScript error when the field is null.
  
  
• If you want an expression that always returns a string, never null, use field.editValue or field.formattedValue
  
  
• I often see code in the form:
  if (field.rawValue == null || field.rawValue == “”) { … }
  This is unnecessary.  Use one of the following:
  if (field.isNull) { … }
  if (field.rawValue === null) { … }
  
  
• If you prefer not to use a validation pattern, you can validate a field using the display picture.  Use this validation script:
  this.formattedValue !== this.rawValue;

Today’s blog is on a topic that I’ve intended to tackle for a long time. I finally got the opportunity to work on this after a customer contact.  The end goal is to add spreadsheet data to a PDF form.  Copy and paste aren’t enough — we need to preserve the formatting of the cell data.

The solution involves converting the spreadsheet to an xml format and then populating the form with the cell data from a nice friendly xml grammar.

To get consumable XML from within Excel, choose save-as “XML Spreadsheet 2003″.
You can read about this format at:

http://en.wikipedia.org/wiki/Microsoft_Office_XML_formats

and

http://msdn.microsoft.com/en-us/library/aa140066%28office.10%29.aspx

The spreadsheet grammar has all the cell and row properties needed for rendering: formatting, styling, widths, heights etc.  The only problem is that the format does not lend itself to populating a form via data binding. There are too many special cases to construct sensible binding expressions.  Instead, I’ve taken the approach of using JavaScript to parse the spreadsheet format and to turn it into corresponding formatted XFA fields.

Before I go further, I should mention that my implementation took a few shortcuts.  I didn’t implement the full XML Spreadsheet definition.  Some parts of it I implemented fairly poorly.  My intention wasn’t to provide a finished product, but rather to give you a sample you could build on.

Here is the sample form: spreadsheet.pdf.  I trolled around looking for interesting spreadsheets to test with and found the financial statement for Colgate.

To use the form, follow these steps (from Acrobat — not Reader):

1. Take an excel spreadsheet, save-as “XML Spreadsheet 2003″.
2. Click “Load XML Spreadsheet” to embed it as an attachment
3. Select the region to display
4. Click “Display”
5. Repeat steps 4,5 with other regions.  Or go back to step 2 with a different spreadsheet.

Note that Acrobat won’t allow you to do step 2 while the spreadsheet is open in Excel. And if you’ve picked a big spreadsheet it will be slow.  Go ahead and try it with a couple of the spreadsheets you have on your hard drive.

The structure of the form is simple:

• A Table subform that gets repeated for every worksheet. It has a conditional break so that every worksheet after the first one occurs on a new page.
• A repeating Row Subform that corresponds to a spreadsheet row
• A repeating Cell Subform with a Data field that repeats for each column
  

The only really tricky part of the layout is that the cells are explicitly positioned — not flowed left-to-right.  The reason for that is so that I could immitate the Z order of Excel.  When cells overflow they overwrite cells to their right. To make this work I needed the cells to be added in order from right-to-left so that the left-most cells are highest in Z-order.  I needed them to be positioned so that when a cell overflows it doesn’t push its neighbors over.

The script is another matter. There’s over 700 lines of script that are fairly complex.

It starts by prompting the user to add the spreadsheet as an attachment (with the doc.importDataObject() method).  Once the spreadsheet is stored as an attachment it extracts the contents into an E4X XML object.  Then most of the work is simply parsing the style and size information from the spreadsheet and applying those to the cells.

Note that the row and cell columns are unbound.  That means if you looked in the form data for the spreadsheet information, you wouldn’t find it.  But not to worry, the data is close by — available in the attached spreadsheet.

This is a follow-up to a previous blog entry that you probably should read first.

After doing the first version of the floating field editor, I tackled some issues/enhancements:

1. A bug in the script where if you tabbed out and tabbed back in, the editor stopped working.
2. Enforce constraints associated with the referenced fields
3. When the editor does not have focus, display the field values using the formatted values of the referenced fields
4. When the editor has focus, display the field values using the edit values of the referenced fields

I have updated the previous sample form — as well as the Editor fragment.

Enforcing Field Constraints

Since the floating fields are all presented inside a single text field, there was originally no constraints on any of the user input.  Now the form will look at the referenced fields and will restrict user input:

• Respect the max chars constraint of text fields (in the sample, they’re all limited to 10 characters)
• For numeric fields, limit input to valid numeric characters
• For choice list fields, limit input to the set of valid choices

Locale-sensitive Numeric Fields

When restricting the set of valid characters for numeric input, it is tempting to just go with the obvious set:
[0-9\-\.]  However for many locales, the radix (decimal) and minus symbols will be different.  In order to know which symbols to use, the form queries the locale definition.  You XML source peepers will be aware of the <localeSet> packet in your XDP files.  This has all the data for the locales that are explicitly referenced on the form. 

The symbols are stored in a format that looks like:

<localeSet xmlns="http://www.xfa.org/schema/xfa-locale-set/2.6/">
   <locale name="de_DE" desc="German (Germany)">
      <calendarSymbols name="gregorian"> … </calendarSymbols>
      <datePatterns> … </datePatterns>
      <timePatterns> … </timePatterns>
      <dateTimeSymbols>GjMtkHmsSEDFwWahKzZ</dateTimeSymbols>
      <numberPatterns> … </numberPatterns>
      <numberSymbols>
         <numberSymbol name="decimal">,</numberSymbol>
         <numberSymbol name="grouping">.</numberSymbol>
         <numberSymbol name="percent">%</numberSymbol>
         <numberSymbol name="minus">-</numberSymbol>
         <numberSymbol name="zero">0</numberSymbol>
      </numberSymbols>
      <currencySymbols> … </currencySymbols>
      <typefaces> … </typefaces>
    </locale>
    <locale> … </locale>
</localeSet>

I was able to extract the number symbols with this function:

function findLocaleNumberSymbols(vRefField) {
    var oSymbols = {decimal: ".",
                    minus: "-"
                   };
    var vLocale = localeSet[vRefField.locale];
    if (typeof(vLocale) !== "undefined") {
        var vNumberSymbols = vLocale["#numberSymbols"].nodes;

        for (var i = 0; i < vNumberSymbols.length; i++) {
            oSymbols[vNumberSymbols.item(i).name] =
                                     vNumberSymbols.item(i).value;
        }
    }
    return oSymbols;
}

Using the numeric symbols the form is able to more accurately restrict input for numeric fields.

Choice List Fields

The last field in the sample is a reference to a choice list with the American states.  Try out the editing experience here.  It’s pretty cool:

• Input characters are limited to the set of valid choices
• As soon as you type enough characters to uniquely identify a state, the rest of the input is completed automatically

Use Formatted and Edit Values

In the updated sample. the editing field now behaves like any other widget.  When you tab in, referenced field values display in their edit format.  When you tab out, the referenced fields display their formatted value.  In the sample you will notice that the currency and date values change when you tab in/out.  This is functionality that happens automatically on normal fields but had to be emulated in script for this sample.

You don’t have to read through all the script to figure out how it works, but it is worth noting that you can access a field value in three different ways:

• field.rawValue — the canonical value as it is stored in the data
• field.formattedValue — the value with the display pattern applied
• field.editValue — the value with the edit pattern applied

Note that if a format or edit picture/mask is not supplied, there are default patterns for numeric and date values.

Hook up via the enter event

Previously, the editor field tapped into the script object by delegating its initialize, change and exit events.  it now also needs to delegate the enter event:

form1.LetterEdit::change – (JavaScript, client)
scEditFF.handleChangeEvent();

form1.LetterEdit::enter – (JavaScript, client)
scEditFF.handleEnter();

form1.LetterEdit::exit – (JavaScript, client)
scEditFF.handleExit();

form1.LetterEdit::initialize – (JavaScript, client)
scEditFF.initialize(this, Letter, "#c0c0c0", 10);

Futures

There are more constraints that could be enforced, e.g. digits before/after decimal but those

I am often asked to take a look at forms that arrive from a variety of sources — customers, quality assurance, sales engineers etc.  Often one of the first things I do is have a look at some summary information about the template.   Having a unix background, I often save the form as an XDP and poke around with some grep commands e.g.: grep "<field" form.xdp | wc -l to find out how many fields are in the form.  But certain types of information are a little difficult to coax out with unix shell commands, so I set out to do something more user friendly.

Since the template definition, is completely accessible to JavaScript, I decided to design a form that would summarize the contents of another form. The result is today’s sample.  The form uses the Acrobat APIs to load and launch a file.  Once launched, we simply iterate over the contents of the template and generate a report from what we find.  The report consists of:

• meta data about the form: File Name, Creator, Template Version, Compatible Version, Strict Scoping setting, Static/Dynamic setting
• Enumerate the referenced fonts (including references found inside rich text fragments)
• List all linked and embedded images (for embedded images, indicate their base64-encoded size)
• Count instances of plain text vs. rich text
• Enumerate scripts, indicating which language (FormCalc or JavaScript) along with what context (event) and how many lines long
• Binding properties — summarize what kinds of data binding are in use
• Picture Formats – enumerate all the picture formats found in the form
• All other properties.  For example, if you want to see how many captions are on the form, note how many times the <caption> element appears.

To use the form, you need to be running Acrobat (not Reader).  Simply press the button, select a form and wait for the report.  Note that for large forms, this can take a few seconds (sometimes more than a few :-).  I’ve attached another sample that has enough of each kind of content to generate an interesting report.  Save this form to disk and select it from the reporter form.  You’ll see each category of the report populated with data.

I have also attached a report generated from a customer form with which I have enjoyed some quality time.

The script in this form is pretty complex.  But if you’re a good JavaScript programmer you could probably extend the script to capture other information that you find interesting.

For form calculations, validations and general scripting problems, our form designers can choose between using FormCalc and JavaScript. FormCalc was designed as an expression grammar targeting forms functionality — and designed to be easy to use for a novice form author familiar with expressions in Excel.  JavaScript on the other hand, is a much more general purpose and powerful scripting language.  Over time, form authors have gravitated more toward JavaScript, mainly because of its familiarity.  However, there are functionality gaps — specifically a set of built in functions that are available in FormCalc but not in JavaScript. 

Many of the built-in FormCalc functions can be easily mimicked in JavaScript.  It doesn’t take a rocket scientist to write a JavaScript script object function to imitate the FormCalc sum() function. However, there are some functions that are not so easily mimicked.  The most notable of these are the Parse() and Format() functions available in FormCalc.  Parse() and Format() are the entry points into the very rich picture clause processing functionality.  When you consider the inherent locale handling, date/time functionality, different calendars etc. it’s plain to see that you don’t want to do this in JavaScript.

But now we have a problem.  Many users are committed to JavaScript because they built frameworks for controlling validations and other form behaviours.  I did the same in the series of blog posts on validations (the most recent version of the framework was in the sample from this post).  The problem is that you cannot directly call FormCalc from JavaScript.  So it would seem that you can’t enjoy both the power and familiarity of JavaScript as well as the built-in functions of FormCalc.  Well, … actually you can.

There are two design patterns I’d like to cover:

1. Validating fields with picture clauses
2. General mechanism for calling FormCalc functions

Validating Fields with Picture Formats

Picture clause validations are a simple, declarative mechanism for form authors to ensure that input data complies with a specific format.  If the field value can successfully be formatted with the validation picture clause, then the field is considered valid.  e.g. if your validation picture clause is "Date{YYYYMMDD}", then the field is considered valid only if its value can be formatted as a date.  If you were to express this validation as a script, you could write this FormCalc expression:

form1.#subform[0].DateTimeField1::validate – (FormCalc, client)
format("YYYYMMDD", $) <> ""

Now the question is how to tap into this functionality from JavaScript.  The short answer:

1. Define a *display* picture format (do not a validation picture format)
2. Write a JavaScript validation script that returns true when the field.rawValue is different from field.formattedValue

Most often when you use a validation picture clause you also use a display picture.  In fact, there’s really no reason why these picture clauses need to be different.  Combine that knowledge with an understanding of how the field.formattedValue property works:  when a field value can be formatted using the display picture, field.formattedValue will return the result of the format operation.  If the format operation fails, field.formattedValue returns the same as field.rawValue.  So to find out if a field was formatted correctly, use this JavaScript validation:

form1.#subform[0].DateTimeField1::validate – (JavaScript, client)
this.rawValue != this.formattedValue;

Mechanism for Calling FormCalc Functions from JavaScript

The solution is to use the execCalculate() method to indirectly cross the bridge between JavaScript and FormCalc.  When you open the attached sample, you will find a subform called "fc" that holds a script object called "func".  "func" has a series of embedded JavaScript functions that mimic the FormCalc functions with the same name.  Each function populates form variables with the function name and input parameters.  It then calls execCalculate() on the fc subform and returns the resulting value:

FCfromJS.fc.#variables[0].func – (JavaScript, client)
function Format(vPicture, vValue)
{
    F.value = "Format";
    P1.value = vPicture;
    P2.value = vValue;
    this.execCalculate();
    return ResultString.value;
}

The subform calculation script looks like this:

 FCfromJS.fc::calculate – (FormCalc, client)
; execute the requested function based on the input
; request parameters
if (F == "WordNum") then
    ResultString = WordNum(P1)

elseif (F == "Parse") then
    ResultString = Parse(P1, P2)

elseif (F == "Format") then
    ResultString = Format(P1, P2)

elseif (F == "Uuid") then
    ResultString = Uuid(P1)

elseif (F == "UnitValue") then
    ResultString = UnitValue(P1, P2)

else
    ResultString = ""
endif

The field that wants to use the format() functionality has a simple calculate script:

FCfromJS.#subform[1].format.Result::calculate – (JavaScript, client)
fc.func.Format(PictureClause.rawValue, Value.rawValue);

This fc subform can easily be incorporated as a custom library object in Designer that can be dragged onto any form.  It should be pretty easy to follow the design pattern if you want to extend the sample and add other FormCalc functions.

One usage note — if you want to call these functions from initialization scripts, then be sure to place the fc subform at the beginning of your template.  This is necessary because the fc subform has an initialization script that creates the necessary form variables.  By placing this subform at the top of the form hierarchy, we’ll be certain that the fc initialization event fires before other initialization events.

August 12, 2009 Update

I have updated the sample form:

• Added access to the formcalc Get() function. 
• Added a version checking mechanism (see this post for details).
• The subform housing this functionality is now available as a downloadable fragment: scFormCalc.xdp
• The functionality is accessed using the subform "enter" event rather than the calculate.  The calculate event was introducing unwanted calculation dependencies.