Testing cross-compiled ActionScript

After discussing optimizing and debugging cross-compiled ActionScript I would like to take a look at testing cross-compiled ActionScript. The short answer I can offer is: testing cross-compiled ActionScript is not much different from testing JavaScript. But of course there is more to it. There is always more…

 

Testing your patience

Be warned, this is a long post. I thought about splitting this post up as I have done in previous posts. But in this case I think everything should stay together. That said, let’s dive in!

In my opinion there are two different testing scenarios:

  1. Testing your cross-compiler.
  2. Testing client code.

In both scenarios we test JavaScript, but as we will see the goals and methods are different. Before we jump right in let me also point out that there are two types of testing, automated testing and manual (interactive) testing. Of course we prefer automated testing, because it is cheaper, less error prone, and can be integrated into Continuous Integration (CI) loops.

For running the Tamarin Acceptance Tests, which tests the cross-compiler, I use Chrome’s d8 (the debugger for V8) and for testing client code and running regression tests I use the default browser (usually Safari on OSX) with Google’s  js-test-driver.

 

Correctness versus Consistency

As mentioned above the goals for testing your cross-compiler and testing client code are different. When you test your cross-compiler you want to ensure that all language constructs in the source language are correctly cross-compiled to the target language. The term “correct” refers to preserving the internal consistency of the source code in the target code, not correctness in an abstract sense. I tried to explain this idea in a previous post, where in ActionScript and JavaScript you’ll get a mathematically incorrect result when adding 0.2 to 0.1:

// yields 0.30000000000000004, and not 0.3 in AS3 and JS.
trace( 0.2 + 0.1 );

I argued that in the context of cross-compiling one language into another it is very important to produce the same results in both worlds (here 0.30000000000000004) even if the results are incorrect. Our test suites need to follow that principle as well. As wrong as it may seem we should include a unit test that tests against 0.2 + 0.1 = 0.30000000000000004.

 

Missing Language Specification

Testing implies that you know the correct and expected results of your tests. When testing your cross-compiler the expected results are defined by the grammar of the source and target languages. What we need in our case are the language specifications of ActionScript and JavaScript. Herein lies a big problem. While the current JavaScript specification is summarized in ECMA-262 you won’t find a language specification for ActionScript. Just to be clear, a “language specification” is a document that describes the language grammar (usually formulated in Backus-Naur Form), so that developers can build a compiler for that language. A language specification is not a tutorial, FAQ, or online help about that language. For example Dart has a language spec, on the other hand Coffeescript’s langage reference does not qualify as a specification.

Neither does ActionScript’s online help. The closest to a language specification that Adobe currently offers is a section in the online help called  Learning ActionScript 3.0 . You might argue that an ActionScript language specification is not really necessary, because the syntax of ActionScript is by now well known and documented. I am afraid you cannot assume that everything about ActionScript is well documented. Just try find the exact syntax for Vectors in Learning ActionScript 3.0. I found this snippet under Array/Advanced Topics (you also find other bits and pieces under Basics of Arrays):

Note: You can use the technique described here to create a typed array. However, a better approach is to use a Vector object. A Vector instance is a true typed array, and provides performance and other improvements over the Array class or any subclass. The purpose of this discussion is to demonstrate how to create an Array subclass.

Even though it’s not really my fault, I truly regret that Adobe has never properly published a language specification for ActionScript. One could even argue that all computer languages can be divided into “hobby languages” and “professional languages”. The latter have language specifications.

 

Tamarin Acceptance Tests

The lack of a language specification for ActionScript has another nasty side effect. Without a language specification you also won’t find a corresponding language validation suite, which tests a compiler against the language specification. But that’s exactly what we need: a test suite that tells us that our cross-compiler complies to the language specification.

The next best thing to a language validation suite are currently the Tamarin Acceptance Tests. Those tests are grouped into different sections:

  • abcasm
  • as3
  • e4x
  • ecma3
  • misc
  • mmgc
  • mops
  • recursion
  • regress
  • spidermonkey
  • versioning.
Of those sections the most important ones are as3 and ecma3. The ecma3 section covers about 40,000 unit tests while as3 only has about 650 unit tests. I suspect the name “ecma3″ refers to the 3rd edition of ECMA-262. In reality, though, the tests seem to validate against the latest edition, which is the 5.1 edition of ECMA-262. Unfortunately we can’t really tell what the ecma3 folder is testing against, because there is no ActionScript language specification, and the Tamarin Acceptance Tests don’t claim to be the language validation suite.

 

Oberflächenverdoppelung

In fact, the Tamarin Acceptance Tests only seem to care about the behavior of the ActionScript VM in the Flash Player not changing in new versions of the Tamarin VM. The definition “ActionScript is what passes the Tamarin Acceptance Tests” does unfortunately not work as we will see in the following chapters.

There is a wonderful, but rarely used German philosophical term for this kind of circular reasoning (ActionScript is what passes the Tamarin Tests, which validate Tamarin VM results, which reflect ActionScript language rules). It is called “Oberflächenverdoppelung“, which I would translate as “surface duplication“. By creating tests that just confirm the results of an existing VM you only reiterate the knowledge of your own belief system. Instead of probing for truth you merely duplicate the surface of what you already know.

 

typeof Number

Assuming that the “ecma3″ section of the Tamarin Acceptance Tests validates against ECMA-262 (even the 3rd edition) there is a cluster of tests, which are in my opinion incorrect. To be precise, I believe ActionScript does not always comply to ECMA-262, and the Tamarin tests just bend to the status quo of ActionScript. For example, test e15_4_2_2_2, which is under ecma3/Array, would fail in every browser, because of this tiny difference between ActionScript 3 and JavaScript:

// ActionScript
typeof(new Number()) == "number"
// JavaScript
typeof(new Number()) == "object"

I asked our specialists here at Adobe why there is this difference and this was one of the replies:

In javascript, new Number(…) evaluates to an Object that wraps a number value. In AS3, new Number(…) return a number value. In both languages the Array constructor creates an Array of length 1 if given an object and an array of length n if given a number, where n is the value of the number.

But does ECMA-262 really allow both points of view? Is it allowed to return a number value for “new Number(…)” without fragmenting the language specified by ECMA-262? The ECMA-262 Language Overview chapter says (highlights added by me):

ECMAScript defines a collection of built-in objects that round out the definition of ECMAScript entities. These built-in objects include the global object, the Object object, the Function object, the Array object, the String object, the Boolean object, the Number object, the Math object, the Date object, the RegExp object, the JSON object, and the Error objects Error, EvalError, RangeError, ReferenceError, SyntaxError, TypeErrorand URIError.

Number is clearly defined as an Object and typeof(new Number()) = "object" seems to be the correct result according to ECMA-262. There is also a whole chapter on Number Objects in ECMA-262. But I couldn’t find any supporting arguments for typeof(new Number()) = "number".

As mentioned earlier it is important to preserve the behavior of the source language in the target language. But I decided that in this case, ActionScript (and the bending Tamarin tests) are incorrect, if you assume that “ecma3″ really does validate against ECMA-262. This would be similar to, say, getting 0.2 + 0.1 = 0.30000000000000004 in AS3 and 0.2 + 0.1 = 0.3 in JS. If you get a different result in AS3 and JS and JS reports the correct result I usually decide against AS3. You can’t rely on the Tamarin Acceptance Test. Their definition of correctness only preserves behavior exhibited in pre-existing C++ implementations of the ActionScript VM.

In either case, this was one of the rare exceptions where I decided to stick with ECMA-262 and not with what the Tamarin tests says is ActionScript (who really knows what ActionScript is without a language spec). As we just learned, one nice thing about language specifications is that there are almost no ambiguities (Number is an Object). I’ll get to the “almost” in a second.

 

function.toString()

There are many more Tamarin tests, that I find questionable and sometimes even absurd. This one is particularly annoying, because the test could have been phrased to serve a broader definition of what function.toString() may return. This code yields different results in ActionScript 3.0 and JavaScript in browsers:

(Array(1,2)).toString;
// Result in ActionScript
"function Function() {}"
// Result in JavaScript
"function toString() { [native code] }"

It turns out that ECMA-262 does not specify what function.toString() should exactly return. But we do know the retuned value is of type String and it should start with “function “. Instead of validating against  “function ” the Tamarin Acceptance Tests unfortunately check for what the Tamarin VM returns, which is “function Function() {}” and therefore triggers numerous false negatives in browser environment.

What shall we do about it? Shall we perhaps decide against ActionScript like we did for “typeof Number” above? This is a tough one, and I don’t think I am allowed to implement the JavaScript version by ignoring the ActionScript version as I did with “typeof Number”, because ECMA-262 is in regards to function.toString() somewhat ambiguous. (At least I try to be consistent in the way I am fragmenting the ActionScript language!)

So here is the compromise that I came up with. I added a new compiler option to the cross-compiler called “-js-extends-dom”, which defaults to “true”. If you compile the Tamarin tests, -js-extends-dom will be true and the cross-compiler will emit this ugly glue code at the top of your generated JavaScript:

// Additions to DOM classes necessary for passing Tamarin's acceptance tests.
// Please use -js-extend-dom=false if you want to prevent DOM core objects changes.
if( typeof(__global["Math"].NaN) != "number" )
{
  /**
  * @const
  * @type {function()}
  */
  var fnToString = function() { return "function Function() {}"; };
  /** @type {object} */
  var proto = {};
  // Additions to Math
  __global["Math"].NaN = NaN;
  // Additions to Array (see e15_4_1_1, e15_4_2_1_1, e15_4_2_1_3, e15_4_2_2_1)
  proto = __global["Array"].prototype;
  proto.toString.toString = fnToString;
  proto.join.toString = fnToString;
  proto.sort.toString = fnToString;
  proto.reverse.toString = fnToString;
  if( Object.defineProperty )
  {
    proto.unshift = (function ()
    {
      /** @type {function()} */
      var f = proto.unshift;
      return function() { return f.apply(this,arguments); };
    })();
    Object.defineProperty( proto, "unshift", {enumerable: false} );
  }
  // Additions to Error (see e15_11_2_1)
  proto = __global["Error"].prototype;
  proto.getStackTrace = proto.getStackTrace || function() { return null; };
  proto.toString = (function ()
  {
    /** @type {function()} */
    var f = proto.toString;
    return function ()
    {
      return (this.name == this.message || this.message == "") ?
      this.name : f.call(this);
    };
  })();
  proto = __global["Object"].prototype;
  // Additions to Object (see e15_11_2_1)
  proto.toString = (function ()
  {
    /** @type {function()} */
    var f = proto.toString;
    return function ()
    {
      return (this instanceof Error) ?
      ("[object " + this.name + "]") : f.call(this);
    };
  })();
}

 

The glue code above contains all the hacks for passing the Tamarin tests. In October 2011 my cross-compiler passed 40930 of the ecma3 Tamarin tests, which is about 95%.

In case you are wondering, of course I tell all my clients to run my cross-compiler with -js-extends-dom=false.

 

Validating versus Testing

In my test suite I differentiate between validating and testing. Validating unit tests generate JavaScript that I diff against a known, correct JavaScript version of the same test. All other tests are just run in the browser or in d8 as mentioned earlier. If the generated JavaScript is different during validation, I’ll get an error. If I accepted the newly generated JavaScript as the correct version then I would copy the new version to the saved version. If I detected a bug in the delta code of the newly generated JavaScript,  I would reject the build. This method has been very successful and helped me keeping my bug regression rate at an astonishing low level.

 

Testing Client Code

I have a few test apps that I always cross-compile as part of my unit tests. One of them is SpriteExample, which is one of the first examples that I ever cross-compiled to JavaScript. You can find that example at the bottom of the Flex online help for flash.display.Sprite .

The SpriteExample is actually not that trivial, because it uses constructor functions (which I very much dislike):

var sprite:Sprite = Sprite(event.target);

In order to get this example to work you would also have to solve event dispatching and drag and drop for Sprites. The drag and drop part can only be tested manually, though.

 

The End

That’s it, I hope! Now I have told you almost everything I know about cross-compiling ActionScript to JavaScript. There are still many topics that I could discuss in more detail. But after almost three month since my first post when I started this series, I think, we all deserve a break.

Next week, I am starting a new series about something completely different. Well, maybe not completely different…