Using AIRBench to Test AIR Across Android Phones

I recently uploaded an application called AIRBench to the Android Market which is designed to compare Adobe AIR performance across devices. We test AIR across as many different devices with as many different configurations as we can, of course, but there are so many different permutations out there now that we decided to release an app to help us understand how different devices compare to each other.

AIRBench performs two types of tests: capability and performance. Capability tests just ensure that a particular API works, and performance tests determine how well your device performs a particular task. Below is a breakdown of all the tests included in the first version of AIRBench:

Capability Tests

  1. Camera: calls Camera.getCamera() and displays the video feed.
  2. CameraUIImageTest: Makes sure you can take a picture using the CameraUI class.
  3. CameraUIVideoTest: Makes sure you can record video with the CameraUI class.
  4. CameraRollAddTest: Adds an image to the camera roll.
  5. CameraRollBrowseTest: Tests the ability for the user to browse the camera roll and choose an image.
  6. GeolocationTest: Makes sure geolocation data can discovered.
  7. AccelerometerTest: Makes sure the Accelerometer APIs work.
  8. MultitouchTest: Has the user place as many fingers on the screen as Multitouch.maxTouchPoints reports, and makes sure those two numbers are equal.
  9. MicrophoneTest: Makes sure sound can be detected from the microphone. (Watch out for interference on this one.)
  10. StageWebViewTest: Loads an HTML page into a StageWebView.

Performance Tests

  1. XMLTest: Parses a pretty big XML file (74K) and reports the time. (It’s my Google Reader feed copied several times.)
  2. MemoryAllocationTest: Creates a ByteArray and sets it length to 8MB, then reports the amount of time it took.
  3. FileAccessTest: Creates a file, writes 1MB of data, reads the file, then deletes the file. Reports the total time.
  4. SQLTest: Creates a new database, creates a table with 5 columns, inserts 500 rows of data, runs a select statement, deletes all the data, and closes the connection. Reports the total time.
  5. StringTest: Runs 9 different string operations on a very big string and reports the total time.
  6. ComputationTest: Hashes a 49K image using SHA256 (to test bitwise operations), then calculates sunrise and sunset times for 10 different locations (Adobe offices) 100 times (to test math operations).
  7. RenderingTest: Animates a square, changing the rotation, scale, and alpha at 60 frames per second. Measures the FPS after a few seconds and reports the number of frames dropped. (Note that this version of the test does not use hardware acceleration — that will be tested in a future test.)

Tests are versioned, so I’ll release new and better tests as we add new capabilities to the runtime, and as I have time to make improvements. Results can be submitted to Adobe (completely anonymously), and once I get back from MAX, I’ll write a series of reports for everyone to check out at airbenchmark.com.

If you want to help contribute data to the AIRBench database, just search for AIRBench in the Android Market, run through the series of tests, and submit the results. AIRBench serves a second purpose by providing a simple demonstration of how all the device-specific APIs work, so if you’re interested in doing some AIR mobile development, check out the code.

9 Responses to Using AIRBench to Test AIR Across Android Phones

  1. Rich Morey says:

    What is the best way to test an application I am building across devices? Does Adobe have a set of pre-configured emulators that can be run through the Android emulator that is part of the Android SDK?

  2. Hi,
    great app, and great that source is shared too, I have only one issue with it, I can’t close it, I have to go to the “manage applications” and force it to close.
    Otherwise is a gem.

  3. Steve Broome says:

    Nice work, testing out now on a Nexus One.

  4. Nice one!
    What did you do to have the option ‘Copy to SD Card’. How can this be accomplished? Is this an option in the -app.xml or in the code?

    Cheers,

    Sidney

  5. Pingback: The Importance of Scaling Images on Mobile Devices « Christian Cantrell

  6. Pingback: How to Use CameraUI in a Cross-platform Way « Christian Cantrell

  7. With the release of AIR 2.6, would be an interesting AIRBench for iOS devices.
    Cantrell will be coming out?

  8. Christian Cantrell says:

    @Jose: I’ve actually run AIRBench on iOS devices. The only issue is that Flex is not fully supported yet; it works, but the performance isn’t where it needs to be with the version of Flex I used (it’s several months old). I’ll release it when Flex is ready.

  9. Thanks @Cantrell.
    Do you know any alternative to debug with IOS simulators?