« Some Background | Main | Release now available »

Preparing the initial release

So far, I have written about 50 test files. I’m trying to explore the axes of my test space: what areas need to be tested, how deeply and specifically do they need to be tested, etc. The tests I’ve written cover C++ language concepts, simple idioms, common use idioms, runtime support, specification conformance, and compiler optimizations. Most of the tests have at least one major compiler performing badly. Of course, that is probably because I’m taking the tests from lists of things that I know compilers don’t do so well. On a positive note: all of the compilers I have tested are doing very well on dead code elimination.

I selected 3 of those tests to go out in the initial release — because people are more likely to read, understand, and discuss 20 pages of code than 2000. This will also give me a chance to get some verification on the approach and style, then clean up the remaining test files before sending those out.

Stepanov Abstraction
An expanded version of the original test, answering “what happens to performance when I wrap a value in curly braces”? Almost all compilers do well on the original summation tests, but they don’t do nearly so well on simple sort routines using the same abstractions.

Loop Invariant Code Motion
A test to see if the compiler will move loop invariant calculations out of the loop. This is something that a lot of developers assume that the compiler does for them. Unfortunately, the compilers I tested have a lot of room for improvement in this area.

Loop Unrolling
This is almost a straightforward test to see if compilers will correctly unroll loops to hide instruction latency. “Almost” because if I hand unrolled the loops it would be several hundred pages of source (I did it, it’s big). So, I used templates to do the unrolling — and found that some compilers have problems with such templates (which is yet another performance bug). Every compiler I’ve tested has a long way to go on correctly unrolling loops.

Machine Info
Plus one utility program to print out information about the compiler version, OS, and machine environment – because it’s nice to know which build of your compiler generated a particular report, and which of the 30 machines in your lab that it was run on.

A few weeks ago, I sent the initial release out to select compiler vendors for review. I received responses from 3 of the major compiler vendors, and 2 compiler teams have already found and fixed a couple of bugs based on my code (Yea!).

My next step is to setup a web site for downloads and send the initial release out to a larger audience of compiler writers.


    Very interesting. I did indeed assume that compilers these days did a great job at invariant code motion and loop unrolling.

    Are you planning to publish test results for common compilers/platforms or are you leaving that up to others?

    Yes, I eventually plan to have results and some analysis posted for common compilers and platforms.

    My primary concerns at the moment are increasing portability of the code, improving documentation, and getting feedback on the style of the benchmarks. In other words: do they compile, do you understand what they’re about and what they’re reporting, and are they useful?

    I’m still trying to figure out how to post results in a way that makes them useful. Then I need to figure out a way to post the analysis of each result in a way that makes those useful.

Sorry, the comment form is closed at this time.

Copyright © 2012 Adobe Systems Incorporated. All rights reserved.
Terms of Use | Privacy Policy and Cookies (Updated)