C++ Performance

A few people have asked how to analyze the numbers that the benchmark code generates. Unfortunately, I don’t have any automated analysis yet. So I wrote up an example analysis of one test.
I’ll post more examples as I get time.

Example Analysis

Posted by Chris Cox at 8:52 PM | Permalink | Comments (0)

The initial release of my benchmark is available now from http://stlab.adobe.com/performance/.

Posted by Chris Cox at 9:34 PM | Permalink | Comments (1)

So far, I have written about 50 test files. I’m trying to explore the axes of my test space: what areas need to be tested, how deeply and specifically do they need to be tested, etc. The tests I’ve written cover C++ language concepts, simple idioms, common use idioms, runtime support, specification conformance, and compiler optimizations. Most of the tests have at least one major compiler performing badly. Of course, that is probably because I’m taking the tests from lists of things that I know compilers don’t do so well. On a positive note: all of the compilers I have tested are doing very well on dead code elimination.

I selected 3 of those tests to go out in the initial release — because people are more likely to read, understand, and discuss 20 pages of code than 2000. This will also give me a chance to get some verification on the approach and style, then clean up the remaining test files before sending those out.

Stepanov Abstraction

An expanded version of the original test, answering “what happens to performance when I wrap a value in curly braces”? Almost all compilers do well on the original summation tests, but they don’t do nearly so well on simple sort routines using the same abstractions.

Loop Invariant Code Motion

A test to see if the compiler will move loop invariant calculations out of the loop. This is something that a lot of developers assume that the compiler does for them. Unfortunately, the compilers I tested have a lot of room for improvement in this area.

Loop Unrolling

This is almost a straightforward test to see if compilers will correctly unroll loops to hide instruction latency. “Almost” because if I hand unrolled the loops it would be several hundred pages of source (I did it, it’s big). So, I used templates to do the unrolling — and found that some compilers have problems with such templates (which is yet another performance bug). Every compiler I’ve tested has a long way to go on correctly unrolling loops.

Machine Info

Plus one utility program to print out information about the compiler version, OS, and machine environment – because it’s nice to know which build of your compiler generated a particular report, and which of the 30 machines in your lab that it was run on.

A few weeks ago, I sent the initial release out to select compiler vendors for review. I received responses from 3 of the major compiler vendors, and 2 compiler teams have already found and fixed a couple of bugs based on my code (Yea!).

My next step is to setup a web site for downloads and send the initial release out to a larger audience of compiler writers.

Posted by Chris Cox at 10:31 PM | Permalink | Comments (2)