Example Analysis

Example Analysis

I selected 3 of those tests to go out in the initial release — because people are more likely to read, understand, and discuss 20 pages of code than 2000. This will also give me a chance to get some verification on the approach and style, then clean up the remaining test files before sending those out.

Stepanov Abstraction

An expanded version of the original test, answering “what happens to performance when I wrap a value in curly braces”? Almost all compilers do well on the original summation tests, but they don’t do nearly so well on simple sort routines using the same abstractions.

Loop Invariant Code Motion

A test to see if the compiler will move loop invariant calculations out of the loop. This is something that a lot of developers assume that the compiler does for them. Unfortunately, the compilers I tested have a lot of room for improvement in this area.

Loop Unrolling

This is almost a straightforward test to see if compilers will correctly unroll loops to hide instruction latency. “Almost” because if I hand unrolled the loops it would be several hundred pages of source (I did it, it’s big). So, I used templates to do the unrolling — and found that some compilers have problems with such templates (which is yet another performance bug). Every compiler I’ve tested has a long way to go on correctly unrolling loops.

Machine Info

Plus one utility program to print out information about the compiler version, OS, and machine environment - because it’s nice to know which build of your compiler generated a particular report, and which of the 30 machines in your lab that it was run on.

A few weeks ago, I sent the initial release out to select compiler vendors for review. I received responses from 3 of the major compiler vendors, and 2 compiler teams have already found and fixed a couple of bugs based on my code (Yea!).

My next step is to setup a web site for downloads and send the initial release out to a larger audience of compiler writers.

Over the years, I've collected a lot of code showing particular problems we've found, and solutions we have found. But that code has only been shared inside Adobe or with compiler vendors with whom Adobe has a Non Disclosure Agreement (NDA).

Several months back, I was writing up some notes on the concept of abstraction in computer science and revisited Alex Stepanov's abstraction penalty benchmark. Alex wrote this benchmark at a time (circa 1994) when C++ compilers weren't doing a great job of optimizing even basic C++ abstraction. Within a few years of the benchmark's release, most compiler vendors had identified and fixed their performance problems that the benchmark exposed.

I wondered if the compiler writers had really done the job well or taken shortcuts, and what the penalties would be for using more complex C++ abstraction. So, I asked Alex (who happens to work just down the hall from me) if anyone had updated his benchmark or was doing active research on it and related penalties. Alex said "No, I don't know of anyone working on that. Why don't you take it over?".

Alex then proceeded to convince me that we really need better benchmarks - ones that don't try to sum up the whole world in one number, but tests that probe specific areas and patterns. They should answer the questions such as: "What is the penalty for using X?" or "Does my compiler perform optimization Y?" Alex argued that such a set of benchmarks, if released as open source (without all the NDA hassles), would benefit all of Adobe's applications by improving the compilers, and benefit all C++ users the same way. He said that it was almost the same thing I had been doing with our internal code, but with wider exposure, and hopefully a few more people contributing. Of course, then we had to convince my manager -- but the idea was good (and backed by several senior researchers who had joined the discussion), so she agreed that I could spend part of my time creating benchmarks.

Now I've got a blank slate, a lot of historical code that can't go out as-is, a long list of complaints about compilers, a longer list of suspicions about compilers, and a lot of things that I've heard other people claim or complain about C++ and compilers (much of which I know not to be true).

Where should I start?