What to release?
Out of the growing list of tests written: what do I pick to release?
Mostly, I pick tests that show the biggest problems. That could be huge problems on one compiler, or significant problems on many compilers. Some tests are intended to provide the developer with information instead of showing problems – those need to be released, but aren’t as urgent. I also have some tests that don’t show problems on any compilers tested – they’re the lowest priority, but will eventually be released as well.
Why only release a few new tests at a time? Because it takes time to clean up the code, time to review the code, time to verify the results, etc. Because people downloading the benchmarks are more likely to read 3 or 4 new files at a time than 50. Also, because compiler vendors are more likely to dig into the details of a few new bugs at a time than when they’re hit with hundreds of new bugs to track down. At least, that’s the theory.
I test the benchmark code (over 90 files right now) on as many compilers and platforms as I have available. Then I create a quick summary of each test’s results across those compilers (sort of a “good, bad or ugly” ranking). From that I select a few candidates for further investigation. I need to verify that the files are testing what I think they are testing, and that the problems shown are real. Now I may need to fix problems, or just clean up the source code. Sometimes I need to profile the code and figure out why unexpected results are occurring. Next comes peer review of the code, and more cleanup. Sometimes reviewers point out redundancies (stuff that could easily be removed), or new things that need to be added to complete the picture of a given test area. And that can lead to more investigation and more review.
Eventually the code settles down and is ready to go out the door. Then I zip it up, post it on the website, and create a blog entry announcing the new release. Then I just hope that someone else is going to read it.