Two! Four! Six! Eight! Numbers we appreciate!

I write every once in a while about Flash Player statistics, which I hope is interesting and useful for you. As product managers for Flash Player, Justin Everett-Church and I are responsible for managing and interpreting the data related to Flash Player. That data includes the penetration study, as well as player download and installation metrics. So, forgive me for bringing up old news, but I was out on vacation when the Firefox 3 Download Day happened. I was (quite happily) checked out enough to not really hear anything about it until I got back in the office this month. I understand a few Adobe folks blogged about how our Flash Player numbers were also world-record worthy, and a lot of different download numbers often show up in blogs and press without any context of time period. Is the number of daily downloads a useful metric? What do these numbers really say?

Don’t get me wrong — I love data, and download numbers ARE interesting. But, as with any data, you need to figure out what the numbers tell you and why they are important. Adobe tends to talk about the install rate of Flash Player quite often, although it may not be the right number for people to focus on since it doesn’t give you context, and you have to take our word for it. For Firefox, the download number is interesting because they wanted to beat their numbers for the previous release, and to generate buzz for the current release by submitting it for a world record — which was verified by a third party.

It’s interesting to note that for Adobe, the number that is quoted is an “install” and not a “download” number. We’ve never said how many downloads happen a day because it is a ridiculously large number AND we know that it’s not that useful metric because those successful downloads don’t all turn into successful installations. In July 2008, successful downloads averaged about 33 million per day, and successful installs averaged around 18 million per day. That seems like a big drop, but consider that ActiveX was about 80% of our installs that month and when you visit a page that triggers the ActiveX install experience the installer is downloaded to the machine before the security warning dialog appears. The user might say “no thanks” to the security warning dialog, and refreshing the page or visiting another page that requires a newer version may download the installer again.

How do we monitor Flash Player download and installation numbers?

We have an internal dashboard application to track the general “health” of Flash Player downloads and installs, and it can chart the data daily, weekly, monthly, etc. It uses XML feeds of our server log data from Akamai, our current CDN. The dashboard is useful in helping us 1) to understand our traffic so we can try to optimize things like install success rate, and catch problems with our installers or CDN delivery in a matter of days, and 2) get a sense of where our penetration might be in the penetration study in the next wave. The penetration study is only run once a quarter, and it’s hard to wait three months to know where we’re at.

For “Player downloads” we count the attempted and successful downloads for all the player installers we post to the CDN (based on the related HTTP status codes), such as “swflash.cab” and “install_flash_player.exe” (note these numbers aren’t unique.) We also count the number of installs, which is the sum of requests for a small text file that a newly installed player requests the first time it is launched in the browser. That number is used as an estimate – we’ve had releases where the text file request wasn’t implemented or working on certain platforms, and since it is only requested once there are a number of reasons why it might not make it all the way to the server. The install number wasn’t ever intended to be a marketing point, although it was an exciting number for us to talk about when we realized that our install average was 8 million a day shortly after Flash Player 9 first launched. The daily average has gone up since then, most likely due to increasing penetration of broadband and improvements at the CDN. And it can spike depending on whether there is a hot new site that is sweeping the web, or dip if we’ve got a CDN or installer problem.

The point is, I know it can be confusing to hear about numbers like 8, 12, 14 or 25 million daily installs when there isn’t any context in which to interpret the meaning. We don’t yet have an internal “standard” for the install number that we are quoting as a company, such as “average daily installs for the past month” or “average daily installs since the last release”, etc. Not that it really matters. The intent behind stating these VERY BIG numbers is to say something about demand. That is a big reason why we have the penetration study, and have been tracking it since Flash Player 3. I’ve talked in the past about the general aspects of the penetration data we post, and how it may or may not apply when you get down to your particular audience. As with any statistics, you should understand the methodology behind the numbers. But the good thing about that number is it is something you or another third party like Forrester, can independently test or verify — which makes it the more interesting and important number for Flash Player.