Posts in Category "statistics"

Two! Four! Six! Eight! Numbers we appreciate!

I write every once in a while about Flash Player statistics, which I hope is interesting and useful for you. As product managers for Flash Player, Justin Everett-Church and I are responsible for managing and interpreting the data related to Flash Player. That data includes the penetration study, as well as player download and installation metrics. So, forgive me for bringing up old news, but I was out on vacation when the Firefox 3 Download Day happened. I was (quite happily) checked out enough to not really hear anything about it until I got back in the office this month. I understand a few Adobe folks blogged about how our Flash Player numbers were also world-record worthy, and a lot of different download numbers often show up in blogs and press without any context of time period. Is the number of daily downloads a useful metric? What do these numbers really say?

Don’t get me wrong — I love data, and download numbers ARE interesting. But, as with any data, you need to figure out what the numbers tell you and why they are important. Adobe tends to talk about the install rate of Flash Player quite often, although it may not be the right number for people to focus on since it doesn’t give you context, and you have to take our word for it. For Firefox, the download number is interesting because they wanted to beat their numbers for the previous release, and to generate buzz for the current release by submitting it for a world record — which was verified by a third party.

It’s interesting to note that for Adobe, the number that is quoted is an “install” and not a “download” number. We’ve never said how many downloads happen a day because it is a ridiculously large number AND we know that it’s not that useful metric because those successful downloads don’t all turn into successful installations. In July 2008, successful downloads averaged about 33 million per day, and successful installs averaged around 18 million per day. That seems like a big drop, but consider that ActiveX was about 80% of our installs that month and when you visit a page that triggers the ActiveX install experience the installer is downloaded to the machine before the security warning dialog appears. The user might say “no thanks” to the security warning dialog, and refreshing the page or visiting another page that requires a newer version may download the installer again.

How do we monitor Flash Player download and installation numbers?

We have an internal dashboard application to track the general “health” of Flash Player downloads and installs, and it can chart the data daily, weekly, monthly, etc. It uses XML feeds of our server log data from Akamai, our current CDN. The dashboard is useful in helping us 1) to understand our traffic so we can try to optimize things like install success rate, and catch problems with our installers or CDN delivery in a matter of days, and 2) get a sense of where our penetration might be in the penetration study in the next wave. The penetration study is only run once a quarter, and it’s hard to wait three months to know where we’re at.

For “Player downloads” we count the attempted and successful downloads for all the player installers we post to the CDN (based on the related HTTP status codes), such as “swflash.cab” and “install_flash_player.exe” (note these numbers aren’t unique.) We also count the number of installs, which is the sum of requests for a small text file that a newly installed player requests the first time it is launched in the browser. That number is used as an estimate – we’ve had releases where the text file request wasn’t implemented or working on certain platforms, and since it is only requested once there are a number of reasons why it might not make it all the way to the server. The install number wasn’t ever intended to be a marketing point, although it was an exciting number for us to talk about when we realized that our install average was 8 million a day shortly after Flash Player 9 first launched. The daily average has gone up since then, most likely due to increasing penetration of broadband and improvements at the CDN. And it can spike depending on whether there is a hot new site that is sweeping the web, or dip if we’ve got a CDN or installer problem.

The point is, I know it can be confusing to hear about numbers like 8, 12, 14 or 25 million daily installs when there isn’t any context in which to interpret the meaning. We don’t yet have an internal “standard” for the install number that we are quoting as a company, such as “average daily installs for the past month” or “average daily installs since the last release”, etc. Not that it really matters. The intent behind stating these VERY BIG numbers is to say something about demand. That is a big reason why we have the penetration study, and have been tracking it since Flash Player 3. I’ve talked in the past about the general aspects of the penetration data we post, and how it may or may not apply when you get down to your particular audience. As with any statistics, you should understand the methodology behind the numbers. But the good thing about that number is it is something you or another third party like Forrester, can independently test or verify — which makes it the more interesting and important number for Flash Player.

Flash Player 9 @ 95.7% in December study

It’s been up for a few weeks now, but the quarterly study results for December are now available. Note that this wave includes emerging markets, which are every other quarter.

The first row is how we present the data on the website. It tells you the percentage of people that can view a SWF of that version. The second row, is the % of the population with that player version installed (the delta between v and v+1).

December 2007 – Mature Markets v6 v7 v8 v9
% able to view content by SWF version 98.8% 98.8% 98.3% 95.7%
Install base by version 0.0% 0.5% 2.6% 95.7%
December 2007 – Emerging Markets v6 v7 v8 v9
% able to view content by SWF version 97.4% 97.3% 95.5% 93.3%
Install base by version 0.1% 1.8% 2.2% 93.3%

Flash Player 9 @ 93.3% in September penetration study

The September stats are now up on Adobe.com. Thanks to Millward-Brown and our web team for giving us a quick turnaround so we could get these live before Adobe MAX.

The first row is how we present the data on the website. It tells you the percentage of people that can view a SWF of that version. The second row, is the % of the population with that player version installed (the delta between v and v+1).

September 2007 – Mature Markets v6 v7 v8 v9
% able to view content by SWF version 99.1% 99.1% 98.4% 93.3%
Install base by version 0.0%* 0.7% 5.1% 93.3%

* Looks like people finally retired their old machines and are upgrading to more current hardware :-)

Need a way to project penetration for player releases? Read on…

This is great question, and I’m happy to say it is not a difficult one for me to answer. We have a lot of historical data on new player releases, and can validate that it follows a fairly consistent path. This is how we have been able to state that historically, new releases took 12 months to reach 80% — but with Flash Player 8 and Flash Player 9, it now takes 9 months to reach 80% and 12 months to reach 90%.

We have our data, but also we hear anecdotal stats from our customers that validate the penetration study we run every quarter. I always try to remind folks that the Millward Brown study is sampling to mirror an general internet-connected PC population, so your site’s demographics are likely to differ. But for large sites with broad reach and a broad customer base, the numbers you get should be fairly consistent with the numbers we publish. Use our numbers as a general guide on the penetration of Flash Player, but the best way for you to know what the penetration is for YOUR particular audience is to check it yourself.

So here is a basic guideline that you can use to project penetration of a particular release:
@3 months = 30 – 40%
@6 months = 55 – 65
@9 months = 80 – 85%
@12 months = 90+%

So working with the release date, you can figure out the penetration of a specific version and later. For example, Flash Player 9r28 with the full screen feature went out in November. That’s about 9 months since initial release. Therefore, PCs that have 9r28 or a later version should be around 80%. Check your site, and see how that matches up for your audience.

Again, I said this is the “typical” trend. If every major site in the world decided to start pushing Moviestar the day we launch, the penetration rate would be faster in the first two periods. The trend tends to flatten out once it hits 80% on it’s way to 90% and beyond.

To find out about the player dot release enhancements, fixes and versions, visit the release notes here. Unfortunately, they don’t have release dates (maybe a good thing to add moving forward, now that I mention it) but I believe I’ve blogged availability for every release so far for Flash Player 9.

Flash Player penetration study page here. Note, the June data is horribly late and I apologize for that. We’re working on getting it posted. The Flash Player 9 penetration was at 90.3% for mature markets, v8 was at 98.5%, and v6 and v7 were at 99.3%. Putting it in other terms, it means:

90.3% of PCs can view SWF9 or below
98.5% of PCs can view SWF8 or below
99.3% of PCs can view SWF 7 or below, and we can drop testing of SWF6 moving forward.

And if you don’t like that presentation, the above can also be read as:
90.3% of PCs have Flash Player 9 installed
8.2% of PCs have Flash Player 8 installed
0.8% of PCs have Flash Player 7 installed

Fun with numbers :-) Hope this helps.

Flash Player 9 at 83.4% in the March 2007 penetration study

With all the CS3 and NAB announcements going out, the March 2007 penetration study results didn’t quite make the website launch. So, I thought I’d go ahead and give you a sneak peek. We made it past the 80% mark in about 9 months, just as we did with Flash Player 8 penetration. The trend is following the v8 ramp up, so we should expect 90% or so by the June study.

The first row is how we usually present the data on the website. It tells you the percentage of people that can view a SWF of that version. I added the second row, which is the % of the population with that player version installed (the delta between v and v+1).

March 2007 – Mature Markets v6 v7 v8 v9
% able to view content by SWF version 98.7% 98.1% 96.3% 83.4%
Install base by version 0.6% 1.8% 12.9% 83.4%

Note: “Mature markets” include US, Canada, UK, Germany, France, and Japan. The “Emerging Markets” are every other wave so they’ll be in the June study.

The site should be updated in the next week or so.

Flash Player stats available for September 2006

Flash Player 9 is at 35.9% penetration in the mature markets, three months after release. This is on track with the Flash Player 8 ramp up and the download rates are the same to slightly up, so you can safely use the v8 ramp up as the projection for v9.

To make a fair comparison between this stat and the first quarter after v8 released, there are two things to note:
1) The first penetration stat for v8 was closer to four months after launch because we had to re-run the December 2005 study in early January 2006 due to a data issue with the original study.
2) The Dec 2005 number from NPD is for the US. So the first v8 stat was 45.2% in the US (Dec 2005)and the first stat for v9 is 40.3% (Sept 2006) in the US. Adjusting for the actual length of time for each and +/- the margin of error, v9 is the same to slightly faster.

As a reminder, we are running the emerging markets study every other quarter and the main number we are reporting is the mature markets results. We have also decided to drop Flash Player 5 from the study, because the v5 and v6 numbers were very similar.

Flash Player 8 is at 86% (June 2006), and info about the methodology update

The latest penetration study has been published, and Flash Player 8 is at 86% penetration. That’s an amazing number considering it only took 9 months to cross the 80% mark — something that historically has taken 12 months to reach. Flash Player 9 launched after the survey was sent out, so we won’t have stats for v9 until the September study. But the download rates are the same as Flash Player 8 so you can use history as your guide. ;-)

Another important aspect of the June study is that we decided to improve upon the methodology we are using, since it has been the same since it was first started in 1999. A new company, Millward Brown, was selected for the updated study. The study itself has not changed; we still survey users with a set of media and content files and ask them “Can you see this fish?” (click here to see a sample of the survey, also linked from the methodology page). But we wanted to improve upon the worldwide sampling to get a better representation for the study. In the new sample, we have broadened it to a multi-country study and based on the World Internet Usage and Population Statistics, the surveyed countries collectively represent 72% of the world’s internet connected population as of June 30, 2006. We will now be reporting a weighted average for “mature markets” as our worldwide number. The “emerging markets” study will be conducted twice a year.

Obviously, making any changes to the methodology does introduce potential continuity issues with the data. We actually did run both versions for the last two quarters to check the results, and the results were very similar and within the margin of error. But, if you really need to compare apples to apples this is still possible with the latest data. In the previous studies with NPD, the primary sample was in the U.S. so you can still trend the US numbers across the studies.

FAQs for other popular questions that popup whenever we release a study:

Q: What is the penetration in corporations/enterprise?
A: Unfortunately, we don’t really have this information. The reason we don’t test for this in a penetration study is that, frankly, it’s not easy to do. The reason is that with “internet households” or “internet population” there are forecasts, data, and census that can be used to understand how to select a representative sample. That doesn’t really exist for companies. The study does ask the user if they are on a home or work computer, but again, since our sample isn’t designed to be representative of enterprise internet access we can’t project that data into percentage penetration.

Q: My company just did a test on our site to see the version penetration of our visitors. Why is our penetration higher/lower than what this study reports?
A: The penetration of Flash for your target audience is likely to differ from the results reported by this study, because the study is a conducted on a representative sample of the world internet connected population. For example, if you have a website that is primarly targeted to young kids who are likely to be on hand-me down PCs and aren’t allowed to install things — you will probably see a lower penetration of the latest Flash Player on your site. If you are targeting the MySpace/YouTube generation — chances are you are seeing 80+% of your site visitors with the latest version. The study provides guidance on the overall penetration of each technology, but applying it to your specific case may require some thought about whether the number is appropriate for your use as-is or if it is one input into your decision about updating versions depending on what you know about your target audience.

Q: Now that Flash Player 9 is out, does that mean the Flash Player 8 penetration ends at 86%?
A: Nope, that part is a bit confusing based on the way it is presented. Since the study is asking the user if they can see a set of files with the plug-ins they currently have installed on their system, what we are truly testing in each question is whether the user can see a SWF of version 5, 6, etc. When we say that Flash Player 5 has 97.3% penetration, this is the same as stating “97.3% of people can view a version 5 SWF.” If you wanted to state the percentage of users that actually have Flash Player 5 installed, it would be the delta between those that can see only SWF6 and up and those that can see SWF5 and up OR [97.3% – 97.2%] = 0.1%! Note that with weighted averages and margins of error, this calculation doesn’t work with some of the data. But you get the general idea of how to get to the individual player penetration.

April 2006 Penetration Stats available — Flash Player 8 at 69.3%

Updated NPD results are up, and Flash Player tested at 69.3% penetration for the April 2006 study.

The study tests the percent of people that can view of SWF of a specific version. So, if you want to know what the penetration of a specific version, it is the delta between two columns.

For example: the percent of people who can view (up to) v8 content is 69.3%, which is the same as saying they have Flash Player 8 installed. The percent of people who can view up to v7 content is 94.8%, which is the same as saying (94.8% – 69.3% =) 25.5% have Flash Player 7 installed. And so on.

Unfortunately, due to some data collection issues the Shockwave Player data is not available for this quarter. We will resume the stats with next quarter’s study.