In a crisis, communication is key

Peleus here. I focus most of my energy working with the Flash Player and AIR teams. The first half of the year had been fairly uneventful for Flash Player with only a small number of responsibly disclosed vulnerabilities reported over the last few months. The break was welcome and allowed us to focus on the more strategic security tasks that we don’t always blog about, but are still important parts of the Adobe Secure Product Lifecycle.
Unfortunately, nothing lasts forever and the last few weeks have been very busy for the Adobe Secure Software Engineering Team. We had to snap back into action on July 10th to handle the first of two different externally reported vulnerabilities that we ended up patching in Flash Player on July 30th.  First, Microsoft notified us of a flaw in their ATL library that had the potential to affect many of Adobe’s products.  The entire ASSET team mobilized to quickly identify which of Adobe’s over 200 products might be vulnerable.  Fortunately for us, the MSVR team at Microsoft was well organized. They were able to share info early on that helped to speed the triage effort inside Adobe.  We were able to identify and fix two vulnerable products (Shockwave and Flash Player) and confirm that many other widely distributed products like Adobe Reader and Connect Pro were NOT vulnerable. We also provided continual feedback into the Microsoft process that they could then, in turn, share with other partners.
Once the triage process for the ATL vulnerability was well under way we received a sample of a new attack in the wild against the Flash Player library within Reader. Although there are reports Adobe had known about this bug for eight months, the reality is that the July 16th report to the Product Security Incident Response Team (PSIRT) was the first time Adobe evaluated the bug as a security issue. The 12/31/08 bug was not caught as a security bug, so our usual Incident Response process was never initiated.
Flash is at the core of many of Adobe’s flagship products and this simultaneous patch effort required coordination between Reader, Flash Player and the AIR development teams.  This is no small task when you consider that we need to distribute reliable patches to over 90% of the Internet and support multiple versions of each product across multiple versions of the Windows, Macintosh and UNIX operating systems. Granted, we have been doing this for some time so we have automation in place to make the technical processes efficient and accurate. However, we still needed the Flash team to coordinate the creation of the patch and provide guidance to the Reader and AIR teams on adopting the solution. One mis-step in coordination could mean that the testing process starts all over again for each of the products resulting in a delay in the patch for our customers.
Since some of these products are among the most widely distributed software on Earth we used every available resource to get software updates out as soon as we could. We managed to execute a plan for updating Shockwave, Flash Player, AIR, Adobe Reader and Acrobat, all before the end of the month.
In addition to the tremendous amount of internal coordination required, PSIRT also managed the external aspects of the Adobe response. This included communicating with CERTs and other partners in the security ecosystem, creating CVE’s, issuing bulletins and advisories, and responding to customers and media inquiries.  Each notification requires attention to detail so that we provide useful information, but not so much that we put more customers at risk.  As with all major incidents, PSIRT is at the center of balancing the needs of Adobe’s customers, security researchers, product teams, partners and Adobe as a whole.
I recently read a
ZDNet Zero Day blog by George Stathakopoulos talking about coordination within the security community and it’s a common theme in my conversations with friends in the security community.  Our coordination with Microsoft on the ATL header vulnerability was definitely key to being ready in time for their release date.  In addition, coordination within Adobe was critical to shipping four products within the same week, all while balancing the needs of both external and internal stakeholders.  We hosted daily security meetings that leveraged Adobe Connect Pro as a 24/7 virtual war room where we could post the latest information, record IM conversations and track issues.  We used Buzzword for collaborative authoring our PSIRT blog posts, Advisories, and Bulletins that are key to communicating our progress with customers and the security ecosystem.  Overall, situations like these highlight the need for security professionals to truly be a community. With all the technological solutions and formal processes we have at our disposal for creating secure products, we sometimes forget how important the little things like communication are to security.