In my role as Senior Quality Leader for Adobe LiveCycle, I have been presenting a quality overview slide deck to many of our enterprise customers showing them our quality standards and methods we use while developing LiveCycle. The most recent presentation was in January 2011 to a major pharmaceutical company that has been a customer of ours for many releases. I wanted to take a few minutes to share with all of our customers the highlights of what I often share in that slide deck.
Adobe LiveCycle is comprised of 30 teams and over 320 people. We are located across the globe with our major development centers in San Jose (United States), Ottawa (Canada), and Noida (India). Other remote sites include Boston, Beijing, and Minnesota. Even though we are located across the globe, one of the key approaches we take is to have the development and quality engineering teams reside in the same location. This helps for effective communication at the most important level (the team) while allowing us to take advantage of the 24/7 nature of a large development team. Many of our quality engineers have completed master’s degrees on quality improvement and test automation.
Adobe uses an overall product life cycle (PLC) framework for all of its projects, and LiveCycle is no different. That PLC process is a documented process with consistent terminology. This common language helps our product reviews go smoothly when presented to senior executives. Product teams are empowered in their decision-making but this also ensures, through regular reviews with senior management, that their project is on track. Governance is managed through a series of weekly/quarterly reviews where our teams document the key decisions for the project and inform the key stakeholders of changes. All of these changes are documented as part of a change control board (CCB) and are archived in a database for future reference. When Adobe LiveCycle was audited in 2008, we received the highest possible ranking awarded to a first-time vendor for our documented process and software delivery mechanism as defined in the PLC framework.
Also starting in 2008, LiveCycle switched from a more traditional waterfall-based software development process, to a Scrum-based agile development process. We have been continuing to refine that process over the last few releases. We know, based on internal survey results, that the teams feel we are improving our ability to deliver quality software to our customers in a shorter amount of time.
We use a combination of a number of four-week development sprints followed by a two-week quality sprint in order to deliver on a number of milestones during a release. For the Scrum purists out there, this might just seem like we are doing a series of mini-waterfalls inside of a Scrum framework. While this might have been more true in 2008 when we started, by constantly refining our definition of done process at the sprint level we are constantly evolving to achieve a more “pure” Scrum approach. The ultimate goal would is to eliminate the quality sprints between the development ones in an effort to truly be ship-ready after each sprint.
From a quality perspective, we divide the program into three areas:
- Sprint-Level Quality – Focusing on things like continuous integration, unit testing, build acceptance tests, functional tests, demoed features that meet the “definition of done,” and coverage across our large platform matrix
- Milestone-Level Quality – Integrated uses cases, upgrading customer images, negative testing, 14-day longevity and performance tests, and about 20 other exit criteria
- Post-GM Quality – Customer quick fixes, service packs, root cause analysis for customer issues
Putting all of this together requires a large effort of tracking and coordination. We have tens of thousands of test cases and millions of test runs per release. But as most software quality professionals know, it is not about the number of tests you have; it is about the coverage rate of those tests. All of our teams set code coverage targets for their code. By focusing our efforts on new code, we can increase your overall coverage rate without getting bogged down in legacy code areas. We then cover the rest of the code through use case and integration testing. As new product teams are formed, their goals are to achieve world-class code coverage metrics of 80%.
From a quality tooling perspective, we use as many open source tools (JUnit, CPPUnit, NUnit) as possible. In addition we use commercial tools like Borland Silktest and Hewlett-Packard Quick Test Professional for functional and user-interface testing. For code coverage we use Clover from Atlassian, and BullseyeCoverage. From a security perspective, we have standardized on IBM AppScan and Veracode.
I hope this gives you a high level overview of the tools and processes LiveCycle uses to delight our customers.
(post updated for grammar – 3/9/2011)