“Automate all the things!” is a popular meme in the cloud community. Many of today’s talks at security conferences discuss the latest, sophisticated automation tool developed by a particular organization. However, adding “automation” to a project does not magically make things better by itself. Any idea can be automated; including the bad ones. For instance, delivering false positives “at scale” is not going to help your teams. This blog will discuss some of the projects that we are currently working on and the reasoning behind their goals.
Computer science has been focused on automation since its inception. The advent of the cloud only frees our ideas from being resource bound by hardware. However, that doesn’t necessarily mean that automation must take up 100 scalable machines. Sometimes simple automation projects can have large impacts. Within Adobe, we have several types of automation projects underway to help us with security. The goals range from business-level dashboards and compliance projects to low level security testing projects.
One large project that we are currently building is a security automation framework focused on security assertions. When you run a traditional web security scanner against a site, it will try to tell you everything about everything on the site. In order to do that effectively, you have to do a lot of pre-configuration (authentication, excluded directories, etc.). Working with Mohit Kalra, the Sr, Security Manager for the ASSET security team, we experimented with the idea of security assertions. Basically, could a scanner answer one true/false question about the site with a high degree of accuracy? Then we would ask that one simple question across all of our properties in order to get a meaningful measurement.
For instance, let’s compare the following two possible automation goals for combating XSS:
(a) Traditional automation: Give me the location of every XSS vulnerability for this site.
(b) Security assertion: Does the site return a Content Security Policy (CSP) header?
A web application testing tool like ZAP can be used to automate either goal. Both of these tests can be conducted across all of your properties for testing at scale. Which goal you choose will decide the direction of your project:
Effort to implement:
(a) Potentially requires effort towards tuning and configuration with a robust scanner in order to get solid results. There is a potential risk to the tested environment (excessive DB entries, high traffic, etc.)
(b) A straight forward measurement with a simple scanner or script. There is a low risk to the tested environment.
Summarizing the result for management:
(a) This approach provides a complex measurement of risk that can involve several variables (reflected vs. persistent, potential value of the site, cookie strategy, etc.). The risk that is measured is a point-in-time assessment since new XSS bugs might be introduced later with new code.
(b) This approach provides a simple measurement of best practice adoption across the organization. A risk measurement can be inferred but it is not absolute. If CSP adoption is already high, then more fine grained tests targeting individual rules will be necessary. However, if CSP adoption is still in the early stages, then just measuring who has started the adoption process can be useful.
Developer interpretation of the result:
(a) Development teams will think in terms of immediate bugs filed.
(b) Development teams will focus on the long term goal of defining a basic CSP.
Both (a) and (b) have merits depending on the needs of the organization. The traditional strategy (a) can give you very specific data about how prevalent XSS bugs are across the organization. However, tuning the tools to effectively find and report all that data is a significant time investment. The security assertion strategy (b) focuses more on long term XSS mitigations by measuring CSP adoption within the organization. The test is simpler to implement with less risk to the target environments. Tackling smaller automation projects has the added value of providing experience that may be necessary when designing larger automation projects.
Which goal is a higher priority will depend on your organization’s current needs. We found that, in playing with the true/false approach of security assertions, we focused more of our energy on what data was necessary versus just what data was possible. In addition, since security assertions are assumed to be simple tests, we focused more of our design efforts on perfecting the architecture of scalable testing environment rather than the idiosyncrasies of the tools that the environment would be running. Many automation projects try to achieve depth and breadth at the same time by running complex tools at scale. We decided to take an intermediate step by using security assertions to focus on breadth first and then to layer on depth as we proceed.
Focused automation within continual deployment
Creating automation environments to scan entire organizations can be a long term project. Smaller automation projects can often provide quick wins and valuable experience on building automation. For instance, continuous build systems are often a single chokepoint through which a large portion of your cloud must pass before deployment. Many of today’s continuous build environments allow for extensions that can be used to automate processes.
As an example, PCI requires that code check-ins are reviewed. Verifying this process is followed consistently requires significant human labor. One of our Creative Cloud security champions, Jed Glazner, developed a Jenkins plugin which can verify each check-in was reviewed. The plugin monitors the specified branch and ensures that all commits belong to a pull request, and that the pull requests were not self merged. This allows for daily, automatic verification of the process for compliance.
Jed worked on a similar project where he created a Maven plug-in that lists all third-party Java libraries and their versions within the application. The plugin would then upload that information to our third-party library tracker so that we can immediately identify libraries that need updates. Since the plug-in was integrated into the Maven build system, the data provided to the third-party library tracker was always based on the latest nightly build and it was always a complete list.
Your organization will eventually build, buy or borrow a large scale automation tool that scales out the enumeration of immediate risk issues. However, before you jump head first into trying to build a robust scanning environment from scratch, be sure to first identify what core questions you need the tools to answer in order to support your program. You might find that starting with smaller automation tasks that track long term objectives or operational best practices can be just as useful to an organization. Deploying these smaller projects can also provide experience that can help your plans for larger automation projects.