Author Archive: Peleus Uhley

OWASP, IR, ML, and Internal Bug Bounties

A few weeks ago, I traveled to the OWASP Summit located just outside of London. The OWASP Summit is not a conference. It is a remote offsite event for OWASP leaders and the community to brain storm on how to improve OWASP.  There were a series of one hour sessions on how to tackle different topics and what OWASP could offer to make the world better. This is a summary of some of the more interesting sessions that I was involved in from my personal perspective. These views are not in any way official OWASP views nor are they intended to represent the views of everyone who participated in the respective working groups.

One session that I attended dealt with incident response (IR). Often times, incident response guidelines are written for the response team. They cover how to do forensics, analyze logs, and other response tasks covered by the core IR team. However, there is an opportunity for creating incident response guidelines which target the security champions and developers that IR team interacts with during an incident. In addition, OWASP can relate how this information ties back into their existing security development best practices. As an example, one of the first questions from an IR team is how does customer data flow through the application. This allows the IR team to quickly determine what might have been exposed. In theory, a threat model should contain this type of diagram and could be an immediate starting point for this discussion. In addition, after the IR event, it is good for the security champion to have a post mortem review of the threat model to determine how the process could have better identified the exploited risk. Many of the secure development best practices recommended in the security development lifecycle support both proactive and reactive security efforts. Also, a security champion should know how to contact the IR team and regularly sync with them on the current response policies. These types of recommendations from OWASP could help companies ensure that a security champion on the development team are prepared to assist the IR team during a potential incident. Many of the ideas from our discussion were captured in this outcomes page from the session: https://owaspsummit.org/Outcomes/Playbooks/Incident-Response-Playbook.html.

Another interesting discussion from the summit dealt with internal bug bounties. This is an area where Devesh Bhatt and myself were able to provide input based on our experiences at Adobe. Devesh has participated in our internal bounty programs as well as several external bounty programs. Internal bug bounties have been a hot topic in the security community. On the one hand, many companies have employees who participate in public bounties and it would be great to focus those skills on internal projects. Employees who participate in public bug bounties often include general developers who are outside of the security team and therefore aren’t directly tied into internal security testing efforts. On the other hand, you want to avoid creating conflicting incentives within the company. If this is a topic of interest to you, Adobe’s Pieter Ockers will be discussing the internal bug bounty program he created in detail at the O’Reilly Security conference in New York this October: https://conferences.oreilly.com/security/sec-ny/public/schedule/detail/62443.

Lastly, there was a session on machine learning. This has been a recent research area of mine since it is the next natural step of evolution for applying the data that is collected from our security automation work. Adobe also applies machine learning to projects like Sensei. Even though the session was on Friday, there was a large turnout by the OWASP leaders.  We discussed if there were ways to share machine learning training data sets and methodologies for generating them using common tools.  One of the observations was that many people are still very new to the topic of machine learning. Therefore, I decided to start by drafting out a machine learning resources page for OWASP. The goal of the page isn’t to copy pages of existing introductory content onto an OWASP page where it could quickly become dated. The page also isn’t designed to drown the reader with a link to every machine learning reference that has ever been created. Instead, it focuses on a small selection of references that are useful to get someone started with the basic concepts. The reader can then go find their own content that goes deeper into the areas that interest them. For instance, Coursera provides an 11 week Stanford course on machine learning but that would overwhelm the person just seeking a high-level overview. The first draft of my straw man proposal for the OWASP page can be found here: https://www.owasp.org/index.php/OWASP_Machine_Learning_Resources. As OWASP creates more machine learning content, this page could eventually be a useful appendix. This page is only a proposal and it is not an official OWASP project at this stage.  Additional ideas from the workshop discussion can be found on the outcomes page: https://owaspsummit.org/Outcomes/machine-learning-and-security/machine-learning-and-security.html.

OWASP is a community driven group where all are invited to participate. If these topics interest you, then feel free to join an OWASP mailing list and start to provide more ideas. There were several other great sessions from the summit and you can find a list of outcomes from each session here: https://owaspsummit.org/Outcomes/.

Peleus Uhley
Principal Scientist

Pwn2Own 2017

The ZDI Pwn2Own contest celebrated its tenth anniversary this year. Working for Adobe over the past ten years, I have seen a lot of changes in the contest as both an observer and as a vendor triaging the reports.

People often focus on the high level of aspects of who got pwned, how many times, in how many seconds, etc. However, very little of the hyped information is relevant to the actual state of security for the targeted application. There are a lot of factors that determine whether a team chooses to target a platform outside of its relative difficulty.  These can include weighing how to maximize points, the time to prepare, personal skill sets, difficulty in customizing for the target environment, and overall team strategy. ZDI has to publish extremely detailed specs for the targeted environment because even minor device driver differences can kill some exploit chains.

For instance, some of the products that were new additions this year were harder for the teams to add to their target list in time for the competition. On the other hand, it was unsurprising that Tencent and Qihoo 360 both competed against Flash Player since they regularly contribute responsible disclosures to Adobe’s Flash Player and Reader teams. In fact, Tencent was listed in our January Flash Player bulletin and we credited two Flash Player CVEs to the Qihoo 360 Vulcan Team in our March Flash Player security bulletin that went out the day before the contest. On the Reader side, Tencent team members were responsible 19 CVEs in the January release. Therefore, both teams regularly contribute to Adobe’s product security regardless of Pwn2Own.

The vendors don’t make it easy for the competitors. For instance, since the contest occurs after Patch Tuesday, there is always the chance that their bugs will collide with the vendors patch release. For instance, Chrome released 36 patches in March, VMWare had two security updates in March, and Flash Player released seven patches in our March release. In addition, new mitigations sometimes coincide with the contest. Last year, Flash Player switched to Microsoft’s Low Fragmentation Heap and started zeroing memory on free in the release prior to the contest. As a result, one of the Pwn2Own teams from that year did not have time to adjust their attack. This year, Flash Player added more mitigations around heap and metadata isolation in the Patch Tuesday release.

Adobe doesn’t target the mitigations for the contest specifically. These occur as part of our Adobe Secure Product Lifecycle (SPLC) process that continually adds new mitigations. For instance, between Pwn2Own last year and Pwn2Own this year, we added zeroing memory on allocation, running Flash Player in a separate process on Edge, blocking Win32k system calls and font access in Chrome, adding memory protections based on the Microsoft MemProtect concept, and several similar mitigations that are too detailed to include in a simple list. Similarly, Mozilla has been working on instrumenting sandboxing for their browser over the last year. Therefore, this is a contest that does not get any easier as time goes on. If you want to try and sweep all the Pwn2Own categories, then you need a team to do it.

In fact, a lot has changed since 2008 when Flash Player was first hacked in a Pwn2Own contest. The list of mitigations that Flash Player currently has in place includes compiler flags such as GS, SEH, DEP, ASLR and CFG. We have added sandboxing techniques such as job controls, low integrity processes, and app containers for Firefox, Chrome, Safari, and Edge. There have been memory defenses added that include constant folding, constant blinding, random NOP insertion, heap isolation, object length checks, replacing custom heaps, and implementing MemProtect. In addition, the code has gone through rounds of Google fuzzing, Project Zero reviews, and countless contributions from the security community to help eliminate bugs in addition to our internal efforts.

While advanced teams such as Qihoo 360 and Tencent can keep up, that security hardening has hindered others who target Flash Player. For instance, ThreatPost recently wrote an article noting that Trustwave measured a 300% drop in exploit kit activity. While exploit kit activity can be influenced by several factors including law enforcement and market ROI, the CVEs noted in exploit kits are for older versions of Flash Player. As we add mitigations, they not only need new bugs but also new mitigation bypass strategies in order to keep their code working. In addition, ThreatPost noted a recent Qualys report measuring a 40% increase in Flash Player patch adoption which helps to limit the impact of those older bugs. Zero days also have been pushed towards targeting a more limited set of environments.

All of that said, nobody is resting on their laurels. Advanced persistent threats (APTs) will select technology targets based on their mission. If your software is deployed in the environment an APT is targeting, then they will do what is necessary to accomplish their mission. Similarly, in Pwn2Own, we see teams go to extraordinary lengths to accomplish their goals. For instance, Chaitin Security Research Lab chained together six different bugs in order to exploit Safari on MacOS.  Therefore, seeing these creative weaponization techniques inspires us to think of new ways that we can further improve our defenses against determined malicious attackers.

The ZDI team has done a tremendous job improving Pwn2Own each year and adding interesting new levels of gamification. It is amazing to watch how different teams rise to the occasion. Due to the increased difficulty added by vendors each year, even winning a single category becomes a bigger deal with each new year. Thanks to everyone who contributed to making Pwn2Own 2017 a great success.

Peleus Uhley
Principal Scientist

Security Automation Part III: The Adobe Security Automation Framework

In previous blogs [1],[2], we discussed alternatives for creating a large-scale automation framework if you don’t have the resources for a multi-month development project. This blog assumes that you are ready to jump in with both feet on designing your own internal automation solution.

Step 1: Research

While we particularly like our design, that doesn’t mean that our design is the best for your organization. Take time to look at designs, such as Salesforce’s Chimera, Mozilla’s Minion, ThreadFix, Twitter SADB, Guantlt, and other approaches. These projects tackle large scale security implementations in distinctly different ways. It is important to understand the full range of approaches in order to select the one that is the best fit for your organization. Projects like Mozilla Minion and Guantlt are open-source if you are looking for code in addition to ideas. Some tools follow specific development processes, such as Guantlt’s adherence to Behavior Driven Development, which need to be considered. The OWASP AppSec Pipeline project provides information on how to architect around solutions such as ThreadFix.

The tools often break down into aggregators or scanners. Aggregators focus on consolidating information from your diverse deployment of existing tools and then trying to make sense of the information. ThreadFix is an example of this type of project. Scanners look to deploy either existing analysis tools or custom analysis tools at a massive scale. Chimera, Minion, and Adobe’s Security Automation Framework take this approach.  This blog focuses on our scanner approach using our Security Automation Framework but we are in the midst of designing an aggregator, as well.

Step 2: Put together a strong team

The design of this solution was not the result of any one person’s genius. You need a group of strong people who can help point out when your idea is not as clever as you might think. This project involved several core people including Mohit Kalra, Kriti Aggarwal, Vijay Kumar Sahu, and Mayank Goyal. Even with a strong team, there was a re-architecting after the version 1 beta as our approach evolved. For the project to be a success in your organization, you will want many perspectives including management, product teams, and fellow researchers.

Step 3: Designing scanning automation

A well thought out design is critical to any tool’s long term success. This blog will provide a technical overview of Adobe’s implementation and our reasoning behind each decision.

The Adobe Security Automation Framework:

The Adobe Security Automation Framework (SAF) is designed around a few core principles that dictate the downstream implementation decisions. They are as follows:

  1. The “framework” is, in fact, a framework. It is designed to facilitate security automation but it does not try to be more than that. This means:
    1. SAF does not care what security assessment tool is being run. It just needs the tool to communicate progress and results via a specified API. This allows us to run any tool, based on any language, without adding hard coded support to SAF for each tool.
    2. SAF provides access to the results data but it is not the primary UI for results data. Each team will want their data viewed in a team specific manner. The SAF APIs allow teams to pull the data and render it as best fits their needs. This also allows the SAF development team to focus their time on the core engine.
  2. The “framework” is based on Docker. SAF is designed to be multi-cloud and Docker allows portability. The justifications for using a Docker based approach include:
    1. SAF can be run in cloud environments, in our internal corporate network, or run from our laptops for debugging.
    2. Development teams can instantiate their own mini-copies of SAF for testing.
    3. Using Docker allows us to put security assessment tools in separate containers where their dependencies won’t interfere with each other.
    4. Docker allows us to scale the number of instances of each security assessment tool dynamically with respect to their respective job size.
  3. SAF is modularized with each service (UI, scheduler, tool instance, etc.) in its own Docker container. This allows for the following advantages:
    1. The UI is separated from the front-end API allowing the web interface to be just another client of the front-end API. While people will initiate scans from the UI, SAF also allows for API driven scan requests.
    2. The scanning environments are independent. The security assessment tools may need to be run from various locations depending on their target. For instance, the scan may need to run within an internal network, external network, a specific geographic location, or just within a team’s dedicated test environments. With loose-coupling and a modular design, the security assessment tools can be run globally while still having a local main controller.
    3. Docker modularity allows for choosing the language and technology stack that is appropriate for that module’s function.
  4. By having each security test encapsulated within its own Docker container, anyone in the company can have their security assessment tools included in SAF by providing an appropriately configured Docker image. Volunteers can write a simple Python driver based on the SAF SDK that translates a security testing tool’s output into compatible messages for the SAF API and provide that to the SAF team as a Docker image. We do this because:
    1. The SAF team does not want to be the bottleneck for innovation. By allowing external contributions to SAF, the number of tools that it can run increases at a far faster rate. Given the wide array of technology stacks deployed at Adobe, this allows development teams to contribute tools that are best suited for their environments.
    2. In certain incident response scenarios, it may be necessary to widely deploy a quick script to analyze your exposure to a situation. With SAF, you could get a quick measurement by adding the script to a Docker image template and uploading it to the framework.
  5. The “security assertion”, or the test that you want to run, should test a specific vulnerability and provide a true/false result that can be used for accurate measurements of the environment. This is similar to the Behavior Driven Development approaches seen in tools like Guantlt. SAF is not designed to run a generic, catch-all web application penetration tool that will return a slew of results for human triage. Instead it is designed for analysis of specific issues. This has the following advantages:
    1. If you run an individual test, then you can file an individual bug for tracking the issue.
    2. You create tests specifically around the security issues that are critical to your organization. The specific tests can then be accurately measured at scale.
    3. Development teams do not feel that their time is being wasted by being flooded with false positives.
    4. Since it is an individual test, the developer in charge of fixing that one issue can reproduce the results using the same tool as was deployed by the framework. They could also add the test to their existing automation testing suite.

Mayank Goyal on our team took the above principles and re-architected our version 1 implementation into a design depicted in the following architecture diagram:

The SAF UI

The SAF UI is simplistic in nature since it was not designed to be the analysis or reporting suite. The UI is a single page based web application which works with SAF’s APIs. The UI focuses on allowing researchers to configure their security assertions with the appropriate parameters. The core components of the UI are:

  • Defining the assertion: Assertions (tests) are saved within an internal GitHub instance, built via Jenkins and posted to a Docker repository. SAF pulls them as required. The Github repository contains the DockerFile, the code for the test, and the code that acts as bridge between the tool and the SAF APIs using the SAF SDK. Tests can be shared with other team members or kept private.
  • Defining the scan: It is possible that the same assertion(test) may be run with different configurations for different situations. The scan page is where you define the parameters for different runs.
  • Results: The results page provides access to the raw results. The results are broken down into pass, fail, or error for each host tested. It is accompanied by a simple blob of text that is associated with each result.
  • Scans can be set to run at specific intervals.

This screenshot demonstrates an assertion that can identify whether the given URL parameter has login forms are available over HTTP. This assertion is stored in Git, is initiated by /src/startup.sh, and it will use version 4 of the configuration parameters.

A Scan is then configured for the assertion which says when the test will be run and which input list of URLs to test. A scan can run more than one assertion for the purposes of batching results.

The SAF API Server

This API server is responsible for preparing the information for the slave testing environments in the next stage. It receives the information for the scan from the UI, an API client, or based on the saved schedule. The tool details and parameters are packaged and uploaded to the job queue in a slave environment for processing. It assembles all the meta information for testing for the task/configuration executor. The master controller also listens for the responses from the queuing system and stores the results in the database. Everything downstream from the master controller is loosely coupled so that we can deploy work out to multiple locations and different geographies.

The Job Queueing system

The Queueing system is responsible for basic queuing and allows the scheduler to schedule tasks based on resource availability and defer them when needed. While cloud providers offer queuing systems, ours is based on RabbitMQ because we wanted to have deployment mobility.

The Task Scheduler

This is the brains of running the tool. It is responsible monitoring all the Docker containers, scaling, resource scheduling and killing rogue tasks. It has the API that receives the status and result messages from the Docker containers. That information is then relayed back to the API server.

The Docker images

The Docker images are based on a micro-service architecture approach. The default baseline is based on Alpine Linux to keep the image footprint small. SAF assertions can also be quite small. For instance, the test can be a small Python script which makes a request to the homepage of a web server and verifies whether an HSTS header was included in the response. This micro-service approach allows the environment to run multiple instances with minimum overhead. The assertion script communicates its status (e.g. keep-alives) and results (pass/fail/error) back to the task executor using the SAF SDK.

Conclusion

While this overview still leaves a lot of the specific details unanswered, it should provide a basic description of our security automation framework approach at the architectural and philosophical level. For a security automation project to be a success at the detailed implementation level, it must be customized to the organization’s technology stack and operational flow. As we progress with our implementation, we will continue to post the lessons that we learn along the way.

Peleus Uhley
Principal Scientist

Security Automation Part II: Defining Requirements

Every security engineer wants to build the big security automation framework for the challenge of designing something with complexity. Building those big projects have their set of challenges. Like any good coding project, you want to have a plan before setting out on the adventure.

In the last blog, we dealt with some of the high level business concerns that were necessary to consider in order to design a project that would produce the right results for the organization. In this blog we will look at the high level design considerations from the software architect’s perspective. In the next blog, we will look at the implementer’s concerns. For now, most architects are concerned with the following:

Maintainability

This is a concern for both the implementer and architect, but they often have different perspectives. If you are designing a tool that the organization is going to use as a foundation of its security program, then you need to design the tool such that the team can maintain it over time.

Maintainability through project scope

There are already automation and scalability projects that are deployed by the development team. These may include tools such as Jenkins, Git, Chef, or Maven. All of these frameworks are extensible. If all you want to do is run code with each build, then you might consider integrating into these existing frameworks rather than building your own automation. They will handle things such as logging, alerting, scheduling, and interacting with the target environment. Your team just has to write code to tell them what you want done with each build.

If you are attempting a larger project, do you have a roadmap of smaller deliverables to validate the design as you progress? The roadmap should prioritize the key elements of success for the project in order to get an early sense if you are heading in the right direction with your design. In addition, while it is important to define everything that your project will do, it is also important to define all the things that your tool will not perform. Think ahead to all of the potential tangential use cases that your framework could be asked to perform by management and customers. By establishing what is out of scope for your project, you can set proper expectations earlier in the process and those restrictions will become guardrails to keep you on track when requests for tangential features come in.

Maintainability through function delegation

Can you leverage third-party services for operational issues?  Can you use the cloud so that baseline network and machine uptime is maintained by someone else? Can you leverage tools such as Splunk so that log management is handled by someone else? What third-party libraries already exist so that you are only inventing the wheels that need to be specific to your organization? For instance, tools like RabbitMQ are sufficient to handle most queueing needs.  The more of the “busy work” that can be delegated to third-party services or code, the more time that the internal developers can spend on perfecting the framework’s core mission.

Deployment

It is important to know where your large scale security framework may be deployed. Do you need to scan staging environments that are located on an internal network in order to verify security features before shipping? Do you need to scan production systems on an external network to verify proper deployment? Do you need to scan the production instances from outside the corporate network because internal security controls would interfere with the scan? Do you want to have multiple scanning nodes in both the internal and external network? Should you decouple the job runner from the scanning nodes so that the job runner can be on the internal network even if the scanning node is external?  Do you want to allow teams to be able to deploy their own instances so that they can run tests themselves? For instance, it may be faster if an India based team can conduct the scan locally than to run the scan from US based hosts. In addition, geographical load balancers will direct traffic to the nearest hosts which may cause scanning blind spots. Do you care if the scanners get deployed to multiple  geographic locations so long as they all report back to the same database?

Tool selection

It is important to spend time thinking about the tools that you will want your large security automation framework to run because security testing tools change. You do not want your massive project to die just because the tool it was initially built to execute falls out of fashion and is no longer maintained. If there is a loose coupling between the scanning tool and the framework that runs it, then you will be able to run alternative tools once the ROI on the initial scanning tool diminishes. If you are not doing a large scale framework and are instead just modifying existing automation frameworks, the same principles will apply even if they are at a smaller scale.

Tool dependencies

While the robustness of tests results is an important criterion for tool selection, complex tools often have complex dependencies. Some tools only require the targeted URL and some tools need complex configuration files.  Do you just need to run a few tools or do you want to spend the time to make your framework be security tool agnostic? Can you use a Docker image for each tool in order to avoid dependency collisions between security assessment tools? When the testing tool conducts the attack on the remote host, does the attack presume that code injected into the remote host’s environment can send a message back to the testing tool?  If you are building a scalable scanning system with dynamically allocated, short-lived hosts that live behind a NAT server, then it may be tricky for the remote attack code to send a message back to the original security assessment tool.

Inputs and outputs

Do the tools require a complex, custom configuration file per target or do you just need to provide the hostname? If you want to scale across a large number of sites, tools that require complex, per-site configuration files may slow the speed at which you can scale and require more maintenance over time. Does the tool provide a single clear response that is easy to record or does it provide detailed, nuanced responses that require intelligent parsing? Complex results with many different findings may make it more difficult to add alerting around specific issues to the tool. They could also make metrics more challenging depending on what and how you measure.

Tool scalability

How many instances of the security assessment tool can be run on a single host? For instance, tools that listen on ports limit the number of instances per server. If so, you may need Docker or a similar auto-scaling solution. Complex tools take longer to run which may cause issues with detecting time outs. How will the tool handle re-tests for identified issues? Does the tool have granularity so that dev team can test their proposed patch against the specific issue? Or does the entire test suite need to be re-run every time the developer wants to verify their patch?

Focus and roll out

If you are tackling a large project, it is important to understand what is the minimum viable product? What is the one thing that makes this tool different than just buying the enterprise version of the equivalent commercial tool? Could the entire project be replaced with a few Python scripts and crontab? If you can’t articulate what extra value your approach will bring over the alternative commercial or crontab approach, then the project will not succeed. The people who would leverage the platform may get impatient waiting for your development to be done. They could instead opt for a quicker solution, such as buying a service, so that they can move on to the next problem.  As you design your project, always ask yourself, “Why not cron?” This will help you focus on the key elements of the project that will bring unique value to the organization. Your roadmap should focus on delivering those first.

Team adoption

Just because you are building a tool to empower the security team, doesn’t mean that your software won’t have other customers. This tool will need to interact with the development teams’ environments. This security tool will produce outputs that will eventually need to be processed by the development team. The development teams should not be an afterthought in your design. You will be holding them accountable for the results and they need methods for understanding the context of what your team has found and being able to independently retest.

For instance, one argument for integrating into something like Jenkins or GIt, is that you are using a tool the development team already understands. When you try to explain how your project will affect their environment, using a tool that they know means that the discussion will be in language that they understand. They will still have concerns that your code might have negative impacts on their environment. However, they may have more faith in the project if they can mentally quantify the risk based on known systems. When you create standalone frameworks, then it is harder for them to understand the scale of the risk because it is completely foreign to them.

At Adobe, we have been able to work directly with the development teams for building security automation. In a previous blog, an Adobe developer described the tools that he built as part of his pursuit of an internal black belt security training certification. There are several advantages to having the security champions on development teams build the development tools rather than the core security team. One is that full time developers are often better coders than the security teams and the developers better understand the framework integration. Also, in the event of an issue with the tool, the development team has the knowledge to take emergency action. Often times, a security team just needs the tool to meet specific requirements and the implementation and operational management of the tool can be handled by the team responsible for the environment. This can make the development team more at ease with having the tool in their environment and it frees up the core security team to focus on larger issues.

Conclusion

While jumping right into the challenges of the implementation is always tempting, thinking through the complete data flow for the proposed tools can help save you a lot of rewriting. It also important that you avoid trying to boil the ocean by scoping more than your team can manage. Most importantly, always keep focus on the unique value of your approach and the customers that you need to buy into the tool once it is launched. The next blog will focus on an implementer’s concerns around platform selection, queuing, scheduling, and scaling by looking at example implementations.

Security Automation Part I: Defining Goals

This is the first of a multi-part series on security automation. This blog will focus on high-level design considerations. The next blog will focus on technical design considerations for building security automation. The third blog will dive even deeper with the specific examples as the series continues to get more technical.

There are many possible approaches for adding automation to your security process. For many security engineers, it is an opportunity to get away from reviewing other engineers’ code and write some of their own. One key difference between a successful automation project and “abandonware” is to create a project that will produce meaningful results for the organization. In order to accomplish that, it is critical to have a clear idea of what problem you are trying to solve at the onset of the project.

Conceptual aspirations

When choosing the right path for designing security automation, you need to decide what will be the primary goals for the automation. Most automation projects can be grouped into common themes:

Scalability

Scalability is something that most engineers instinctively go to first because the cloud has empowered researchers to do so much more. Security tools designed for scalability often focus on the penetration testing phase of the software development lifecycle. They often involve running black or grey box security tests against the infrastructure in order to confirm that the service was deployed correctly. Scalability is necessary if you are going to measure every host in a large production environment. While scalability is definitely a powerful tool that is necessary in the testing phase of your security development lifecycle, it sometimes overshadows other important options that can occur earlier in the development process and that may be less expensive in terms of resources.

Consistency

There is a lot of value in being able to say that “Every single release has ___.” Consistency isn’t considered as challenging of a problem as scalability because it is often taken for granted. Consistency is necessary for compliance requirements where public attestations need to have clear, simple statements that customers and auditors can understand. In addition, “special snowflakes” in the environment can drown an organization’s response when issues arise. Consistency automation projects frequently focus on the development or build phase of the software development lifecycle. They include integrating security tasks into build tools like Jenkins, Chef, Git, or Maven. By adding security controls into these central tools in the development phase, you can have reasonable confidence that machines in production are correctly configured without scanning each and every one individually.

Efficiency

Efficiency projects typically focus on improving operational processes that currently involve too much human interaction.  The goal is to refocus the team’s manual resources to more important tasks. For instance, many efficiency projects have the word “tracking” somewhere in their definition and involve better leveraging tools like JIRA or Sharepoint. Often times, efficiency automation is purchased rather than built because you aren’t particularly concerned with how the problem gets solved, so long as it gets solved and that you aren’t the one who has to maintain the code for it. That said, SalesForce’s open-source VulnReport.io project (http://vulnreport.io) is an example of a custom built efficiency tool which they claim improved operational efficiency and essentially resulted “in a ‘free’ extra engineer for our team.”

Metrics

Metrics gathering can be a project in itself or it can be the byproduct of a security automation tool. Metrics help inform and influence management decisions with concrete data. That said, it is important to pick metrics that can guide management and development teams towards solving critical issues. For instance, development teams will interpret the metrics gathered as the key performance indicator (KPI) that they are being measured against by management.

In addition, collecting data almost always leads to requests for more detailed data. This can be useful in helping to understand a complex problem or it can be a distraction that leads the team down a rabbit hole. If you take time to select the proper metrics, then you can help keep the team focused on digging deeper into the critical issues.

Operational Challenges

If your scalable automation project aims to run a web application penetration tool (WAPT) across your entire enterprise, then you are basically creating an “enterprise edition” for that tool. If you have used enterprise edition WAPTs in the past and you did not achieve the success that you wanted, then recreating the same concept with a slightly different tool will most likely not produce significantly different results when it comes to action within the enterprise. The success or failure of tools are typically hinged on the operational process surrounding the tool more than the tool itself. If there is no planning for handling the output from the tool, then increasing the scale at which the tool is run doesn’t really matter. When you are designing your automation project, consider operational questions such as:

Are you enumerating a problem that you can fix?

Enumerating an issue that the organization doesn’t have the resources to address can sometimes help justify getting the funding for solving the problem. On the other hand, if you are enumerating a problem that isn’t a priority for an organization, then perhaps you should refocus the automation on issues that are more critical. If no change occurs as the result of the effort, then the project will stop iterating because there is no need to re-measure the issue.

In some situations, it may be better to tackle the technical debt of basic issues before tackling larger issues. Tests for basic technical debt issues are often easier to create and they are easier for the dev team to address. As the dev team addresses the issues, the project will also iterate in response. While technical debt isn’t as exciting as the larger issues, addressing it may be a reasonable first step towards getting immediate ROI.

Are you producing “noise at scale”?

Running a tool that is known for creating a high level of false positives at scale will produce “noise at scale”. Unless you have an “at scale” triage team to eliminate the false positives, then you are just generating more work for everyone. Teams are less likely to take action on metrics that they believe are debatable due to the fear that their time might be wasted. A good security tool will empower the team to be more efficient rather than drown them in reports.

How will metrics guide the development team?

As mentioned earlier, the metric will be interpreted as a KPI for the team and they will focus their strategy around what is reported to management. Therefore, it makes sense not to bother measuring non-critical issues since you don’t want the team to get distracted by minor issues. You will want to make sure that you are collecting metrics on the top issues in a way that will encourage teams towards the desired approach.

Often times there are multiple ways to solve an issue and therefore multiples ways to measure the problem. Let’s assume that you wanted to create a project to tackle cross-site scripting (XSS). Creating metrics that count the number of XSS bugs will focus a development team on a bug fixing approach to the problem. Alternatively, counting the number of sites with content security policy headers deployed will focus the development team on security mitigations for XSS. In some cases, focusing the team on security mitigations has more immediate value than focusing on bug fixing.

What metrics does management need to see?

One method to determine how your metrics will drive development teams and inform management, is to phrase them in terms of an assertion. For instance, “I assert that the HSTS header is returned by all our sites in order to ensure our data is always encrypted.”  By phrasing it as an assertion, you are phrasing the test in terms of a simple true/false terms that can be reliably measured at scale. You are also phrasing the test in terms of its desired outcome using simplistic terms. This makes it easier to determine if the goal implied by the metric’s assertion meets with management’s goals.

From a management perspective, it is also important to understand whether your measurement of change or a measurement of adoption. Measuring the number of bugs in an application is often measuring an ongoing wave. If security programs are working, the height of the waves will trend down overtime. Although, that means you have to watch the wave through multiple software releases before you can reliably see a trend in its change. If you measure security mitigations, then you are measuring the adoption rate of a project that will end with a state of relative “completeness.” Tracking wave metrics overtime is valuable because you need to see when things are getting better or worse. However, since it is easy to procrastinate on issues that are open ended,  adoption-style projects that can be expressed as having a definitive “end” may get more immediate traction from the development team because you can set a deadline that needs to be met.

Putting it all together

With these ideas and questions in mind, you can mentally weigh which types of projects to start with for immediate ROI and the different tools for deploying them.

For instance, counting XSS and blind SQL injection bugs are hard tests to set up (authentication to the application, crawling the site, etc.), these tests frequently have false positives, and they typically result in the team focusing on bug fixing which would require in-depth monitoring overtime because it is a wave metric. In contrast, a security project measuring security headers, such as X-Frame-Options or HSTS, are simple tests to write, they have low false-positive rates, they can be defined as “(mostly) done” once the headers are set, and they focus the team on mitigations. Another easy project might be writing scalable tests that confirm the SSL configuration meets the company standards. Therefore, if you are working on a scalability project, starting with a simple SSL or security header projects can be quick wins that demonstrate the value of the tool. From there, you can then progress to measuring the more complex issues.

However, let’s say you don’t have the compute resources for a scalability project. An alternative might be to turn the projects into consistency style projects earlier in the lifecycle. You could create Git or Jenkins extensions that search the source code for indicators that the team has deployed security headers or proper SSL configurations. You would then measure how many teams are using the build platform extensions and review the collected results from the extension. It would have a similar effect as testing the deployed machines without as much implementation overhead. Whether this will work better overall for your organization will depend on where you are with your security program and its compliance requirements.

Conclusion

While the technical details of how to build security automation is an exciting challenge for any engineer, it is critical to build a system that will empower an organization. Your project will have a better chance of success if you spend time considering how the output of your tool will help guide progress. The scope of the project in terms of development effort and project coverage by carefully considering where in the development process you will deploy the automation.  By spending time on defining how the tool can best serve the team’s security goals, you can help ensure you are building a successful platform for the company.

The next blog will focus on the technical design considerations for building security automation tools.

Peleus Uhley
Principal Scientist, Security

Preparing for Content Security Policies

Deploying Content Security Policies (CSPs) can help increase the security of your website. Therefore, it is an easy recommendation that most security professionals make when working with development teams. However, the process of helping a team go from the simple recommendation to a successfully deployed policy can be complicated. As a security professional, you need to be aware of the potential challenges along the way in order to help teams navigate their way to a successful deployment.

Let’s start with the initial challenges that developers may face. One, is that many authoring tools that support accelerated web design, do not produce CSP compliant code. While some of the major JS frameworks support CSP, the extensions to those frameworks may not. Also, there are trade-offs to running in CSP compliant mode. For instance, AngularJS has a CSP-compliant mode that you can enable. (https://docs.angularjs.org/api/ng/directive/ngCsp) However, there are speed trade offs in enabling that flag. Security people may think a 30% speed trade-off is acceptable but, depending on how dependent the page is on AngularJS, the developers may take notice.

Lastly, a CSP compliant coding style may be counter-intuitive to some long time web developers. For instance, let’s take a simple button tag as an example. In a pre-CSP design approach, you would write it as:

<input type=”button” id=”myButton” value=”Click Me!” onClick=”doCoolStuff();”/>

To enable CSP, we have to tell the developer to remove the onClick statement and put it in a separate JS file. This may be counterintuitive because the developer may wonder why making two Internet requests and dynamically adding the event handler might be safer than just hard coding the event inline within the original request. From a developer’s perspective, the extra web request adds latency to the page load. Also, it will make the code harder to read because you have to do more cross-referencing in order to understand what the button does.

In terms of moving the code out of the HTML, it is more complicated than just copying and pasting. For instance, you can’t add an event handler to the button until the button has been added to the DOM. Therefore, you may need to add an additional onLoad event handler just to add the onClick event handler to the button. The same race condition would apply anytime you dynamically add content to an innerHTML property.

For cascading style sheets, you also need to remove any style properties from the code. Fortunately, while many developers still use inline style statements, the deprecation of several style properties in tags by the HTML5 movement has already forced the migration of many style settings to separate files. That said, there may be templates and libraries that developers use with style statements still in the HTML.

Understanding the changes required to support CSP is important because it gives you perspective on the scope of what you are asking teams to change. It is more than just enumerating everywhere that you load content and putting it in a header. You are likely asking teams to recode large amounts of working, validated code. This is similar to the “banned function” movement that happened in C/C++ code several years ago where developers had to re-architect existing code to use newer C/C++ functions.

Before you start a CSP deployment with teams, you should be prepared to answer questions such as:

  • Do I deploy CSP Level 1, Level 2, or Level 3?
    • Is Level 2 better than Level 1 in terms of security or does it just have more features?
    • What is the browser adoption for each level? I want to use the nonce and hash features of CSP Level 2 for my inline scripts but I also have to be compatible with older browsers.
    • I heard that the “strict-dynamic” property in Level 3 is going to make deployment easier. Should we just wait for that?
  • I read one site where it said I should start with the policy “default ‘self'” but another site said I should start with “default-src ‘none’; script-src ‘self’; connect-src ‘self’; img-src ‘self’; style-src ‘self’;”.  Which should I use?
  • Do I have to set a unique, restricted CSP for each page or can I just set one for the entire site that is a little more permissive?
  • How do I handle the CSP report-only reports? What is the process for ensuring someone reviews the reports? Is there a pre-made module available via gem/pip/npm that handles some of this and automatically puts the reports in a database?
  • Do I need to deploy the CSP reporting in production or is staging enough?
  • What are the common mistakes people make when deploying CSPs?
  • How do I debug when the CSP is breaking content? The CSP report doesn’t give me the line number of the violation and sometimes it doesn’t even give me the full path for the request. Do the builtin browser developer consoles help with debugging violations? Which is best?
  • I load JS content from a third-party provider and their content is causing issues with the recommended CSP settings. What should I do?
  • How bad is it to allow unsafe-inline? Can I use the nonce approach instead? Are there any prewritten/preapproved npm/pip/gem modules for generating nonces? How does that compare to hash source?  How do I know whether to use nonces or hashes? I heard that nonces don’t work well with static content. Is that true? Is there a way to have the server generate the hash at runtime so I don’t have to update CSPs every time I change a file?
  • How do we change our developer guides to help ensure that new code is written in a CSP compliant manner? Are there plugins for our IDEs to detect these things? Will our static analysis tools flag these issues? Are there pre-written guidelines which enumerate what you can and can’t do? Is there a way to style the code so that it is easier to handle the added complexity of cross-referencing between files?
  • Are there any published stats on rendering time when implementing CSP? What is the performance impact?
  • Is it OK to specify things like media-src, object-src or font-src and set them to ‘self’ even though we aren’t currently using them on the page? I want to limit how often I have to adjust the policy files in production. As long as they are set to ‘self’, then it shouldn’t be that big of a risk, right?

As you can see, the questions arising from deployment can get complicated quickly. For instance, it is trivial to define what would be the ideal policy and how to set the header. However, content security policies are the lowest common denominator of all your page dependencies. If the team integrates with multiple providers, then the policy is going to be whatever lowest common denominator is necessary to support all the providers that are linked by that page. How are you going to handle the situation where their third-party library requires “unsafe-inline unsafe-eval data: …”?  Are your policies immutable or will there be an exception process? What is the exception process?

While I can’t provide all the answers in a single blog, here are some thoughts on how to approach it:

  • Work on changing coding guidelines so that new code is CSP compliant. You will never catch up to where you need to be if new code continues to follow the existing development practices.
  • You can get a feel for the scope of the effort by setting up a test server with a few web pages from your site and experimenting with adding CSPs locally.
  • Rather than going after the entire site, start with critical pages. The simple process of getting your first page to work will likely require a decent amount of effort such as training the team on what the rules mean, deciding what approach should be used by the server to add the headers, learning how to debug issues, identifying patterns, etc. For instance, a login page is critical in terms of security and it is likely to have fewer third-party dependencies than other pages. Once that first page is established, you can build on your CSP deployment incrementally from there.
  • Track the policies necessary to support different third-party providers and libraries. This will allow teams to share rules and limit the amount of debugging necessary for third-party code.
  • Search for blogs and talks where people talk about the actual deployment process rather than just the technical definitions. One example is this blog by Terrill Dent of Square : https://corner.squareup.com/2016/05/content-security-policy-single-page-app.html For people who prefer full presentations, Michele Spagnuolo and Lukas Weichselbaum have conducted a lot research in this area. Their HITB Amsterdam presentation detailed common mistakes made in CSP deployments: https://conference.hitb.org/hitbsecconf2016ams/materials/D1T2%20-%20Michele%20Spagnuolo%20and%20Lukas%20Weichselbaum%20-%20CSP%20Oddities.pdf (slides) https://www.youtube.com/watch?v=eewyLp9QLEs (video) They also recently presented, “Making CSP great again” at OWASP AppSecEU 2016: https://www.youtube.com/watch?v=uf12a-0AluI&list=PLpr-xdpM8wG-Kf1_BOnT2LFZU8_SXfpKL&index=37

It is trivial to email a team telling them they should deploy content security policies and describing all the wonderful benefits that they can bring. However, the next step is to dig in and help the teams through the implementation process. In Spagnulo and Weichselbaum’s AppSecEU presentation, they mentioned that they analyzed 1.6 million policies from the web and they estimated that they were able to bypass the whitelist of at least 90% of them. Therefore, simply deploying CSPs does not guarantee security.  You will need to be prepared for all the design questions and policy decisions that will result from the effort in order to deploy them well. That preparation will pay dividends in creating and executing a successful CSP deployment plan with a development team.

Peleus Uhley
Principal Scientist

A Vendor Perspective on Crowd Sourced Penetration Tests

Bug bounties, also known as crowd sourced penetration tests, are becoming increasingly popular. New programs are announced every month. At NullCon this year, there was an entire track dedicated to the topic where vendors and researchers could meet. For a security researcher, there are a ton of options for participating ranging from the self-run programs, such as Google’s, to participating on consolidated platforms like BugCrowd and HackerOne. However, for the vendor, the path into bug bounties can be somewhat complex and the most significant benefits are not always obvious. Here are some tips on how to get more from your bug bounty.

Preparation

You should pick a team that has gone through several traditional penetration tests and where the ROI from those tests is trending down. If traditional consultants are still finding numerous bugs and architectural issues, your time and money would be better spent addressing the known issues and strengthening the architecture. Testing against a more mature development team can also benefit in other ways as you will soon see. A good crowd-sourced penetration test will involve both sides, researchers and development teams, being active in the bounty program.

If you have never done a bounty before, starting with short-term, private bounties will allow you to experience a few hiccups in a controlled situation. Be sure that you have planned out how to issue accounts to a large number of users and that the environment works when testing from outside your corporate environment. Try testing from home just to make sure it works.

Bounty guidelines

The large number of public bounties can serve as a baseline template for your test rules. As you review them, be sure to take note of their differences and consider what may have lead to those differences. A good set of bounty rules will be tailored to the service being tested. One of the less obvious components of a bounty announcement is how you describe your service to the tester. While the service may be extremely popular within your social circles, a researcher across the globe may have never heard of it. Therefore, be sure your bounty description provides an easy-to-understand description of what they are testing and perhaps a link to a short YouTube video that has your product pitch. The less time a researcher has to spend figuring out the goal of the service, the more time they can spend finding quality bugs.

Thematic issues

Penetration tests are typically scoped to a certain set of new features. However, crowd sourced penetration tests are often scoped across the entire service. Since traditional penetration tests are often focused on specific areas, they will not find issues in the connective code between features. Also, since the researchers are testing across the entire service, they are testing across the entire development team and not just within individual sprint teams. This may allow you to pick up on things that the overall team is consistently missing which can guide you as to where to focus energy going forward. For instance, if you have several authorization bugs, then is there a way to better consolidate authorization checking within the platform or is there a way to enable the quality team to better test these issues?

Critical bugs

Since the bounty hunters usually want to get top dollar for their efforts, they will often find more critical bugs. A critical bug is often the result of multiple issues that aren’t mentioned in the initial write-up. For instance, if they send you your password file, then there should be multiple questions beyond what type of injection was used in the attack. A few examples: Would egress filters on the network help? Do we need host monitoring solution to detect when the server process touches unexpected files? It is important to remember that these critical bugs aren’t just theoretical issues found through a code review. These vulnerabilities were successfully exploited issues found via black box testing of your infrastructure from a remote location.

Variant testing

If you have developers on hand during the bounty, then the developers can push the patch to the staging environment before the end of the program. You can then reach out to researcher and say, “Bet you can’t do that twice!”  You basically offer the researcher a separate bounty if they can find a variant or the same bug in a different API. It often isn’t difficult for the researcher to re-test something they have already tested. For the developer, they can get immediate feedback on the patch while the issue is still fresh in their minds. In my experiments at Adobe, losing that bet with the researcher is more valuable than the money it costs us because it typically identifies some broader issue with the platform or the process. This can be key for critical bugs.

Red Team/Blue Team

With a crowd sourced penetration test, you are likely testing against your staging environment or a dedicated server in order to minimize risk to your production network. A staging environment typically has low traffic volumes since only the product team is using it. However, during the testing period, you will have people from across the globe testing that environment and reporting the vulnerabilities that they are finding. For your response teams, this is an excellent opportunity to see what your logs captured about the attack. In theory, identifying the attack should be straight forward since the staging environment is low volume, you know what attack occurred, and you have a rough estimate of when the attack occurred. If you can’t find an attack in your logs under those conditions, then that is clear feedback about how your logging and monitoring can be improved. If you can save the logs until after the bounty has ended, this type of analysis can be done post-assessment if you don’t have the resources to play along real time.

A crowd-sourced penetration test can change up the routine you have established for finding issues. Like any change in routine, there can be a few challenges at first. However, when done well, they can provide a vendor with insights that they may have never obtained through the existing status quo. These are not a replacement for traditional consultants. Rather, the new insights into the platform can help you re-focus the consultants more effectively to get a higher ROI.

 

Peleus Uhley
Principal Scientist

Reflections on Pwn2Own

Returning from CanSecWest left me reflecting on how the Pwn2Own competition has evolved over time. A lot has changed in the Pwn2Own competition over the years. The event has grown in attendance, competitors, and complexity, just as the industry has grown.

For the first contest in 2007, no one competed on the first day. Shane McCauley called fellow security researcher Dino Dai Zovi in NYC that night and urged him to compete. Dino was able to write the exploit over night and win the contest on the second day. Visually, the attacks seem no different from year to year. The contestant sits down at the machine, and “seconds” later the calculator (or notepad in this year’s case), pops up on the screen.  However, the preparation for the attacks from 2007 to 2016 are now drastically different, with contestants preparing attacks weeks in advance.

Media coverage around security advisories is often just a run down of how many CVEs a vendor released that month. People often imply from these articles that all the CVEs are easily exploitable and trivially weaponizable. This can lead to false perceptions that exploiting the software is a simple task. In reality, a CVE from a vendor is not a guarantee of exploitability. Even if the CVE can give the attacker the ability to overwrite memory, that is not a guarantee that it can be weaponized. Technologies such as ASLR (Address Space Layout Randomization) weren’t even released for Mac OS X when Dai Zovi competed in 2007. Today’s attackers have to work around defenses such as CFG (Control Flow Guard), Isolated Heaps, and a number of other technologies designed to prevent a crash from becoming an exploit.

In addition, competitors have to deal with the fact that the contest frequently occurs after Patch Tuesday where a vendor’s security improvements could interfere with their attack. Adobe has been aggressively adding mitigations to Flash Player over the last few months. In our release the week before the contest, we added changes to zero memory more often and leveraged the Windows’ low fragmentation heap. Both Adobe and Google found that some of the contestant’s entries overlapped with recent security reports. Part of the reason for the increasing payouts for the winners comes from the fact that the targets for the competition all have active security teams and external communities that are constantly working to improve the platform. As the community and vendors continue to mature their software, bug or mitigation collisions become more of a material risk for competitors. While it may not seem like it on the surface, the attackers are trying to hit moving targets.

The Flash Player updates prior to the contest led to some failed attempts at the competition. The failed attempts were by teams who had already won under different categories. Therefore, the competitors were clearly highly skilled and had already proven themselves to be capable of weaponizing exploits for the target platform. The reality is that what they are attempting to do is not easy and the failures serve to remind us how hard it is to get those wins. When any competition gets to a certain level, even the most skilled players are going to experience some losses.

That said, the contest always has its share of winners. Those wins demonstrate that there is always more that vendors can do in order to improve security. While the individual bugs help, Pwn2Own is truly valuable because it shows how different researchers will try to bypass the existing mitigations to create the fully weaponized exploit. That insight into different attack approaches inspires us as vendors to come up with the next generation of defenses.

Like many pros, the winning contestants always make it look easy. Although, as an industry we are often too quick to lump everything in the same “it is easy because it is completely vulnerable” bucket. Companies like Adobe, Microsoft, and Google are in a constant sprint with attackers. The security industry has progressed from the days of just trying to write clean, well validated code. Today, we are adding in large platform features that serve no other purpose than trying to thwart attackers. These types of features are added at an increasingly frequent basis. The companies who are on the front line of this battle will change and grow over time. It is important for those vying to one day be on the defender’s side of Pwn2Own to understand the current table stakes.

Overall, Pwn2Own is a fun contest to interact with security researchers and to push the industry forward. Beneath the high level pageantry of smoking jackets and large prizes, is a low-level escalation between offensive and defensive strategists. While the visual results from watching in the room seem similar from year to year, the innovation and challenge required to achieve those results increases every year.

Peleus Uhley
Principal Scientist

Community Collaboration Enhances Flash

With the December release of Flash Player, we introduced several new security enhancements. Just like the Flash Player mitigations we shipped earlier this year, many of these projects were the result of collaboration with the security community and our partners.

Adobe has spent the year working with Google and Microsoft on proactive mitigations. Some of the mitigations were minor tweaks to the environment: such as Google’s Project Zero helping us to add more heap randomization on Windows 7 or working with the Chrome team to tweak our use of the Pepper API for better sandboxing. There have also been a few larger scale collaborations.

For larger scale mitigations we tend to take a phased, iterative release approach. One of the advantages of this approach is that we can collect feedback to improve the design throughout implementation. Another advantage is that moving targets can increase the complexity of exploit development for attackers who depend on static environments for exploit reliability.

One example of a larger scale collaboration is our heap isolation work. This project initially started with a Project Zero code contribution to help isolate vectors. Based on the results of that release and discussions with the Microsoft research team, Adobe then expanded that code to cover ByteArrays. In last week’s release, Adobe deployed a rewrite of our memory manager to create the foundation for widespread heap isolation which we will build on, going forward. This change will limit the ability for attackers to effectively leverage use-after-free vulnerabilities for exploitation.

Another example of a larger scale mitigation this year was – with the assistance of Microsoft – our early adoption of Microsoft’s new Control Flow Guard (CFG) protection. Our first roll out of this mitigation was in late 2014 to help protect static code within Flash Player. In the first half of this year, we expanded our CFG usage to protect dynamic code generated by our Just-In-Time (JIT) compiler. In addition, Microsoft also worked with us to ensure that we could take advantage of the latest security controls for their new Edge browser.

Throughout 2015, vulnerability disclosure programs and the security community have been immensely helpful in identifying CVE’s. Approximately one-third of our reports this year were via Project Zero alone. Many of these were non-trivial as many of the reported bugs required significant manual research into the platform. With the help of the security community and partners like Microsoft and Google, Adobe has been able to introduce important new exploit mitigations into Flash Player and we are excited about what we are queuing up for next year’s improvements. Thank you to everyone who has contributed along the way.

Peleus Uhley
Principal Scientist

Better Security Through Automation

Automation Strategies

“Automate all the things!” is a popular meme in the cloud community. Many of today’s talks at security conferences discuss the latest, sophisticated automation tool developed by a particular organization. However, adding “automation” to a project does not magically make things better by itself. Any idea can be automated; including the bad ones. For instance, delivering false positives “at scale” is not going to help your teams. This blog will discuss some of the projects that we are currently working on and the reasoning behind their goals.

Computer science has been focused on automation since its inception. The advent of the cloud only frees our ideas from being resource bound by hardware. However, that doesn’t necessarily mean that automation must take up 100 scalable machines. Sometimes simple automation projects can have large impacts. Within Adobe, we have several types of automation projects underway to help us with security. The goals range from business-level dashboards and compliance projects to low level security testing projects.

 

Defining goals

One large project that we are currently building is a security automation framework focused on security assertions. When you run a traditional web security scanner against a site, it will try to tell you everything about everything on the site. In order to do that effectively, you have to do a lot of pre-configuration (authentication, excluded directories, etc.). Working with Mohit Kalra, the Sr, Security Manager for the ASSET security team, we experimented with the idea of security assertions. Basically, could a scanner answer one true/false question about the site with a high degree of accuracy? Then we would ask that one simple question across all of our properties in order to get a meaningful measurement.

For instance, let’s compare the following two possible automation goals for combating XSS:

(a) Traditional automation: Give me the location of every XSS vulnerability for this site.

(b) Security assertion: Does the site return a Content Security Policy (CSP) header?

A web application testing tool like ZAP can be used to automate either goal. Both of these tests can be conducted across all of your properties for testing at scale. Which goal you choose will decide the direction of your project:

Effort to implement:

(a) Potentially requires effort towards tuning and configuration with a robust scanner in order to get solid results. There is a potential risk to the tested environment (excessive DB entries, high traffic, etc.)

(b) A straight forward measurement with a simple scanner or script. There is a low risk to the tested environment.

Summarizing the result for management:

(a) This approach provides a complex measurement of risk that can involve several variables (reflected vs. persistent, potential value of the site, cookie strategy, etc.). The risk that is measured is a point-in-time assessment since new XSS bugs might be introduced later with new code.

(b) This approach provides a simple measurement of best practice adoption across the organization. A risk measurement can be inferred but it is not absolute. If CSP adoption is already high, then more fine grained tests targeting individual rules will be necessary. However, if CSP adoption is still in the early stages, then just measuring who has started the adoption process can be useful.

Developer interpretation of the result:

(a) Development teams will think in terms of immediate bugs filed.

(b) Development teams will focus on the long term goal of defining a basic CSP.

Both (a) and (b) have merits depending on the needs of the organization. The traditional strategy (a) can give you very specific data about how prevalent XSS bugs are across the organization. However, tuning the tools to effectively find and report all that data is a significant time investment. The security assertion strategy (b) focuses more on long term XSS mitigations by measuring CSP adoption within the organization. The test is simpler to implement with less risk to the target environments. Tackling smaller automation projects has the added value of providing experience that may be necessary when designing larger automation projects.

Which goal is a higher priority will depend on your organization’s current needs. We found that, in playing with the true/false approach of security assertions, we focused more of our energy on what data was necessary versus just what data was possible. In addition, since security assertions are assumed to be simple tests, we focused more of our design efforts on perfecting the architecture of scalable testing environment rather than the idiosyncrasies of the tools that the environment would be running. Many automation projects try to achieve depth and breadth at the same time by running complex tools at scale. We decided to take an intermediate step by using security assertions to focus on breadth first and then to layer on depth as we proceed.

 

Focused automation within continual deployment

Creating automation environments to scan entire organizations can be a long term project. Smaller automation projects can often provide quick wins and valuable experience on building automation. For instance, continuous build systems are often a single chokepoint through which a large portion of your cloud must pass before deployment. Many of today’s continuous build environments allow for extensions that can be used to automate processes.

As an example, PCI requires that code check-ins are reviewed. Verifying this process is followed consistently requires significant human labor. One of our Creative Cloud security champions, Jed Glazner, developed a Jenkins plugin which can verify each check-in was reviewed. The plugin monitors the specified branch and ensures that all commits belong to a pull request, and that the pull requests were not self merged. This allows for daily, automatic verification of the process for compliance.

Jed worked on a similar project where he created a Maven plug-in that lists all third-party Java libraries and their versions within the application. The plugin would then upload that information to our third-party library tracker so that we can immediately identify libraries that need updates. Since the plug-in was integrated into the Maven build system, the data provided to the third-party library tracker was always based on the latest nightly build and it was always a complete list.

 

Your organization will eventually build, buy or borrow a large scale automation tool that scales out the enumeration of immediate risk issues. However, before you jump head first into trying to build a robust scanning environment from scratch, be sure to first identify what core questions you need the tools to answer in order to support your program. You might find that starting with smaller automation tasks that track long term objectives or operational best practices can be just as useful to an organization. Deploying these smaller projects can also provide experience that can help your plans for larger automation projects.

 

Peleus Uhley
Principal Scientist