Adobe Document Cloud is now PCI DSS 3.1 compliant

The Payment Card Industry Data Security Standard (PCI DSS) prescribes certain security controls for organizations that accept payments via credit card.  The standard is designed to help reduce fraud by increasing controls around cardholder data.

On June 30, 2016, Adobe Document Cloud (which includes Adobe Sign and PDF Services) achieved compliance with PCI DSS 3.1 as a merchant and a service provider.

The Adobe Document Cloud’s PCI compliant status as a service provider helps our customers meet PCI requirements for safe handling of cardholder data.

The Adobe Common Controls Framework (CCF) and the underlying Security Compliance strategy helped us meet the current PCI requirements.  Any changes to the PCI standard are proactively incorporated into the CCF to ensure on-going compliance for all Adobe businesses.

More information about our Common Controls Framework and compliance efforts can be found on Adobe.com.

Abhi Pandit
Sr. Director, Risk Advisory and Assurance Services (RaaS)

Join Members of our Security Team at AppSec Europe and Security of Things World

Our director of secure software engineering, Dave Lenoe, will be speaking at the upcoming Security of Things World conference in Berlin, Germany, June 27 – 28. In addition, two more members of our security team will also be speaking at the upcoming OWASP AppSec Europe conference in Rome, Italy, June 27 – July 1.

First up is Dave at Security of Things World. He will be speaking about how Adobe engages with the broader security community for both proactive and reactive assistance in finding and resolving vulnerabilities in our solutions. You can join him on Monday, June 27, at 2:30 p.m.

Next up will be Julia Knecht, security analyst for Adobe Marketing Cloud, at OWASP AppSec Europe to share lessons learned from developing and employing an effective Secure Product Lifecycle (SPLC) process for our Marketing Cloud solutions. This session will give you on-the-ground knowledge that may assist you in developing your own SAAS-ready SPLC that helps break down silos in your organization, making it more agile and effective at building secure solutions. Julia’s session will be on Thursday, June 30th, at 3:00 p.m.

Finally, Vaibhav Gupta, security researcher, will be leading a “lightning training” on the OWASP Zed Attack Proxy (ZAP) tool at OWASP AppSec Europe. ZAP is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers. It helps you automatically find security vulnerabilities in your web applications while you are developing and testing them. This training is focused on helping you with ZAP automation to enable better integration of it into your DevOps environment. Vaibhav’s session will be on Friday, July 1st, at 10:20 a.m.

If you will be at either of these conferences next week, we hope you can join our team for their sessions and conversation after or in the hallways throughout the event.

Striving for a Security Culture

Security is a discipline that is forever changing. There are always new threats on the horizon and we are constantly warning people that software is never as secure as we might like it as we cannot predict the future. Be alert! Never let your guard down! Think like an attacker! These are well tread refrains.

The truth is, however, none of us can be a perfect sentry. Yes, there is a lot of security work that can be automated and integrated into the product lifecycle, but there will always be a human element and humans are not perfect. They can get bored, complacent, and rely too much on crutches that have worked for them in the past. Also, there are limits to the amount of information that a person can take in. A focus on security can get lost amongst things like needing to create new features or meeting deadlines.

So how do we make sure that security awareness is not lost? That all the training we spend precious treasure creating doesn’t get lost in the noise?

The answer is to systematically work towards creating a culture of security.

A healthy security culture is one where folks have internalized your security protocols to the extent that desirable security actions are the default. For example: suspicious activity on a server? Immediately report it to the incident response team.

Culture is what you do with the information, education and tools that you have been exposed to. Creating an organizational culture strong in security principles and awareness requires identifying deliberate behaviors, creating opportunities for these behaviors to exist, rewarding them, and measuring the result. This can often be achieved through “gamification”.

So, how did we use this technique effectively? We wanted to foster a culture of open communication about security. We want everyone to be able to join the conversation, but we knew we were going to need to find allies in order to make the conversation authentic and genuine. We created an e-mail distribution list and we started inviting everyone who took our training to join. We also encouraged our security champions to join as watchers and helpers for any issues on the thread.

After a while, it became clear that our black and brown belts (the most advanced trainees) were very active on the list, researching and answering questions that would have cost our dedicated Security Researchers valuable time. We maintain a program where we give spontaneous positive feedback to managers, and the brown and black belt holders are consistently at the top of that list for their participation.

Over the years, the list has grown considerably, it is self-mediated, and remains a thriving community where anyone can ask a question. It is the first place we go to make announcements about our security efforts, where we learn from across the company tips and tricks to help keep things more secure, and where we share the latest security news.

Let’s break down further what we did here:

  1. Declared goal – open communication
  2. Created e-mail list and advertised it to champions and thought leaders
  3. Let people interact without over moderating
  4. Reward participants with positive feedback, engage in their topics, and encourage public praise
  5. Keep track of subscriber growth

Here are a few of the many dividends that have come from this tool to encourage our open security culture:

  • Security issues or potential problems within our products can be more quickly identified
  • We are often made aware of cutting edge industry and academic research that might be useful
  • There is a sense of “community” amongst members of this list
  • It has become easier to spread information to the right people very quickly

As mentioned above, security is an ever moving target. Security culture efforts need to adapt to meet that target. The culture of openness example above is a broad target designed to change a complex set of behaviors. You can use the same strategy to address more simple behaviors – like badge surfing or locking computer screens. Prioritize the behaviors you want to change, then focus on them, reward people for even the smallest of victories and measure the difference in the occurrence of the behavior.

Josh Kebbel-Wyen
Sr. Security Awareness and Training Program Manager

Look Back on 10 Years of Incident Response at Adobe @ FIRST 2016

This coming Tuesday, June 14th, from 1:00 – 2:30 p.m., join Tom Cignarella, Director of our Security Coordination Center (SCC), and Dave Lenoe, Director of Product Security, at the FIRST Conference in Seoul, Korea, for an insightful look back at the past 10 years of our work in incident response. Incident response at Adobe started off when the Product Security team was first formed working mostly on coordinated disclosure (called ‘responsible disclosure’ back then) of vulnerabilities from security researchers and partners. After a couple of years, relying on coordinated disclosure became more challenging as exploits in the wild against Adobe products began to proliferate. As the product and threat landscape evolved further, with hosted services entering into the mix, we began to see that vulnerability response and traditional network incident response were overlapping, and a new approach was required to tackle these changes.

Tom and Dave will talk about our journey, lessons we’ve learned along the way, detail our new approach now that we are a cloud services-based company, and where we see our incident response programs going in the future. You can also get a preview from Tom and Dave on this talk via a podcast from the FIRST team. If you are attending the FIRST Conference in Seoul, please join us or grab Tom or Dave in the hallway to share stories.

Scaling Security Controls Across the Enterprise

Adobe is changing the world through digital experiences. We do that in an incredibly creative and innovative environment where we also take our customers’ data security and privacy very seriously.  As a large and growing cloud company, achieving these things at the same time takes a sound strategy focused on two key pillars:

  • Establish Effective, Enforceable Security Standards
  • Implement and Adopt Simple, Scalable Security Services

Establish Effective, Enforceable Security Standards

The Common Controls Framework (CCF) is a comprehensive set of simple control requirements, rationalized from the alphabet soup of several different industry information security and privacy standards. To help ensure that our standards effectively meet our customers’ expectations, we are constantly refining this framework based on industry requirement changes, customer asks, and internal feedback.

In a recent update, Abhi Pandit (Adobe’s Sr. Director of Risk Advisory & Assurance Services) stated that “[Adobe] has made significant investments across the company to harmonize various security functions, compliance and governance processes, and technologies.” The CCF is the backbone of our corporate security policies and standards.

Implement and Adopt Simple, Scalable Security Services

With a common vision for customer data security and privacy set, we can look to our talented engineers to develop and implement the solutions that help meet our objectives.  Engineering teams collaborate to develop scalable solutions that implement the CCF requirements in the most efficient and elegant way possible. Other teams leverage those services and help improve them over time.

A common pitfall seen with initiatives like the CCF is that teams may try to implement all of the standards on their own with little or no reliance on cross-functional services. Not only does this undermine the entire purpose of a common standard, it can often result in some undesirable outcomes. Examples include:

  • Team resources overwhelmed by compliance initiatives and ability to deliver on product features may be hindered.
  • Standalone compliance initiatives only partly address requirements and security may be compromised.
  • Exhausted team resources lead to operational failure of compliance responsibilities and security may be compromised.

At Adobe, we make a concerted effort to help product teams use simple, scalable services to help prevent these undesirable outcomes. These services are a combination of internally developed services and reliable third party tools. Teams focus their effort on adopting those security services helping to free up their time to focus more on features that deliver a delightful, more secure user experience.

Practical Example of Implementation

Adobe uses four different types of control “roles” to organize our compliance efforts to meet the CCF standards: Driver, Subscriber, Contributor, and Standalone.

A driver is ultimately responsible for developing a service, including controls, that will mitigate a particular risk associated with a process and address the CCF requirements. For example, a driver implements a robust Identity Management system that can provide automated workflows for logical access control.

A subscriber is responsible to integrate their systems and processes in a way that will take advantage of the driver’s service. Continuing with the example above, a subscriber makes sure the only way to access their system is through the driver’s identity management system. As long as this continues, they share the risk with the driver.

A contributor is like a subscriber, integrated with a driver’s solution, but they may have a more active role in executing the control. A great example is a periodic access review. The driver’s identity management system gives contributors a notification that it’s time to review the access privileges in their respective groups so that they can certify the access is appropriate. They use a tool that makes it easy and effective—much easier and effective than a manual process that leverages a ticket or email.

If a team decided to tackle these problems as a standalone, they would need to set up their own identity management system or manual procedures to handle logical access control. These tend to be more manual and at a higher risk for control failure. The advantage of the CCF framework is these necessary core services and controls are provided and teams do not need to tackle these issues on their own, helping to lessen the overall risk.

The table below outlines a summary of these different roles and how they are currently viewed at Adobe.

Adobe CCF Control Roles

Role Business Objective Risk Responsibility
Driver Implement robust service for all teams to satisfy control requirements. Reduce instances of services to the minimum possible and improve the services over time. Mitigate
Subscriber Adopt robust service provided by Driver Transfer/Share
Contributor Leverage robust service provided by Driver and work with them to meet requirements Transfer/Share
Standalone Implement a process to satisfy control requirements. Reduce instances to the minimum possible. Mitigate

Conclusion

As Adobe grows, it is the hope that the number of compliance procedures should not grow at the same rate. The simple concepts explained in this post make up some of the secret sauce of how to leverage a compliance program to more effectively mitigate information security and privacy risk in a scalable way.

Kenny Scott
Manager, I.T. and Information Security, Risk Advisory and Assurance Services

Building a Team of Digital Marketing Security Champions

Two years ago, I joined the Digital Marketing Product Security Team and took on the responsibility of establishing and managing the Secure Product Lifecycle (SPLC) process for Digital Marketing Product Engineering. There are currently eight Digital Marketing Solutions with engineering teams located all over the world.  Many of these solutions came to Adobe by way of acquisition.  I work with differing stacks, languages, company cultures, and time zones.  I knew some of the engineers from having run our 3rd Party Penetration Testing program for three years – however, I was mostly starting the process from scratch. My main goal was to lower security overhead in the product development cycle and leverage existing processes.

I am very passionate about quickly making improved security an integrated part of our product development and leveraging as many existing processes and tools as possible.   In order to promote security knowledge throughout the large Digital Marketing engineering organization.  I created a human “botnet” of security champions. These champions come from positions all over the organization and coordinate with our security team to facilitate ongoing management and enforcement of our SPLC process.

Security, admittedly, has a bit of an “image problem” among development teams.  It is something that developers often think of as this big, scary set of tasks intended to make their jobs more difficult or less enjoyable.  We placed a big emphasis on changing this perception. The Digital Marketing Security team is focused on being a supportive, service organization – a far cry from the perception that we can be a terrible force of nature leaving engineers feeling like they’ve been hit by a truck or would like to be.  Rather than coming in with the metaphorical hammer, we thought, “can we get people to actually enjoy their interactions with our security team?  How can we make this incredibly important, but often dreaded, piece of software development an integral and easier to implement piece of the existing process?”

The first thing I did was to meet with the solution owners and program managers to learn about how these teams develop and deliver software for these SAAS offerings.  Adobe has an incredible program management network, and an existing Service Lifecycle program that I was able to leverage and adapt to help meet requirements of our Secure Product Lifecycle.  I worked with the program managers to figure out how we could best add SPLC steps to their development and release process. I also ensured we had a clear process for adding security requirements and checkpoints to the release process. I worked with solution engineering directors to identify Security Champions on their engineering teams who would work with me to continue to improve our approach to security for the solutions.

A Security Champion is: ‘An advocate of security and the Digital Marketing Security team’s point of contact for the solution.  The champion has a good understanding of the technology, an interest in ensuring better security for their offering, and a strong personal network in the engineering organization.’  Once this human “botnet” of Security Champions was established, the heavy lifting began.  I set key performance indicators (KPIs) for the different elements of the SPLC around security training, threat modeling, static/dynamic analysis and penetration testing. The very first KPI that we focus on, for the purpose of enabling the proper background for having security conversations with the engineers, is technical security training.  Adobe’s corporate secure software engineering team (known as “ASSET”) has created a fantastic training program that focuses on technical security topics and awards certifications in the form of white and green belts, similar to karate training. Each of the program managers have added this training to the new engineer onboarding steps and they and the security champions have helped to develop strong measurements for the other KPIs.

My Security Champions helped increase the pervasiveness of our “security culture” more than I could have imagined when first starting this program.  They are one of the driving forces in helping to further improve security across Adobe’s Digital Marketing solutions.  They have been an amazing force multiplier helping to prioritize security practices in their teams’ design process, roadmap development, and mindset.

About 6 months after kicking off the Security Champions program, Digital Marketing Engineering had grown their base of security knowledge to have over 95% of their engineers white and green belt certified.  We’d also increased the number of threat models, penetration tests, ongoing security projects, and automated security testing. Our metrics against these initiatives have continued to increase and improve. The teams are more proactively involving the Digital Marketing and corporate security teams in their design discussions helping to ensure better security implementations throughout the process.

Messages like this from the teams show it’s working and make it all worth it:

Picture1

We’re committed to building and maintaining the trust of our Digital Marketing customers by developing and providing them with the most secure software possible – solutions that help meet business demands and allow configurations to help meet their security and compliance needs.  The SPLC and Security Champions program have helped to broaden the security knowledge and awareness of the Digital Marketing engineering teams.  We will continue to raise that bar by continuing to iterate and improve on these programs.

Julia Knecht
Security Analyst, Digital Marketing

Adobe CCF Enables Quicker Adherence to Updated PCI Standards

The Adobe.com e-commerce store has been a PCI level 1 certified merchant for the last few years.  Adobe has significantly reduced its Card Holder Data environment (CDE) scope for this environment by using an external tokenization solution and maintains PAN-free environment by not storing any Primary Account Numbers (PAN) in its internal network. Adobe has implemented its Common Controls Framework (CCF) within the Card Holder Data environment which allows it to use the same set of controls to meet with requirements set forth by Payment Card Industry Data Security Standard PCI DSS V3.1 and many other security/compliance frameworks like ISO27001:2013, SOC2, among others. CCF is a set of approximately 250 controls designed specifically for Adobe’s business and provides the benefit by rationalizing the overlapping requirements across 10 different compliance and security frameworks.

PCI Security Standards Council (PCI SSC) recently released the latest version of the Data Security Standard V3.2. One of the notable changes in the PCI DSS V3.2 is the additional clarification provided around the use of multi-factor authentication for all administrative and remote access to the CDE.

PCI DSS V3.2 reference:

“8.3 Secure all individual non-console administrative access and all remote access to the CDE using multi-factor authentication.”

 By implementing CCF within the CDE, Adobe has already established a baseline control which requires all remote VPN sessions and production environments to be accessed via multi – factor authentication. This baseline control was adopted to meet with the requirements established by the more stringent of the compliance frameworks, hence allowing for Adobe to already be complaint with the clarifications provided in the PCI DSS v3.2 around multi-factor authentication.

Prasant Vadlamudi
Manager, Risk Advisory and Assurance Services (RAAS)

Fingerprinting a Security Team

The central security team in a product development organization plays a vital role in implementing a secure product lifecycle process.  It is the team that drives the central security vision for the organization and works with individual teams on their proactive security needs.   I lead the technical team of proactive security researchers in Adobe. They are all recognized security experts and are able to help the company adapt to the ever changing threat landscape.  Apart from being on top of the latest security issues and potential mitigations that may need to be in place, the security team also faces challenges of constant skill evolution and remaining closely aligned to the business.

This post focuses on the challenges faced by the security team and potential ways to overcome them.

Increase in technologies as a function of time.

A company’s product portfolio is a combination of its existing products, new product launches, and acquisitions intended to help bridge product functionality gaps or expand into new business areas.  Over time, this brings a wide variety of technologies and architectures into the company.  Moreover, the pace of adoption of new technologies is much higher than the pace of retiring older technologies.  Therefore, the central security team needs to keep up with the newer technology stacks and architectures being adopted while also maintaining a manageable state with existing ones. An acquisition can further complicate this due to an influx of new technologies into the development environment in a very short period of time.

Security is not immune to business evolution.

The cloud and mobile space have forced companies to rethink how they should offer products and services to their customers.  Adobe went through a similar transformation from being a company that offers desktop products to one that attempts to strike the right balance between desktop, cloud, and mobile.  A security team needs to also quickly align with such business changes.

Multi-platform comes with a multiplication factor.

When the same product is offered on multiple operating systems, on multiple form factors (such as mobile and desktop), or deployed on multiple infrastructures, security considerations can increase due to the unique qualities of each platform. The central security team needs to be aware of and fluent in these considerations to provide effective proactive advice.

Subject matter expertise has limitations.

Strong subject matter expertise helps security teams’ credibility in imparting sound security advice to teams.  For security sensitive areas, experts in the team are essential to providing much deeper advice.  That said, any one individual cannot be an expert on every security topic.  Expertise is something that needs to be uniformly distributed through a team.

These challenges can be addressed by growing the team organically and through hiring.  Hiring to acquire new skills alone is not the best strategy – the skills required today will soon be outdated tomorrow.  A security team therefore needs to adopt strategies that allow it to constantly evolve and stay current. A few such strategies are discussed below.

T-Shaped skills.

Security researchers in a security team should aim for a T-Shaped skill set.  This allows for a fine balance between breadth and depth in security. The breadth is useful to help cover baseline security reviews.  The depth helps researchers become specific security subject matter experts. Having many subject experts strengthens the overall team’s skills because other team members learn from them and they are also available to provide guidance when there is a requirement in their area of expertise.

Strong Computer Science foundations.

Product security is an extension of engineering work.  Security requires understanding good design patterns, architecture, code, testing strategies, etc. Writing good software requires strong foundations in computer science irrespective of the layer of technology stack one ends up working on. Strong computer science skills can also help make security skills language and platform agnostic.  With strong computer science skills, a security researcher can learn new security concepts once and then apply to different platforms as and when needed.  With such strong fundamentals, the cost of finding out the “how” on new platforms is relatively small.

Hire for your gaps but also focus on ability to learn quickly.

A working product has so many pieces & processes that make it work.  If you can make a mental image of what it takes to make software, you can more clearly see strengths and weaknesses in your security team.  For example, engineering a service requires a good understanding of code (and the languages of choice), frameworks, technology stacks (such as queues, web server, backend database, third party libraries), infrastructure used for deploying, TLS configurations, testing methodologies, the source control system, the overall design and architecture, the REST interface, interconnection with various other services, the tool chain involved – the list is extensive. When hiring, one facet to evaluate in a candidate is whether he or she brings security strengths to the team through passion and past job experience that can fill the team’s existing gaps.  However, it can be even more important to evaluate the candidate’s willingness to learn new skills.  The ability to learn, adapt, and not be held captive to one existing skill set is an important factor to look for in candidates during hiring.  The secondary goal is to add a variety of security skills to the team and try to avoid duplicating the existing the skill set already in the team.

“Skate where the puck’s going, not where it’s been.”

To stay current with the business needs and where engineering teams are headed, it is important for a security team to spend a portion of their time investigating the security implications of newer technologies being adopted by the product teams.  As Wayne Gretzky famously said, “you want to skate where the puck’s going, not where it’s been.” However, security teams need to cover larger ground. You do have to stay current with new technologies being adopted. Older technologies still get used in the company as only some teams may move away from them. So it would be wise not to ignore those older technologies by maintaining expertise in those areas, while aiming to move teams away from those technologies as they become more difficult to effectively secure.  Predicting future areas of investment is difficult.  Security teams can make that task easier by looking at the industry trends and by talking to engineering teams to find out where are they headed.  The managers of a security team also have a responsibility to stay informed about new technologies, as well as future directions their respective companies may go in, in order to invest in newer areas to grow the team.

Go with the flow.

If a business has taken a decision to invest in cloud or mobile or change the way it does business, a security team should be among the first in the company to detect this change and make plans to adapt early.  If the business moves in a certain direction and the security team does not, it can unfortunately label a team as being one that only knows the older technology stack.  Moreover, it is vital for the security team to show alignment with a changing business. It is primarily the responsibility of the security team’s leadership to detect such changes and start planning for them early.

Automate and create time.

If a task is performed multiple times, the security team should evaluate if the task can be automated or if a tool can do it more efficiently.  The time reduced through automation and tooling can help free up time and resources which can then be used to invest in newer areas that are a priority for the security team.

Growing a security team can have many underlying challenges that are not always obvious to an external observer.  The industry’s primary focus is on the new threat landscapes being faced by the business.  A healthy mix of organic growth and hiring will help a security team adapt and evolve continuously to the changes being introduced by factors not in their direct control.  It is the responsibility of both security researchers and the management team to keep learning and to spend time detecting any undercurrents of change in the security space.

Mohit Kalra
Sr. Manager, Secure Software Engineering

A Vendor Perspective on Crowd Sourced Penetration Tests

Bug bounties, also known as crowd sourced penetration tests, are becoming increasingly popular. New programs are announced every month. At NullCon this year, there was an entire track dedicated to the topic where vendors and researchers could meet. For a security researcher, there are a ton of options for participating ranging from the self-run programs, such as Google’s, to participating on consolidated platforms like BugCrowd and HackerOne. However, for the vendor, the path into bug bounties can be somewhat complex and the most significant benefits are not always obvious. Here are some tips on how to get more from your bug bounty.

Preparation

You should pick a team that has gone through several traditional penetration tests and where the ROI from those tests is trending down. If traditional consultants are still finding numerous bugs and architectural issues, your time and money would be better spent addressing the known issues and strengthening the architecture. Testing against a more mature development team can also benefit in other ways as you will soon see. A good crowd-sourced penetration test will involve both sides, researchers and development teams, being active in the bounty program.

If you have never done a bounty before, starting with short-term, private bounties will allow you to experience a few hiccups in a controlled situation. Be sure that you have planned out how to issue accounts to a large number of users and that the environment works when testing from outside your corporate environment. Try testing from home just to make sure it works.

Bounty guidelines

The large number of public bounties can serve as a baseline template for your test rules. As you review them, be sure to take note of their differences and consider what may have lead to those differences. A good set of bounty rules will be tailored to the service being tested. One of the less obvious components of a bounty announcement is how you describe your service to the tester. While the service may be extremely popular within your social circles, a researcher across the globe may have never heard of it. Therefore, be sure your bounty description provides an easy-to-understand description of what they are testing and perhaps a link to a short YouTube video that has your product pitch. The less time a researcher has to spend figuring out the goal of the service, the more time they can spend finding quality bugs.

Thematic issues

Penetration tests are typically scoped to a certain set of new features. However, crowd sourced penetration tests are often scoped across the entire service. Since traditional penetration tests are often focused on specific areas, they will not find issues in the connective code between features. Also, since the researchers are testing across the entire service, they are testing across the entire development team and not just within individual sprint teams. This may allow you to pick up on things that the overall team is consistently missing which can guide you as to where to focus energy going forward. For instance, if you have several authorization bugs, then is there a way to better consolidate authorization checking within the platform or is there a way to enable the quality team to better test these issues?

Critical bugs

Since the bounty hunters usually want to get top dollar for their efforts, they will often find more critical bugs. A critical bug is often the result of multiple issues that aren’t mentioned in the initial write-up. For instance, if they send you your password file, then there should be multiple questions beyond what type of injection was used in the attack. A few examples: Would egress filters on the network help? Do we need host monitoring solution to detect when the server process touches unexpected files? It is important to remember that these critical bugs aren’t just theoretical issues found through a code review. These vulnerabilities were successfully exploited issues found via black box testing of your infrastructure from a remote location.

Variant testing

If you have developers on hand during the bounty, then the developers can push the patch to the staging environment before the end of the program. You can then reach out to researcher and say, “Bet you can’t do that twice!”  You basically offer the researcher a separate bounty if they can find a variant or the same bug in a different API. It often isn’t difficult for the researcher to re-test something they have already tested. For the developer, they can get immediate feedback on the patch while the issue is still fresh in their minds. In my experiments at Adobe, losing that bet with the researcher is more valuable than the money it costs us because it typically identifies some broader issue with the platform or the process. This can be key for critical bugs.

Red Team/Blue Team

With a crowd sourced penetration test, you are likely testing against your staging environment or a dedicated server in order to minimize risk to your production network. A staging environment typically has low traffic volumes since only the product team is using it. However, during the testing period, you will have people from across the globe testing that environment and reporting the vulnerabilities that they are finding. For your response teams, this is an excellent opportunity to see what your logs captured about the attack. In theory, identifying the attack should be straight forward since the staging environment is low volume, you know what attack occurred, and you have a rough estimate of when the attack occurred. If you can’t find an attack in your logs under those conditions, then that is clear feedback about how your logging and monitoring can be improved. If you can save the logs until after the bounty has ended, this type of analysis can be done post-assessment if you don’t have the resources to play along real time.

A crowd-sourced penetration test can change up the routine you have established for finding issues. Like any change in routine, there can be a few challenges at first. However, when done well, they can provide a vendor with insights that they may have never obtained through the existing status quo. These are not a replacement for traditional consultants. Rather, the new insights into the platform can help you re-focus the consultants more effectively to get a higher ROI.

 

Peleus Uhley
Principal Scientist

Reflections on Pwn2Own

Returning from CanSecWest left me reflecting on how the Pwn2Own competition has evolved over time. A lot has changed in the Pwn2Own competition over the years. The event has grown in attendance, competitors, and complexity, just as the industry has grown.

For the first contest in 2007, no one competed on the first day. Shane McCauley called fellow security researcher Dino Dai Zovi in NYC that night and urged him to compete. Dino was able to write the exploit over night and win the contest on the second day. Visually, the attacks seem no different from year to year. The contestant sits down at the machine, and “seconds” later the calculator (or notepad in this year’s case), pops up on the screen.  However, the preparation for the attacks from 2007 to 2016 are now drastically different, with contestants preparing attacks weeks in advance.

Media coverage around security advisories is often just a run down of how many CVEs a vendor released that month. People often imply from these articles that all the CVEs are easily exploitable and trivially weaponizable. This can lead to false perceptions that exploiting the software is a simple task. In reality, a CVE from a vendor is not a guarantee of exploitability. Even if the CVE can give the attacker the ability to overwrite memory, that is not a guarantee that it can be weaponized. Technologies such as ASLR (Address Space Layout Randomization) weren’t even released for Mac OS X when Dai Zovi competed in 2007. Today’s attackers have to work around defenses such as CFG (Control Flow Guard), Isolated Heaps, and a number of other technologies designed to prevent a crash from becoming an exploit.

In addition, competitors have to deal with the fact that the contest frequently occurs after Patch Tuesday where a vendor’s security improvements could interfere with their attack. Adobe has been aggressively adding mitigations to Flash Player over the last few months. In our release the week before the contest, we added changes to zero memory more often and leveraged the Windows’ low fragmentation heap. Both Adobe and Google found that some of the contestant’s entries overlapped with recent security reports. Part of the reason for the increasing payouts for the winners comes from the fact that the targets for the competition all have active security teams and external communities that are constantly working to improve the platform. As the community and vendors continue to mature their software, bug or mitigation collisions become more of a material risk for competitors. While it may not seem like it on the surface, the attackers are trying to hit moving targets.

The Flash Player updates prior to the contest led to some failed attempts at the competition. The failed attempts were by teams who had already won under different categories. Therefore, the competitors were clearly highly skilled and had already proven themselves to be capable of weaponizing exploits for the target platform. The reality is that what they are attempting to do is not easy and the failures serve to remind us how hard it is to get those wins. When any competition gets to a certain level, even the most skilled players are going to experience some losses.

That said, the contest always has its share of winners. Those wins demonstrate that there is always more that vendors can do in order to improve security. While the individual bugs help, Pwn2Own is truly valuable because it shows how different researchers will try to bypass the existing mitigations to create the fully weaponized exploit. That insight into different attack approaches inspires us as vendors to come up with the next generation of defenses.

Like many pros, the winning contestants always make it look easy. Although, as an industry we are often too quick to lump everything in the same “it is easy because it is completely vulnerable” bucket. Companies like Adobe, Microsoft, and Google are in a constant sprint with attackers. The security industry has progressed from the days of just trying to write clean, well validated code. Today, we are adding in large platform features that serve no other purpose than trying to thwart attackers. These types of features are added at an increasingly frequent basis. The companies who are on the front line of this battle will change and grow over time. It is important for those vying to one day be on the defender’s side of Pwn2Own to understand the current table stakes.

Overall, Pwn2Own is a fun contest to interact with security researchers and to push the industry forward. Beneath the high level pageantry of smoking jackets and large prizes, is a low-level escalation between offensive and defensive strategists. While the visual results from watching in the room seem similar from year to year, the innovation and challenge required to achieve those results increases every year.

Peleus Uhley
Principal Scientist