Author Archive: mkalra

Evolving an Application Security Team

A centralized application security team, similar to ours here at Adobe, can be the key to driving the security vision of the company. It helps implement the Secure Product Lifecycle (SPLC) and provide security expertise within the organization.  To stay current and have impact within the organization, a security team also needs to be in the mode of continuous evolution and learning. At inception of such a team, impact is usually localized more simply to applications that the team reviews.  As the team matures, the team can start to influence the security posture across the whole organization. I lead the team of security researchers at Adobe. Our team’s charter is to provide security expertise to all application teams in the company.  At Adobe, we have seen our team mature over time. As we look back, we would like to share the various phases of evolution that we have gone through along the way.

Stage 1:  Dig Deeper

In the first stage, the team is in the phase of forming and acquires the required security skills through hiring and organic learning. The individuals on the team bring varied security expertise, experience, and a desired skillset to the team. During this stage, the team looks for applicability of security knowledge to the applications that the engineering teams develop.  The security team starts this by doing deep dives into the application architecture and understanding why the products are being created in the first place. Here the team understands the organization’s application portfolio, observes common design patterns, and then starts to build the bigger picture on how applications come together as a solution.   Sharing learnings within the team is key to accelerating to the next stage.

By reviewing applications individually, the security team starts to understand the “elephants in the room” better and is also able to prioritize based on risk profile. A security team will primarily witness this stage during inception. But, it could witness it again if it undergoes major course changes, takes on new areas such as an acquisition, or must take on a new technical direction.

Stage 2: Research

In the second stage, the security team is already able to perform security reviews for most applications, or at least a thematically related group of them, with relative ease.  The security team may then start to observe gaps in their security knowhow due to things such as changes in broader industry or company engineering practices or adoption of new technology stacks.

During this phase, the security team starts to invest time in researching any necessary security tradeoffs and relative security strength of newer technologies being explored or adopted by application engineering teams. This research and its practical application within the organization has the benefit of helping to establish security experts on a wider range of topics within the team.

This stage helps security teams stay ahead of the curve, grow security subject matter expertise, update any training materials, and helps them give more meaningful security advice to other engineering teams. For example, Adobe’s application security team was initially skilled in desktop security best practices. It evolved its skillset as the company launched products centered around the cloud and mobile platforms. This newly acquired skillset required further specializationwhen the company started exploring more “bleeding edge” cloud technologies such as containerization for building micro-services.

Stage 3: Security Impact

As security teams become efficient in reviewing solutions and can keep up with technological advances, they can then start looking at homogeneous security improvements across their entire organization.  This has the potential of a much broader impact on the organization. Therefore, this requires the security team to be appropriately scalable to match possible increasing demands upon it.

If a security team member wants to make this broader impact, the first step is identification of a problem that can be applied to a larger set of applications.  In other words, you must ask members of a security team to pick and own a particularly interesting security problem and try to solve it across a larger section of the company.

Within Adobe, we were able to identify a handful of key projects that fit the above criteria for our security team to tackle. Some examples include:

  1. Defining the Amazon Web Services (AWS) minimum security bar for the entire company
  2. Implementing appropriate transport layer security (TLS) configurations on Adobe domains
  3. Validating that product teams did not store secrets or credentials in their source code
  4. Forcing use of browser supported security flags (i.e. XSS-Protection, X-Frame-Options, etc.) to help protect web applications.

The scope of these solutions varied from just business critical applications to the entire company.

Some guidelines that we set within our own team to achieve this were as follows:

  1. The problem statement, like an elevator pitch, should be simple and easily understandable by all levels within the engineering teams – including senior management.
  2. The security researcher was free to define the scope and choose how the problem could be solved.
  3. The improvements made by engineering teams should be measurable in a repeatable way. This would allow for easier identification of any security regressions.
  4. Existing channels for reporting security backlog items to engineering teams must be utilized versus spinning up new processes.
  5. Though automation is generally viewed as a key to scalability for these types of solutions, the team also had flexibility to adopt any method deemed most appropriate. For example, a researcher could front-load code analysis and only provide precise security flaws uncovered to the application engineering team.  Similarly, a researcher could establish required “minimum bars” for application engineering teams helping to set new company security standards. The onus is then placed on the application engineering teams to achieve compliance against the new or updated standards.

For projects that required running tests repeatedly, we leveraged our Security Automation Framework. This helped automate tasks such as validation. For others, clear standards were established for application security teams. Once a defined confidence goal is reached within the security team about compliance against those standards, automated validation could be introduced.

Pivoting Around an Application versus a Problem

When applications undergo a penetration test, threat modeling, or a tool-based scan, teams must first address critical issues before resolving lower priority issues. Such an effort probes an application from many directions attempting to extract all known security issues.  In this case, the focus is on the application and its security issues are not known when the engagement starts.  Once problems are found, the application team owns fixing it.

On the other hand, if you choose to tackle one of the broader security problems for the organization, you test against a range of applications, mitigate it as quickly as possible for those applications, and make a goal to eventually eradicate the issue entirely from the organization.  Today, teams are often forced into reactively resolving such big problems as critical issues – often due to broader industry security vulnerabilities that affect multiple companies all at once.  Heartbleed and other similar named vulnerabilities are good examples of this.  The Adobe security team attempts to resolve as many known issues as possible proactively in an attempt to help lessen the organizational disruption when big industry-wide issues come along. This approach is our recipe for having a positive security impact across the organization.

It is worth noting that security teams will move in and out of the above stages and the stages will tend to repeat themselves over time.  For example, a new acquisition or a new platform might require deeper dives to understand.  Similarly, new technology trends will require investment in research.  Going after specific, broad security problems complements the deep dives and helps improve the security posture for the entire company.

We have found it very useful to have members of the security team take ownership of these “really big” security trends we see and help improve results across the company around it. These efforts are ongoing and we will share more insights in future blog posts.

Mohit Kalra
Sr. Manager, Secure Software Engineering

Fingerprinting a Security Team

The central security team in a product development organization plays a vital role in implementing a secure product lifecycle process.  It is the team that drives the central security vision for the organization and works with individual teams on their proactive security needs.   I lead the technical team of proactive security researchers in Adobe. They are all recognized security experts and are able to help the company adapt to the ever changing threat landscape.  Apart from being on top of the latest security issues and potential mitigations that may need to be in place, the security team also faces challenges of constant skill evolution and remaining closely aligned to the business.

This post focuses on the challenges faced by the security team and potential ways to overcome them.

Increase in technologies as a function of time.

A company’s product portfolio is a combination of its existing products, new product launches, and acquisitions intended to help bridge product functionality gaps or expand into new business areas.  Over time, this brings a wide variety of technologies and architectures into the company.  Moreover, the pace of adoption of new technologies is much higher than the pace of retiring older technologies.  Therefore, the central security team needs to keep up with the newer technology stacks and architectures being adopted while also maintaining a manageable state with existing ones. An acquisition can further complicate this due to an influx of new technologies into the development environment in a very short period of time.

Security is not immune to business evolution.

The cloud and mobile space have forced companies to rethink how they should offer products and services to their customers.  Adobe went through a similar transformation from being a company that offers desktop products to one that attempts to strike the right balance between desktop, cloud, and mobile.  A security team needs to also quickly align with such business changes.

Multi-platform comes with a multiplication factor.

When the same product is offered on multiple operating systems, on multiple form factors (such as mobile and desktop), or deployed on multiple infrastructures, security considerations can increase due to the unique qualities of each platform. The central security team needs to be aware of and fluent in these considerations to provide effective proactive advice.

Subject matter expertise has limitations.

Strong subject matter expertise helps security teams’ credibility in imparting sound security advice to teams.  For security sensitive areas, experts in the team are essential to providing much deeper advice.  That said, any one individual cannot be an expert on every security topic.  Expertise is something that needs to be uniformly distributed through a team.

These challenges can be addressed by growing the team organically and through hiring.  Hiring to acquire new skills alone is not the best strategy – the skills required today will soon be outdated tomorrow.  A security team therefore needs to adopt strategies that allow it to constantly evolve and stay current. A few such strategies are discussed below.

T-Shaped skills.

Security researchers in a security team should aim for a T-Shaped skill set.  This allows for a fine balance between breadth and depth in security. The breadth is useful to help cover baseline security reviews.  The depth helps researchers become specific security subject matter experts. Having many subject experts strengthens the overall team’s skills because other team members learn from them and they are also available to provide guidance when there is a requirement in their area of expertise.

Strong Computer Science foundations.

Product security is an extension of engineering work.  Security requires understanding good design patterns, architecture, code, testing strategies, etc. Writing good software requires strong foundations in computer science irrespective of the layer of technology stack one ends up working on. Strong computer science skills can also help make security skills language and platform agnostic.  With strong computer science skills, a security researcher can learn new security concepts once and then apply to different platforms as and when needed.  With such strong fundamentals, the cost of finding out the “how” on new platforms is relatively small.

Hire for your gaps but also focus on ability to learn quickly.

A working product has so many pieces & processes that make it work.  If you can make a mental image of what it takes to make software, you can more clearly see strengths and weaknesses in your security team.  For example, engineering a service requires a good understanding of code (and the languages of choice), frameworks, technology stacks (such as queues, web server, backend database, third party libraries), infrastructure used for deploying, TLS configurations, testing methodologies, the source control system, the overall design and architecture, the REST interface, interconnection with various other services, the tool chain involved – the list is extensive. When hiring, one facet to evaluate in a candidate is whether he or she brings security strengths to the team through passion and past job experience that can fill the team’s existing gaps.  However, it can be even more important to evaluate the candidate’s willingness to learn new skills.  The ability to learn, adapt, and not be held captive to one existing skill set is an important factor to look for in candidates during hiring.  The secondary goal is to add a variety of security skills to the team and try to avoid duplicating the existing the skill set already in the team.

“Skate where the puck’s going, not where it’s been.”

To stay current with the business needs and where engineering teams are headed, it is important for a security team to spend a portion of their time investigating the security implications of newer technologies being adopted by the product teams.  As Wayne Gretzky famously said, “you want to skate where the puck’s going, not where it’s been.” However, security teams need to cover larger ground. You do have to stay current with new technologies being adopted. Older technologies still get used in the company as only some teams may move away from them. So it would be wise not to ignore those older technologies by maintaining expertise in those areas, while aiming to move teams away from those technologies as they become more difficult to effectively secure.  Predicting future areas of investment is difficult.  Security teams can make that task easier by looking at the industry trends and by talking to engineering teams to find out where are they headed.  The managers of a security team also have a responsibility to stay informed about new technologies, as well as future directions their respective companies may go in, in order to invest in newer areas to grow the team.

Go with the flow.

If a business has taken a decision to invest in cloud or mobile or change the way it does business, a security team should be among the first in the company to detect this change and make plans to adapt early.  If the business moves in a certain direction and the security team does not, it can unfortunately label a team as being one that only knows the older technology stack.  Moreover, it is vital for the security team to show alignment with a changing business. It is primarily the responsibility of the security team’s leadership to detect such changes and start planning for them early.

Automate and create time.

If a task is performed multiple times, the security team should evaluate if the task can be automated or if a tool can do it more efficiently.  The time reduced through automation and tooling can help free up time and resources which can then be used to invest in newer areas that are a priority for the security team.

Growing a security team can have many underlying challenges that are not always obvious to an external observer.  The industry’s primary focus is on the new threat landscapes being faced by the business.  A healthy mix of organic growth and hiring will help a security team adapt and evolve continuously to the changes being introduced by factors not in their direct control.  It is the responsibility of both security researchers and the management team to keep learning and to spend time detecting any undercurrents of change in the security space.

Mohit Kalra
Sr. Manager, Secure Software Engineering