A Vendor Perspective on Crowd Sourced Penetration Tests

Bug bounties, also known as crowd sourced penetration tests, are becoming increasingly popular. New programs are announced every month. At NullCon this year, there was an entire track dedicated to the topic where vendors and researchers could meet. For a security researcher, there are a ton of options for participating ranging from the self-run programs, such as Google’s, to participating on consolidated platforms like BugCrowd and HackerOne. However, for the vendor, the path into bug bounties can be somewhat complex and the most significant benefits are not always obvious. Here are some tips on how to get more from your bug bounty.


You should pick a team that has gone through several traditional penetration tests and where the ROI from those tests is trending down. If traditional consultants are still finding numerous bugs and architectural issues, your time and money would be better spent addressing the known issues and strengthening the architecture. Testing against a more mature development team can also benefit in other ways as you will soon see. A good crowd-sourced penetration test will involve both sides, researchers and development teams, being active in the bounty program.

If you have never done a bounty before, starting with short-term, private bounties will allow you to experience a few hiccups in a controlled situation. Be sure that you have planned out how to issue accounts to a large number of users and that the environment works when testing from outside your corporate environment. Try testing from home just to make sure it works.

Bounty guidelines

The large number of public bounties can serve as a baseline template for your test rules. As you review them, be sure to take note of their differences and consider what may have lead to those differences. A good set of bounty rules will be tailored to the service being tested. One of the less obvious components of a bounty announcement is how you describe your service to the tester. While the service may be extremely popular within your social circles, a researcher across the globe may have never heard of it. Therefore, be sure your bounty description provides an easy-to-understand description of what they are testing and perhaps a link to a short YouTube video that has your product pitch. The less time a researcher has to spend figuring out the goal of the service, the more time they can spend finding quality bugs.

Thematic issues

Penetration tests are typically scoped to a certain set of new features. However, crowd sourced penetration tests are often scoped across the entire service. Since traditional penetration tests are often focused on specific areas, they will not find issues in the connective code between features. Also, since the researchers are testing across the entire service, they are testing across the entire development team and not just within individual sprint teams. This may allow you to pick up on things that the overall team is consistently missing which can guide you as to where to focus energy going forward. For instance, if you have several authorization bugs, then is there a way to better consolidate authorization checking within the platform or is there a way to enable the quality team to better test these issues?

Critical bugs

Since the bounty hunters usually want to get top dollar for their efforts, they will often find more critical bugs. A critical bug is often the result of multiple issues that aren’t mentioned in the initial write-up. For instance, if they send you your password file, then there should be multiple questions beyond what type of injection was used in the attack. A few examples: Would egress filters on the network help? Do we need host monitoring solution to detect when the server process touches unexpected files? It is important to remember that these critical bugs aren’t just theoretical issues found through a code review. These vulnerabilities were successfully exploited issues found via black box testing of your infrastructure from a remote location.

Variant testing

If you have developers on hand during the bounty, then the developers can push the patch to the staging environment before the end of the program. You can then reach out to researcher and say, “Bet you can’t do that twice!”  You basically offer the researcher a separate bounty if they can find a variant or the same bug in a different API. It often isn’t difficult for the researcher to re-test something they have already tested. For the developer, they can get immediate feedback on the patch while the issue is still fresh in their minds. In my experiments at Adobe, losing that bet with the researcher is more valuable than the money it costs us because it typically identifies some broader issue with the platform or the process. This can be key for critical bugs.

Red Team/Blue Team

With a crowd sourced penetration test, you are likely testing against your staging environment or a dedicated server in order to minimize risk to your production network. A staging environment typically has low traffic volumes since only the product team is using it. However, during the testing period, you will have people from across the globe testing that environment and reporting the vulnerabilities that they are finding. For your response teams, this is an excellent opportunity to see what your logs captured about the attack. In theory, identifying the attack should be straight forward since the staging environment is low volume, you know what attack occurred, and you have a rough estimate of when the attack occurred. If you can’t find an attack in your logs under those conditions, then that is clear feedback about how your logging and monitoring can be improved. If you can save the logs until after the bounty has ended, this type of analysis can be done post-assessment if you don’t have the resources to play along real time.

A crowd-sourced penetration test can change up the routine you have established for finding issues. Like any change in routine, there can be a few challenges at first. However, when done well, they can provide a vendor with insights that they may have never obtained through the existing status quo. These are not a replacement for traditional consultants. Rather, the new insights into the platform can help you re-focus the consultants more effectively to get a higher ROI.


Peleus Uhley
Principal Scientist

Comments are closed.