Author Archive: Peleus Uhley

Top 10 Hacking Techniques of 2013: A Few Things to Consider in 2014

For the last few years, I’ve been a part of the annual ranking of top 10 web hacking techniques organized by WhiteHat Security. Each year, it’s an honor to be asked to participate, and this year is no different. Not only does judging the Top 10 Web Hacking Techniques allow me to research these potential threats more closely, it also informs my day-to-day work.

WhiteHat’s Matt Johansen and Johnathan Kuskos have provided a detailed overview of the top 10 with some highlights available via this webinar.  This blog post will further describe some of the lessons learned from the community’s research.

1. XML-based Attacks Will Receive More Attention

This year, two of the top 15 focused on XML-based attacks. XML is the foundation of a large portion of the information we exchange over the Internet, making it an important area of study.

Specifically, both researchers focused on XML External Entities. In terms of practical applications of their research, last month Facebook gave out their largest bug bounty yet for an XML external entity attack. The Facebook attack demonstrated an arbitrary file read that they later re-classified as a potential RCE bug.

Advanced XML features such as XML external entities, XSLT and similar options are very powerful. If you are using an XML parser, be sure to check which features can be disabled to reduce your attack surface. For instance, the Facebook patch for the exploit was to set libxml_disable_entity_loader(true).

In addition, JSON is becoming an extensively used alternative to XML. As such, the JSON community is adding similar features to the JSON format. Developers will need to understand all the features that their JSON parsers support to ensure that their parsers are not providing more functionality than their APIs are intended to support.

2. SSL Takes Three of the Top 10 Spots

In both the 2011 and 2012 Top 10 lists, SSL attacks made it into the top spot.  For the 2013 list, three attacks on SSL made it into the top 10: Lucky 13, BREACH and Weaknesses in RC4. Advances in research always lead to more advances in research. In fact, the industry has already seen our first new report against SSL in 2014.  It will be hard to predict how much farther and faster research will advance, but it is safe to assume that it will.

Last year at BlackHat USA, Alex Stamos, Thomas Ptacek, Tom Ritter and Javed Samuel presented a session titled “The Factoring Dead: Preparing for the Cryptopocalypse.” In the presentation, they highlighted some of the challenges that the industry is facing in preparing for a significant breach of a cryptographic algorithm or protocol. Most systems are not designed for cryptographic agility and updating cryptography requires a community effort.

These three Top 10 entries further highlight the need for our industry to improve our crypto agility within our critical infrastructure. Developers and administrators, you should start examining your environments for TLS v1.2 support. All major browsers currently support this protocol. Also, review your infrastructure to determine if you could easily adopt future versions of TLS and/or different cryptographic ciphers for your TLS communication. The OWASP Transport Layer Protection Cheat Sheet provides more information on steps to hard your TLS implementation.

3. XSS Continues to Be a Common Concern for Security Professionals

We’ve known about cross-side scripting (XSS) in the community for over a decade, but it’s interesting that people still find innovative ways to both produce and detect it. At the most abstract level, solving the problem is complex because JavaScript is a Turing-complete language that is under active development. HTML5 and CSS3 are on the theoretical edge of Turing-Completeness in that you can implement Rule 110 so long as you have human interaction. Therefore, in theory, you could not make an absolute statement about the security of a web page without solving the halting problem.

The No. 1 entry in the Top 10 this year demonstrated that this problem is further complicated due to the fact that browsers will try to automatically correct bad code. What you see in the written code is not necessarily what the browser will interpret at execution. To solve this, any static analysis approach would not only need to know the language but also know how the browser will rewrite any flaws.

This is why HTML5 security advances such as Content Security Policies (CSP) and iframe sandboxes are so important (or even non-standards-based protections such as X-XSS-Protection).  Static analysis will be able to help you find many of your flaws. However, due to all the variables at play, they cannot guarantee a flawless site. Additional mitigations like CSP will lessen the real world exploitability of any remaining flaws in the code.

These were just a few of the things I noticed as a part of the panel this year. Thanks to Jeremiah Grossman, Matt Johansen, Johnathan Kuskos and the entire WhiteHat Security team for putting this together. It’s a valuable resource for the community – and I’m excited to see what makes the list next year.

Peleus Uhley

Lead Security Strategist

 

Mass Customization of Attacks Talk at RSA

Business consultant Stanley Davis defined mass customization as the “customization and personalization of products and services for individual customers at a mass production price.” Anyone who has ever ordered a custom PC is no stranger to mass customization: that particular combination of components wasn’t assembled into a PC until the customer initiated an order.

As we responded to zero-day exploits in the past couple of years, we took stock of some of the properties that separated them from mass malware, which affect older, patched vulnerabilities. For example, we noticed zero-day attacks starting to target more than one version of a platform on one or more operating systems. In addition, we observed that zero-day attacks contain more than one exploit possibly affecting multiple vendors’ products. Our thesis can be stated as follows: The exploit creation industry is maturing; by combining the features of mass malware with multiple zero-day exploits, they can create mass-customized attacks.

 masscustomizedattacks

 

We expand on this thesis in our upcoming talk at the RSA 2014 conference and use several case studies to prove it.

If you’re going to be attending RSA on Tuesday, Feb. 25, please swing by our talk at 2:40 p.m. in the West Room 3006. We look forward to sharing our research and the conversations with our friends and partners in the industry!

Peleus Uhley, Platform Security Strategist
Karthik Raman, Security Researcher

Flash Player Sandbox Now Available for Safari on Mac OS X

Over the last few years, Adobe has protected our Flash Player customers through a technique known as sandboxing. Thus far, we have worked with Google, Microsoft and Mozilla on deploying sandboxes for their respective browsers. Most recently, we have worked with Apple to protect Safari users on OS X. With this week’s release of Safari in OS X Mavericks, Flash Player will now be protected by an OS X App Sandbox.

For the technically minded, this means that there is a specific com.macromedia.Flash Player.plugin.sb file defining the security permissions for Flash Player when it runs within the sandboxed plugin process. As you might expect, Flash Player’s capabilities to read and write files will be limited to only those locations it needs to function properly. The sandbox also limits Flash Player’s local connections to device resources and inter-process communication (IPC) channels. Finally, the sandbox limits Flash Player’s networking privileges to prevent unnecessary connection capabilities.

Safari users on OS X Mavericks can view Flash Player content while benefiting from these added security protections. We’d like to thank the Apple security team for working with us to deliver this solution.

Peleus Uhley
Platform Security Strategist

Flash Player Security with Windows 8 and Internet Explorer 10

With the launch of Internet Explorer 10 on Windows 8 last year, customers have experienced improved Flash Player capabilities. Adobe worked closely with Microsoft to integrate Flash Player into Internet Explorer 10 for the Windows 8 platform, but some of our customers are still unaware of the full benefit of the security enhancements. We’d like to take the opportunity to discuss how this integration introduced several new changes that have increased end-user security.

The first significant change is that Flash Player updates for IE 10 on Windows 8 are now distributed through Windows Update. End-users are no longer prompted by the Flash Player auto-updater to update Internet Explorer. This also means that enterprises can now distribute Flash Player updates for Windows 8 through their existing Windows OS patch management workflows. For IE 10 users on Windows 7, you will continue to be updated through Flash Player’s existing update mechanisms.

Windows 8 and IE 10 bring a new level of security known as Enhanced Protected Mode (EPM). In immersive mode, EPM is enabled by default. End users can enable Enhanced Protected Mode on the desktop by selecting Tools > Internet Options > Advanced and checking “Enable Enhanced Protected Mode.”

EPM on IE 10 provides several new protections. One is that all content processes will run as 64-bit processes. This means that Flash Player will also be run as a 64-bit process which will make heap sprays more difficult. The larger address space makes it more difficult to predict the memory location of the spray with a decent statistical likelihood.

The Windows 8 OS security model also utilizes AppContainers for Windows Store. The AppContainer for Internet Explorer 10 is an improvement on the existing idea of Integrity levels. The IE 10 AppContainer brokers both read and write access to most of the operating system. This is an improvement over traditional Protected Mode where only write access was limited. Since Flash Player will be executing as a low privileged process, it will not be able to read user-owned data without user interaction. In addition, the IE 10 AppContainer enforces certain network restrictions which are described here. Since Flash Player is integrated into IE 10, Flash Player is sandboxed by the same AppContainer broker as Internet Explorer.

One aspect of the new AppContainer brokers is that Internet Explorer 10 has an unique cookie store for each mode. Browser cookies for immersive surfing will be placed in the IE 10 AppContainer storage location. Cookies created while surfing Internet-zone content in IE 10 on the desktop will be placed in the Low Integrity Level (LowIL) cookie location. Flash Player acknowledges this paradigm for Local Shared Objects (LSOs), as well. This means that any data stored from your Flash Player gaming in immersive mode will not be available to Flash Player when you are surfing with IE on the desktop. More information on how IE 10 handles cookies on Windows 8 can be found in this blog.

Overall, these new protections serve to further improve security for our Windows 8 customers while also delivering a more streamlined update workflow. Adobe will continue to work with Microsoft to better improve security for our mutual customers going forward.

Peleus Uhley
Platform Security Strategist

Reflections on Black Hat & DefCon

This year the ASSET security team along with security engineers from several other Adobe teams travelled to Vegas to attend the summer’s largest security conferences – Black Hat and DefCon. The technical talks can typically range from “cool bugs” to “conceptual issues that require long term solutions.” While the bugs are fun, here’s my take on the major underlying themes this year.

One major theme is that our core cryptographic solutions such as RSA and TLS are beginning to show their age. There was more than one talk about attacking TLS and another presentation by iSEC Partners focused on advances related to breaking RSA. The iSEC team made a valid case that we, as an industry, are not prepared for easily deploying alternative cryptographic solutions. Our industry needs to apply the principles of “crypto agility” so that we can deploy alternative solutions in our core security protocols, should the need arise.

Another theme this year was the security issues with embedded systems. Embedded systems development used to be limited to small bits of assembly code on isolated chips. However, advances in disk storage, antenna size, and processors has resulted in more sophisticated applications powering more complex devices. This exposed a larger attack surface to security researchers at Black Hat and DefCon who then found vulnerabilities in medical devicesSIM cardsautomobilesHVAC systemsIP phonesdoor locksiOS chargersSmart TVsnetwork surveillance cameras, and similar dedicated devices. As manufacturing adopts more advanced hardware and software for devices, our industry will need to continue to expand our security education and outreach to these other industries.

In traditional software, OS enforced sandboxes and compiler flags have been making it more difficult to exploit software. However, Kevin Snow and Lucas Davi showed that making additional improvements to address space layout randomization (ASLR), known as “fine-grained ASLR,” will not provide any significant additional levels of security. Therefore, we must rely on kernel enforced security controls and, by logical extension, the kernel itself. Mateusz Jurczyk and Gynvael Coldwind dedicated significant research effort into developing tools to find kernel vulnerabilities in various operating system kernels. In addition, Ling Chuan Lee and Chan Lee Yee went after font vulnerabilities in the Windows kernel. Meanwhile, Microsoft offered to judge live mitigation bypasses of their kernel at their booth. With only a small number of application security presentations, research focus appears to be shifting back toward the kernel this year.

Ethics and the law had an increased focus this year. In addition to the keynote by General Alexander, there were four legal talks at Black Hat and DefCon from the ACLU, EFF and Alex Stamos. Paraphrasing Stamos’ presentation, “The debate over full disclosure or responsible disclosure now seems quaint.” There were no easy answers provided; just more complex questions.

Regardless of the specific reason that drew you to Vegas this year, the only true constant in our field is that we must continue learning. It is much harder these days to be an effective security generalist. The technology, research and ethics of what we do continues to evolve and forces deeper specialization and understanding. The bar required to wander into a random Black Hat talk and understand the presentation continues to rise. Fortunately, walking into a bar at Black Hat and offering a fellow researcher a drink is still a successful alternative method of learning.

Peleus Uhley
Platform Security Strategist

The Evolution of Exploit Sophistication

When we look at the exploits that Adobe patched from February and March of this year, it is clear that today’s zero-day exploits are increasingly more sophisticated. This increase in sophistication is not limited to the skills needed to find and exploit the vulnerability. The code used to exploit the environment is also more robust in terms of code quality and testing. In short, exploit creation today requires the same level of rigor as professional software engineering projects.

Today’s advanced exploits need to be written to work in any target environment. For instance, February’s Reader 0-day supported 10 different versions of Reader with 2 sub-versions dependent on the end-user’s language. In addition, Flash Player CVE-2013-0634 had shell code for Windows XP, Vista, Windows 7, Server 2003, Server 2003 R2, Server 2008 and Server 2008 R2 as well as supporting six versions of Flash Player. Variants of CVE-2013-0634 also supported Firefox and Safari on Mac OS X. An exploit developer would need a robust testing environment to ensure that the exploit would work in that many different environments for each version of Flash Player. The exploit writers even took into account different CPU architectures by including a signed 32-bit payload and a 64-bit payload. This reflects the fact that these exploits are written with professional code quality and stability requirements for distribution across a dynamic target base.

As vendors are increasing software defenses through techniques such as sandboxing, attackers are now combining multiple vulnerabilities from different vendors to achieve their goals.When I look at the reports from Pwn2Own and some of the recent zero-day reports such as CVE-2013-0643, attacks are moving toward combining vulnerabilities from multiple products, some of which are from different vendors. We are moving away from the model of single vulnerability exploits.

This is all a part of the natural evolution of the threat landscape and the commercialization of exploits. This will require an equal evolution on the part of vendors in their software defences. Karthik Raman and I will be discussing this topic, “Security Response in the Age of Mass Customized Attacks,” in more detail at the upcoming Hack in the Box Conference (HITB) Amsterdam next week. Please stop by our talk if you would like to discuss this further.

Peleus Uhley
Platform Security Strategist

Top 10 Web Hacking Techniques of 2012

I was honored that Jeremiah Grossman asked me to serve again on the panel for the Top 10 Web Hacking Techniques of 2012. During the process, I get to take a look at the year’s most interesting research in greater detail than what is normally allowed in my day-to-day schedule. It makes me question what I thought I knew about existing attack techniques and how this year’s top research reflects on the industry.

Ranking the entries is challenging when you are comparing an SSL cryptography bug vs. the server side request forgery attacks vs. attacks against Chrome add-ons. To do so, requires that you read the full research, review the source code that was written and try to determine what impact the research had on the community. I typically start by considering factors such as:

  • Is the research completely new or is it just an incremental improvement of a known issue?
  • How many end-users are impacted?
  • Is this an individual bug affecting one vendor or an industry-wide issue?
  • What does successful exploitation yield (XSS, file access, OS control, etc.)?
  • Is the exploit practical to conduct?
  • What was the depth of research (a white paper, a tool, etc.)?
  • Extra points for style or creativity?

Reviewing the research to this depth then leads to uncovering smaller bits that are interesting in their own right. For instance, the Blended Threats and JavaScript research leveraged HTML5 to create cross-site file upload capabilities. You will find links to tools that back up the research in entries such as Blended Threats, Attacking OData, Bruteforcing PHPSESSID, and the Chrome add-on research. In some cases, the research reminds you about how many of the old tricks can be re-purposed against new technologies.

As an example, in July, 1998 Microsoft issued security bulletin MS98-004 which allowed, “Unauthorized ODBC Data Access with RDS and IIS”. In 1999, new information was published regarding the bug and Bugtraq ID 529 said exploiting the vulnerability could result in, “obtaining access to non-public servers or effectively masking the source of an attack on another network”. Today, we would classify this type of attack as, “Server Side Request Forgery” (SSRF). Modern SSRF attacks can include both old and new attack techniques. For instance, the ERPScan paper discussed how to use the gopher: protocol in an XML External Entity to attack SAP gateways. Over a decade later, we now have a formal taxonomy for classifying these types of bugs. However, the core issue of effectively sandboxing a web application to a finite set of resources is not much easier than it was 15 years ago. If anything, it has become more difficult as the number of supported communication channels between endpoints has increased.

Finally, you begin to think about why these top 10 bugs are critical to the industry. The panel ranked,”Compression Ratio Info-leak Made Easy” (CRIME) in the top slot due to the overall difficulty of the research, the vendor action required to address it, and the importance of SSL to the security ecosystem. As we move more data to the cloud where we will access it via our mobile or tablet devices, we will increasingly rely on SSL to protect it in transmission. Another interesting trend in this year’s top list was the multiple SSRF entries. As I mentioned, it wasn’t a completely new concept but rather a concept that is seeing a renewed interest as more data moves to the cloud. There were three SSRF entries in the Top 15 and the SSRF entry that was ranked number 2 includes research by two separate groups. As our dependence on cloud storage grows, SSRF will become an increasingly used attack vector for reaching all the critical data that lies behind those front end cloud servers.

As everyone reviews the Top 15, I encourage you to find time to read through each entry. The final list represents great information by talented researchers. Also, I would recommend sharing the list with your engineers. When I shared the Top 15 list with our internal Adobe community a few months ago, I can attest that at least one developer recognized a flaw that existed in his system and corrected the issue. Thanks to Jeremiah Grossman for putting this together each year and allowing me to be a part of it.

Peleus Uhley
Platform Security Strategist

Firefox Click-to-Play Helps Protect Our Customers

The Adobe team has worked hard to improve patch adoption by delivering background updaters for Flash Player and Adobe Reader. In addition, we have worked with partners, such as Microsoft and Google, to reduce update fatigue by delivering patches through existing update mechanisms. However, one of the hardest challenges in protecting end users is reaching what is sometimes referred to as the “long tail” in an update graph. These are the users who, for various reasons, have not updated their systems in several months or even years. Reaching these last few end users can be difficult if they have disabled their update mechanisms. Unfortunately, they are also the users who are most likely to be successfully attacked.

Yesterday, Mozilla announced an update to the Firefox click-to-play feature that will warn users when they try to play plugin content with an out-of-date browser plugin. Since Mozilla will now be assisting plugin vendors in reminding these users to update, we will hopefully be able to convert more of them to patched versions. At the same time, Mozilla is helping to protect these users from those who would use older vulnerabilities to infect their systems. We support Mozilla in their efforts to protect and inform our mutual customers.

ColdFusion 10 Provides Powerful New Security Tools

Today marks the release of ColdFusion 10. This release redefines many aspects of the ColdFusion 10 security model and incorporates the principles of the Adobe Secure Product Lifecycle (SPLC). With this release, we’ve worked to improve three major areas: Our goals were to improve patch adoption, improve the default configuration, and to make it easier for developers to create secure ColdFusion applications.

One of the most common reasons for a successful attack against a ColdFusion server is that it doesn’t have the latest security updates. In all fairness, this is not completely the administrator’s fault. Updating a ColdFusion server can be difficult due to the number of manual steps involved. Also, it is easy to miss a security update announcement. With ColdFusion 10, we make both of these steps easier. The ColdFusion 10 administration interface now incorporates a simple “Check For Updates” button. Alternatively, the server can be configured to automatically check for updates and send an email to the administrator once one becomes available.  Finally, the interface allows the developer to apply the patch through a single button click in the administrator interface. These features help make updating the server much more straightforward.

The second major area of improvement focused on making it easier for administrators to securely deploy ColdFusion 10. One of the most attractive characteristics of ColdFusion is that it has always been a simple development environment. Therefore, there were several features that favored making the early phases of development easier by leaving the complicated aspects disabled by default. The cost of this choice was that once developers were ready to deploy to production, they had to review a 35-page lockdown guide to enable and/or configure those more complicated features appropriately. With today’s release, we offer the option of starting the server in a secure-by-default configuration. This greatly simplifies the process of making a server production-ready with a secure configuration.

The last area of improvement focused on providing developers with an increased number of tools for creating secure ColdFusion applications. One example is that we have provided integrated OWASP ESAPI support in the platform. We originally started to include ESAPI in ColdFusion 9 just for our internal needs of addressing cross-site scripting (XSS). Once developers noticed the library in the update, they quickly developed several blogs on how to unofficially start using it in your ColdFusion code. Today’s release formally exposes several aspects of ESAPI through ColdFusion API’s to help developers avoid cross-site scripting vulnerabilities.

We also improved the session management capabilities in ColdFusion–another aspect of making it easier for developers to create ColdFusion applications. We have improved APIs to make it easier to set the HttpOnly and Secure flags on cookies. Session rotation has been improved through new SessionRotate and SessionInvalidate APIs.  To combat cross-site request forgery (CSRF) with active sessions, the ColdFusion team added an API for generating unique tokens for form requests. The team also added support for protecting against clickjacking attacks on active users by adding support for the X-FRAME-OPTIONS header.

ColdFusion 10 is a significant advancement in helping ColdFusion customers improve their secure product lifecycle processes. It is even easier to create secure content, deploy the content on a secure server and manage the server updates once it is deployed. This is only an introduction to the major security enhancements in ColdFusion 10. For more information on all the new security APIs for developers, please see the ColdFusion documentation on Security Enhancements in ColdFusion 10.  ColdFusion administrators should review the Administering Security and the Server Update sections for a complete list of server improvements.

A Basic Distributed Fuzzing Framework for FOE

Last week, CERT released a Python-based file format fuzzer for Windows called Failure Observation Engine (FOE). It is a Windows port of their Linux-based fuzzer, Basic Fuzzing Framework(BFF). CERT provided Adobe with an advanced copy of FOE for internal testing, and we have found it to be very useful. One of the key features of FOE is its simplicity. The configuration file is very straightforward, which makes it easy to introduce to new teams. We have also used the “copy” mode of FOE to help automate triaging large sets of external reports. It is a great tool to have for dumb fuzzing. For this blog, I am going to discuss a simple Python wrapper I created during my initial testing of the tool which helped to coordinate running FOE across multiple machines. This approach allows you to pull seed files from a centralized location. You can also view the status of all of the fuzzing runs and their results from the same location. If you are not interested in writing a distributed fuzzing framework, then you might want to stop reading because the rest of this blog is all about code. :-)

The goal of this distributed fuzzing framework design was to create something simple, lightweight since I was experimenting with a new tool. I set a personal limit of keeping the project to around 1,000 lines of code in order to scope my time investment. That said, I also wanted to build something that I could easily scale later in the event that I liked it enough to invest more time. For the client-side code, I used Python since that was already required for FOE. On the server side, I had a Linux/Apache/MySQL/Perl (LAMP) server. Knowing that everyone has their own preference for server-side authoring, I am only going to describe the server-side architecture rather than providing the Perl source. Nothing in the server-side code is so complicated that a Web developer couldn’t figure out how to do an implementation in the language of their choice from this description. While I designed this for testing the FOE fuzzer, only one file in the entire system is FOE-specific, which makes the infrastructure reusable for other fuzzers. The current name of the main script is “dffiac.py” because I thought of this project as a, “Distributed Fuzzing Framework in a Can”.

For this design, all of the tracking logic is consolidated on the centralized server. The Python script will issue requests for data using simple GETs and POSTs over HTTP. The server will respond to the requests with basic XML. The fuzzing seed files are hosted on the server in a public web server directory from which they can be downloaded. Identified crashes will be uploaded to the server and placed in a public web server directory. Both the client-side and server-side codes are agnostic with regards to the format of the seed files and the targeted application. Therefore, this should be relatively easy to set up in any infrastructure.

 

The database design

In this design, the mySQL server coordinates the runs across all the different machines. You first need a table containing all the files that you want to fuzz. At a bare minimum, it needs a unique primary key (fid), the name of the file and its location on the web server. I currently have a database of more than 60,000 SWF files that are sub-categorized based on type so that I can focus fuzzing to specific types of SWF files. However, name and location will get you started with fuzzing.

 

seed_files

Field Type Description
fid Integer (primary key, autoincrement) The unique File ID for this entry
name VARCHAR The filename
location VARCHAR The relative web directory for the file (e.g. “/fuzzing/files/”)

 

The next thing that you will need is a table to track all of the fuzzing runs. A “run” is defined as one or more servers testing with the same FOE configuration file against a defined set of seed files. There are multiple ways in which you can define the selected seed files for the run. For instance, you may want to use FOE against multiple types of applications. For this scenario, you might have a different seed_files for each file type. To support the need for different seed_files tables, the design of run_records requires that you provide the “table_name” that will be used for this run. Once a seed_files table is selected, it may be necessary to further restrict the run to a subset of files within the seed_files tables. For instance, you may only want to select a subset of files within the given table. Therefore, the design requires that you provide a “type” parameter which denotes the method for selecting files from the seed_files table. The value of type can include values such as “all”, “range” or any other sub-category you want to define. As an example, this particular run may be a “range” type that starts at start_fid and stops at end_fid.

 

run_records

Field Type Description
rid Integer (primary key, autoincrement) The unique ID for this run
name VARCHAR The human readable name for the run
description VARCHAR A description for the run (e.g. config or mutation used, # of iterations, etc.)
type VARCHAR Values can include (all, range, etc)
table_name VARCHAR The name of the seed_files table that will be used for testing
start_fid Integer The first fid from seed_files to be fuzzed in this run
end_fid Integer The last fid from seed_files to be fuzzed in this run
current_fid Integer This tracks the next fid to be tested during the run

 

For every run, you will have multiple servers running FOE. For each server instance, it will be necessary to track the server name, when it started, the current status of the server, and when it last provided an update. The status will include values such as “running” and “complete.”  You can infer whether a machine has died based on whether it has been too long since the timestamp for the last_update field was modified.

 

server_instances

Field Type Description
siid Integer (primary key, autoincrement) The unique server instance ID
server_name VARCHAR The name of the server (e.g hostname + IP address)
status VARCHAR Is it running or has it completed.
start_time timestamp When did this instance start?
last_update timestamp When was the last request from this instance?
rid Integer What run_record is this instance associated with?

 

Lastly, you will need a table to record the results. The script will record the server_instance ID (siid) where the crash was found in case there are issues with reproducing the crash. This will allow a QA to retest on the original machine where the crash occured. It is also necessary to track which run was able to identify the crash. The rid is not recorded because it can already be extrapolated from the siid. According to database normalization rules, redundant information should not be stored in tables. In this design, the script will record a result in fuzz_records regardless of whether a crash was identified.  This allows you to track which files have been tested against which FOE configurations. If a crash is identified, the web server directory where the crash result was stored is also recorded.

 

fuzz_records

Field Type Description
frid Integer (primary key, autoincrement) The unique fuzz record ID
fid Integer The seed_files ID for this entry
siid Integer The server instance ID for this entry
crash Boolean Whether a crash was recorded during this test
location VARCHAR Where the crash result was stored (e.g. /results/run_id/)

 

The config file

You will start the Python script by providing a simple configuration file in the command line: “python dffiac.py dffiac.cfg”. The configuration file is in the same format as the FOE configuration file and contains the following:

 

dffiac.cfg

[foeoptions]
python_location=C:Python26python.exe
config_location=C:FOEconfigsmy_foe_config.cfg

 

[runoptions]
run_id=1
web_server=http://my.internal.server.com
upload_cgi=/fuzzers/crash_uploader.cgi
action_cgi=/fuzzers/action_handler.cgi

 

[logoptions]
log_dir=C:dffiaclogs

 

The foeoptions section tells the script where to find the Python executable and the location of the FOE config script you will use for this run. The runoptions section provides the run id (rid) the database is using to track this run along with the location of the web server, the path to the action_handler.cgi and the path to the CGI that will handle the file uploads. The logoptions allows you to specify where the script will log local information regarding the run. The logs directory needs to exist prior to starting the script. The config_location and run_id are likely the only two elements that will change from run to run.

 

The transaction flow

For this next section, we will review the transactions between the dffiac.py script and the web server. The web server will read in the GET parameters, execute the relevant SQL query and return the results as XML. All but one request is handled by the action_handler defined in the dffiac.cfg config file. The upload of the crash results is handled by the upload_cgi defined in the dffiac.cfg config file.

Once dffiac.py has started and been initialized by the config file, the script will begin sending requests to the server. An “action” parameter informs the action_handler CGI which query to perform. The server will always respond to the Python script with the relevant information for the request in a simple XML format.

 

The first HTTP request from the Python code will be to gather all the information regarding the run_id provided in the config file:

GET /fuzzers/action_handler.cgi?action=getRunInfo&rid=1

 

The web server will then perform this SQL query with the rid that was provided:

select run_type,start_fid,end_fid from run_records where rid = ?

 

The results from the query will be used to return the following XML (assuming the run is defined as the range of fids from 1-25):

<xml>
  <run_type>range</run_type>
  <start_fid>1</start_fid>
  <end_fid>25</end_fid>
</xml>

 

Now that dffiac.py has the information for the run, it will then inform the web server that the run is starting:

GET /fuzzers/action_handler.cgi?action=recordServerStart&rid=1&serverName=server1

 

This HTTP request will result in the following SQL query:

insert into server_instances (server_name,status,start_time,rid) values (?,'running',NOW(),?)

 

The insert_id from this query (siid) becomes the unique identifier for this instance and is returned for use in later queries:

<xml>
  <siid>1</siid>
</xml>

 

Now that this instance has officially registered to contribute to this run, the Python script will begin requesting individual files to test:

GET /fuzzers/action_handler.cgi?action=getNextFid&rid=1&run_type=range

 

The corresponding SQL query will vary depending on how you have defined your run. For this example, we will assume that this is a basic run that will incrementally walk through the file IDs in the seed_files table. To accomplish this, we create an SQL variable called “value” and assign it the current_fid. By recording the value of the current fid and incrementing the “value” in a single statement, we can avoid a race condition when multiple servers are running.

update run_records set current_fid = current_fid + 1 where rid = ? and @value := current_fid;

 

At this point, “@value” is set to 1 which is the fid the Python script will test and the current_fid in the database table has been incremented to 2. The web server can then fetch “@value with the following SQL command:

select @value;

 

Since the process of asking for the next fid will automatically increment the value of current_fid, the value of current_fid will eventually exceed the value of the end_fid in the database table. While it may seem weird, it doesn’t hurt the process. This can be allowed to occur or you can add a little more server-side logic to have the server return -1 as the current_fid to stop the run when end_fid is reached.

 

The “select @value” result will be returned to Python script as the current_fid available for testing:

<xml>
  <current_fid>1</current_fid>
</xml>

 

The Python script will then compare the current_fid with the end_fid that it received earlier to determine whether to stop testing.

 

Once we have the fid of the file that we will test, we can then fetch the information for that specific file:

GET /fuzzers/action_handler.cgi?action=getFileInfo&rid=1&fid=1

 

Using the rid, the web server can query the run_records table to find the table_name that contains the seed files.

select table_name from run_records where rid = ?

 

Assuming the result of that query will be saved as the variable, “$table_name”, the web server can construct the query to retrieve the file name and the directory location that corresponds to the file id:

"select name, location from" . $table_name . "where fid = ?"

 

The web server will return the file name and location with the following XML:

<xml>
  <name>seed.txt</name>
  <location>/fuzzers/files/</location>
</xml>

 

Now, that the location of the seed file is known, it can be downloaded by dffiac.py and saved in the FOE seeds directory. The FOE fuzzer is then started, and dffiac.py waits for FOE to finish testing that seed file. Once FOE testing has completed, the result will need to be recorded by sending the fid and a boolean value indicating whether a crash was identified with that test:

GET /fuzzers/action_handler.cgi?action=recordResult&siid=1&fid=1&crash=1

 

This will result in the following query:

insert into fuzz_records (siid,fid,crash) values (?,?,?)

 

The web server will also record that it has received an update from this fuzzing server instance in the server_instances table to let us know that it is still alive and processing:

update server_instances set lastUpdate = NOW() where siid = ?

 

The result is recorded regardless of success or failure so that you can track which files have been successfully tested with which configs. You could infer this from the run_records, but if a machine dies, a file might be skipped. The server-side code will take the insert_id from the fuzz_records statement (frid) and return the following XML:

<xml>
  <frid>1</frid>
</xml>

 

If there was a crash, the Python script will zip up the crash directory, base64 encode the file and POST it to the upload_cgi identified in the dffiac configuration file. The script will leave the zip file on the fuzzing server if an error is detected during the upload. Along with the zip file, it will send the rid and frid. The rid is used to store files in a web server directory unique to that run. The frid is sent so that the action_handler can update the fuzz_records entry with the location of the uploaded crash file (e.g. “/results/1/zip_file_name.zip”) in the following SQL query:

update fuzz_records set location = ? where frid = ?

 

A successful upload will result in the following XML:

<xml>
  <success>1</success>
</xml>

 

A failed upload can return the description of the error to the client with the following XML:

<xml>
  <error>Replace me with the actual error description</error>
</xml>

 

The dffiac.py script will then continue retrieving new files and testing them with FOE until the end_fid is reached. Then the final call to the web server will record that this fuzzing server instance has completed its run and has stopped:

GET /fuzzers/action_handler.cgi?action=recordRunComplete&siid=1

 

The web server will record the completion with the following SQL query:

update server_instances set status='complete', lastUpdate=NOW() where siid = ?

 

The web server will respond to this last request with the following XML:

<xml>
  <success>1</success>
</xml>

 

The last XML response is currently ignored by the Python script but a more robust implementation could double-check for errors.

 

The Python code

The logic for the distributed fuzzing framework is split into one main file (dffiac.py) and three libraries that are contained in a /libs directory. We’ll start with the three libraries in the /libs directory. The code below is the library that contains the utilities for creating the zip file of the crash result.

 

ZipUtil.py (30 lines)

import zipfile
import os

 

class ZipUtil:

 

#Create a zip file and add everything in path_ref
def createZipFile(self, path_ref, filename):
  zip_file = zipfile.ZipFile(filename, 'w')

 

  #Check to see if path_ref is a file or folder
  if os.path.isfile(path_ref):
    zip_file.write(path_ref)
  else:
    self.addFolder(zip_file, path_ref)

 

  zip_file.close()

 

#Recursively add folder contents to the zip file
def addFolder(self, zip_file, folder):
  for file in os.listdir(folder):

 

    #Get path of child element
    child_path = os.path.join(folder, file)

 

    #Check to see if the child is a file or folder
    if os.path.isfile(child_path):
      zip_file.write(child_path)
    elif os.path.isdir(child_path):
      self.addFolder(zip_file, child_path)

 

The second library will base64 encode the zip file prior to uploading it to the web server via a POST method.  On the server side, you will need to base64 decode the file before writing it to disk.

 

PostHandler.py (77 lines)

import mimetools
import mimetypes
import urllib
import urllib2
import base64

 

class PostHandler(object):

 

  def __init__(self,webServer,uploadCGI):
    self.web_server = webServer
    self.upload_cgi = uploadCGI
    self.form_vars = []
    self.file_attachments = []
    self.mime_boundary = mimetools.choose_boundary()
    return

 

  #Add a form field to the request
  def add_form_vars(self, name, value):
    self.form_vars.append((name, value))
    return

 

  #Get the mimetype for the attachment
  def get_mimetype(self,filename):
    mimetype = mimetypes.guess_type(filename)[0] or 'application/octet-stream'
    return(mimetype)

  #Add a base64 encoded file attachment
  def append_file(self, var_name, filename, file_ref, mimetype=None):
    raw = file_ref.read()
    body = base64.standard_b64encode(raw)
    if mimetype is None:
      mimetype = self.get_mimetype(filename)
    self.file_attachments.append((var_name, filename, mimetype, body))

  #Get the body of the request as a string
  def get_request_body(self):
    lines = []
    section_boundary = '--' + self.mime_boundary

 

    # Add the form fields
    for (name, value) in self.form_vars:
      lines.append(section_boundary)
      lines.append('Content-Disposition: form-data; name="%s"' % name)
      lines.append('')
      lines.append(value)

 

    # Add the files to upload
    for var_name, filename, content_type, data in self.file_attachments:
      lines.append(section_boundary)
      lines.append('Content-Disposition: file; name="%s"; filename="%s"' % 
        (var_name, filename))
      lines.append('Content-Type: %s' % content_type)
      lines.append('Content-Transfer-Encoding: Base64')
      lines.append('')
      lines.append(data)

 

    #Add the final boundary
    lines.append('--' + self.mime_boundary + '--')
    lines.append('')

 

    #Combine the list into one long string
    CRLF = 'rn'
    return CRLF.join(lines)
  #Send the final request
  def send_request(self):
    request = urllib2.Request(self.web_server + self.upload_cgi)
    content_type = 'multipart/form-data; boundary=%s' % self.mime_boundary
    request.add_header('Content-type',content_type)

 

    form_data = self.get_request_body()
    request.add_header('Content-length',len(form_data))
    request.add_data(form_data)

 

    result = urllib2.urlopen(request).read()
    return result

 

 

The third library handles the communication between the client and server. It will generate the GET requests and parse the XML responses.

 

actionHandler.py (94 lines)

import urllib
import urllib2
from xml.dom.minidom import parseString

 

class ActionHandler:

 

  #Initialize with the information from the config file
  def __init__(self,options,localLog):
    self.webServer = options['runoptions']['web_server']
    self.uploadCGI = options['runoptions']['upload_cgi']
    self.actionCGI = options['runoptions']['action_cgi']
    localLog.write("Configured web servern")

 

  #Parse the XML for the requested text value
  def getText(self,nodelist):
    rc = []
    for node in nodelist:
      if node.nodeType == node.TEXT_NODE:
        rc.append(node.data)
    return ''.join(rc)

 

  #Make a web request to the server with the provided GET parameters
  def retrieveInfo(self,values):
    url = self.webServer + self.actionCGI
    data = urllib.urlencode(values)
    response = urllib2.urlopen(url,data)
    xml = response.read()
    response.close()
    return(xml)

 

  #Get the information for the rid provided in the config file
  def getRunInfo(self,rid):
    values = {'action':'getRunInfo',
      'rid': rid}
    xml = self.retrieveInfo(values)
    dom = parseString(xml)
    run_type = self.getText(dom.getElementsByTagName("run_type")[0].childNodes)
    start_fid = self.getText(dom.getElementsByTagName("start_fid")[0].childNodes)
    end_fid = self.getText(dom.getElementsByTagName("end_fid")[0].childNodes)
    return (run_type,start_fid,end_fid)

 

  #Record that this server instance is starting a run
  def recordServerStart(self,rid,serverName):
    values = {'action':'recordServerStart',
      'rid': rid,
      'serverName':serverName}
    xml = self.retrieveInfo(values)
    dom = parseString(xml)
    lastrowid = self.getText(dom.getElementsByTagName("siid")[0].childNodes)
    return (lastrowid)

 

  #Record that the server is now complete with its tests
  def recordRunComplete(self,siid):
    values = {'action':'recordRunComplete',
      'siid': siid}
    xml = self.retrieveInfo(values)

 

  #Get the fid for the next file to be fuzzed
  def getNextFid(self,rid,fid,run_type):
    values = {'action':'getNextFid',
      'fid':fid,
      'run_type':run_type,
      'rid':rid}
    xml = self.retrieveInfo(values)
    dom = parseString(xml)
    current_id = self.getText(dom.getElementsByTagName("current_fid")[0].childNodes)
    return current_id

 

  #Get the file name and location for the selected fid
  def getFileInfo(self,rid,fInfo):
    values = {'action':'getFileInfo',
      'rid':rid,
      'fid':fInfo.fid}
    xml = self.retrieveInfo(values)
    dom = parseString(xml)
    fInfo.name = self.getText(dom.getElementsByTagName("name")[0].childNodes)
    fInfo.location = self.getText(dom.getElementsByTagName("location")[0].childNodes)

 

#Record the result from the fuzzing test
  def recordResult(self,siid,fid,result):
    values = {'action':'recordResult',
      'siid':siid,
      'fid':fid,
      'crash':result}
    xml = self.retrieveInfo(values)
    dom = parseString(xml)
    frid = self.getText(dom.getElementsByTagName("frid")[0].childNodes)
    return frid

 

Finally, we get to the main file which is responsible for reading the config file and driving the fuzzing run. This is the only file that is specific to the FOE fuzzer.

 

dffiac.py (177 lines)

import os
import shutil
import socket
import subprocess
import sys
import urllib2
import ConfigParser
import time

 

sys.path.append("libs")

 

from ZipUtil import ZipUtil
from PostHandler import PostHandler
from ActionHandler import ActionHandler

 

#This will track the fid, and location of the file
class FileInfo:
  pass

 

#Convert the options in the config file to lists
def parse_options(config):
  options = {}
  for section in config.sections():
    options[section] = {}
    for (option, value) in config.items(section):
      options[section][option] = value
  return options

#Create a local text file for logging
def openLog(options):
  localLogDir = options['logoptions']['log_dir']
  runName = options['runoptions']['run_id']
  timestamp = int(time.time())
  localLog = open(localLogDir + runName + '_' + str(timestamp) + '.txt', 'w')
  localLog.write("Starting run: " + runName + " at " + str(timestamp) + "n")
  return localLog

 

#Close the local text file log
def closeLog(localLog):
  localLog.write("COMPLETEn")
  localLog.close()

 

#Download the next file to be fuzzed
def getNextFile(fInfo, options, foe_options, localLog):
  u = urllib2.urlopen(options['runoptions']['web_server'] + fInfo.location + fInfo.name)
  localFile = open(foe_options['runoptions']['seedsdir'] + "\" + fInfo.name, 'wb')
  localFile.write(u.read())
  localFile.close()
  localLog.write ('Created file: ' + foe_options['runoptions']['seedsdir'] + "\" + fInfo.name + 'n')

 

#Store the results in a zip file
def createZip(outputDir,filename):
  zipTool = ZipUtil()
  zipTool.toZip(outputDir,filename)
  zipFile = open(filename,'rb')
  return zipFile

 

#Post the zip file to the server
def postZip(options,frid,rid,filename,zipFile):
  form = PostHandler(options['runoptions']['web_server'], options['runoptions']['upload_cgi'])
  form.add_form_vars('frid',frid)
  form.add_form_vars('rid',rid)
  form.append_file('fname',filename,zipFile)
  result = form.send_request()
  return result

 

if __name__ == "__main__":
  if (len(sys.argv) < 2):
    print "usage: %s <runconfig.cfg>" % sys.argv[0]
    exit(1)

 

  #Read the dffiac config file
  configFile = sys.argv[1]
  if not os.path.exists(configFile):
    print "config file doesn't exist: %s" % configFile
    exit(1)
  config = ConfigParser.SafeConfigParser()
  config.read(configFile)

 

  #Read the foe config file
  options = parse_options(config)
  config2 = ConfigParser.SafeConfigParser()
  config2.read (options['foeoptions']['config_location'])
  foe_options = parse_options(config2)

 

  #Set up logging
  localLog = openLog(options)

 

  #Configure the web server
  aHandler = ActionHandler(options, localLog)

 

  #Get the information for this run
  rid = options['runoptions']['run_id']
  (run_type,start_fid,end_fid) = aHandler.getRunInfo(rid)

 

  #Record server start
  hostName = socket.gethostname()
  hostIP = socket.gethostbyname(hostName)
  serverName = hostName + "_" + hostIP
  siid = aHandler.recordServerStart(rid, serverName)
  localLog.write("Starting as server instance: " + siid + "n")

 

  #Get the first file to be processed
  fInfo = FileInfo()
  fInfo.fid = aHandler.getNextFid(rid,start_fid,run_type)
  localLog.flush()

 

  #loop until done
  while (int(fInfo.fid) <= int(end_fid)):
    #Get the location information for the current file
    aHandler.getFileInfo(rid,fInfo)

 

    #Download and store the file
    getNextFile(fInfo,options,foe_options,localLog)

    outputDir = foe_options['runoptions']['outputdir'] + "\" + foe_options['runoptions']['runid']

 

    #Run fuzzer
    exitCode = subprocess.call(options['foeoptions']['python_location'] + " " + options['foeoptions']['foe_location'] + " " + options['foeoptions']['config_location'], shell=True)

 

    #Check for completion of a succesful run
    if exitCode != 0:
      localLog.write("Error running foe on fid " + fInfo.fid + "n")
    else:
      dirList = os.listdir(outputDir)

 

      #Detect whether bugs were found
      if len(dirList) > 2:

 

        #Record the result in fuzz_records
        frid = aHandler.recordResult(siid,fInfo.fid,1)
        localLog.write("Recording frid: " + frid + "n")

 

        #Store the results in a zip file
        filename = frid + "-" + fInfo.name + ".zip"
        file_path = os.getcwd() + filename
        zipFile = createZip(outputDir,file_path)

 

        #Post the zip file back to the server
        result = postZip(options,frid,rid,filename,zipFile)
        zipFile.close()

 

        #Make sure the file got there OK
        if result.find("error") == -1:
          localLog.write("Results successfully uploaded.n")
          os.remove(file_path)
        else:
          localLog.write("There was an error in the upload: " + result + "n")

 

        localLog.write("Found bugs with " + fInfo.fid + "n")
      else:
        #Record no bugs found in the directory
        aHandler.recordResult(siid,fInfo.fid,0)
        localLog.write("No bugs found with " + fInfo.fid + "n")

 

    #The if len(dirlist) check on the results is complete
    #Erase files so that FOE starts clean on the next run
    os.remove(foe_options['runoptions']['seedsdir'] + "\" + fInfo.name)
    shutil.rmtree(outputDir)
    localLog.flush()

 

    #Get the next FID
    fInfo.fid = aHandler.getNextFid(rid,fInfo.fid,run_type)

 

  #The while loop is complete
  #Record this run instance as being complete
  aHandler.recordRunComplete(siid)

 

  #Close the local file log
  closeLog(localLog)

 

This blog is only meant to describe how you can stand up a basic distributed fuzzing framework based on FOE fairly quickly in approximately 1,000 lines of code. The client-side code turned out to be 378 lines, my server-side action_handler CGI was 150 lines and the upload CGI was 72 lines of Perl. That is enough to get the script to run based on information from a database. With the remaining 400 lines, I created a CGI to display the status of my runs and a CGI to generate a run. You will also want to write a script to mirror the dffiac.cfg and FOE configuration file across machines. Over time, I expect that you would make this design more robust for your particular infrastructure and needs. You can also expand this infrastructure for your other fuzzers with some modifications to the main file. What I provide here is just enough to help you get started performing distributed fuzzing with a small amount of coding and the FOE fuzzer.

 

Permission for this blog entry is granted as CCplus, http://www.adobe.com/communities/guidelines/ccplus/commercialcode_plus_permission.html