Hi folks, Allen Householder here. In addition to the recent introduction of our new Failure Observation Engine (FOE) fuzzing framework for Windows and Linux Triage Tools, we have updated the CERT Basic Fuzzing Framework (BFF) to version 2.5. This post highlights the significant changes.

BFF now runs on both OS X and Linux

BFF was originally developed for Linux. With the development of FOE, we gained a fuzzing framework for Windows. BFF 2.5 adds support for Mac OS X using the same underlying python code as on Linux. The main difference is that on OS X we use CrashWrangler as the debugger instead of gdb. An installer for Mac can be found on the BFF download page.

Support for multiple seed files

One of our most-requested features was to allow BFF to fuzz multiple seed files. BFF now applies a machine learning technique to observe the campaign results and can adjust its efforts to focus on the seed files that produce the most unique crashes. This all happens automatically as the campaign progresses. A similar technique is applied to the rangefinder feature introduced in BFF 2.0.

Crashes found during minimization get analyzed as well

After we added the improved crash minimization feature to BFF in our previous release, we noticed that the minimizer was encountering other new unique crashes while doing its thing. In BFF 2.0, those crashes were simply ignored. Now they are processed along with the crashes found using zzuf. In our experience developing BFF 2.5, this has improved our crash yield significantly.

Minimizer tuned for performance

The more crashes BFF finds, the more time it can wind up spending in the crash minimization cycle. Because of this, it's important that the minimization process be as efficient as it can be. We've made some algorithmic improvements to the minimizer to optimize it for speed.

Optional minimization-to-string feature

Say you have a crashing test case, but you really need to get it to a proof-of-concept exploit. The problem is when you load the crash into your debugger you can't easily tell which registers, stack values, or memory locations are under your control. But what if you could change the crashing test case so that it had only the bytes required to cause that crash, and the rest were all masked out with a fixed value, say "x" (0x78)? Then you'd know that if you saw EIP=0x78787878, you may already be a winner. The minimize-to-string option does just that.

The top image shows a crashing PDF test case prior to being minimized and the image below it shows the minimized version filled with mostly 0x78s. Since (a) it tends to be a more time consuming process than simply minimizing to the seed file and (b) it's not strictly necessary to analyze the crash, we have disabled this feature by default. It can be enabled in the BFF configuration file or when running the minimizer in standalone mode.

Callgrind for all unique crashers

When analyzing a crashing test case, it can be helpful to know what functions are being called in the test case. The callgrind tool for valgrind gives you a view of the call tree for a crash. BFF generates callgrind output for every crash it finds.

Basic crash clustering using callgrind coverage analysis

The callsim tool (analysis/callsim.py) takes the callgrind output for each crash and produces a dendrogram showing how the crashes relate to each other relative to their call tree similarity.

Because we want to focus our attention on the salient differences and not the calls they all have in common, we borrowed the Term Frequency - Inverse Document Frequency (TF-IDF) method from the text analysis world. TF-IDF is used to de-emphasize common words appearing throughout a corpus of documents relative to less common words when comparing two documents. We've applied that technique to function calls in order to compare crashing test cases. The result is that crashes appearing next to each other on the tree are likely to be related.

Improved crash uniqueness determination on Linux

We noticed that we could do better on crash uniqueness determination if we checked things like whether or not EIP was in a loaded module. We've also incorporated a blacklist for certain common library functions so that they are ignored when generating a crash signature. This improves our uniqueness determination since many crash backtraces end in libc functions (for example) even though that isn't where the actual problem lies.

Virtual machine upgrades

The DebianFuzz Virtual Machine now includes python 2.7 and gdb 7.2

Getting started with BFF

To get started with BFF, simply follow these steps:

1. Download BFF 2.5 at https://www.sei.cmu.edu/about/divisions/cert/index.cfm
2. Unzip BFF-2.5.zip to c:\fuzz
3. Unzip DebianFuzz-2.5.zip to a directory of your choice
4. Open DebianFuzz.vmx with VMware (e.g., Workstation, Fusion, Player, or compatible virtualization software)
5. Create a snapshot in VMware
6. Power on the VM

You may need to verify that the shared folder (c:\fuzz -> /mnt/hgfs/fuzz) is enabled in the VM preferences. Other virtualization products may work with some additional configuration. See the README file in BFF-2.5.zip for more details.

We provide the DebianFuzz VM for convenience, but BFF can be installed on other Linux systems as well. Additional information can be found in the INSTALL file in BFF-2.5.zip.

NOTE: We strongly recommend fuzzing in a virtual machine. BFF periodically clears the contents of the temporary directory, and when run on Linux it also activates a software watchdog on that will reboot the machine if a fuzzing campaign stops. Fuzzing may also trigger operating system bugs that could cause kernel panics.