Recently, Dan Kaminsky published a blog entry that compared the fuzzing resiliency of Microsoft Office and Oracle OpenOffice. This blog entry contains the results from a similar test that I performed in November 2010. Also included are some other aspects of the Office suites that can affect the software's security.

Background

Fuzz testing is a dynamic software testing technique that can be used to find bugs that result in the crashing of an application. Every bug that results in a crash has the potential of being a vulnerability. Depending on the specific circumstances of a crash, these bugs may also result in vulnerabilities that allow an attacker to execute arbitrary code. Fuzz testing can be used as one measure of the number of vulnerabilities that an application may contain.

The Test

I used a Python-based mutational fuzzing framework for Microsoft Windows. The fuzzer takes a starting, or "seed," file, mutates it, and opens it using the target application while monitoring that application for a crash. Microsoft's !exploitable Crash Analyzer debugger extension analyzes any crashes and assigns them each a hash identifier. The fuzzing framework uses the hash that was generated to determine if a crash is unique. Note that !exploitable may assign different hashes to the same underlying software defect. I configured the fuzzing framework to use a set of 19 different DOC files. The fuzzer used a random byte mutation strategy, which set the value of a range of bytes to a random value. The range used for this test was to mutate 0.001% to 1% of the seed file for each iteration. Each seed file was mutated in 10,000 different ways, resulting in a 190,000-iteration fuzzing campaign for each target application. I used the same set of 190,000 mutated files to test each Office suite.

The Targets

I tested currently supported Microsoft Office products:

• Microsoft Office XP Professional (released March 5, 2001)
• Microsoft Office 2003 Professional (released November 17, 2003)
• Microsoft Office 2007 Professional (released January 30, 2007)
• Microsoft Office 2010 Professional (File validation disabled)
• Microsoft Office 2010 Professional (released June 15, 2010)

All Microsoft Office products were fully patched with updates as of November 2010.

I tested Microsoft Office 2010 with file validation both enabled and disabled. The default configuration for Office 2010 has a feature called "Gatekeeper," which performs some preliminary validation of input files before opening them. If a file fails the validation, the user receives a warning but can continue to open the file. The standard fuzz testing run for Office 2010 indicates files that would crash the application without any additional user interaction. The "Office 2010 with File Validation disabled" fuzz testing run indicates files that would crash the application if the user proceeds to open the file despite the warning.

I also tested the following versions Oracle OpenOffice:

• OpenOffice 3.2.1 (released June 4, 2010)
• OpenOffice 3.3.0 RC7 (released November 26, 2010)

Because the fuzzing campaign focused on DOC file parsing, I used Microsoft Word in the Microsoft Office suite and OpenOffice Writer in the Oracle OpenOffice suite.

The Results

This graph shows the results from all of the products. As indicated by the first five columns, there is a clear decrease in the number of unique crashes with the Microsoft Office products. Office XP has the most unique crashes of any of the Office suites. However, there is not a one-to-one mapping of unique crash hashes to software bugs. More specifically, the !exploitable extension produces a Major hash and a Minor hash. Unique Major hashes are more likely to be unique bugs, while unique Minor hashes may be variations of the same bug.

Here is the same data, but only focusing on the crash Major value:

By looking only at the crash Major value, the Office XP bar is less of an outlier. An investigation of the crash hashes revealed that a large number of the unique crashes for Office XP contained the same Major value but a different Minor. Those results suggest that there is a good chance that they are all the same underlying bug. To narrow the scope of the data even further, let's look at only crashes that are reported to be either EXPLOITABLE or PROBABLY_EXPLOITABLE by the Microsoft !exploitable extension:

With this particular fuzzing campaign, Oracle OpenOffice produced more unique crash Major values that are tagged as EXPLOITABLE or PROBABLY_EXPLOITABLE.

While you can determine some trends with these graphs, it's important to realize that every Office suite contains bugs that can easily be discovered with simple fuzzing techniques. Because software vendors will never eliminate 100% of the bugs in their software, it is important to consider the features that can help prevent an attacker from exploiting the bugs.

Mitigations

Eliminating software defects should be a priority for software vendors, but software can never be 100% defect-free. Therefore, it is also important that software vendors take advantage of runtime exploit mitigations available in the operating system. Runtime exploit mitigations make it more difficult for an attacker to execute code when exploiting a vulnerability. For example, if runtime exploit mitigations are in place when a user opens a malicious Office document, they may cause the application to crash instead of allowing malicious code to be executed. The following are some runtime exploit mitigations:

• ASLR (address space layout randomization) - This feature prevents libraries from loading at predictable locations in order to mitigate exploits that rely on return-to-library or "return-oriented programming" techniques.
• DEP (data execution prevention) - This feature prevents execution of code that resides in memory pages that aren't marked as executable. To be truly effective, DEP must be combined with ASLR because there are several strategies for bypassing DEP that leverage predictable locations of libraries.
• Preliminary File Validation - This feature checks that a file conforms to an application's expectations. Software bugs are often triggered by malformed data. If a malformed file fails validation, then the target application may not open the file.

Further information about DEP and ASLR can be found in the Microsoft Security Research & Defense Blog.

We tested which mitigations the Office suites use. In the table below, if there is a "Y" in the Permanent DEP column, the application uses SetProcessDEPPolicy to enable DEP for the life of the process. The ASLR column indicates the percentage of libraries in the application's directory that are linked with the /DYNAMICBASE flag, which indicates compatibility with ASLR. Any library that is not linked with /DYNAMICBASE can potentially be used to bypass DEP. There are two caveats for this column:

1. Although a library exists in the application's install directory, the application does not necessarily use the library.
2. An application may install and load a library from a location outside of the application's install directory.

The percentages reported for ASLR can be used as a rough approximation of the level of ASLR compliance. Finally, the Preliminary File Validation column indicates whether the application performs validation of files before opening them for full processing.

In the table, the colors represent how well the product incorporates the mitigations. Green indicates the ideal, yellow indicates some conformance, and red indicates little or no conformance.

Mandatory Mitigations

The information in the previous table indicates that none of the Office suites fully employ all of the runtime exploit mitigations available in Windows. Microsoft has provided a tool called the Enhanced Mitigation Experience Toolkit (EMET), which can be used to force an application to use various runtime exploit mitigations. An example of how to use EMET to mitigate an Adobe Reader vulnerability was provided in the Microsoft SRD Blog. A similar method can protect the various components of the software suites identified in this document. The full user guide for EMET is available in PDF form.

EMET can be used to enable DEP and ASLR on a system-wide or per-application basis. Note that Microsoft Windows Vista or later is required to support ASLR.

Software Updates

The final aspect of the Office suites that I investigated was how the applications received updates. Unlike robustness to fuzzing, the method for receiving updates is not a direct indicator of code quality. However, the mechanism through which an application uses to receive updates can affect the security of the system that runs the application.

In the following table, green indicates the ideal answers.

Two important aspects of checking for software updates:

1. Does the check occur over SSL or another secured channel?
2. Does the software validate the signature of the downloaded software?

If the software fails to perform those two aspects, then it may be a security risk. Oracle OpenOffice does not use a secured channel to check for updates, and the software that is downloaded is not digitally signed. Therefore, OpenOffice is vulnerable to the EvilGrade attack. In fact, this vulnerability has been known since 2008.

Conclusion

All of the Office suites that I tested were vulnerable to a number of crashes due to malformed input. The fuzzing campaign focused solely on DOC file parsing. However, each Office suite supports many types of data. For example, Microsoft Office supports more than 100 different file types. The numbers shown in the graphs are a small subset of the total number of unique crashes that can easily be discovered through random mutation fuzzing.

Of the Microsoft Office suites, Office XP has the most unique crashes. It is important to note that the extended support end date for Office XP is July 12, 2011. After this point, Microsoft will not provide security updates for the product. Office 2010 has the fewest number of unique crashes, uses permanent DEP, and has the highest number of libraries that are compatible with ASLR. Office 2010 file validation, which is enabled by default, provides additional protection against malformed input.

Of the Oracle OpenOffice products, the RC7 prerelease for OpenOffice 3.3.0 shows a small decrease in unique crashes when compared with the release version 3.2.1.

We are currently coordinating the results of the fuzz tests and other security analysis with the affected vendors.