Hi folks, it's Will. Recently I have been investigating man-in-the-middle (MITM) techniques for analyzing network traffic generated by an application. In particular, I'm looking at web (HTTP and HTTPS) traffic. There are plenty of MITM proxies, such as ZAP, Burp, Fiddler, mitmproxy, and others. But what I wanted was a transparent network-layer proxy, rather than an application-layer one. After a bit of trial-and-error investigation, I found a software combination that works well for this purpose. I'm happy to announce the release of CERT Tapioca (Transparent Proxy Capture Appliance), which is a preconfigured VM appliance for performing MITM analysis of software.

From a software analysis perspective, network sniffing and MITM proxies can uncover several types of software issues:

I wanted an environment where I could easily investigate all of these scenarios. One of the important requirements is that the MITM proxy operates on the network layer rather than the application layer. Why? I need to be able to test applications and devices in a way that the target under test is not aware that it is being proxied. This ability allows me to transparently test applications and devices that do not support proxies.

Network-Layer MITM Proxying

To perform network-layer proxying, you need two things:

1. A proxy application that supports transparent proxying.
2. An operating system network configuration that supports redirection based on the NAT target port.

To achieve network-layer proxying, I took the UbuFuzz VM that we provide for BFF and customized it. UbuFuzz VM has two network adapters: one for the internet/uplink side and one for the local side. On the local side, it provides NAT, DHCP, and DNS capabilities. For the MITM proxy, I chose to use the excellent mitmproxy software. The scripts that launch mitmproxy configure iptables to redirect network traffic to the local proxy. (For more information about how mitmproxy works, check out its documentation.)

At this point, we have a VM that provides internet capability to the local side, and any traffic that is destined for port 80 or 443 (HTTP, and HTTPS, respectively) is proxied with mitmproxy. What you connect to the local side is completely up to you. It could be connected to a virtual network so that software running in other virtual machines will be MITM proxied. Or you can bridge a wireless access point to the local side so that you have a wireless network where any connected client will have its web traffic proxied.

CERT Tapioca

Let's look at what we get when we power up the CERT Tapioca VM:

Here we have a basic Xfce desktop environment. The default fluxbox window manager that comes with UbuFuzz is a bit too lightweight for our needs. In the basic environment, mitmproxy and tcpdump are both started automatically, logging data to the ~/logs directory.

On the bottom toolbar, there are two icons that you will likely use the most:

If you want to bypass the mitmproxy software, click the red stop sign icon. This function adjusts the iptables rules to simply perform NAT. This function can also allow you to visit https sites and have a valid certificate. The green arrow button clears out the logs and restarts both mitmproxy and tcpdump, configuring iptables to route web traffic through the proxy.

CERT MITM Proxy in Use

Let's try using the CERT MITM proxy VM by connecting another VM to the same virtual network as the local side of the proxy VM. We fire up our horribly outdated Internet Explorer 6 browser and go to www.google.com:

Here we see several http GET requests in the mitmproxy window. What if we try to visit https://www.google.com instead?

It's good that we get this warning. It means that our client software is checking the validity of SSL certificate chains when making HTTPS network connections. In this particular case, it's saying that the certificate was not issued by a root certificate authority that the browser already trusts. If we don't accept this warning, then mitmproxy will not log any HTTPS requests.

If we click Yes, then mitmproxy will log our traffic:

Then we can actually dive into any of the requests to see the details:

Here we see the content that was transferred over the encrypted https connection. We see the traffic because mitmproxy communicates securely with the web server, decrypts the traffic, and then re-encrypts the traffic using a dynamically generated certificate using its own root CA certificate.

Interpreting Results

As I mentioned previously, if you ever see an https:// URL in mitmproxy, you're either dealing with an application that fails to validate SSL certificate chains or you have manually accepted the invalid certificate. We can easily check for this programmatically:

grep "scheme,5:https" ~/logs/flows.log

This command determines whether the client successfully sent or retrieved data through the HTTPS connection.

Intercepting All HTTPS Traffic

Rather than investigating which applications properly validate SSL certificate chains, if our goal is to investigate all HTTPS traffic coming from an application, we need to change our client platform by importing the mitmproxy root CA certificate. The root CA certificate used by mitmproxy is available in several formats in the ~/certs directory in the CERT Tapioca VM. The folks at CAcert have provided guidelines for how to import a root CA certificate in a variety of platforms.

Trusting HTTPS

Once you start performing MITM testing of HTTPS traffic, you hopefully will have a better idea of the level of trust that you should be giving HTTPS. Let's consider the situation where you type https:// and the domain name in your web browser's address bar for the site that you want to visit. If you don't get a certificate warning, what does that mean? It really just means that the certificate provided by the server was issued by any one of the root CAs trusted by your browser. For example, the current version of Mozilla Firefox comes pre-loaded with over 90 trusted root CAs. Any one of these sites may provide a dozen or more individual trusted root CA certificates.

If you're using a system that you don't manage and you're relying on HTTPS to keep your web traffic from prying eyes, you may want to think twice. Just like we can silently intercept HTTPS traffic by having the mitmproxy root CA certificate installed, you must assume that enterprises with managed desktop systems are employing similar techniques to monitor traffic. It's their network after all.

But it's also worth noting the impact the compromise of a single root CA can have. It's happened in the past with DigiNotar and Comodo. Without getting too sidetracked here, Moxie Marlinspike's blog entry SSL And The Future Of Authenticity goes into a good amount of detail about the problems with SSL and trust.

Getting CERT Tapioca

CERT Tapioca is available in OVA format:

CERT_Tapioca.ova (sig)

This OVA should be compatible with a range of virtualization products, including VMware, VirtualBox, and others. Just take note which network adapter connects to which network. The adapter labeled "NAT" is the uplink, so by default it will share the internet connection of your host system. The adapter labeled "Custom" is the local side, so you will want to connect this adapter to whatever system you wish to investigate. Depending on which virtualization software you use, the uplink adapter may not be the first network adapter in the virtual machine properties.

Happy bug hunting!