Hi, it's Timur Snoke of the CERT NetSA group, posting on behalf of Deana Shick and Angela Horneman. It's not every day that 9.6 terabytes of data is released into the public domain for further research. The Internet Census 2012 project scanned the entire IPv4 address space using the Nmap Scripting Engine(NSE) between March and December of 2012. The engineer of this data set (identity unknown) saved and released the collected data in early 2013. The data is broken down into seven types of scan results: ICMP ping, reverse DNS, service probes, host probes, syncscan queries, TCP/IP fingerprints, and traceroute.

This information has proved valuable in our research in understanding aspects of devices associated with various sets of IP addresses, as shown in our upcoming tech report "Investigating Advanced Persistent Threat 1." This vast source of information also has potential for many other research projects.

The NSE allows users to write unique scripts to augment Nmap for their individual needs. Instead of scanning a personal network, the engineer used a personal NSE script to port scan the /0 address space. The engineer conducted the scan by developing a botnet, the Carna botnet, that deployed a small binary onto a group of non-secure sample machines. These non-secure machines were used to build a port scanner for the entire IPv4 address space.

The botnet consisted of a central server for data collection and analysis, middle nodes to transmit large pieces of data to the central server, and many devices for data collection. The data collection machines scanned other machines via a slew of Python scripts from this binary on Telnet port 23, which transferred data back to middle nodes. The middle nodes forwarded the information to the main server for data collection and analysis.

Because the Internet Census 2012 data was collected with the use of a botnet and scans done without permission, there are some ethical issues that surround the use of this data set in research. With the internet census, and any other data set obtained in what may be viewed an illegitimate manner, researchers must decide if they have the right to use the information. In some cases, the answer may be clear cut. For example, it's clear cut if the data set contains legally restricted information, but other cases are not so clear cut and are a judgment call. Researchers responsible for determining the impact of a botnet or remediating issues arising from a botnet must look at information related to that botnet, including what it collected, to map out appropriate responses and learn how to prevent future botnet infections. Other researchers may benefit by using the information collected from botnets in other types of studies. However, if they choose to use the information, they have the responsibility to use the data in a manner that protects the people impacted in collection from further harm. Some ways to do this are to only use information in aggregate, anonymized data and to be selective in publication.

As far as we can tell, this data set does not contain legally restricted information, like classified information, personally identifiable information, or trade secrets.

When an ICMP ping is sent to a device, it measures the time the device takes to return a response. The response will return one of the following: network unreachable, host unreachable, communication administratively prohibited, alive, alive from A.B.C.D, or unreachable from A.B.C.D. For the purposes of the IPv4 scan, the ICMP ping tells if the address was reachable, and if not, what IP address was used for the scan. This scan sent an ICMP ping to every address in the IPv4 space every few days.

The ICMP ping data provided in the internet census is divided by CIDR /8 netblock; each /8 netblock contains its own unique file. Besides the target IP address and status, the ICMP ping records also include a Unix format timestamp. Below are examples of IP address listings in an internet census ICMP file.

Individual IP addresses may or may not have multiple entries within the ICMP data. Since the scans were done multiple times within the March through December 2012 time frame, these entries indicated that only a portion of the responses collected from the census were released. Since this is the case, we cannot tell if a particular IP address never had an "alive" response. However, it may be useful to know that an IP address has at least one "alive" entry in the data.

This scan queried every IP address in the IPv4 address space for related domain names by sending requests to the top 16 DNS servers. Once compiled, DNS records were sent back to the collection server. The reverse DNS data provided in the internet census is divided by CIDR /8 netblock; each /8 netblock contains its own unique file. The rDNS entries also contain a Unix format timestamp for when the rDNS response was received. Below are examples of IP address listings in an internet census rDNS file.

The rDNS data has a large number of entries that have a numeric response, indicating that some error, such as record not found, occurred in the request. For the entries that have domain names in the responses, these may be helpful to use as a snapshot of domain names at various time periods in the second half of 2012. However, keep in mind that this data comes from only the 16 largest DNS servers and that the domain name to IP address mapping can change often.

Service probe scans look for responses on various ports that can be attributed to certain services. The probed IP address returns a state of 1 (open), 2 (open filtered), 3 (reset), 4 (closed/reset), or 5 (timeout). Although 175 billion probes were saved, only a small minority of the data has a return state of 1. For IP addresses returning a state of 2-5, the file lists only the number and no further explanation. If the IP address returns a state of 1, then the response to the probe is recorded and, in many cases, it is in an encoded or unreadable format. The data also includes a Unix format timestamp. Below are examples of IP address listings in an internet census service probes file.

The way the NSE scan handles service probe scans is to use the returned responses where the state is 1 to determine the most likely service that is running on that port. If this determination was done as part of the service probe scans from the Carna botnet, the results were not released as part of the data set.

Within our short time frame, we did not come up with a way to use the data provided to make the determination on our own. For the very small minority of cases where a response returns readable information, there may be some important data, but the analyst must go through the responses manually because the format is not standard, and the content varies widely. We have not developed an automated way to go through this part of the data set, although using a text search tool like grep with regular expressions may enable the identification of entries that may contain things like phone numbers or email addresses.

Host probe scans relay to a researcher whether the IP address is up or down and the reason for each state.This scanning was done as a first step of the ICMP ping scans to determine if a more in-depth scan should be tried against the IP address. The host probe data provided in the internet census is divided by CIDR /8 netblock; each /8 netblock contains its own unique file. The host probe data contains the target IP address, a Unix format timestamp for when the data was received, an indication of whether the device tied to the IP address is up or down, and the reason for the determination. Below are examples of IP address listings in an internet census host probe file.

The syncscan queries inform a researcher if a particular IP address has open, open-filtered, or closed ports. This scan contains every IP address in the IPv4 address space. The syncscan data provided in the internet census is divided by CIDR /8 netblock; each /8 netblock contains its own unique file. The data lists all of the open, open-filtered, and closed ports associated with each IP address in the /8 netblock. In many cases, an IP address has all three statuses, occurring at various times, and therefore has three different lines in the file. The syncscan query also provides the type of packet sent to a particular IP address, a Unix format timestamp for when the data was received, and the type of packet sent to each port. Below are examples of IP address listings in an internet census syncscan file.

This scan attempts to gather information that identifies the type of device and the operating system (OS) running on the machine. The data includes the target IP address, a Unix format timestamp, and the fingerprint of the target device. Below are examples of IP address listings in an Internet Census TCP/IP fingerprint file.

It appears the Nmap tool used in the scan was modified to store the response used in the determination of the fingerprint. The standard Nmap program captured the operating system results, but does not keep the actual response used in fingerprinting. If the operating system results were captured during the internet census scans, they were not provided in the internet census data.

It is possible to write a program that compares the stored fingerprints with the Nmap nmap_os_db file to get an idea of what type of device may be tied to an IP address. However, not all the fingerprints have a high percentage of fields that match any template in the Nmap file. Also, a few of the stored fingerprints are malformed, not matching the standard format, and some also had random extraneous data embedded.

The traceroute scan provides the path a data packet took from a source address to the destination. Unlike the other scans, all traceroute data is stored in a single file. This makes sense as this data set has the least amount of entries. Each entry contains multiple IP addresses, usually from multiple CIDR /8 netblocks. The data includes a Unix format timestamp, the IP address of the device initiating the traceroute scan, the destination IP address, an indication of whether the scan was done using ICMP or UDP, and the route results. Below are examples of IP address listings in the traceroute file.

Most of the traceroutes were not able to capture a full route. Many capture only the first hop, often a 10/8 address, and many others just seem to peter out after several hops. There are relatively few that appear to be complete. It may be interesting to map out the complete traceroutes, but we have not yet found a reason to use this data in our analysis.

Recap

The Internet Census 2012 data set contains lots of data points. Several of the scan types may be interesting, but require much manual sifting or provide duplicate data with other scans.

• Service probes did not store the most useful part of the data: the service running on a port.
• Host probes data is redundant with ICMP ping.
• Traceroutes have relatively few complete routes between different IP addresses; compared to other scans, relatively few IP addresses are represented.

The other scans are easier to use with automated analysis or provide significant data.

• ICMP ping data can be used to quickly find IP addresses that were alive on at least one date.
• Reverse DNS can be used to get an idea of IP addresses associated with specific domains or vice versa.
• Syncscans provide an interesting set of IP addresses that have open ports.
• TCP/IP fingerprints can be used to determine likely operating systems or devices tied to an IP address.