The network time protocol (NTP) synchronizes the time of a computer client or server to another server or within a few milliseconds of Coordinated Universal Time (UTC). NTP servers, long considered a foundational service of the Internet, have more recently been used to amplify large-scale Distributed Denial of Service (DDoS) attacks. While 2016 did not see a noticeable uptick in the frequency of DDoS attacks, the last 12 months have witnessed some of the largest DDoS attacks, according to Akamai's State of the Internet/Security report. One issue that attackers have exploited is abusable NTP servers. In 2014, there were over seven million abusable NTP servers. As a result of software upgrades, repaired configuration files, or the simple fact that ISPs and IXPs have decided to block NTP traffic, the number of abusable servers dropped by almost 99 percent in a matter months, according to a January 2015 article in ACM Queue. But there is still work to be done. It only takes 5,000 abusable NTP servers to generate a DDoS attack in the range of 50-400 Gbps. In this blog post, I explore the challenges of NTP and prescribe some best practices for securing accurate time with this protocol.

How NTP Works

Segal's Law states

A man with a watch knows what time it is. A man with two watches is never sure.

Computer scientist David L. Mills created NTP in the early 1980s to synchronize computer clocks to a standard time reference. Since the inception of NTP, a group of volunteers with the NTP pool project have maintained a large, publicly available "virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients" around the world for many Linux distributions and network appliances.

As detailed at NTP.org, NTP works in a hierarchical fashion by passing time from one stratum to another. For example, Stratum 0 serves as a reference clock and is the most accurate and highest precision time server (e.g., atomic clocks, GPS clocks, and radio clocks.) Stratum 1 servers take their time from Stratum 0 servers and so on up to Stratum 15; Stratum 16 clocks are not synchronized to any source. The time on a client is established through an exchange of packets with one or more stratum servers. These packets put timestamps on each message and the time taken to transmit the messages are factors in the algorithm for establishing the consensus for time that should be on the client.

NTP can provide an accurate time source through consensus with multiple input servers. It can also identify which available time servers are inaccurate. One challenge is that NTP was built during a time when the Internet community was friendlier. During NTP's inception, NTP servers weren't concerned with user verification. In the early 1980s, it was common for NTP servers to be publicly available as a resource developers could use to troubleshoot and confirm their own NTP solution.

The standard protocol for NTP is User Datagram Protocol (UDP). This design decision created opportunities for abuse. UDP, which is connectionless, is a best-effort protocol and, therefore, more susceptible to spoofing and losing packets than Transmission Control Protocol (TCP). While NTP pool project volunteers have continued to advance NTP, the motivation for many network administrators and business owners to patch their own gear is not as strong. Since its inception, the NTP networking protocol has been integrated into countless systems, many of which haven't been patched and could be running code that is 20 revisions old or more.

Attacks Related to NTP Vulnerabilities

Cory Doctorow recently wrote about the potential fallout of outdated NTP for Boing/Boing:

NTP is how virtually every computer you interact with keeps its clock accurate, which is a function so fundamental to the functioning of the Internet that it can't be overstated... What's more, vulnerabilities in NTP had turned the Internet's many time-servers into force-multipliers for Denial of Service attacks, making merely punishing attacks into nearly unstoppable ones.

The method being employed for recent DDoS attacks does not rely on vulnerabilities but poor configuration. NTP services are responding to a query request for a list of monitored servers. A single small request with a spoofed source can generate a list of 600 servers and be sent to a target. Even though there is no vulnerability, when hundreds or thousands of these servers are being redirected at an unwitting target, the victims do not care about the semantics of the issue causing them pain; they just want relief.

Time synchronization on systems, or lack thereof, can be a significant contributing factor to vulnerabilities that can compromise a system's basic functions. In addition to NTP abuse contributing to DDoS attacks, lack of time synchronization on a network creates an opportunity for replay attacks (i.e., playback attacks) involving the fake or malevolent repeated delay of an authentic data transmission.

For example, a replay attack can occur when a user attempts to provide proof of identity to another user. A malicious actor in the middle would intercept the message and prevent it from getting to its intended target. The malicious user then sends a request for identity confirmation and includes the stolen proof as a validation. If the time is not synchronized, the window that the exchange is allowed can be increased beyond what is considered safe and allows the subterfuge. As a result, valid users can be tricked into thinking that they have successfully confirmed the identity of an imposter posing as a legitimate user. In all fairness, this type of replay attack is uncommon and extremely challenging to execute successfully without access to the network, the communications path, and a compromised machine in that path.

Many security experts, such as Shaun Kelly, have noted that NTP has been used to manipulate logs and change the time on a computer system, altering the sequence of events. When clocks aren't synchronized, network analysts have a much harder time performing log correlation across disparate systems. The manipulation of NTP can make the identification of network activities and sequence of events leading up to an attack much harder to pinpoint.

Other applications that would be at risk because time is not working right include high-speed trading and security cameras. Many time-sensitive encryption algorithms involving key exchanges and tokens are also at risk because of NTP weaknesses.

NTP Best Practices

The remainder of this post details best practices for configuring your own NTP server and requesting a public NTP server.

Use Public NTP for external hosts. If an enterprise is building capabilities, services, or other embedded platforms that are intended to be deployed outside of the enterprise, network administrators might want to consider requesting a public NTP server from the pool of available servers mentioned earlier.

It's important to note that most of the public NTP servers specify rules of engagement. If an enterprise has multiple devices within the enterprise that will be using NTP, it would make sense to set up their own hierarchy to synchronize with instead of competing for access to the publicly available servers.

Configure your own Internal NTP hierarchical service for your network. It is possible to purchase Stratum 1 or Stratum 0 NTP appliances to use internally for less than the cost of a typical server. It is also possible to set up a private NTP server at a very low cost. The feasibility of setting up a commercial off the shelf (COTS) NTP server is evidenced in a recent effort to configure a Raspberry Pi computer as a Stratum-1 server. If you do decide to configure you own, please consider the following best practices:

• Standardize to UTC time. Within an enterprise, standardize all systems to coordinated universal time (UTC). Standardizing to UTC simplifies log correlation within the organization and with external parties no matter what time zone the device being synchronized is located in.
  
  
• Securing the network time service. Restrict the commands that can be used on the stratum servers. Do not allow public queries of the stratum servers. Only allow known networks/hosts to communicate with their respective stratum servers.
  
  
• Consider the business need for cryptography. Many administrators try to secure their networks with encrypted communications and encrypted authentication. I would introduce a note of caution here because although there are cryptographic services associated with NTP for securing NTP communications, the use of encryption introduces more sources for problems, such as requiring key management, and it also requires a higher computational overhead.

• Remember Segal's Law. Ideally, it would work to have three or more Stratum 0 or Stratum 1 servers and use those servers as primary masters. Remember Segal's Law: having two NTP servers makes it hard to know which one is accurate. Two Stratum 0 servers would provide a more accurate timestamp because they are using a time source that is considered definitive.

The presence of three or more time sources would allow the network to maintain accurate time even if one of the primary masters fails. Ideally, NTP servers would be located in three geographically disparate locations. This group of primary masters would be the source for time for the enterprise. They would be considered hidden masters because they would only provide services to the secondary stratum servers. This configuration would allow those servers to provide time to collocated secondary masters that are actually providing services to an organization. The primary masters remain hidden and are only accessed by the NTP infrastructure that provides services elsewhere. That supply chain should allow you to provide accurate time across your organization and have multiple sources corroborating an accurate time source.

Locations that have more devices needing to have their time synchronized can add additional Stratum 2 or Stratum 3 servers and have them rely on the secondary masters as well as each other to further distribute the load on a system and providing services to a larger group of NTP clients.

By setting up an internal NTP service on the latest revision of stable code and standardizing its use, the viability of time-based network attacks or processes that are dependent on time are harder to co-opt. The identification of the order of events in a compromise becomes easier because the times in the logs can now be systems of record. For law enforcement and other investigative agencies, accurate NTP services can be very constructive in evaluating evidence and sequencing a chain of events.

Wrapping Up and Looking Ahead

As attacks become more sophisticated, our team of network analysts at CERT increasingly finds Internet-facing services that aren't well deployed within a network. As Mark Langston wrote in his recent post on DNS Best Practices, many of these services make up the foundation for the security and operation of internal and external network applications.

This is the latest in a series of blog posts offering best practices on these foundational structures to help government agencies and other enterprises address hidden sources of vulnerabilities within their networks. Our team lead, Rachel Kartch, published the first post in this series, Distributed Denial of Service Attacks: Four Best Practices for Prevention and Response.

We welcome your feedback in the comments section below.

Additional Resources

Read Mark Langston's blog post Six Best Practices for Securing a Robust Domain Name System (DNS) Infrastructure.

Read Rachel Kartch's blog post Distributed Denial of Service Attacks: Four Best Practices for Prevention and Response.