Hi, this is Vijay Sarvepalli, Information Security Architect in the CERT Division. On what seemed like a normal day at our vulnerability coordination center, one of my colleagues asked me to look into a vulnerability report for pppd, an open source protocol. At first glance, this vulnerability had the potential to affect multiple vendors throughout the world. These widespread coordination cases usually have a prolonged coordination timeline. They typically involve multiple vendors on the one end and a security researcher (or "Finder" in the language of the CERT Guide to Coordinated Vulnerability Disclosure) on the other end, each with competing expectations and priorities.

Assessing the Vulnerability

The vulnerability was discovered in the Point-to-Point Protocol (PPP) daemon, or pppd. PPP is a Layer 2 protocol used for establishing connections over dial-up modems, DSL connections, and many other physical networks including mobile networks. PPP has been included in and expanded upon by additional protocols, such as the Point-to-Point Tunneling Protocol (PPTP) that is used in Virtual Private Networks (VPNs) to provide encrypted connections. To learn more about this vulnerability, read our advisory VU#782301.

In this case, SEI CERT's coordination team worked with security researcher Ilja Van Sprundel (IOActive) who discovered this vulnerability and software developer Paul Mackerras (OZlabs) who maintains the source code to quickly analyze the problem and find a solution. The problem involved a buffer overflow in the source code of pppd, caused by a simple error in a Boolean expression and the execution of conditional statements that followed it. The statement below can be tricked into accepting input of unspecified length and copying it to a stack buffer. This is typically known as buffer overflow or stack buffer overflow.

if (vallen >= len + sizeof(rhostname)) { // Copy to buffer rhostname

The fix for the vulnerability was to simply change the above statement to the Boolean logic below.

if (len-vallen >= sizeof(rhostname)) { // Copy to buffer rhostname

Paul obtained CVE-2020-8597 for this vulnerability and proceeded to patch it in the source code over which he had control. The code change to fix the vulnerability is a small one that involves only a few lines of code. However, this vulnerable code exists in thousands of software project repositories. It was implemented by more than 100 vendors that provide network connectivity products ranging from home routers to enterprise-scale network equipment. Because this vulnerability affects both PPP clients and servers, it also impacts Internet Service Providers (ISPs). A quick look using the Shodan search engine for Internet of Things (IoT) devices provided an estimate of over 2 million Internet-accessible devices that could potentially be affected by this vulnerability.

Our role in this vulnerability coordination process was not simply to notify vendors about this patch. We also had to find out how many software projects and entire devices were using the vulnerable pppd source code. Once I started searching for this code, I soon found that it was present at over 5,000 locations globally in numerous repositories. As I looked into these repositories and where their code was stored, I found that the top location where this code had been copied was GitHub. Well over 4,000 affected repositories (4,019 distinct repositories, to be exact) were served out of the GitHub software platform. After researching GitHub's latest security efforts, I realized that GitHub Security Lab had good potential for solving this problem.

GitHub Security Lab to the Rescue

As I explored this new security endeavor from GitHub, I tried to find ways to use GitHub's API and CodeQL approaches to solve the issue and employ automation to recommend the patch to software repository owners. I also reached out to our government champion of security efforts, Allan Friedman (Director of Cybersecurity Initiatives at the National Telecommunications and Information Administration (NTIA) in the U.S. Department of Commerce), who had gathered a number of organizations, including GitHub, for an effort in building a Software Bill of Materials (SBOM). Allan referred me to people at GitHub who are committed to security; they then put me in touch with the head of GitHub Security Lab, Nico Waisman.

Nico and his international team at GitHub Security Lab quickly found a way to apply their security patching process to this problem. They activated an automated "robot" code that reached out to the owners of all repositories that were affected by this vulnerability. Repository owners only needed to follow a few simple steps to patch and protect their copied or forked version of the pppd software, remediating the vulnerability. This group effort led us to a point where software was being patched by software. It provided a scalable, timely solution to adopt source code changes for security improvements. Within four days of the automated announcements by GitHub Security Lab, 1,896 repository owners received details of the vulnerability and were given the ability to patch it with a few clicks. At least 42 of these repository owners accepted the automated patch; 13 more stated that the issue had already been patched. Without automation, notifying the affected repository owners to patch their software would have taken many days longer.

DoD's Challenge and CERT's Role in the Future of Software

As a federally funded research and development center (FFRDC), Carnegie Mellon University's Software Engineering Institute (SEI) and its CERT Division are constantly confronted by the challenges the U.S. Department of Defense (DoD) faces in cyberspace. DoD CIO Terry Halvorsen, a renowned cybersecurity evangelist, said that "cyber defensive actions and counteractions will occur in milliseconds" in his address at the AFCEA conference. These desired cyber defensive actions cannot be done manually or through cumbersome communication processes. They must be delivered via software and automated as much as possible to limit issues with current human-in-the-loop patching models.

In future vulnerability coordination efforts, we hope to find scenarios where we can take advantage of opportunities (such as this collaboration with GitHub Security Lab) to automate the patching of source code against cybersecurity vulnerabilities. While we acknowledge this does not solve every problem with securing software and is not a replacement for secure coding practices, we also know that vulnerabilities will continue to be discovered in software after its release. As software becomes ubiquitous in our daily lives, we can only succeed in securing software by timely detection of vulnerabilities--and whenever possible, automating both detection and response.