While the Internet has enabled modernization in parts of the developing world, it has also introduced new cybersecurity challenges. Many developing countries are unprepared for large-scale cyber attacks and ongoing threats posed by hackers. A July 2017 New York Times article notes that developing countries have become an ideal testing ground for hackers. These attacks caught the attention of the Cote d'Ivoire (Ivory Coast) computer security incident response (CSIRT) team, who reached out to the SEI through the U.S. Department of State Office of the Coordinator for Cyber Issues (S/CCI) to request a collaborative workshop to help Cote d'Ivoire address the threats of distributed denial-of-service (DDoS) attacks and botnets. Through this engagement, we developed a set of training documents and guides that can help other CSIRTs in developing countries.

As my colleague Angel Hueca wrote in a September 2018 blog post introducing CSIRTs and their role in building cyber capacity across the globe, the SEI CERT Division's International Cybersecurity Initiatives (ICI) Team and the S/CCI collaborate with the global national CSIRT community; stakeholders from the government, private sector, and technical community; and relevant regional and international organizations. We travel to different nations and train officials and staff on best practices and what to look for when training cybersecurity teams. Working side by side with national CSIRT teams and partnering governments, we pinpoint where further development is needed, conducts workshops on CSIRT functions, and provide guidance for developing internal workflows, policies, and standards. This post is the second in a series about our efforts.

The Cote D'Ivoire side Ministry of Telecommunications gathered a well-prepared team for the workshop. It included representatives from the Cote d'Ivoire Computer Emergency Response Team (CI-CERT); the Cote D'Ivoire Internet Exchange; .CI, the Internet country code top-level domain for Cote D'Ivoire; country-level domain name system root operators; and a number of local Internet service providers (ISPs). This combination of government and commercial entities helped drive discussions toward practical approaches to DDoS detection and botnet mitigation.

Below I will walk through our experience "jour-par-jour" (day-by-day) to overview how the workshop helped identify the best path forward for cyber capacity building in this developing country. I will also describe another outcome of the workshop: a valuable document that will steer efforts in Cote d'Ivoire and elsewhere to improve cybersecurity, starting with DDoS mitigation and expanding to other relevant topics in internet security.

The Journal of Our Week

Jour 1: Network analysis background and understanding network abuse

On Day 1, we reviewed network analysis and common abuses of the Internet. We then discussed the challenges that enable such abuses: The Internet enables anonymity, and its lack of clear borders leads to ambiguity about which laws apply and how to enforce them when attacks can occur from around the world. We also explored challenges specific to developing countries, such as lack of regulation, weak enforcement of the law, and a lack of tools to track network abuse. Participants discussed pragmatic technical solutions to these issues. For example, the team collaboratively created a chart of possible network failures that would indicate a DDoS event is in progress and mitigation methods that they could apply in each case.

Jour 2: Intrusion detection, anomaly detection, and security monitoring

On Day 2, we focused on increasing workshop participants' technical understanding of intrusion detection. The group participated in exercises to improve monitoring, increase transparency, and facilitate collaborative reporting. For example, in one exercise a simulated mock event caused two of the major 15 physical cables to be flooded with unknown traffic. Participants analyzed this mock event and wrote an after-action report about it.

We then introduced some open-source tools for detecting intruders in large-scale operations centers such as Cote d'Ivoire's. These included MRTG, Cacti, OpenNMS, and IXP Manager. We also explored other freely available resources for security monitoring, such as Spoofer, a tool developed by the Center for Applied Internet Data Analysis (CAIDA) that assesses anti-spoofing practices; RIPEstat, which provides information about specific IP addresses; and DNS Stats Collector (DSC), which collects statistics from DNS servers. Open-source tools and public resources can help teams who work on a shoestring budget to address cybersecurity more efficiently and effectively.

Jour 3: DDoS concepts and botnets

Our assembled group was eager to understand the challenges related to DDoS and the growing concern of botnets. Despite technical language barriers, we were able to collaborate to understand the great outreach of botnets into developing countries and the necessity of international cooperation to address this problem. For example, workshop participants looked at a recent State of the Internet report on DDoS attacks by Akamai; then they used a CAIDA country report to investigate whether Cote d'Ivoire computers were sources of this unwanted traffic. As the participants explored this case, they became acutely aware that our interconnectedness continued to be abused by nefarious individuals to perform large-scale attacks.

Jour 4: Mitigating DDoS and botnet attacks

Now that we had introduced DDoS concepts, we turned to approaches to mitigate DDoS attacks. We began with some exercises to practice identifying botnets that grow through self-propagation, finding command-and-control (C2) infrastructure taking it down, and finally identifying DNS amplification attacks.

As the ISP representatives collaborated to perform these exercises, they found many technical and operational ways to mitigate botnet risk and DDoS threats on an ongoing basis. For example, ISPs were able to identify components of their infrastructure that could be abused for nefarious activities. This exercise helped them expand their analysis to identify resources with similar misconfiguration or improper protection and repair or replace these devices before they can be used in future attacks. As the community members collected ideas and activities, we filled a 20-page easel pad with insights from cross-functional organizations. Some of these are captured in the DDoS guide for NatCSIRTs, described below.

Jour 5: Sustaining secure operations to prevent DDoS and botnet attacks

Now that the team had mastered the ins and outs of DDoS attacks--seeing them from the perspectives of both the victims and unintended participants--it was time to bring all the workshop participants' new knowledge together into a strategy for sustainable operations. The CERT CI Team diligently captured the local knowledge that was developed from this busy 5-day work with the nationals. The outcome was to build a customized quick-reference guide or a cheat sheet that will help many developing countries' NatCSIRTs.

A Way Forward: A DDoS Guide for NatCSIRTs

The workshop revealed both technical insights and the need for process development and ongoing training to address the challenging issues of DDoS. We captured some practical steps in the DDoS Quick Reference Guide for NatCSIRTs that can be downloaded from the SEI online library. I will highlight some of these because they were developed at this workshop and will likely drive ongoing efforts to mitigate the risk of botnets beyond sub-Saharan Africa:

1. NatCSIRTs should adopt perimeter security monitoring at large-scale enterprises and IXPs (Internet exchange points).
2. NatCSIRTs should coordinate DDoS exercises (e.g., hackathons) and evaluations.
3. ISPs should implement Quality of Service (QoS) and rate limiting and prioritize critical communications such as VoIP.
4. Vendors and local manufacturers should pursue secure code and establish Product Security Incident Response Teams (PSIRTs) to address security vulnerabilities.
5. ISPs and mobile providers should block or otherwise remediate botnets in a timely fashion with coordinated service-level agreements.

Looking Ahead

Our workshop with Cote d'Ivoire spawned many hard discussions as commercial partners, non-profit organizations, and the government tried to balance policy, productivity, and commercial interests. We will use these discussions and our experience working with Cote d'Ivoire to improve our workshop and continue to help NatCSIRTs in other developing countries.

I would like to thank the U.S. Department of State and the Cote d'Ivoire NCSIRT team for making it possible for us to engage with a vibrant community and impact the security of the Internet. The topic of DDoS and botnets is both relevant and well suited for a collaborative workshop. These efforts will continue to evolve as the threat of DDoS from botnets continues to target various weaknesses in the open and collaborative Internet platform. The SEI's CERT Division is committed to ongoing efforts with developing countries to address this very critical issue.

Additional Resources

State Department S/CCI overview: https://www.state.gov/s/cyberissues

Internet Society report on Africa and sub-Saharan Africa:
https://www.internetsociety.org/internet/history-of-the-internet-in-africa/

Impact of the Internet on the economy in Africa:
https://www.theguardian.com/world/2016/jul/25/can-the-internet-reboot-africa

Internet user statistics for Cote d'Ivoire:
http://www.internetlivestats.com/internet-users/cote-d-ivoire/

World Bank Internet user and economics profile of Cote d'Ivoire:
https://data.worldbank.org/indicator/IT.NET.USER.ZS?locations=CI&name_desc=false

ITU Google open data on the Cote d'Ivoire Internet: https://goo.gl/cRpXY1

SEI CSIRT information page:https://www.sei.cmu.edu/education-outreach/computer-security-incident-response-teams/index.cfm