icon-carat-right menu search cmu-wordmark

6 Lessons Learned from Hosting the President's Cup Cybersecurity Competition

John DiRicco

A strong cybersecurity defense is vital to most public- or private-sector activities in the United States. In 2019, Executive Order 13870 stated that, “America’s cybersecurity workforce is a strategic asset that protects the American people, the homeland, and the American way of life.” One outcome of this executive order is an effort to foster cybersecurity education through competitions. These events allow participants to tackle real-world cybersecurity problems in a timed, competitive, safe environment through hands-on challenges that assess and build cybersecurity skills. Competitors sign up for individual or team tracks or both to strengthen their abilities and be tested by offensive or defensive challenges. A cybersecurity competition is an ideal environment for these professionals to dive into and explore realistic scenarios.

The Department of Homeland Security’s Cybersecurity & Infrastructure Security Agency (CISA) was tasked with holding a cybersecurity competition for the federal cyber workforce. It chose to partner with the SEI to develop and run the President’s Cup Cybersecurity Competition, a national cyber competition that identifies, recognizes, and rewards the best cybersecurity talent in the federal executive workforce.

In six years more than 8,000 people have taken part in the President’s Cup. In that same time span nearly 4,000 help-desk tickets concerning questions with challenges/the platform/registration, etc., were created to support the President’s Cup. While designing high-level challenges is the most important part of a cybersecurity competition, having a support team that can process and resolve competitors’ concerns in a timely fashion is a critical part of making a cybersecurity competition successful. In this post we present lessons learned from six years of hosting President’s Cup Cybersecurity Competitions including the necessity of competition support staffing.

Support Team Purpose and Structure

Even though participants are competing, they still occasionally need the assistance of a support team. The goal of the support team is to help competitors experience a seamless event. The support team doesn’t offer hints to the individuals and teams when they get stuck during a challenge; it serves as an intermediary between competitors and the competition’s platform and challenge engineers whenever challenge questions and/or platform issues arise. Sometimes a technical issue with a challenge or the competition environment needs repair, while other times a competitor seeks clarity about a particular part of the challenge. The support team keeps the wheels moving.

Support Hours and Tiers

The first decision when planning support for a competition is deciding when to offer live support. Some competitions offer live support 24/7, while others offer live support for specific times during the rounds. Either way, it’s important to clearly communicate the hours when competitors can and cannot expect live assistance.

The President’s Cup Cybersecurity Competition is run through Gameboard, an open source application, where users access the challenges and reach out to the support team through the Gameboard-hosted ticketing system. Before tickets start arriving, it’s a good idea to break the support team into three tiers to best triage and resolve issues.

  • Tier 1. During the competition, Tier 1 support staff are responsible for fielding initial support tickets, acknowledging action is being taken and communicating with the competitors until the issue is resolved. The emphasis is to resolve help requests as soon as possible since there are time constraints during the competitive rounds. Some examples of Tier 1 issues include registration questions, profile updates, and general questions about rules.
  • Tier 2. Sometimes an issue must be escalated to developers for resolution. Perhaps a feature requires troubleshooting, or an engineer must determine if the challenge is working correctly. These engineers form the Tier 2 part of the support team. Some other examples of Tier 2 issues include problems with grading, virtual machines that fail to launch, or clarification questions about wording in challenge documentation.
  • Tier 3. Tier-3 problems, such as infrastructure outages or bugs, can be the most serious to handle because of their potential severity. For example, if virtual machines for all challenges are suddenly unable to start, the entire competition grinds to a halt until the problem is rectified. Therefore, infrastructure experts must be available or on call in case an urgent situation emerges.

How do competitors reach support, and how is a problem funneled to the correct tier for resolution?

Support Workflow and Responses

During the President’s Cup, users submit support tickets through the Gameboard application. The support interface automatically captures the specific President’s Cup challenge, the user’s PlayerID, and a support code that helps the support team pinpoint the issue. When the Tier 1 team receives the ticket, they triage the situation either for resolution or elevation to Tier 2. Either way, the Tier 1 group communicates with the competitors that they’ve received their request and will keep them informed of progress toward resolution. It’s important to rapidly communicate with competitors and try to resolve most tickets within 15 minutes since the competitors only have a certain amount of time to take part in each round.

While stock answers to typical user questions can serve as a general starting point for support-team responses, it’s best to approach each support ticket individually so that users know their specific question is getting addressed. The goal is not to answer questions in a rote fashion but to respond to each competitor’s situation in a satisfactory way.

Weekend and after-hour responses present unique situations. If competitors can participate during times when live support won’t be staffed, the unavailability of support must be communicated clearly (users can access the President’s Cup site 24/7 to read the competition’s rules and FAQ section, however).

Adjudication Issues

Some issues are unusual enough (e.g., a competitor discovers an unexpected way to solve a challenge) or severe enough (e.g., an infrastructure outage causes a competition delay) to require immediate or post-round adjudication.

Occasionally a user’s support ticket reveals an unknown problem or infrastructure issue. If, after investigation, administrators determine that a problem with the challenge or other competition infrastructure was the cause, they may award extra time in the player's session or points for answers that the player found.

Extra time is awarded to a competitor when a problem with a challenge or competition infrastructure prevented the competitor from making progress on a challenge. The extra time is generally awarded according to how much time administrators believe the competitor lost due to the error.

Awarding points as part of an adjudication is rare. Points should only be awarded if administrators determine that players submitted an answer that should be considered correct but was graded as incorrect by the challenge. This can happen in infrequent circumstances when a challenge inadvertently has multiple correct answers that were not accounted for during challenge design, QA, and grading.

Support reporting area
Figure 1: Gameboard Support Field and Filter Reporting Options

The President’s Cup Gameboard reporting features provide helpful data to the support team. Support reports summarize information about the support tickets handled during the competition. They can be filtered for a specific round, a specific challenge and/or other parameters such as labels. Labels are tags added to individual tickets that allow the support team to easily identify, classify and search all tickets. Tickets can be labeled by round, issue (e.g., VM-outage), or any parameter the support team decides to use. Once tickets are tagged with labels, it’s easy to run reports. Reports allow the support team to highlight competition problem areas or issues that must be addressed before an ensuing round. Reports can also serve as a starting point for the planning of future competitions.

Six Lessons Learned in Supporting Cybersecurity Competitions

  • Understand Your Limits. Consider your plan for support when offering a cybersecurity competition. If 24/7 support will be offered, don’t promote that to potential competitors and assume you can fill the support schedule later. It’s harder than you think to secure staffing for every time slot, especially overnight. Keep in mind that if you follow a tiered-support strategy, at least two people must be scheduled for every shift. Make sure you have enough team members who possess the skills and availability to handle support assignments.
  • Analyze Data. Use your support site’s reporting features during and after a competition to look at data. With the President’s Cup, CISA and the SEI use Gameboard’s extensive, built-in reporting features to glean key information about competition challenges and logistics (such as increasing support staff during certain hours or realizing support isn’t needed as much as originally thought during overnight hours). Using reporting data can help determine a competition’s staffing needs.
  • Ensure a strong challenge-review process. A strong challenge testing-and-review process as highlighted in the Designing Great Challenges for Cybersecurity Competition blog post is integral to a successful competition. The goal here is to identify and fix any challenge issues before the competition even starts. Think of this process as offering support before support is even necessary. More challenge-testing before a competition results in
    • Fewer challenge-specific support tickets during the competition,
    • happier participants,
    • and a more satisfactory customer experience for the competition owner.

Another area where a competition site's reporting capabilities can provide useful information is challenge development. Investigate what challenges drew the most support tickets. Are there common threads to some of the problems highlighted in the tickets? For example, if Security Onion takes a long time to start when used in the challenge environment, it might be beneficial to future competitors to highlight that reality in the challenge documentation so that they know the challenges that utilize Security Onion are operating as expected.

  • Maintain an active backup staffing plan. Have backup plans in case someone on your support team is unable to handle their shift. Whether it’s a formal backup schedule or an on-call list, have a plan handy for when life interferes with your competition.
  • Have an easy communication method that your support team can use. In today’s work environment it’s not likely your support team will physically be in the same room during competitive rounds (especially after business hours and on weekends). Collaborative tools such as Mattermost and Microsoft Teams are ideal for allowing real-time communication among your team members. Video-communication platforms like Zoom are also useful for emergency situations that require impromptu meetings (such as a sudden problem with your competition’s cloud provider).
  • Keep a running support-team lessons learned list throughout the competition to help you evolve your support process for upcoming rounds and future competitions. Approach any comments or internal suggestions about your support methodology through the eyes of your competitors and customers. Keep the following questions in mind:
    • How can we improve our competition to better satisfy our customer’s needs?
    • How can we make our support process better for competitors?

It’s also a good idea to keep lessons learned tracking in mind not just for your support process but for all aspects of your cybersecurity competition.

Support Audiences – Who Benefits?

These support practices are the result of the SEI’s experience working with CISA's President’s Cup Cybersecurity Competition. Support strategy works in tandem with challenge development when planning a cybersecurity competition, so support-team considerations are not incidental to achieving the strategic goal of developing and strengthening America’s cybersecurity personnel. A support team that’s able to handle issues that arise during a competition while serving as concierge to competitors satisfies three audiences: competition participants, competition stakeholders, and those who want the United States to have a superior cybersecurity workforce.

Additional Resources

Learn more about the President’s Cup Cybersecurity Competition.

Read the SEI blog post Designing Great Challenges for Cybersecurity Competitions by Jarrett Booz, Josh Hammerstein, and Matt Kaar.

Read the SEI news story SEI Support for President’s Cup Leaves Lasting Legacy.

Watch a video of Matt Kaar, a cyber security exercise developer and trainer, discussing the President’s Cup Cybersecurity Competition during the 2020 SEI Research Review.

Get updates on our latest work.

Each week, our researchers write about the latest in software engineering, cybersecurity and artificial intelligence. Sign up to get the latest post sent to your inbox the day it's published.

Subscribe Get our RSS feed