Security Operations (SecOps) team members within the SEI’s CERT Division travel frequently to work with international organizations, national Computer Security Incident Response Teams (CSIRTs), and security operations centers (SOCs) with the goal of building capacity, capability and sharing information. In 2020, this all changed with the onset of the COVID-19 global pandemic. As countries and organizations implemented measures to curb the spread of the virus that causes COVID-19, the SecOps team also had to pivot in operational posture. Obvious choices in how to conduct engagements include that of remote customer engagements and training workshops. However, virtual engagements have been unfit or impossible in some cases, specifically where networks are siloed and classified data must remain stationary. We chronicle one such case, where members of the SecOps team travelled abroad on multiple occasions to assess and build a security operations center for a foreign military partner in the CENTCOM area of responsibility work, which is part of SecOps support of DoD Program Executive Office (PEO) PMW 740. This blog post provides insight into the SecOps SOC assessment process and highlights challenges our team faced while conducting an international cybersecurity assessment amidst travel bans during the COVID-19 global pandemic.

The Assessment Processes

Having a sound process to assess and act upon is a key component of establishing or maturing a SOC team. The primary focus of projects such as this is to understand and develop the people, process, and technology aspects of SOC implementations. Other factors can also impact the success of a SOC team implementation and may only arise when an assessment team arrives on location.

For example, physical factors, such as determining where the SOC personnel will be located, may require an assessment team to design a physical space for the SOC to operate in. Soft skills, such as understanding the personalities of all project stakeholders, may require the assessment team to adapt their approach to communications about the assessment. In addition, the assessment team will need to be ready to ask important questions to verify baseline capabilities, organizational security controls, and any available tools or documentation required to help the SOC mature.

The assessment process utilized during this project consists of four basic phases: scoping the assessment, conducting the assessment, analyzing the outcomes, and acting on those outcomes. Each of these phases helps establish milestones and highlights achievements throughout the project lifecycle, which often requires flexibility and transparency for assessment activities.

January 2021—Scoping the Assessment

One of the most important aspects of any assessment is to determine the boundaries of operation. The scope typically is established when the project is contracted, which is no different from the project assigned to the SecOps team. However, limitations on travel during the pandemic prevented the team from understanding the full scope of need from customers for these types of assessments.

Remote effort did prove fruitful for some of the soft requirements, such as stakeholder introductions, but technical details and confidential policy information simply could not be obtained or shared outside of the isolated bounds of the customer network. As a critical requirement of these projects, our team needs to understand the network environment and policy. When working with international customers, confidentiality often prevents specific details from being shared outside of in-person exchanges. Therefore, while summary information can be obtained remotely, specific details such as IP address, ports, and services cannot.

In one specific instance, our team wrote and delivered a program to generate a network map containing vital technical details. Without remote access to the isolated customer resources, SecOps team members created a lab environment to mimic the customer network to evaluate the program. The results of the tests were then used to document the impact of the program and provide precise instruction to the customer.

At the request of the customer, the team was cleared to travel on-site to the CENTCOM AOR to conduct critical on-site activities. However, traveling during a pandemic proved to be hard. Fluctuating travel requirements, COVID infection rates, and even U. S. Department of State warnings all presented unique challenges to the trip. Some challenges were easier to address than others, and the team often found that relying on contingency travel plans and setting appropriate expectations resolved most of the challenges.

During one specific trip, team members were required to register with a mobile phone app for contract tracing and infection status. Upon arrival, the team found that registering the app was only possible with a non-U.S. phone carrier. Further complicating the matter, the mobile app had to be shown to authorities at all public venues, including hotels and airports, which required the team to locate a local phone carrier to obtain compatible devices and convince officials that their app was non-functional before entering the carrier location. Despite the set-back, the team was able to successfully register their mobile devices to conduct meetings with the customer, tour facilities, and review policy documentation to clearly identify the scope of the assessment. All the above activities were socially distanced, masked, and contact traced as required at the time.

Information from the scoping engagement enabled the team to return home and begin work on formulating further assessment plans and even begin building some artifacts to be used to establish the SOC. Most importantly, the parameters within which the assessment was to be conducted had been defined, and our team began to fully understand the customer’s cybersecurity challenges and identify which of those would hold priority when defining the capabilities of the SOC.

August 2021 —Conducting the Assessment

Conducting formal assessments, when building either SOCs or incident response teams, commonly rests upon three pillars: people, processes, and technology. The intersection of these pillars allows a team to function as a cohesive unit with applicable knowledge and skill, create policies that back SOC initiatives, and maintain available technology to complete mission objectives. Frameworks such as the SEI’s Sector CSIRT Framework and OpenCSIRT Foundation’s SIM3 model outline the standards by which capability is measured and allow assessments to be quantified for later improvement.

Each of these pillars falls into the scope of SecOps assessments. The process pillar is straightforward and aims to determine whether the organization has policies in place for factors such as security operations, security controls, and risk assessment. The policy also aims to assess whether the organization can identify the proper scope of what the SOC will protect and how to protect it.

Technology complements the policy aspect of a SOC. Operational scope depends on available technology for the SOC, along with the scope of technology that the SOC must protect. Technical factors, such as number of assets, protocols, ports, and network segmentation, all go into building requirements for any security tools to be purchased and implemented.

Finally, without people, there is no one to leverage applicable technology to protect and defend the network according to the policies. People and their roles are the final link tying the two components together. It is therefore important to have a properly identified scope of defense within an environment to identify how many people are needed and what each person’s responsibility will be.

Following the January 2021 scoping engagement, the SecOps team was able to make offsite progress by providing templates and drafts for missing policies discovered while on location. While the drafts required customization, this effort allowed the team to make progress without being on location. Moreover, the team obtained appropriate scoping information for networks and assets, which also allowed them to formulate required roles and responsibilities for the SOC. In preparation for the next visit, the team built training modules for critical capabilities that SOC personnel would conduct and plotted a course of action for finalizing policy.

In August 2021, the team returned to the customer site armed with training materials and a full assessment plan. While the visit was initially slated to focus largely on training, once on site the SEI team found that no SOC personnel had been selected to staff the newly formed roles. Given the challenges of traveling during a pandemic and the absence of on-site SOC personnel, SecOps team members reevaluated their objectives and pivoted to focus on technology and policy.

With a plan of action formed, the team began requesting and reviewing policy documentation and forming interview questions for the assessment. In parallel, the team was also able to aggregate the output of network scans that had also recently been conducted, providing key technical data for the assessment. When the two-weeklong engagement had ended, the team had enough information to begin analyzing the assessment findings and producing results.

January 2022 – Analyzing Assessment Results and Acting

During the August 2021 visit the SecOps assessment team was able collect enough information to build out requirements for people, policy, and technology within the SOC. These requirements are then used to define goals and identify solutions needed to achieve the mission. The requirements can be boiled down into several distinct categories to ensure consistent outcomes: procedural, functional, technical, output, and miscellaneous.

With the assessment specifics and requirements obtained from the August 2021 visit, it was time for the SecOps team to aggregate their findings and provide a path forward for the organization to begin building the SOC. With the policy templates already established, the team focused on aiding the customers in drafting their own version of policy documentation and have it presented to senior leadership in the organization.

One challenge the team faced is that tool design, implementation planning, and staff training all needed to be conducted on-site. Slated to return on-site in early 2022, the team only had a few short months to plan software implementation for multiple tools and sensors and develop a training workshop for the SOC staff. Prior to the trip the team worked to develop recommendations for sensor placements on the customer network and formalize the requirements that would eventually turn into a request for purchase (RFP) for the customer to procure goods and services. Moreover, the team also produced training modules for both the customer’s SOC and network operation center (NOC) teams with the help of the CERT Cyber Workforce Development (CWD) team.

Back on location again in January 2022, the team had two weeks to conduct two separate training workshops, one for network fundamentals and the other for security essentials. Topics we presented spanned network basics to advanced security topics such as penetration testing. Another challenge we faced is that these topics use technical language that is often hard to translate. Under normal circumstances the SecOps team would leverage the aide of translators, however time constraints and travel restrictions for the project did not allow for this option. Therefore the team had to continuously adapt the training curriculum to suit the cultural variances and language barriers. Experience has shown that engaging bilingual training participants and prompting them for assistance throughout the course will normally aide in course execution. In our case, we were fortunate to have several individuals who assisted with explaining complex topics.

In parallel, other members of the SecOps team discussed the selection, implementation, and architecture of security solutions with the organization’s senior leadership. This vital endeavor laid the groundwork for the team and senior leadership to construct the RFP and begin to select critical cybersecurity tools and sensors for the SOC to use. By the end of the two-week engagement, the team had prepped the staff with technical basics to operate the SOC and provided them with the initial components produce evaluate tools and begin to form playbooks.

Although the work had completed, the team was faced again with another challenge. This time, they needed to find an appropriate COVID-19 testing center within 24 hours required to make their 2:00 AM flight back to the U.S. Thinking ahead, team members decided to book an on-site test to occur the afternoon of departure at the hotel, allowing ample time before leaving for the airport. However, at test time, the testing center nurse never showed up to the hotel. Despite calls to the testing center, no tester would be available to come to the hotel to conduct the test and have results available in time for departure. Recalling prior trips to the country, the team booked appointments at two more testing centers, with an optional third test an hour away. When the first testing center opened at 7:00 PM local time, the team members were able to get tested and anxiously awaited results. With only a few hours to spare before takeoff, the team received their negative test results and were able to depart to the airport for their return home.

Lessons Learned

Work continues on the development of the SOC for the DoD’s foreign partner. Additional travel is expected, but with each in-person engagement our SecOps team has realized several lessons. The first and most important takeaway from these engagements has been to always plan for contingencies. Whether for travel or customer deliverables, appropriate backup plans are a critical component of international engagements. If your team cannot continuously travel to a specific region, design tasks and duties to be completed by the customer to help meet the project objectives.

The second lesson is to always remain flexible with planning. On many occasions, cultural differences may dictate different working hours, meeting participants, or even location. Plan accordingly. If you are unable to conduct a training workshop for eight-hour days, adjust your material to accommodate the schedule, and respect the host’s requirements.

The last lesson is to properly manage expectations. This lesson applies to customers as well as fellow team members. While this lesson is obvious when establishing communication channels during customer engagements, the challenges of travel and delivery of objectives make setting expectations even more important. Clearly defining and communicating scope and project boundaries ensures that all stakeholders of the project are properly informed and can make concise decisions when needed.