Addressing cybersecurity for a complex system, especially for a cyber-physical system of systems (CPSoS), requires a strategic approach during the entire lifecycle of the system. Examples of CPSoS include rail transport systems, power plants, and integrated air-defense capability. All these systems consist of large physical, cyber-physical, and cyber-only subsystems with complex dynamics. In the first blog post in this series, I summarized 12 available threat-modeling methods (TMMs). In this post, I will identify criteria for choosing and evaluating a threat-modeling method (TMM) for a CPSoS.

A CPSoS is a system whose components operate and are managed independently. Its components must be able to function fully and independently even when the system of systems is disassembled. These components are typically acquired separately and integrated later, and they may have a physical, cyber, or mixed nature.

The definition of a CPSoS implies a diversity of potential threats that can compromise the integrity of the system, targeting different aspects ranging from purely cyber-related vulnerabilities to the safety of the system as a whole. The traditional approach used to identify threats is to employ one or more TMMs early in the development cycle. Choosing a TMM can be a challenging process by itself. The TMM you choose should be applicable to your system and to the needs of your organization.

CPSoS are connected through one or more cyber networks and run by one or more human operators. The components of those systems are often distributed and are sometimes partially autonomous, with multi-level control and management. Since CPSoSs are safety and/or life critical, threat modeling for these kinds of systems should address the full spectrum of threats: kinetic, physical, cyber-physical, cyber-only, supply chain, and insider threats.

The TMMs we summarized in our first blog post in this series are

1. STRIDE
2. The Process for Attack Simulation and Threat Analysis (PASTA)
3. LINDDUN (linkability, identifiability, nonrepudiation, detectability, disclosure of information, unawareness, noncompliance)
4. Common Vulnerability Scoring System (CVSS)
5. Attack Trees
6. Persona Non Grata (PnG)
7. Security Cards
8. Hybrid Threat Modeling Method (hTMM)
9. Quantitative Threat Modeling Method (Quantitative TMM)
10. Trike
11. Visual, Agile, and Simple Threat (VAST) Modeling
12. Operationally Critical Threat, Asset, and Vulnerability Evaluation (OCTAVE)

Evaluation Criteria

For more detailed information about evaluating TMMs for cyber-physical systems, I encourage you to read our SEI white paper on the same topic. In this post, I describe criteria for evaluation and make recommendations about adopting a TMM. The criteria are: strengths and weaknesses, adoptability, tailorability, application to cyber-physical systems, and automation.

Strengths and Weaknesses

Although a number of TMMs are used in the field, there is no one perfect method. Each method was developed from different points of view and addresses different priorities. Some are focused on assets, others on attackers, and still others on risks. Each method therefore has strengths and weaknesses relating to which types of threats it is best suited to discover, which threats it might miss (false negatives) or mistakenly identify (false positives), and how thorough it is. Each method has its own level of maturity and time to implement. Each deals with its own mitigation strategies. These aspects determine how useful a method may be for a given situation.

All the methods in question are designed to detect potential threats, except for CVSS (which is a scoring method). The number and types of threats will vary considerably, as will the quality and consistency of the methods. Since there is no comprehensive study involving all of these methods, we can only speculate about how effective and efficient they are, based on published studies of their use.

Here is a table that compares the strengths and weaknesses of the 12 TMMs:

Adoptability

Implementing a comprehensive methodology in an organization imposes some burden on everyone involved, so choosing an easy-to-use solution is important. When such an implementation is added to a learning curve for the implementers of the methodology and associated changes to already existing processes, the overall cost of adopting the method may become prohibitive. Availability or absence of comprehensive and useful documentation and support can be critical for successfully adopting the method.

The importance of the adoptability of a method cannot be overstated. There are few, if any, easy TMMs. The successful implementation of a TMM requires a deep understanding of the system and extensive knowledge of cybersecurity. However, the intuitiveness of the method can ease the effort needed to learn and use it. The adoption process can be streamlined if the method employs techniques (such as architecture diagrams or brainstorming) that are already well understood and used in the field.

Table 2 summarizes the evaluation of the main attributes that contribute to the adoptability of the method.

Tailorability

No two organizations have identical development processes, no matter how similar they may be otherwise. A good TMM candidate should therefore be flexible so it can be tailored to the type of system, the organization's priorities, and the system development lifecycle (SDLC) without compromising the quality of the method. This flexibility should include whether the candidate method can be integrated into a development process, and specifically into the increasingly prevalent Agile development process. Methods intended for use on cyber-physical systems (CPS) must be scalable and able to meet the needs of large and often distributed systems.

All methods except OCTAVE are designed for application in the beginning of the SDLC, during the requirements and design phases, and can be integrated into any development lifecycle that includes these phases. Some methods (such as PnG, Trike, and VAST) integrate with the Agile development process better than others.

Since none of these methods were designed for application to a specific type of system, all may be applied to any kind of system. OCTAVE was initially designed for large organizations; PASTA and VAST modeling were designed for large systems. The remainder of the methods can be scaled to accommodate large systems or systems of systems relatively easily.

Applicability to CPS

One of the main characteristics of CPS is complexity. Applying the TMMs recursively is critical. This approach helps account for the relationships between subsystems, address hardware-software dependencies, and address safety-security interdependencies.

Studies of railway communication networks, drone systems, and the automotive industry for connected cars indicate that combinations of two or more TMMs seem to perform better. Combining methods and adding domain-specific techniques allows for deeper analysis of the system, and thus, better threat discovery.

CPS require focused attention not only on the application or system-software-related threats, but also on hardware and physical threats. Malware installed on a hardware component or physical tampering with a component can cause cyber or cyber-physical impact and put a system into an undesirable state.

Automation

It is important to ask the following questions when considering a TMM:

• Does any automation of the method or supporting tools exist?
• What is its utility?

Many cyber-physical systems belong to publicly or privately owned critical infrastructure or to government-developed weapon systems. To an organization that requires a higher level of security, portability of the tools to a standalone mode is another important feature.

Few of the TMMs examined were automated. In fact, most of them exist only as a framework of instructions, questionnaires, and checklists. Table 3 compares our 12 TMMs with respect to automation and portability.

Use PASTA as Basis of Framework

Evaluation of existing TMMs showed that no one method can cover all pieces. Therefore, a framework that employs a combination of methods and techniques should be used. Our recommendation is to apply the PASTA modeling method as the basis of this framework.

PASTA provides the most detailed guidance for the process of threat modeling, including resources that can be easily adapted to different kinds of systems. It can be incorporated into the existing SDLC and allows for easy addition or removal of activities from stages as needed. PASTA also mitigates the threat-explosion weakness of STRIDE and LINDDUN by utilizing risk and impact analysis. This flexibility makes this combination a foundation for a comprehensive framework.

Some modification should be done to this combination of methods to accommodate the scope of the problem. To start, PASTA should be implemented for the whole system using high-level architecture and treating subsystems as black boxes. This initial round will not require the user to go through every activity, but it should effectively define all inputs and outputs for each subsystem.

PASTA should then be implemented recursively for each subsystem--and, in turn, each subsystem of the subsystems. All discoveries from a higher level should be passed to the next level as an input. Expect to encounter quite a few levels of subsystems, depending on the complexity of the system.

In addition to the base PASTA stages, we provide in our white paper a list of activities that should be added to address the full spectrum of threats. In addition to PASTA, we recommend using components of STRIDE and LINDDUN. We also recommend using other tactics that address threat aspects not covered by these three models, such as

• identify system-critical dependencies from the supply chain, including dependencies from trusted third-party systems
• identify physical boundaries (direct and indirect access) to the system's components
• apply internal threat-identification methods
• generate attack ports

Recommendations for Adopting Threat-Modeling Methods

The following recommendations will help with the process of adopting a TMM:

• Threat modeling works best if applied in early stages of the project, i.e., the requirements and design phase.
• Threat modelling is an ongoing process. It is hard to perfect it on the first run, and you cannot refine it indefinitely. You need milestones along the way. It does not stop after your system is delivered. Some steps must be repeated when the system changes.
• In threat modeling, it is dangerous to concentrate exclusively on threats. Modeling users and attackers, and controlling impact on requirements and mitigations, are just as important.
• Threat modeling is not an innate skill. It is learnable and improves with practice. With each iteration, it become better and deeper.

We believe that combining components of PASTA, STRIDE, and LINDDUN with tactics that address additional aspects of CPSoS will provide better coverage of threats than any one model by itself. Adoption of the proposed framework will be a laborious and time-consuming process, but will allow for the creation of a flexible and comprehensive structure for modeling a wide range of threats.

Additional Resources

Read the SEI White Paper, Threat Modeling for Cyber-Physical System-of-Systems: Methods Evaluation, on which this blog post is based.

Read the first blog post in this series, Threat Modeling: 12 Available Methods

Read the SEI White Paper, Threat Modeling: A Summary of Available Methods.

Read the SEI Technical Note, A Hybrid Threat Modeling Method by Nancy Mead and colleagues.

Read Evaluation of Threat Modeling Methodologies by Forrest Shull.

Read the SEI blog post The Hybrid Threat Modeling Method by Nancy Mead and Forrest Shull.

Browse SEI OCTAVE-related information.

Read an SEI Technical Report about Security Quality Requirements Engineering (SQUARE).