search menu icon-carat-right cmu-wordmark

Get the Latest Research in Software Engineering, Cybersecurity,
and AI Engineering

Get email notifications of new blog posts from the SEI Blog.

SEI Insigts

Recent Posts

Cybersecurity Maturity Model Certification (CMMC) Part 2: Process Maturity's Role in Cybersecurity

Cybersecurity Maturity Model Certification (CMMC) Part 2: Process Maturity's Role in Cybersecurity

• SEI Blog
Andrew Hoover

Process maturity represents an organization's ability to institutionalize their practices. Measuring process maturity determines how well practices are ingrained in the way work is defined, executed, and managed. Process maturity represents an organization's commitment to and consistency in performing these practices. A higher degree of process institutionalization contributes to more stable practices that are able to be retained during times of stress. In the case of cybersecurity, having mature cybersecurity processes will improve an organization's ability to prevent and respond to a cyberattack. In the first blog post in this series, we introduced the Cybersecurity Maturity Model Certification, or CMMC....

Read More
The Latest Work from the SEI: DevSecOps, Artificial Intelligence, and Cybersecurity Maturity Model Certification

The Latest Work from the SEI: DevSecOps, Artificial Intelligence, and Cybersecurity Maturity Model Certification

• SEI Blog
Douglas C. Schmidt

As part of an ongoing effort to keep you informed about our latest work, this blog post summarizes some recently published SEI reports, podcasts, conference papers, and webcasts highlighting our work in DevSecOps, cybercrime and secure elections, software architecture, trustworthy artificial intelligence, and Cybersecurity Maturity Model Certification (CMMC). We have also included a webcast of a recent discussion on Department of Defense (DoD) software advances and future SEI work. These publications highlight the latest work of SEI technologists in these areas. This post includes a listing of each publication, author(s), and links where they can be accessed on the SEI...

Read More
Three Risks in Building Machine Learning Systems

Three Risks in Building Machine Learning Systems

• SEI Blog
Benjamin Cohen

Machine learning (ML) systems promise disruptive capabilities in multiple industries. Building ML systems can be complicated and challenging, however, especially since best practices in the nascent field of AI engineering are still coalescing. Consequently, a surprising fraction of ML projects fail or underwhelm. Behind the hype, there are three essential risks to analyze when building an ML system: 1) poor problem solution alignment, 2) excessive time or monetary cost, and 3) unexpected behavior once deployed. In this post I'll discuss each risk and provide a way of thinking about risk analysis in ML systems....

Read More
Situational Awareness for Cyber Security Architecture: Tools for Monitoring and Response

Situational Awareness for Cyber Security Architecture: Tools for Monitoring and Response

• SEI Blog
Tim Shimeall

Visibility into the activities within assets enables network security analysts to detect network compromises. Analysts monitor these activities directly on the device by means of endpoint visibility and in the communications going to and from the device on the network. In our earlier blog posts on cyber situational awareness (SA) for the enterprise, we discussed endpoint visibility and network visibility. However, endpoint and network visibility will do little good if analysts don't have tools to help them analyze the collected data, respond to any identified issues, and document the analysis, the issues, and the response. In this blog post, we...

Read More
Stop Wasting Time: Manage Time as the Limiting Resource

Stop Wasting Time: Manage Time as the Limiting Resource

• SEI Blog
Bill Nichols

Lost time is never found. - Ben Franklin Driven by a competitive marketplace, software developers and programmers are often pressured to adhere to unrealistically aggressive schedules across multiple projects. This pressure encourages management to spread the staff across all the critical work, trying to make progress everywhere at once. This trend helped to spawn the myth of the "x10 programmers"--programmers who are so much more productive than others that they will exert an outsized influence on their organizations' success or failure. Such magical thinking ignores the real problems of productivity. In the first blog post in this series, I challenged...

Read More
System Resilience Part 7: 16 Guiding Principles for System Resilience

System Resilience Part 7: 16 Guiding Principles for System Resilience

• SEI Blog
Donald Firesmith

Adverse events and conditions can disrupt a system, causing it to fail to provide essential capabilities and services. As I outlined in previous posts in this series, resilience is an essential quality attribute of most systems because they provide critical capabilities and services that must continue despite the inevitable adversities. These adversities are often unavoidable and come in many forms. Typical examples include coding defects (robustness), hazards and acccidents (safety), vulnerabilities and attacks (cybersecurity and suvivability), excessive loads (capacity), long lifespans (longevity), and lost communication (interoperability). In the first post in this series, I defined system resilience as the degree...

Read More
System Resilience Part 6: Verification and Validation

System Resilience Part 6: Verification and Validation

• SEI Blog
Donald Firesmith

Adverse events and conditions can disrupt a system, causing it to fail to provide essential capabilities and services. As I outlined in previous posts in this series, resilience is an essential quality attribute of most systems because they provide critical capabilities and services that must continue despite the inevitable adversities. In the first post in this series, I defined system resilience as the degree to which a system rapidly and effectively protects its critical capabilities from harm caused by adverse events and conditions. The second post identified the following eight subordinate quality attributes that categorize the adversities that can disrupt...

Read More
Reviewing Formalized DevOps Assessment Findings and Crafting Recommendations: Sixth in a Series

Reviewing Formalized DevOps Assessment Findings and Crafting Recommendations: Sixth in a Series

• DevOps Blog
Jose Morales

Reviewing DevOps assessment findings and formalizing them into a final list is critical to precisely identifying obstacles to the client. Drafting the appropriate recommendation is key to improving the organization's software development capabilities. This blog post series, based on a paper by me and my colleagues Hasan Yasar and Aaron Volkmann, discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the software development lifecycle (SDLC) within highly regulated environments (HREs)....

Read More
Automatically Detecting Technical Debt Discussions with Machine Learning

Automatically Detecting Technical Debt Discussions with Machine Learning

• SEI Blog
Robert Nord

Technical debt (TD) refers to choices made during software development that achieve short-term goals at the expense of long-term quality. Since developers use issue trackers to coordinate task priorities, issue trackers are a natural focal point for discussing TD. In addition, software developers use preset issue types, such as feature, bug, and vulnerability, to differentiate the nature of the task at hand. We have recently started seeing developers explicitly use the phrase "technical debt" or similar terms such as "design debt" or "architectural smells." Although developers often informally discuss TD, the concept has not yet crystalized into a consistently applied...

Read More
7 Quick Steps to Using Containers Securely

7 Quick Steps to Using Containers Securely

• SEI Blog
Thomas Scanlon

Richard Laughlin coauthored this blog post. The use of containers in software development and deployment continues to trend upwards. There is good reason for this climb in usage as containers offer many benefits, such as being lightweight, modular, portable, and scalable, all while enabling rapid and flexible deployments with application isolation. However, as use of this technology increases, so does the likelihood that adversaries will target it as a means to compromise systems. Such concerns are amplified in organizations where technical staff are implementing containers on-the-fly (i.e., deploying containers while simultaneously still learning about them). To help software developers deploy...

Read More
An Introduction to the Cybersecurity Maturity Model Certification (CMMC)

An Introduction to the Cybersecurity Maturity Model Certification (CMMC)

• SEI Blog
Katie C. Stewart

Andrew Hoover co-authored this blog post. A recent study predicted that business losses due to cybercrime will exceed $5 trillion by 2024. The threat to the Defense Industrial Base (DIB)--the network of more than 300,000 businesses, organizations, and universities that research, engineer, develop, acquire, design, produce, deliver, sustain, and operate military weapons systems--is especially alarming due to current cyber warfare activities by cybercriminals and state-sponsored actors. A cyber attack within the DIB supply chain could result in devastating losses of intellectual property and controlled unclassified information (CUI). To bolster cybersecurity posture within the DIB supply chain, SEI researchers have spent...

Read More
Snake Ransomware Analysis Updates

Snake Ransomware Analysis Updates

• CERT/CC Blog
Kyle O'Meara

In January 2020, Sentinel Labs published two reports on Snake (also known as Ekans) ransomware.[1][2] The Snake ransomware gained attention due to its ability to terminate specific industrial control system (ICS) processes. After reading the reports, I wanted to expand the corpus of knowledge and provide OT and IT network defenders with increased defense capabilities against Snake. The key takeaways from the Sentinel Labs’ reports for additional analysis were the hash of the ransomware and the string decoder script from sysopfb.[3] Two questions I pursued were: Can I find more samples of the Snake ransomware? If yes, do these samples...

Read More
Situational Awareness for Cybersecurity Architecture: Network Visibility

Situational Awareness for Cybersecurity Architecture: Network Visibility

• SEI Blog
Timur Snoke

Network compromises cannot be detected without visibility into the activities within assets. Network security analysts can view these activities in one of two places (or sometimes both): directly on the device by means of endpoint visibility and in the communications going to and from the device; in other words, on the network. In our earlier blog post on cyber situational awareness (SA) for the enterprise, we discussed endpoint visibility. In this post, we turn our attention to the other component required in monitoring and defending against network compromises, network visibility....

Read More
Bridging the Gap Between Research and Practice

Bridging the Gap Between Research and Practice

• CERT/CC Blog
Leigh Metcalf

A fundamental goal for a federally funded research and development center (FFRDC) is to bridge the gap between research and practice for government customers. At the CERT Division of the Software Engineering Institute (SEI), we've taken a step beyond that and decided that, in cybersecurity, we should be bridging the gap for all researchers and practitioners. To help achieve this goal, I decided that a journal would be an important step. The Association for Computing Machinery (ACM) agreed that that was a worthy goal and helped me create ACM Digital Threats: Research and Practice (DTRAP) with my co-Editor-in-Chief, Arun Lakhotia...

Read More
Functional Requirements for Insider Threat Tool Testing

Functional Requirements for Insider Threat Tool Testing

• Insider Threat Blog
Robert M. Ditmore

Derrick Spooner co-authored this post. Because of the scope and scale of the insider threat, the SEI recommends that organizations adopt a use-case-based approach to insider risk mitigation. In such an approach, organizations iteratively deploy capabilities to prevent, detect, and respond to the greatest threats to their most critical assets. However, the tools modern insider threat programs rely on to collect and analyze data do not adapt themselves to the organization or its changing insider threat landscape. A sound testing environment for insider threat tools allows an organization to quickly, responsively, and effectively deploy various capabilities. This blog post presents...

Read More
Using Machine Learning to Detect Design Patterns

Using Machine Learning to Detect Design Patterns

• SEI Blog
Robert Nord

This post was co-written by Zachary Kurtz. Software increasingly serves core DoD functions, such as ship and plane navigation, supply logistics, and real-time situational awareness. The complexity of software, however, makes it hard to evaluate software quality. The ability to evaluate software is critical both for software developers and for DoD program managers who are responsible for software acquisitions. The quality of software can make or break a program budget. Quality attributes such as reliability, security, and modifiability are just as important as making sure the software computes the right answer. Any major design approach chosen by the developer will...

Read More
Security Automation Begins at the Source Code

Security Automation Begins at the Source Code

• CERT/CC Blog
Vijay Sarvepalli

Hi, this is Vijay Sarvepalli, Information Security Architect in the CERT Division. On what seemed like a normal day at our vulnerability coordination center, one of my colleagues asked me to look into a vulnerability report for pppd, an open source protocol. At first glance, this vulnerability had the potential to affect multiple vendors throughout the world. These widespread coordination cases usually have a prolonged coordination timeline. They typically involve multiple vendors on the one end and a security researcher (or "Finder" in the language of the CERT Guide to Coordinated Vulnerability Disclosure) on the other end, each with competing...

Read More
Five Reasons the Cybersecurity Field Needs Trusted Data Sets and Meaningful Metrics

Five Reasons the Cybersecurity Field Needs Trusted Data Sets and Meaningful Metrics

• SEI Blog
Bobbie Stempfley

Matthew Butkovic co-authored this blog post. Cybersecurity is a domain rich with data, but regrettably often only poor insights can be drawn from this richness. CISOs ask questions about how best to allocate resources to address threats, practitioners ask questions about how to measure the effectiveness of one solution over another, senior organizational leaders strive to identify and quantify organizational risks, and public officials work to inform organizational or national policy. Answers often involve anecdotes, small exemplars often generalized beyond their intended use, or weakly coupled analogies. Progress is hard to track at any level, which often impedes the willingness...

Read More
Designing Trustworthy AI for Human-Machine Teaming

Designing Trustworthy AI for Human-Machine Teaming

• SEI Blog
Carol Smith

Artificially intelligent (AI) systems hold great promise to empower us with knowledge and enhance human effectiveness. As Department of Defense (DoD) warfighters partner with AI systems more frequently, we will identify more opportunities to clarify the limits of AI and to set realistic expectations for these types of systems. As a senior research scientist in human-machine interaction at the SEI's Emerging Technology Center, I am working to further understanding of how humans and machines can better collaborate (i.e., team) to solve important problems and also understand our responsibilities and how that work continues once AI systems are operational. This blog...

Read More
Summarizing and Searching Video with Machine Learning

Summarizing and Searching Video with Machine Learning

• SEI Blog
Edwin Morris

The U.S. relies on surveillance video to determine when activities of interest occur in a location that is under surveillance. Yet, because automated tools are not available to help analysts monitor real-time video or analyze archived video, analysts must dedicate full attention to video data streams to avoid missing important information about ongoing activities and patterns of life. In tactical settings, warfighters miss critical information that would improve situational awareness because dedicating full attention to video streams is not feasible; for example, they may miss a telltale muzzle flash that gives away the location of an adversary while scanning the...

Read More
Automated Code Repair to Ensure Memory Safety

Automated Code Repair to Ensure Memory Safety

• SEI Blog
Will Klieber

Memory-safety vulnerabilities are among the most common and most severe types of software vulnerabilities. In early 2019, a memory vulnerability in the iPhone iOS, reportedly exploited by the Chinese government, allowed attackers to take control of a phone when the user visited a malicious website. A similar vulnerability discovered in the Android Stagefright library allowed an attacker to gain control simply by sending a Multimedia Messaging Service (MMS) message to a vulnerable phone. For each of the past three years, the Common Weakness Enumeration (CWE) category for spatial memory violations (CWE-119) have been the most or second most frequent type...

Read More
Formalizing DevOps Assessment Findings and Crafting Recommendations: Fifth in a Series

Formalizing DevOps Assessment Findings and Crafting Recommendations: Fifth in a Series

• DevOps Blog
Jose Morales

Reviewing DevOps assessment findings and formalizing them into a final list is critical to precisely identifying obstacles to the client. Drafting the appropriate recommendation is key to improving the organization's software development. We will dicuss both topics in this blog post. This blog post series, based on a paper by me and my colleagues Hasan Yasar and Aaron Volkmann, discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the software development lifecycle (SDLC) within highly regulated environments (HREs). The first post explored challenges and goals of implementing DevOps in HREs. The second post discussed the first step...

Read More
System Resilience Part 5: Commonly-Used System Resilience Techniques

System Resilience Part 5: Commonly-Used System Resilience Techniques

• SEI Blog
Donald Firesmith

If adverse events or conditions cause a system to fail to operate appropriately, they can cause all manner of harm to valuable assets. As I outlined in previous posts in this series, system resilience is important because no one wants a brittle system that cannot overcome the inevitable adversities. In the first post in this series, I addressed these questions by providing the following, more detailed, and nuanced definition of system resilience: A system is resilient to the degree to which it rapidly and effectively protects its critical capabilities from harm caused by adverse events and conditions. The second post...

Read More
Comments on NIST IR 8269: A Taxonomy and Terminology of Adversarial Machine Learning

Comments on NIST IR 8269: A Taxonomy and Terminology of Adversarial Machine Learning

• CERT/CC Blog
Jonathan Spring

The U.S. National Institute of Standards and Technology (NIST) recently held a public comment period on their draft report on proposed taxonomy and terminology of Adversarial Machine Learning (AML). AML sits at the intersection of many specialties of the SEI. Resilient engineering of Machine Learning (ML) systems requires good data science, good software engineering, and good cybersecurity. Our colleagues have suggested 11 foundational practices of AI engineering. In applications of ML to cybersecurity, we have suggested seven questions decision-makers should ask. A solid understanding of AML is a key element for decision makers in both situations. NIST IR 8269 is...

Read More
Engineering for Cyber Situational Awareness: Endpoint Visibility

Engineering for Cyber Situational Awareness: Endpoint Visibility

• SEI Blog
Phil Groce

This post was co-written by Timur Snoke. In this post, we aim to help network security analysts understand the components of a cybersecurity architecture, starting with how we can use endpoint information to enhance our cyber situational awareness. Endpoints collect a wealth of information valuable for situational awareness, but too often this information goes underutilized....

Read More
System Resilience Part 4: Classifying System Resilience Techniques

System Resilience Part 4: Classifying System Resilience Techniques

• SEI Blog
Donald Firesmith

A system resilience technique is any architectural, design, or implementation technique that increases a system's resilience. These techniques (e.g., mitigations, such as redundancy, safeguards, and cybersecurity countermeasures) either passively resist adversities, actively detect adversities, react to them, or recover from the harm they cause. System resilience techniques are the means by which a system implements its resilience requirements. Resilience techniques can also be viewed as architecture, design, or implementation patterns or idioms. This post begins by clarifying the relationships between resilience requirements and resilience techniques. Because system-, software, and specialty engineers have many techniques that can be used to increase...

Read More
Maturing Your Insider Threat Program into an Insider Risk Management Program

Maturing Your Insider Threat Program into an Insider Risk Management Program

• Insider Threat Blog
Daniel Costa

Having trouble clearly stating the scope of your insider threat program? Struggling with measuring the program's effectiveness? Failing to provide actionable intelligence to the program stakeholders? Lacking consensus regarding your organization's current security posture against insider threats? These are signs that your insider threat program may not be properly integrated with a risk management program within your organization. In this blog post, we will discuss the benefits of grounding insider threat program operations in the principles of risk management, and we will present strategies for successfully maturing your insider threat program into an insider risk management program....

Read More
Programmer Moneyball: Challenging the Myth of Individual Programmer Productivity

Programmer Moneyball: Challenging the Myth of Individual Programmer Productivity

• SEI Blog
Bill Nichols

A pervasive belief in the field of software engineering is that some programmers are much, much better than others (the times-10, or x10, programmer), and that the skills, abilities, and talents of these programmers exert an outsized influence on that organization's success or failure. This topic is the subject of my recent column in IEEE Software, The End to the Myth of Individual Programmer Productivity....

Read More
Performing the DevOps Assessment: Fourth in a Series

Performing the DevOps Assessment: Fourth in a Series

• DevOps Blog
Jose Morales

The overall purpose of a DevOps assessment is to help improve the software development lifecycle (SDLC). Applying DevOps in highly regulated environments (HREs), be they academic, government, or industrial, can be challenging. HREs are mandated by policies for various reasons, most often general security and protection of intellectual property. The restrictions of these policies make the sharing and open access principles of DevOps that much harder to apply. This blog post series, based on a paper by me and my colleagues Hasan Yasar and Aaron Volkmann, discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the SDLC...

Read More
Anti-Phishing Training: Is It Working? Is It Worth It?

Anti-Phishing Training: Is It Working? Is It Worth It?

• Insider Threat Blog
Mike Petock

Phishing attacks target human, rather than technical, vulnerabilities. Some organizations, companies, government agencies, educational institutions, and individuals put on blinders and hope it doesn't happen to them. Others try to prevent the problem by paying for anti-phishing training. Speaking from a cybersecurity trainer's perspective, good training should change user behavior and reduce the primary problem: in this case, an incident or breach initiated by a successful phishing attack. Even for effective training, the cost should be significantly lower than the cost of cleaning up after a breach. So, does anti-phishing training work? Is it worth the effort? The answers can...

Read More
The Latest Work from the SEI: Penetration Testing, Artificial Intelligence, and Incident Management

The Latest Work from the SEI: Penetration Testing, Artificial Intelligence, and Incident Management

• SEI Blog
Douglas C. Schmidt

As part of an ongoing effort to keep you informed about our latest work, this blog post summarizes some recently published SEI reports, podcasts, conference papers, and webcasts highlighting our work in penetration testing, designing trustworthy AI, fielding AI-enabled systems in the public sector, incident management, machine learning in cybersecurity, and cyber hygiene. These publications highlight the latest work of SEI technologists in these areas. This post includes a listing of each publication, author(s), and links where they can be accessed on the SEI website....

Read More
System Resilience Part 3: Engineering System Resilience Requirements

System Resilience Part 3: Engineering System Resilience Requirements

• SEI Blog
Donald Firesmith

At its most basic level, system resilience is the degree to which a system continues to perform its mission in the face of adversity. While critical to operational continuity, the system's services (capabilities) are only some of the assets the system must protect to continue to perform its mission. The system must detect adversities, react to them, and recover from the harm to critical assets that they cause. System resilience at a deeper level is therefore the degree to which a system rapidly and effectively protects itself and its continuity-related assets from harm caused by adverse events and conditions....

Read More
Technology Trends in Data Exfiltration

Technology Trends in Data Exfiltration

• Insider Threat Blog
Alex Pickering

One of our goals at the CERT National Insider Threat Center (NITC) is to monitor the shifting landscape of insider threat to identify tools and techniques insiders may use to harm to their organization. Our expanding repository of insider incidents shows that the tools and techniques insiders use to exploit vulnerabilities change rapidly as new technologies emerge and organizations evolve how they protect their assets. This blog post will look at the emergence of technologies and the frequency with which they have been used by insiders over the last 20 years....

Read More
The Top 10 Blog Posts of 2019

The Top 10 Blog Posts of 2019

• SEI Blog
Douglas C. Schmidt

Every January on the SEI Blog, we present the 10 most-visited posts of the previous year. This year's list of top 10 is presented in reverse order and features posts published between January 1, 2019, and December 31, 2019. -->10. Evaluating Threat-Modeling Methods for Cyber-Physical Systems9. Managing the Consequences of Technical Debt: 5 Stories from the Field8. The Vectors of Code: On Machine Learning for Software 7. Business Email Compromise: Operation Wire Wire and New Attack Vectors 6. Six Free Tools for Creating a Cyber Simulator 5. Operation Cloud Hopper Case Study 4. Deep Learning and Satellite Imagery: DIUx Xview...

Read More
4 Elements in Securing the Telecommunications Supply Chain

4 Elements in Securing the Telecommunications Supply Chain

• SEI Blog
Bobbie Stempfley

On September 27, 2019, the Subcommittee on Communications and Technology of the U.S. House of Representatives Committee on Energy and Commerce convened a hearing on "Legislating to Secure America's Wireless Future." The hearing focused on how the telecommunications industry can use cutting-edge technology to improve the power of our airwaves while securing our nation's networks. Doing this, said Energy and Commerce Chairman Frank Pallone, Jr., and Communications and Technology Subcommittee Chairman Mike Doyle, means "pushing ahead with legislation to root-out suspect network equipment nationwide and explore ways to improve coordination and management of spectrum resources to better serve the American...

Read More
Achieving the Quantum Advantage in Software

Achieving the Quantum Advantage in Software

• SEI Blog
Jason Larkin

Daniel Justice coauthored this blog post. The Department of Defense (DoD) faces a number of computationally challenging software engineering problems, including machine learning and artificial intelligence (AI) along with validating and verifying increasingly complex software systems. Finding the ideal solution to these challenges, known as combinatorial optimization problems, is non-deterministic polynomial hard and, with classical computing paradigms, could take billions of years to solve. In the SEI's Emerging Technology Center (ETC), we are working to apply quantum computing to solve these mission-critical problems for the DoD. As described in this post, our latest effort focuses on near-term quantum computing for...

Read More
Measuring Resilience in Artificial Intelligence and Machine Learning Systems

Measuring Resilience in Artificial Intelligence and Machine Learning Systems

• Insider Threat Blog
Alexander Petrilli

Shing-Hon Lau co-authored this post. Artificial intelligence (AI) and machine learning (ML) systems are quickly becoming integrated into a wide array of business and military operational environments. Organizations should ensure the resilience of these new systems, just as they would any other mission-critical asset. However, the "black box" decision-making processes that can make AI and ML systems so useful may also make the measurement of their resilience different than traditional measures. This blog posts describes the challenges of measuring the resilience of AI and ML systems with the current set of resilience assessment tools. While the discussion in this post...

Read More
AI Engineering: 11 Foundational Practices for Decision Makers

AI Engineering: 11 Foundational Practices for Decision Makers

• SEI Blog
Ipek Ozkaya

This post is also authored by Angela Horneman and Andrew Mellinger. Artificial intelligence (AI) is driving advances throughout modern society, including in the fields of medicine, transportation, education, and finance. In the government space, the Department of Defense (DoD) has made the advancement of AI a priority "to maintain its strategic position to prevail on future battlefields and safeguard a free and open international order." In this landscape, AI systems are being developed at a rapid-fire pace, yet the discipline of AI engineering is still relatively new and evolving. The very nature of AI systems makes foundational practices that are...

Read More
System Resilience Part 2: How System Resilience Relates to Other Quality Attributes

System Resilience Part 2: How System Resilience Relates to Other Quality Attributes

• SEI Blog
Donald Firesmith

To most people, a system is resilient if it continues to perform its mission in the face of adversity. In other words, a system is resilient if it continues to operate appropriately and provide required capabilities despite excessive stresses that can or do cause disruptions. System resilience is not an isolated quality attribute. As this post, the second in a series on system resilience, details, it is directly related to robustness, safety, cybersecurity, anti-tamper, survivability, capacity, longevity, and interoperability. It is less closely related to adaptability, availability, performance, reliability, and reparability....

Read More
Machine Learning in Cybersecurity

Machine Learning in Cybersecurity

• CERT/CC Blog
Jonathan Spring

We recently published a report that outlines relevant questions that decision makers who want to use artificial intelligence (AI) or machine learning (ML) tools as solutions in cybersecurity should ask of machine-learning practitioners to adequately prepare for implementing them. My coauthors are Joshua Fallon, April Galyardt, Angela Horneman, Leigh Metcalf, and Edward Stoner. Our goal with the report is chiefly educational, and we hope it can act like an ML-specific Heilmeier catechism and serve as a kind of checklist for decision makers to pursue and acquire good ML tools for cybersecurity....

Read More
System Resilience: What Exactly is it?

System Resilience: What Exactly is it?

• SEI Blog
Donald Firesmith

Over the past decade, system resilience (a.k.a., system resiliency) has been widely discussed as a critical concern, especially in terms of data centers and cloud computing. It is also vitally important to cyber-physical systems, although the term is less commonly used in that domain. Everyone wants their systems to be resilient, but what does that actually mean? And how does resilience relate to other quality attributes, such as availability, reliability, robustness, safety, security, and survivability? Is resilience a component of some or all of these quality attributes, a superset of them, or something else? If we are to ensure that...

Read More
Situational Awareness for Cybersecurity: Three Key Principles of Effective Policies and Controls

Situational Awareness for Cybersecurity: Three Key Principles of Effective Policies and Controls

• SEI Blog
Angela Horneman

Security measures are most effective when it is clear how assets are supposed to be used and by whom. When this information is documented in clearly written organizational policies, these policies can then be implemented in the form of enforceable security controls. In this third post in our series of blog posts on cyber situational awareness for the enterprise, I discuss how policies and controls contribute to asset protection and to the know what should be component of situational awareness. I also present three key principles that underscore the importance of effective policies and controls to the overall security posture...

Read More
VPN - A Gateway for Vulnerabilities

VPN - A Gateway for Vulnerabilities

• CERT/CC Blog
Vijay Sarvepalli

Virtual Private Networks (VPNs) are the backbone of today's businesses providing a wide range of entities from remote employees to business partners and sometimes even to customers, with the ability to connect to sensitive corporate information securely. Long gone are the days of buying a leased line or a dedicated physical network (or fiber) for these types of communications. VPNs provide a simple way to take advantage of the larger public internet by creating virtual encrypted communications. However, in recent months a number of VPN vulnerabilities have been discovered and are known to be actively exploited (Cybersecurity Requirements Center Advisory),...

Read More
Six Best Practices for Developer Testing

Six Best Practices for Developer Testing

• SEI Blog
Robert V. Binder

Code coverage represents the percent of certain elements of a software item that have been exercised during its testing. As I explained in my first post in this series on developer testing, there are many ideas about which code elements are important to test and therefore many kinds of code coverage. In this post, the second post in the series, I explain how you can use coverage analysis to routinely achieve consistently effective testing....

Read More
Could Blockchain Improve the Cybersecurity of Supply Chains?

Could Blockchain Improve the Cybersecurity of Supply Chains?

• SEI Blog
Eliezer Kanal

A September 2018 report to the President, Assessing and Strengthening the Manufacturing and Defense Industrial Base and Supply Chain Resiliency of the United States, raised concerns about cybersecurity in light of the reliance on complex supply chains in defense applications. Gaps in the cybersecurity sector lead to pervasive and persistent vulnerabilities to the industrial base, [...] unauthorized access to any facet of manufacturing information could create rippling effects and cause innumerable negative economic and national security situations. [...] Cybersecurity risks impact all facets of manufacturing supply chain operations, from product and process data flowing within and across factories, to supply...

Read More
Mapping Cyber Hygiene to the NIST Cybersecurity Framework

Mapping Cyber Hygiene to the NIST Cybersecurity Framework

• Insider Threat Blog
Matthew Trevors

In honor of Cybersecurity Awareness Month, I decided to put fingers to keys and share some basic practices that every organization should consider for their cyber hygiene initiatives. This blog post will describe a process to determine if 41 foundational practices from the CERT Resilience Management Model (CERT-RMM) are part of your NIST Cybersecurity Framework v1.1 target profile....

Read More
Network Traffic Analysis with SiLK: Profiling and Investigating Cyber Threats

Network Traffic Analysis with SiLK: Profiling and Investigating Cyber Threats

• SEI Blog
Paul Krystosek

Tim Shimeall and Nancy Ott co-authored this post. Cyber threats are on the rise, making it vitally important to understand what's happening on our computer networks. But the massive amount of network traffic makes this job hard. How can we find evidence of unusual, potentially hostile activity in this deluge of network data? One way is to use SiLK (System for Internet Level Knowledge), a highly-scalable tool suite for capturing and analyzing network flow data. SiLK's data retrieval and analysis tools enable us to spot trends and anomalies that could indicate unfriendly activity. The publication Network Traffic Analysis with SiLK...

Read More
It's Time to Retire Your Unsupported Things

It's Time to Retire Your Unsupported Things

• CERT/CC Blog
Will Dormann

"If it ain't broke, don't fix it." Why mess with something that already works? This is fair advice with many things in life. But when it comes to software security, it's important to realize that there can be severe consequences to using software or hardware after the vendor stops supporting it. In this blog post, I will discuss a number of examples of products, including Microsoft Windows and D-Link routers, whose continued use beyond their support periods may put you at unnecessary risk. Support periods and information about consumer goods, including software, are regulated differently among jurisdictions, but this post...

Read More
How to Build a Trustworthy Free/Libre Linux Capable 64-bit RISC-V Computer

How to Build a Trustworthy Free/Libre Linux Capable 64-bit RISC-V Computer

• SEI Blog
Gabriel Somlo

The attack surface for commercial hardware now spans all stages of the development lifecycle. Even in the presence of secure, bug-free software, the growing threat of hardware Trojans and backdoors enables adversaries to compromise a system in its entirety or execute a privilege escalation attack. This reality became painfully evident in the wake of Spectre/Meltdown attacks. These two vulnerabilities, which came to light in 2018, affected a wide swath of microprocessors that allowed attackers to access sensitive system data. With the rapid growth of the computing and IT technology market, the government is now deploying commercially produced, civilian market hardware...

Read More
Situational Awareness for Cybersecurity: Assets and Risk

Situational Awareness for Cybersecurity: Assets and Risk

• SEI Blog
Angela Horneman

This post was co-written by Lauren Cooper. When key business assets are not adequately protected from cybersecurity breaches, organizations can experience dire consequences. Lumin PDF, a PDF editing tool, recently had confidential data for its base of 24.3 million users published in an online forum. The personal data of almost every citizen of Ecuador was also recently leaked online. Data breaches exposed 4.1 billion records in the first six months of 2019, and data breaches in the healthcare industry in 2019 have already doubled all of those last year. The purpose of situational awareness (SA) is to protect organizations from...

Read More
Don't Play Developer Testing Roulette: How to Use Test Coverage

Don't Play Developer Testing Roulette: How to Use Test Coverage

• SEI Blog
Robert V. Binder

Suppose someone asked you to play Russian Roulette. Although your odds of surviving are 5 to 1 (83 percent), it is hard to imagine how anyone would take that risk. But taking comparable risk owing to incomplete software testing is a common practice. Releasing systems whose tests achieve only partial code coverage--the percentage of certain elements of a software item that have been exercised during its testing--is like spinning the barrel and hoping for the best, or worse, believing there is no risk. This post is partly a response to questions I'm frequently asked when working with development teams looking...

Read More
Managing the Risks of Ransomware

Managing the Risks of Ransomware

• Insider Threat Blog
David Tobar

This blog post was co-authored by Jason Fricke. Ransomware poses a growing threat to both businesses and government agencies. Though no strategy can fully eliminate these risks, this post provides recommendations, and links to additional best practices, on better managing ransomware risks....

Read More
Artificial Intelligence in Practice: Securing Your Code Using Natural Language Processing

Artificial Intelligence in Practice: Securing Your Code Using Natural Language Processing

• SEI Blog
Eliezer Kanal

Many techniques are available to help developers find bugs in their code, but none are perfect: an adversary needs only one to cause problems. In this post, I'll discuss how a branch of artificial intelligence called natural language processing, or NLP, is being applied to computer code and cybersecurity. NLP is how machines extract information from naturally occurring language, such as written prose or transcribed speech. Using NLP, we can gain insight into the code we generate, and can find bugs that aren't visible to existing techniques. While this field is still young, advances are coming rapidly, and I will...

Read More
Bolstering Security with Cyber Intelligence

Bolstering Security with Cyber Intelligence

• SEI Blog
Jared Ettinger

Stephen Beck co-wrote this blog post. A maxim for intelligence operators and military and special operations communities is "get off the X." The expression, once reserved for combat situations in reference to getting out of "the kill zone, point of attack, minefield, sniper crosshairs or other danger zone" has been adopted by the intelligence communities to convey the danger of a static approach to organizational security. As Michele Rigby Assad, a former intelligence officer in the CIA, explains, "the X refers to the site of an attack. This is the location in which the attackers have the greatest advantage because...

Read More
Insider Threat Incident Analysis: Court Outcome Observations

Insider Threat Incident Analysis: Court Outcome Observations

• Insider Threat Blog
Nick Miller

In the United States, legal cases may be tried in criminal court or civil court. According to data in the CERT National Insider Threat Center (NITC) incident corpus, the type of court makes a big difference in the legal outcomes of insider attack cases. This blog post analyzes these differences, specifically sentencing and restitution in criminal cases and findings of liability in civil cases. This blog post does not, and is not intended to, constitute legal advice. Please consult legal counsel on any specific matter....

Read More
Helping the Federal Government Achieve the Cyber Advantage

Helping the Federal Government Achieve the Cyber Advantage

• SEI Blog
Bobbie Stempfley

The world we live in is increasingly digital, synthetic, and fueled by data. The software it is built on is developed with such speed and automation that we must think about security in a new way. And in today's age of artificial intelligence (AI), cyber adversaries operate with speed and dexterity in a world of ever-changing attack surfaces. In light of this constantly evolving cyber landscape, our researchers work to secure our infrastructure and resources and gain a cyber advantage over our adversaries. As this blog post will detail, this challenge requires that we transcend the capabilities of our adversaries...

Read More
Impacts and Recommendations for Achieving Modular Open Systems Architectures --Fifth Post in a Series

Impacts and Recommendations for Achieving Modular Open Systems Architectures --Fifth Post in a Series

• SEI Blog
Nickolas Guertin

This post was co-written by Douglas Schmidt and William Scherlis. In this series of blog posts, adapted from a recently published paper, we sought to demonstrate how layered business and technical architectures can leverage modular component design practices to establish new approaches for capability acquisition that are more effective for the Department of Defense (DoD) than existing system of systems (SoS) strategies. The aim of these posts is to help the DoD establish an acquisition environment that is more efficient and capable of delivering higher quality, with far greater innovation, in a fraction of the time. Our first post proposed...

Read More
Improving Insider Threat Detection Methods Through Software Engineering Principles

Improving Insider Threat Detection Methods Through Software Engineering Principles

• Insider Threat Blog
Daniel Costa

Tuning detective controls is a key component of implementing and operating an insider threat program, and one we have seen many organizations struggle with. Our work helping organizations with their insider threat programs has revealed common challenges with any tool that generates alerts of potential insider risk, such as user activity monitoring (UAM), security information event management (SIEM), or user and entity behavioral analytics (UEBA) tools. In this blog post, we will discuss some of the challenges and best practices for tuning detective controls....

Read More
7 Guidelines for Being a TRUSTED Penetration Tester

7 Guidelines for Being a TRUSTED Penetration Tester

• Insider Threat Blog
Karen Miller

The best way to learn is by doing. But when it comes to penetration testing, learners risk legal implications and bad habits if they don't follow ethical, safe procedures. Those wishing to develop penetration testing skills are often unaware of the number of resources available for legally and safely testing penetration tools and techniques. In this blog post, I'll describe seven general practices, outlined in the acrostic "TRUSTED," that pen testing learners and professionals should follow to avoid legal consequences and earn trust. I'll also provide resources for learning how to pen test....

Read More
What Engineers Need to Know About Artificial Intelligence

What Engineers Need to Know About Artificial Intelligence

• SEI Blog
Thomas Longstaff

Artificial intelligence (AI) systems by their nature are software-intensive. To create viable and trusted AI systems, engineers need technologies and standards, similar to those in software engineering. At the Software Engineering Institute (SEI)--a federally funded research and development center tasked with advancing the field of software engineering and cybersecurity--we are leading a movement to establish a professional AI Engineering discipline. As we begin a national conversation on AI Engineering, we have identified several key aspects and elements of AI that engineers must understand to work with emerging systems....

Read More
Update on the CERT Guide to Coordinated Vulnerability Disclosure

Update on the CERT Guide to Coordinated Vulnerability Disclosure

• CERT/CC Blog
Allen Householder

It's been two years since we originally published the CERT Guide to Coordinated Vulnerability Disclosure. In that time, it's influenced both the US Congress and EU Parliament in their approaches to vulnerability disclosure. I wanted to provide an update on how the Guide is evolving in response to all the feedback we received....

Read More
Situational Awareness for Cybersecurity: An Introduction

Situational Awareness for Cybersecurity: An Introduction

• SEI Blog
Angela Horneman

Situational awareness (SA) helps decision makers throughout an organization have the information and understanding available to make good decisions in the course of their work. It can be focused specifically on helping people and organizations protect their assets in the cyber realm or it can be more far reaching. SA makes it possible to get relevant information from across an organization, to integrate that information, and to disseminate it to help people make better decisions. This blog post is the first in a series that explores the concepts of cyber SA as they apply to the enterprise....

Read More
The Dangers of VHD and VHDX Files

The Dangers of VHD and VHDX Files

• CERT/CC Blog
Will Dormann

Recently, I gave a presentation at BSidesPGH 2019 called Death By Thumb Drive: File System Fuzzing with CERT BFF. (The slides from my presentation are available in the SEI Digital Library.) Although my primary goal was to find bugs in kernel file-system-parsing code, a notable part of my research was investigating attack vectors. In particular, I focused on VHD and VHDX files on Windows systems. In this post, I describe some of the risks associated with these two file types....

Read More
September Is National Insider Threat Awareness Month

September Is National Insider Threat Awareness Month

• Insider Threat Blog
Daniel Costa

September 2019 has been declared National Insider Threat Awareness Month by the National Insider Threat Task Force, the National Counterintelligence and Security Center, the Federal Bureau of Investigation, the Office of the Under Secretary of Defense (Intelligence), the Department of Homeland Security, and the Defense Counterintelligence and Security Agency. This blog post outlines the CERT National Insider Threat Center's activities in support of this effort....

Read More
The Latest Work from the SEI: AI, Deepfakes, Automated Alert Handling, and Cyber Intelligence

The Latest Work from the SEI: AI, Deepfakes, Automated Alert Handling, and Cyber Intelligence

• SEI Blog
Douglas C. Schmidt

As part of an ongoing effort to keep you informed about our latest work, this blog post summarizes some recently published SEI reports, podcasts, and presentations highlighting our work in artificial intelligence, STEM careers, deepfakes, automated alert handling (here and here), systems and software engineering, and cyber intelligence. These publications highlight the latest work of SEI technologists in these areas. The SEI also made available an online version of the 2018 Year in Review, which highlights the recent work of the institute. This post includes a listing of each publication, author(s), and links where they can be accessed on the...

Read More
Cybersecurity Engineering for Legacy Systems: 6 Recommendations

Cybersecurity Engineering for Legacy Systems: 6 Recommendations

• SEI Blog
Susan Crozier Cox

Harry Levinson co-authored this blog post. Legacy systems continue to play a key role across many organizations. Engineering cybersecurity into these legacy systems presents some unique challenges. In many cases, the original design team is no longer available, leaving the current team with the challenge of changing poorly- and/or un-documented designs and software. Over the years, these systems can become so outdated that they are unable to keep up with new software patterns and development technologies, including the ability to patch known security or design flaws. This blog contains six recommendations to help keep legacy software secure....

Read More
Patterns and Trends in Insider Threats Across Industry Sectors (Part 9 of 9: Insider Threats Across Industry Sectors)

Patterns and Trends in Insider Threats Across Industry Sectors (Part 9 of 9: Insider Threats Across Industry Sectors)

• Insider Threat Blog
Daniel Costa

In previous posts of our series analyzing and summarizing insider incidents across multiple sectors, we presented up-to-date statistics from the CERT National Insider Threat Center (NITC) Incident Corpus and looked closely at which types of insider incidents are prevalent within certain types of organizations. From there, we presented statistics on what types of assets those insider attacks target, the time frames associated with those attacks, and the tactics, techniques, and procedures the insiders used to carry them out. In this final post of the series, we summarize what we have learned or reinforced in these recent explorations of insider incidents...

Read More
Mission Thread Analysis Using End-to-End Data Flows - Part 2

Mission Thread Analysis Using End-to-End Data Flows - Part 2

• SEI Blog
Donald Firesmith

The first blog post in this series provided an overview of the E2E Mission Thread Data Flow Analysis (EMDA) method, an approach that analyzes the flow of data as they traverse end-to-end mission threads through the architecture components of a system of systems. That post addressed relevant challenges that EMDA helps system and software architects face and outlined the work products produced by the method. This second blog post discusses the process used to create and verify the method's work products, the benefits of the method, the challenges must be addressed while implementing the method, and lessons learned during the...

Read More
Why Software Architects Must Be Involved in the Earliest Systems Engineering Activities

Why Software Architects Must Be Involved in the Earliest Systems Engineering Activities

• SEI Blog
Sarah Sheard

Suzanne Miller, Bill Nichols, Don Firesmith, and Mike Phillips contributed to this post. Today's major defense systems rely heavily on software-enabled capabilities. However, many defense programs acquiring new systems first determine the physical items to develop, assuming the contractors for those items will provide all needed software for the capability. But software by its nature spans physical items: it provides the inter-system communications that have a direct influence on most capabilities, and thus must be architected intelligently, especially when pieces are built by different contractors. If this architecture step is not done properly, a software-reliant project can be set up...

Read More
Mission Thread Analysis Using End-to-End Data Flows  - Part 1

Mission Thread Analysis Using End-to-End Data Flows - Part 1

• SEI Blog
Donald Firesmith

Although the vast majority of military missions require the successful collaboration of multiple cyber-physical systems within an overall system of systems (SoS), almost all system and software architects work on programs developing or sustaining individual systems and subsystems. Often, they do not sufficiently understand the ramifications of how their system interoperates with these other systems to accomplish the overall mission. The lack of an end-to-end (E2E) mission thread analysis leads to numerous difficulties, such as integration problems that are not identifiable if one merely looks at one's own system and the specifications of its individual interfaces. This is the first...

Read More
Expectations of Windows RDP Session Locking Behavior

Expectations of Windows RDP Session Locking Behavior

• CERT/CC Blog
Will Dormann

This post was co-written by Will Dormann and Joe Tammariello. Recently, CERT researchers published a vulnerability note (VU#576688 - Microsoft Windows RDP can bypass the Windows lock screen). In this blog post, we provide a little more insight into how the vulnerability was discovered and what it may mean to people who use Microsoft Windows RDP. The following steps reproduce VU#576688: Use a Microsoft Windows RDP client to connect to Windows Server 2019 or Windows 10 build 1803 or newer. Manually lock the remote Windows session. Disconnect the network on the RDP client system. Reconnect the network. After performing these...

Read More
The Promise of Deep Learning on Graphs

The Promise of Deep Learning on Graphs

• SEI Blog
Oren Wright

A growing number of Department of Defense (DoD) data problems are graph problems: the data from sources such as sensor feeds, web traffic, and supply chains are full of irregular relationships that require graphs to represent explicitly and mathematically. For example, modern test and evaluation produces massive, heterogeneous datasets, and analysts can use graphs to reveal otherwise hidden patterns in these data, affording the DoD a far more complete understanding of a system's effectiveness, survivability, and safety. But such datasets are growing increasingly large and increasingly complex, demanding new approaches for proper analysis. Machine learning seems to recommend itself to...

Read More
Cybersecurity Governance, Part 1: 5 Fundamental Challenges

Cybersecurity Governance, Part 1: 5 Fundamental Challenges

• Insider Threat Blog
Seth Swinton

This post was co-authored by Stephanie Hedges. Most organizations have good enterprise-level security policies that define their approach to maintaining, improving, and securing their information and information systems. However, once the policies are signed by senior leadership and distributed throughout the organization, significant cybersecurity governance challenges remain. This blog post examines five fundamental challenges of cybersecurity governance that, while not exhaustive, are essential to establishing and maintaining an effective cybersecurity governance program....

Read More
An Application Programming Interface for Classifying and Prioritizing Static Analysis Alerts

An Application Programming Interface for Classifying and Prioritizing Static Analysis Alerts

• SEI Blog
Lori Flynn

This post was co-written by Ebonie McNeil and Aubrie Woods. In this post, we describe the Source Code Analysis Integrated Framework Environment (SCAIFE) application programming interface (API). SCAIFE is an architecture for classifying and prioritizing static analysis alerts. It is designed so that a wide variety of static analysis tools can integrate with the SCAIFE system using the API. The API is pertinent to organizations that develop or research static analysis alert auditing tools, aggregators, and frameworks....

Read More
Using OOAnalyzer to Reverse Engineer Object Oriented Code with Ghidra

Using OOAnalyzer to Reverse Engineer Object Oriented Code with Ghidra

• SEI Blog
Jeffrey Gennari

Object-oriented programs continue to pose many challenges for reverse engineers and malware analysts. C++ classes tend to result in complex arrangements of assembly instructions and sophisticated data structures that are hard to analyze at the machine code level. We've long sought to simplify the process of reverse engineering object-oriented code by creating tools, such as OOAnalyzer, which automatically recovers C++-style classes from executables. OOAnalyzer includes utilities to import OOAnalyzer results into other reverse engineering frameworks, such as the IDA Pro Disassembler. I'm pleased to announce that we've updated our Pharos Binary Analysis Framework in Github to include a new plugin...

Read More
Selecting Measurement Data for Software Assurance Practices

Selecting Measurement Data for Software Assurance Practices

• SEI Blog
Carol Woody

Measuring the software assurance of a product as it is developed and delivered to function in a specific system context involves assembling carefully chosen metrics. These metrics should demonstrate a range of behaviors to confirm confidence that the product functions as intended and is free of vulnerabilities. The Software Assurance Framework (SAF) is a collection of cybersecurity practices that programs can apply across the acquisition lifecycle and supply chain to promote the desired assurance behaviors. The SAF can be used to assess an acquisition program's current cybersecurity practices and chart a course for improvement, ultimately reducing the cybersecurity risk of...

Read More
Three Architecture Recommendations for Sustainment Organizations

Three Architecture Recommendations for Sustainment Organizations

• SEI Blog
Susan Crozier Cox

In a March 2019 report, the Defense Innovation Board (DIB)--a group of advisors focused on bringing the technical advantages employed by Silicon Valley to the Department of Defense (DoD)--noted that the United States faces threats that are evolving at an ever-increasing pace. The DIB also noted that the DoD's ability to adapt and respond to these threats is now determined by its ability to develop and deploy software to the field rapidly. As the DIB and other reports have noted, the DoD's current approach to software development is broken and a leading source of risk: "it takes too long, is...

Read More
Keeping an Eye Out for Positive Risk

Keeping an Eye Out for Positive Risk

• Insider Threat Blog
Mary Beth Chrissis

We commonly think about risks having negative consequences. With each month bringing new cybersecurity threats, breaches, and vulnerabilities, sound risk management practices are necessary to protect your organization. However, when performing risk management, do organizations unnecessarily limit themselves by only thinking about risks as negative effects and not looking at positive effects, too?...

Read More
Testing Concurrent Systems: Concurrency Defects, Testing Techniques, and Recommendations

Testing Concurrent Systems: Concurrency Defects, Testing Techniques, and Recommendations

• SEI Blog
Donald Firesmith

Concurrency, which exists whenever multiple entities execute simultaneously, is a ubiquitous and an unavoidable fact of life in systems and software engineering. It greatly increases system and software complexity, which directly impacts testing. Concurrency leads to nondeterministic behavior and numerous types of concurrency defects that require specialized approaches to uncover. At the SEI, we are often called upon to review development planning documents including Test and Evaluation Master Plans (TEMPs) and Software Test Plans (STPs). We are also frequently tasked to evaluate developmental testing including software and system integration laboratories (SILs) and other test environments. One common observation is that...

Read More