Every January on the SEI Blog, we present the 10 most-visited posts of the previous year. This year’s top 10 list highlights the SEI’s work in software acquisition, artificial intelligence, large language models, secure coding, insider risk mitigation, and enterprise risk management. The posts, which were published between January 1, 2024, and December 31, 2024, are presented below in reverse order based on the number of visits.

#10 5 Recommendations to Help Your Organization Manage Technical Debt

by Ipek Ozkaya and Brigid O’Hearn

The fiscal year 2022 National Defense Authorization Act (NDAA) Section 835, “Independent Study on Technical Debt in Software-Intensive Systems,” required the Secretary of Defense to engage a federally funded research and development center (FFRDC) “to study technical debt in software-intensive systems.” To satisfy this requirement and lead this work, the Department of Defense (DoD) selected the Carnegie Mellon University (CMU) Software Engineering Institute (SEI), which is a recognized leader in the practice of managing technical debt. According to NDAA Section 835, the purpose of the study was to provide, among other things, analyses and recommendations on quantitative measures for assessing technical debt, current and best practices for measuring and managing technical debt and its associated costs, and practices for reducing technical debt.

Our team spent more than a year conducting the independent study. The report we produced describes the conduct of the study, summarizes the technical trends observed, and presents the resulting recommendations. In this SEI Blog post, we summarize several recommendations that apply to the DoD and other development organizations seeking to analyze, manage, and reduce technical debt. You can find a complete discussion of the study methodology, findings, and recommendations in the SEI’s Report to the Congressional Defense Committees on National Defense Authorization Act (NDAA) for Fiscal Year 2022 Section 835 Independent Study on Technical Debt in Software-Intensive Systems.

Read the post in its entirety.

#9 Applying Large Language Models to DoD Software Acquisition: An Initial Experiment

by Douglas Schmidt and John E. Robert

There is considerable interest in using generative AI tools, such as large language models (LLMs), to revolutionize industries and create new opportunities in the commercial and government domains. For many Department of Defense (DoD) software acquisition professionals, the promise of LLMs is appealing, but there’s also a deep-seated concern that LLMs do not address today’s challenges due to privacy concerns, potential for inaccuracy in the output, and lack of confidence or uncertainty about how to use LLMs effectively and responsibly. This blog post is the second in a series dedicated to exploring how generative AI, particularly LLMs such as ChatGPT, Claude, and Gemini, can be applied within the DoD to enhance software acquisition activities.

Our first blog post in this series presented 10 Benefits and 10 Challenges of Applying LLMs to DoD Software Acquisition and suggested specific use cases where generative AI can provide value to software acquisition activities. This second blog post expands on that discussion by showing specific examples of using LLMs for software acquisition in the context of a document summarization experiment, as well as codifying the lessons we learned from this experiment and our related work on applying generative AI to software engineering.

Read the post in its entirety.

#8 10 Lessons in Security Operations and Incident Management

by Robin Ruefle

Incident response is a critical need throughout government and industry as cyber threat actors look to compromise critical assets within organizations with cascading, often catastrophic, effects. In 2021, for example, a hacker allegedly accessed a Florida water treatment plant’s computer systems and poisoned the water supply. Within the U.S. critical national infrastructure, 77 percent of organizations have seen a rise in insider-driven cyber threats over the last three years. The 2023 IBM Cost of a Data Breach report highlights the crucial role of having a well-tested incident response plan. Companies without a tested plan in place will face 82 percent higher costs in the event of a cyber attack, compared to those that have implemented and tested such a plan.

Researchers in the SEI CERT Division compiled 10 lessons learned from our more than 35 years of developing and working with incident response and security teams throughout the globe. These lessons are relevant to incident response teams contending with an ever-evolving cyber threat landscape. In honor of the CERT Division (also referred to the CERT Coordination Center in our work with the Forum of Incident Response and Security Teams) celebrating 35 years of operation, in this blog post we take a look back at some of the lessons learned from our Cyber Security Incident Response Team (CSIRT) capacity building experiences that also apply to other areas of security operations.

Read the post in its entirety.

#7 CERT Releases 2 Tools to Assess Insider Risk

by Roger Black

According to a 2023 Ponemon study, the number of reported insider risk incidents and the costs associated with them continues to rise. With more than 7,000 reported cases in 2023, the average insider risk incident cost organizations over $600,000. To help organizations assess their insider risk programs and identify potential vulnerabilities that could result in insider threats, the SEI CERT Division has released two tools available for download on its website. Previously available only to licensed partners, the Insider Threat Vulnerability Assessment (ITVA) and Insider Threat Program Evaluation (ITPE) toolkits provide practical methods to assess your organization’s ability to manage insider risk. This post describes the purpose and use of the toolkits, with a focus on the workbook components of the toolkits that are the primary methods of program assessment.

Read the post in its entirety.

#6 What Recent Vulnerabilities Mean to Rust

by David Svoboda

In recent weeks several vulnerabilities have rocked the Rust community, causing many to question the safety of the borrow checker, or of Rust in general. In this post, we examine two such vulnerabilities: the first is CVE-2024-3094, which involves some malicious files in the xz library, and the second is CVE-2024-24576, which involves command-injection vulnerabilities in Windows. How did these vulnerabilities arise, how were they discovered, and how do they involve Rust? More importantly, might Rust be susceptible to more similar vulnerabilities in the future?

Last year we published two blog posts about the security provided by the Rust programming language. We discussed the memory safety and concurrency safety provided by Rust’s borrow checker. We also described some of the limitations of Rust’s security model, such as its limited ability to prevent various injection attacks, and the unsafe keyword, which allows developers to bypass Rust’s security model when necessary. Back then, our conclusion was that no language could be fully secure, yet the borrow checker did provide significant, albeit limited, memory and concurrency safety when not bypassed with the unsafe keyword. We also examined Rust through the lens of source and binary analysis, gauged its stability and maturity, and realized that the constraints and expectations for language maturity have slowly evolved over the decades. Rust is moving in the direction of maturity today, which is distinct from what was considered a mature programming language in 1980. Furthermore, Rust has made some notable stability guarantees, such as promising to deprecate rather than delete any crates in crates.io to avoid repeating the Leftpad fiasco.

Read the post in its entirety.

#5 Generative AI and Software Engineering Education

by Ipek Ozkaya, Douglas Schmidt, and Michael Hilton

The initial surge of excitement and fear surrounding generative artificial intelligence (AI) is gradually evolving into a more realistic perspective. While the jury is still out on the actual return on investment and tangible improvements from generative AI, the rapid pace of change is challenging software engineering education and curricula. Educators have had to adapt to the ongoing developments in generative AI to provide a realistic perspective to their students, balancing awareness, healthy skepticism, and curiosity.

In a recent SEI webcast, researchers discussed the impact of generative AI on software engineering education. SEI and Carnegie Mellon University experts spoke about the use of generative AI in the curriculum and the classroom, discussed how faculty and students can most effectively use generative AI, and considered concerns about ethics and equity when using these tools. The panelists took questions from the audience and drew on their experience as educators to speak to the critical questions generative AI raises for software engineering education.

This blog post features an edited transcript of responses from the original webcast. Some questions and answers have been rearranged and revised for clarity.

Read the post in its entirety.

#4 OpenAI Collaboration Yields 14 Recommendations for Evaluating LLMs for Cybersecurity

by Jeff Gennari, Shing-hon Lau, and Samuel J. Perl

Large language models (LLMs) have shown a remarkable ability to ingest, synthesize, and summarize knowledge while simultaneously demonstrating significant limitations in completing real-world tasks. One notable domain that presents both opportunities and risks for leveraging LLMs is cybersecurity. LLMs could empower cybersecurity experts to be more efficient or effective at preventing and stopping attacks. However, adversaries could also use generative artificial intelligence (AI) technologies in kind. We have already seen evidence of actors using LLMs to aid in cyber intrusion activities (e.g., WormGPT, FraudGPT, etc.). Such misuse raises many important cybersecurity-capability-related questions including

• Can an LLM like GPT-4 write novel malware?
• Will LLMs become critical components of large-scale cyber-attacks?
• Can we trust LLMs to provide cybersecurity experts with reliable information?

The answer to these questions depends on the analytic methods chosen and the results they provide. Unfortunately, current methods and techniques for evaluating the cybersecurity capabilities of LLMs are not comprehensive. Recently, a team of researchers in the SEI CERT Division worked with OpenAI to develop better approaches for evaluating LLM cybersecurity capabilities. This SEI Blog post, excerpted from a recently published paper that we coauthored with OpenAI researchers Joel Parish and Girish Sastry, summarizes 14 recommendations to help assessors accurately evaluate LLM cybersecurity capabilities.

Read the post in its entirety.

#3 10 Benefits and 10 Challenges of Applying Large Language Models to DoD Software Acquisition

by John E. Robert and Douglas Schmidt

Department of Defense (DoD) software acquisition has long been a complex and document-heavy process. Historically, many software acquisition activities, such as generating Requests for Information (RFIs), summarizing government regulations, identifying relevant commercial standards, and drafting project status updates, have required considerable human-intensive effort. However, the advent of generative artificial intelligence (AI) tools, including large language models (LLMs), offers a promising opportunity to accelerate and streamline certain aspects of the software acquisition process.

Software acquisition is one of many complex mission-critical domains that may benefit from applying generative AI to augment and/or accelerate human efforts. This blog post is the first in a series dedicated to exploring how generative AI, particularly LLMs like ChatGPT-4, can enhance software acquisition activities. In this post we present 10 benefits and 10 challenges of applying LLMs to the software acquisition process and suggest specific use cases where generative AI can provide value. Our focus is on providing timely information to software acquisition professionals, including defense software developers, program managers, systems engineers, cybersecurity analysts, and other key stakeholders, who operate within challenging constraints and prioritize security and accuracy.

Read the post in its entirety.

#2 Using ChatGPT to Analyze Your Code? Not So Fast

by Mark Sherman

The average code sample contains 6,000 defects per million lines of code, and the SEI’s research has found that 5 percent of these defects become vulnerabilities. This translates to roughly 3 vulnerabilities per 10,000 lines of code. Can ChatGPT help improve this ratio? There has been much speculation about how tools built on top of large language models (LLMs) might impact software development, more specifically, how they will change the way developers write code and evaluate it.

In March 2023 a team of CERT Secure Coding researchers—the team included Robert Schiela, David Svoboda, and myself—used ChatGPT 3.5 to examine the noncompliant software code examples in our CERT Secure Coding standard, specifically the SEI CERT C Coding Standard. In this post, I present our experiment and findings, which show that while ChatGPT 3.5 has promise, there are clear limitations.

Read the post in its entirety.

#1 The Top 10 Skills CISOs Need in 2024

by Greg Touhill

The role of the chief information security officer (CISO) has never been more important to organizational success. The present and near-future for CISOs will be marked by breathtaking technical advances, particularly those associated with the inclusion of artificial intelligence technologies being integrated into business functions, as well as emergent legal and regulatory challenges. Continued advances in generative artificial intelligence (AI) will accelerate the proliferation of deepfakes designed to erode public trust in online information and public institutions. Furthermore, these challenges will be amplified by an unstable global theater in which nefarious actors and nation states chase opportunities to exploit any potential organizational weakness. Some forecasts have already characterized 2024 as a pressure cooker environment for CISOs. In such an environment, skills are critical. In this post I outline the top 10 skills that CISOs need for 2024 and beyond. These recommendations draw upon my experience as the director of the SEI’s CERT Division, as well as my service as the first federal chief information security officer of the United States, leading cyber operations at the U.S. Department of Homeland Security, and my lengthy military service as a communications and cyberspace operations officer.

Read the post in its entirety.

Looking Ahead in 2025

We publish a new post on the SEI Blog weekly. In the coming months, look for posts highlighting the SEI’s work in artificial intelligence, machine learning, cybersecurity, software engineering, and more.