The fiscal year 2022 National Defense Authorization Act (NDAA) Section 835, “Independent Study on Technical Debt in Software-Intensive Systems,” required the Secretary of Defense to engage a federally funded research and development center (FFRDC) “to study technical debt in software-intensive systems.” To satisfy this requirement and lead this work, the Department of Defense (DoD) selected the Carnegie Mellon University (CMU) Software Engineering Institute (SEI), which is a recognized leader in the practice of managing technical debt. According to NDAA Section 835, the purpose of the study was to provide, among other things, analyses and recommendations on quantitative measures for assessing technical debt, current and best practices for measuring and managing technical debt and its associated costs, and practices for reducing technical debt.

Our team spent more than a year conducting the independent study. The report we produced describes the conduct of the study, summarizes the technical trends observed, and presents the resulting recommendations. In this SEI Blog post, we summarize several recommendations that apply to the DoD and other development organizations seeking to analyze, manage, and reduce technical debt. You can find a complete discussion of the study methodology, findings, and recommendations in the SEI’s Report to the Congressional Defense Committees on National Defense Authorization Act (NDAA) for Fiscal Year 2022 Section 835 Independent Study on Technical Debt in Software-Intensive Systems.

Technical Debt Study Methodology

We based our analyses and recommendations on a series of activities led by the SEI and executed according to a roadmap agreed to with the Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD(A&S)). These activities included

• Literature review—The study team completed a literature review that summarizes the state of the practice.
• Interviews—The SEI led 16 engagements, which included interviewing stakeholders from the U.S. federal government and industry, to gain a broad view of the state of the practice.
• Deep dives on program data—SEI subject matter experts engaged with DoD programs outside of this study to examine their practices, data, and decision making related to technical debt in more depth.
• Report for program stakeholders—The SEI developed a report describing the state of the practice, issues to be aware of at the program level, and examples of technical debt’s cybersecurity impact.

For the purposes of this study, we applied the definition of technical debt specified in NDAA Section 835: “an element of design or implementation that is expedient in the short term, but that would result in a technical context that can make a future change costlier or impossible.” This definition aligns with the SEI’s definition of the term, which is based on a substantial body of work with both industry and the DoD. It also conforms to the definition in Department of Defense Instruction (DoDI) 5000.87, Operation of the Software Acquisition Pathway:

Consists of design or implementation constructs that are expedient in the short term but that set up a technical context that can make a future change costlier or impossible. Technical debt may result from having code issues related to architecture, structure, duplication, test coverage, comments and documentation, potential bugs, complexity, coding practices, and style which may accrue at the level of overall system design or system architecture, even in systems with great code quality.

These definitions also conform to the notion that delayed upgrades, technology refresh, and sustainment items also become technical debt.

5 Recommendations for Managing Technical Debt

Though we prepared the following high-level recommendations for the upper levels of the DoD, most of them contain valuable information for any software development organization. The first recommendation, for example, provides a concrete starting point for implementing recommended practices, including bringing visibility to existing technical debt, establishing goals, and establishing tooling and measurement environments. Programs in any of these three stages can use the recommendations to help manage their technical debt intentionally. The following recommendations distill information applicable to the wider software development community from more detailed information specific to the DoD.

1. Share Best Practices

Development organizations should empower programs to incorporate technical debt management into software development lifecycle activities as one of the core software engineering practices.

The effective management of technical debt is critical for modern software practice, especially when it comes to sustaining an appropriate cadence for deploying capabilities. A key aspect of technical debt management is bringing visibility to instances of technical debt and making tradeoffs explicit for the long-term mitigation of it. Development organizations should therefore look for opportunities to make it easy for programs to incorporate technical debt management practices into their software development life cycle.

Resource challenges are often not easy to resolve and handing down mandates for new practices and metrics gathering are not likely to generate positive change. We therefore recommend a phased approach that relies on the following established practices:

• Stage 1: Bring visibility to existing technical debt. While it may be relatively easy to put tools in place to scan software code, doing so may identify an overwhelming number of technical debt issues. Instead, do the following:
• Configure existing issue tracking and management tools to include a technical debt category, so that these instances can be tracked and handled separately.
• During design and architecture reviews, explicitly capture technical debt, including remediation strategies.
• During development, empower developers to manually document as technical debt any issues that are hard to resolve and that require further tradeoff and root cause analysis.
• As part of regular release reviews, capture technical debt items, including remediation strategies. These technical debt items may include overarching concerns (e.g., end-of-life of software, hardware, operating systems) that will require substantial rework.
• Pair recurring examples of technical debt, which indicate overall technical risks, with risk management practices to ensure appropriate priorities are assigned and resources are secured.
• Stage 2: Establish goals. We found that there are successful DoD programs actively managing their technical debt. They clearly identified and related their technical debt items to Agile enabler stories, reviewed these technical debt stories regularly during sprints and other reviews, and prioritized these stories alongside other capability priorities. This approach allowed the teams to apply measures, such as
• percentage of resources allocated to managing quality and technical debt per delivery increment (e.g., sprint, iteration, gate, release)
• percentage of technical debt items in the backlog, which enables the program to visualize the technical debt that is carried
• Stage 3: Establish tooling and measurement environments. After an understanding of the level of existing technical debt and where it accumulates starts to emerge, programs can assess their existing tooling to manage technical debt and incorporate other tools as needed. Special attention should be given to assessing where these emerging tools may fit.

2. Update Existing Policy to Include Technical Debt Management Practices

Organizations wishing to get a handle on technical debt should continue to update their existing software development best practice guidelines and recommendations, policy, and guidance to include technical debt management practices. Updated bets practices, policy, and guidance should provide important information about how technical debt management can be instantiated in your organization. The best practices, policy, and guidance update should be based on lessons learned from real programs with mature practices. At a minimum, these updates should include the following:

• Employ both automated mechanisms (e.g., static code analysis and other tool scans) and manual mechanisms for identifying technical debt (e.g., opportunities for developers to add technical debt items to the backlog and tag them as technical debt when intentionally taking on debt or identify technical debt in design reviews).
• Track technical debt items on the backlog separate from other types of items, such as vulnerabilities and defects.
• Allocate appropriate effort during iteration capacity planning for resolving technical debt items, and they must ensure that this effort is protected from the pressure to focus on new capabilities.
• Include the effort for managing technical debt in program roadmaps to ensure that it is planned and that effort is allocated to it over time.
• To avoid having unintentional quality issues creep in that may result in technical debt, ensure software at least passes a code quality scan and unit tests before allowing check-ins when developing in a continuous integration and continuous delivery/continuous deployment (CI/CD) environment.

These activities will generate data that when analyzed will allow organizations to also identify common areas where most technical debt exist and areas where most likely to accumulate technical debt. Best practice recommendations and guidelines should include this information and evolved as needed.

3. Encourage Technical Debt Management Training

Training can help your organization institutionalize important technical debt practices by making the issue visible to more stakeholders and ensuring that these stakeholders are armed with the practices and strategies needed to manage technical debt effectively. The goal should be to ensure that technical debt management is part of the commonly expected baseline of software management for your organization.

Look for or institute role-based technical debt training for executives, program managers, and development teams. Providing targeted training for these roles will enable them to use consistent vocabulary, concepts, and practices. Some of this training can also be made available to contractors to ensure that everyone involved in a program uses the same vocabulary.

Depending on the targeted role, training content should include, but not be limited to, the following:

• explaining what technical debt is and reviewing representative examples,
• differentiating between causes of technical debt and actual technical debt that needs to be monitored within systems,
• the relationship between technical debt items and enablers, vulnerabilities, defects, and new capabilities,
• selecting best-fit tools and customizing their detection and reporting capabilities to a program’s needs,
• understanding the role of qualitative (e.g., developers’ perceptions of existing technical debt) and quantitative (e.g., mean time to resolution) measures,
• establishing data analysis pipelines from issue trackers and scan results,
• recognizing technical debt during design reviews, and
• conducting tradeoff analysis, which feeds into prioritizing which debt to resolve and which to carry forward.

4. Require Continuous Collection of Technical-Debt-Related Data and Metrics

The programs we studied that are managing technical debt successfully use metrics similar to those used for defect and vulnerability management, such as mean time to resolution, duration open, rate of recurrence, and density. By using metrics like these, successful programs map technical debt items discovered to the number of issues identified, prioritized, and addressed over a given delivery tempo. Likewise, each technical debt item is sized according to its scope and allocated to a sprint or iteration based on its scope and system context.

For example, our study noted that Google explored 117 metrics, including technical-system-quality-related metrics, as indicators of common areas of technical debt identified in its quarterly surveys (e.g., dependencies, code quality, migration, code degradation). Google’s analysis showed that no single metric predicts reported categories of technical debt. Additional evidence from our industry interviews and other research aligns with the conclusion that no single generalizable metric can be used to understand system specific leading indicators of technical debt.

Teams must select specific metrics for their specific system context based on their business priorities and technical challenges. Our study results also show that the design implications of technical debt are different in different contexts. For example, coupling and cohesion are two widely used system-modularity-related design metrics, where loosely coupled software is expected to be easier to modify. In systems where high performance is desired, however, compromises from modularity must be made. Robust technical debt management practices that do not rely solely on metrics will allow the clear expression of these tradeoffs and their implications.

We recommend programs use programmatic metrics for quantifying technical debt while using technical metrics to provide insights for system-level quality and design issues by contextualizing them based on program tempo, high-priority architectural concerns, areas of change, and refactoring costs. Other data, such as existing design concerns, rework and refactoring costs, and data about technical debt items, should also be collected and regularly analyzed.

5. Ensure Greater Access to Modern Development, Analysis, and CI/CD Tools and Practices

Several programs we interviewed for our study had already embraced DevSecOps approaches to incorporating code quality and security analysis tools (e.g., CheckMarx, Fortify, SonarQube, and CAST) into their environments. These tools, and others that assist with development activities (e.g., integrated development, automated code review, automated unit and integration testing), are essential to ensuring timely quality development and avoiding unintentional technical debt. They also enable the timely detection of implementation errors. However, these tools must be configured to ensure that false positives are minimized and high-priority issues are detected appropriately.

The Ongoing Challenge of Technical Debt

Technical debt creates added development costs and quality risks if not planned for and managed in any domain, not just in the DoD. When planned for, however, it can be an intentional investment that accelerates development. Like many development organizations, the DoD faces the constant challenges of limited resources and an accelerating demand for new features and capabilities. Software-reliant systems play an ever-growing role not only in our nation’s defense but in almost all areas of human endeavor. Development organizations must therefore manage technical debt to improve the modernization of software-driven capability that rapidly delivers value and stays ahead of emerging threats.