Software engineers face a universal problem when developing software: weighing the benefit of an approach that is expedient in the short-term, but which can lead to complexity and cost over the long term. In software-intensive systems, these tradeoffs can create technical debt, which is a design or implementation construct that is expedient in the short term, but which sets up a technical context that can make future changes more costly or even impossible.

Research has shown that if technical debt is not paid back in a timely manner, it correlates with greater likelihood of defects, unintended rework, and increased time for implementing new system capabilities in software. But does technical debt also correlate with an increase in security vulnerabilities? To answer this question this blog post describes research we conducted that tested the relationship between software vulnerabilities and technical debt.

Foundations of Our Work

Our rationale for conducting this research relies on the logic that technical debt often begins with design shortcuts, which are symptoms of development practices that can make vulnerabilities more likely, yet harder to find. Empirical evidence shows that defects and vulnerabilities do not behave similarly as systems evolve, especially in terms of likelihood of detection and the factors that affect the rate of discovery. Additional empirical research should therefore be conducted on vulnerability data to understand these behaviors.

Technical debt can have observable adverse consequences on software security, meaning that allowing debt to accumulate may be even more costly than expected. Some vulnerabilities may be inadvertently introduced as the result of technical debt. For example, a vulnerability may be fixed in one location but is not fixed in a similar duplicated code fragment or a programmer studying overly complex code may make a mistake when trying to reason about whether a dangerous corner-case condition is feasible or not. Alternatively, as we show below, technical debt can also be caused by addressing a vulnerability's symptoms rather than its root cause. Both these relationships motivate a better understanding of the complex relationship between software vulnerabilities and technical debt.

Our Approach

To test the relationship between technical debt and software vulnerabilities, we began by working with a data set from the Chromium open source project. Chromium is a complex web-based application that operates on sensitive information and allows untrusted input from both web clients and servers. We use Chromium as a representative test bed of typical technical debt issues and types of vulnerabilities. The Chromium open source project we used for this analysis was released as Version 17.0.963.46 on February 8, 2012. This release contained 18,730 files. From February 1, 2010 to February 8, 2012, there were 14,119 bugs reported as fixed.

Our approach involves the following steps:

Identify software vulnerabilities. We enumerated issues in the Chromium issue tracker that have a security label and classified each issue in terms of the Common Weakness Enumeration maintained by MITRE. For each issue, we also identified the set of files changed by commits that reference the issue.

1. Identify technical debt. In this step, we classified issues for technical debt and classified the type of design problem and rework based on the issue description. Finally, we looked for design flaws that co-exist in the same files changed to fix the issues labeled security.
2. Model the relationships between technical debt issues and vulnerabilities in the common artifacts they represent. In this step, we tested whether technical debt indicators correlate with the number of vulnerabilities reported. We then investigated how selected vulnerabilities were influenced by the correlated technical debt indicators.

Preliminary Findings

Preliminary findings from our analysis of the Chromium open source project are outlined in our paper Can Knowledge of Technical Debt Help Identify Software Vulnerabilities?, which we presented at the Workshop on Cyber Security Experimentation and Test (CSET 16). To identify vulnerabilities, we used the 79 issues labeled security. There are no such labels for technical debt, so we looked to the issues and code for insight. To classify issues, we applied a classification approach we developed to tag issues as technical debt based on developer discussions and reported 21 of the 79 issues as describing technical debt. To analyze source code files, we used the results of a study that analyzed the same Chromium data set and reported 289 files associated with design flaws that can be detected in the code.

We will use Issue 10977: Crash due to large negative number as an example to illustrate the data we analyzed to prioritize security issues from a design perspective. Getting to the root cause of an issue helped us identify related vulnerabilities in the backlog of issues and prevent new symptoms from surfacing in duplicate vulnerability reports. This example illustrates that such analysis necessitates using code, issue trackers, and commit history in concert.

In Issue 10977, evidence of an integer overflow vulnerability can be traced to a design concern with external dependency. This vulnerability is an occurrence of CWE-703: Improper Check or Handling of Exceptional Conditions. Developers also discuss the consequences of alternative solutions to motivate looking beyond the local fix to the root cause:

"We could just fend off negative numbers near the crash site or we can dig deeper and find out how this -10000 is happening."

"Time permitting, I'm inclined to want to know the root cause. My sense is that if we [only] patch it here, it will pop-up somewhere else later."

Finding: When software developers address security issues, they use concepts related to technical debt, such as getting to the root cause, understanding the underlying design issue, recording symptoms where changes are taking longer than usual or problems are reoccurring, predicting consequences of immediate patches over the longer term, and building evidence for a more substantial fix to address security issues.

Technical debt indicators include the number and type of design flaws, the number of traditional bugs, the number of bugs labeled security, and the lines of code that change to fix a bug (bug churn) during development. Table 1 shows the results of the study referenced above that computed the Pearson correlation coefficient between design flaws and number of bugs, bug churn, number of security bugs, and security churn for the issues and files in the data set we studied. The issues labeled security (security bugs) and the number of changes required in the corresponding files (security churn) show a correlation with design flaws.

Our further analysis shows that for three of the four types of design flaws--modularity violation, clique, and improper inheritance--files with vulnerabilities are also more likely to have design flaws. The more types of design flaws a file is involved in, the higher the likelihood of it also having vulnerabilities.

Finding: We see evidence of correlations between vulnerabilities and technical debt indicators, such as design flaws and code churn. Files with vulnerabilities also tend to have more code churn.

Returning to our example of Issue 10977, we see from the developer comments in the issue tracker that it took some time to analyze the problem. Three users submitted reports of the crash, which were eventually merged into a single issue. Developers posed a local fix to the related files involved in the integer overflow that caused the crash. But treating issues one at a time can lead developers to create a localized patch for each similar issue. Developers were aware of these matters and expressed their concerns. Weeks later, they noted additional reports of crashes:

"There have been 28 reports from 7 clients ... 18 reports from 6 clients."

"Hmm ... reopening. The test case crashes a debug build, but not the production build. I have confirmed that the original source code does crash the production build, so there must be multiple things going on here."

Apparently the root cause of the problem was not found and fixed. Defective files seldom exist alone in large-scale software systems. Instead, they are usually architecturally connected, so a fix to one source file may cause a problem in another part of the system.

In our data sample, we classified multiple issues related to integer overflow as technical debt. We traced the design cause of several of these issues to an external package used by Chromium whose API calls inject an out-of-bounds number resulting in crashes related to integer overflows. Our analysis supplemented developer knowledge recorded in the issue tracker and detected that one of the files participates in design flaws of cross-module cycles and improper hierarchy that violate architecture principles. The improper hierarchy also has a causal relationship with bug churn and provides additional evidence of technical debt.

Knowing the root cause, the developers can fix the problem once at the design source, rather than patching the files where the API calls create the crash only to see the problem resurface in other files where more API calls create more crashes.

Finding: A time-consuming relationship between vulnerabilities and technical debt is tracing the vulnerability to its root cause when it is caused by technical debt.

Wrapping Up and Looking Ahead

The goal of this preliminary analysis was to answer the question "Are software components with accrued technical debt more likely to be vulnerability-prone?" Due to the small number of security issues in the Chromium sample, correlating raw numbers was not particularly useful. The preliminary analyses presented here are based on differences between files linked to vulnerability issues and those that are not. Measurement requires quantitative static analysis of anomalies in code and qualitative issue classification of design consequences. Combining evidence can provide context and help prioritize the number of issues generated from static analysis in the effort to reduce the number of false positives and focus on what is important.

Refining these conclusions and identifying the most useful predictors of technical debt require more analysis on a larger data set. We will look further into the characteristics of the data to answer the following questions: Are some forms of technical debt more closely related to vulnerabilities than others? What types of vulnerabilities correlate with technical debt?

We will experiment with the modifiability and security taxonomies to understand which taxonomies are feasible and supply a useful level of insight. Ultimately we would like to codify known sources of security-related technical debt as design flaws that tools can analyze for with increasing accuracy and reduced manual effort.

This study is part of a wider SEI effort on technical debt. If you would like to collaborate on a study of managing technical debt, please get in touch.

Additional Resources

Read about our recent work on managing technical debt in the following publications:

• Avgeriou, P., Kruchten, P., Nord, R., Ozkaya, I., and Seaman, C. 2016. Reducing Friction in Software Development. IEEE Software 33, 1 (2016), 66−73.
• Bellomo, S., Nord, R. L., Ozkaya, I., and Popeck, M. Got Technical Debt? Surfacing Elusive Technical Debt in Issue Trackers. Mining Software Repositories 2016, co-located with ICSE 2016, in Austin, Texas, May 2016.
• Ernst, N., Bellomo, S., Ozkaya, I., Nord, R. L., and Gorton, I. Measure It? Manage It? Ignore It? Software Practitioners and Technical Debt. In Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 50−60. ACM, 2015.