Posted on by Technical Debtin
By Rick Kazman Research Scientist Software Solutions Division
Recent research has demonstrated that in large scale software systems, bugs seldom exist in isolation. As detailed in a previous post in this series, bugs are often architecturally connected. These architectural connections are design flaws. Static analysis tools cannot find many of these flaws, so they are typically not addressed early in the software development lifecycle. Such flaws, if they are detected at all, are found after the software has been in use; at this point they are far more costly and time-consuming to address.
In our first post in this series, we presented a tool that supports a new architecture model that can identify structures in the design (based on an analysis of a project's code base) that have a high likelihood of containing bugs. Typically, investment in refactoring to remove such design flaws has been difficult for architects to justify or quantify to their managers. The costs of refactoring are immediate and up-front. The benefits have, in the past, been vague and long-term. And so managers typically refuse to invest in refactoring.
In this post, which was excerpted from a recently published paper that I coauthored with Yuanfang Cai, Ran Mo, Qiong Feng, Lu Xiao, Serge Haziyev, Volodymyr Fedak, and Andriy Shapochka, we present a case study of our approach with SoftServe Inc., a leading software outsourcing company. In this case study we show how we can represent architectural technical debt (hereinafter, architectural debt) concretely, in terms of its cost and schedule implications (issues of importance that are easily understood by project managers). In doing so, we can create business cases to justify refactoring to remove root causes of the architectural debt.
The Design Rule Space Analysis Approach
Our recent research has shown that defective files seldom exist alone in large-scale software systems. They are usually architecturally connected, and their architectural structures exhibit significant design flaws that propagate bugginess among files. We call these structures the architectural roots of bugginess. These architectural roots are the prime sources of technical debt in a project.
The practice of locating architectural debt, as it is practiced in industry, is typically an informal practice based on experience and intuition. For this reason it does not always successfully locate the architectural issues or design flaws that propagate bugs among multiple files.
This post presents an economic model for use of a design representation, known as Design Rule Space (DRSpace), and associated set of tools (Titan) that allow us to reverse engineer an existing code base, extract architectural design information, manipulate and organize that information, and then analyze that manipulated form of the information to discover architectural flaws. The approach we have developed is highly automated, which will save developers and researchers much-needed resources and time.
Economic Analysis: A Return on Investment for Refactoring
Removing the architecture roots of bugginess requires refactoring, but the benefits of refactoring have historically been difficult for architects to quantify or justify. Working with researchers from Drexel University's Computer Science Department, we recently applied our approach to identify and quantify such architecture debts in a large-scale industrial software project. Using the data collected from this case study we were able to provide convincing evidence of the economic benefits of refactoring.
As detailed in our earlier post, using the DRSpace approach and the Titan tool chain we have been able to analyze the important design structures in software projects. These structures are led by design rules: important interfaces or abstractions. Using this analysis, we can identify the buggiest areas of the project. Given this information, we can contrast parts of a project with zero design flaws and calculate the bug and change rates in these parts. Next, we can examine the bug and change rates in problematic areas of a project--those with design flaws. This analysis allows us to estimate the benefits that would accrue if these design flaws were removed. In much the same way that economic analyses extrapolate current economic data to estimate the future value of financial investments, we can estimate the future value of "investing" in software refactoring to remove architectural debt.
High-Cost and Mounting Problems with Technical Debt
The software development organization that we worked with in applying our approach was SoftServe Inc., a leading outsourcing company with more than 3,500 employees and 200 active projects in eight countries. The company, however, was facing high and mounting problems with technical debt in a project that we selected for our case study. Working with commercial tools such as SonarQube, Understand, and Structure 101, SoftServe developers and architects had been maintaining their system for nearly two years and were actively trying to improve the maintainability of the code base, remove dead and cloned code, and rationalize its architecture.
SoftServe architects had maintained a list of technical debt items in a web portal system that were of multiple types including the following:
These technical debt items were not, however, providing value to SoftServe because they did not quantify the debts, nor did they indicate how to remove the debts or estimate the expected value of removing the debts.
Using the DRSpaces approach detailed in our earlier post, we identified instances of architectural flaws within these DRSpaces:
In our analysis we identified three DRSpaces that were particularly problematic, as shown in the figure below. We named these three DRSpaces after their leading files, Pear.java, Apple.java, and Bean.java (for the purposes of the paper all important file names were obfuscated). These three DRSpaces led clusters of 139, 158, and 165 files respectively.
The files in these DRSpaces were both highly buggy and their relations exhibited numerous design flaws. And no metrics or code analysis techniques could have found such flaws.
Estimating the Benefit from Refactoring
As detailed in our paper, as a basis for our estimates of the benefits of refactoring, we used existing project annual averages shown in cells B11-B13. An average file in this project experienced .33 defects annually (i.e., there were 265 defects affecting a total of 797 files) and 2.9 changes annually (i.e., 2,332 changes over 797 files) requiring an average of 169.95 lines of code to resolve (there were a total of 135,453 lines of code committed for the project's 797 files).
By contrast, the 291 distinct files in these three DRSpaces experienced about 238 bugs and 2,332 changes annually, a rate of .82 bugs per file and 8.01 changes per file.
For our economic analysis we made the assumption that the refactoring of the three DRSpaces would bring them down to project averages. This is, we feel, a conservative assumption for two reasons:
Thus we feel that using existing project averages as our target for improvement is very conservative. Our follow-on longitudinal study will allow us to test the validity of this assumption.
Based on this assumption, we calculated the expected benefit from refactoring these three DRSpaces to remove the design flaws. Cells L2-L4 list the expected number of defects that would affect each of the existing problematic DRSpaces--34, 34, and 10--assuming that the refactoring brought them down to project averages. Similarly cells M2-M4 and N2-N4 show the normalized expected numbers of changes and committed lines of code under the same assumption--that the refactored DRSpaces exhibit project average behaviors. The totals for the expected numbers of defects, changes, and lines of code are presented in cells L6, M6, and N6.
These calculations allow us to calculate the expected benefit from these refactorings. We calculate the expected benefit as the difference between the actual numbers of committed lines of code and the expected numbers of committed lines of code. These expected "savings" are given in cells L8, M8, and N8. We ignored the loss of time and reputation due to bugs that are avoided (L8) and focused purely on the lines of code that we expect the project will not have to commit, due to refactoring (N8). Our analysis revealed that the project can conservatively expect to save 24,808 lines of code annually by refactoring these three problematic spaces. Next we take the company average productivity numbers and use this to calculate the expected person-months of effort avoided by refactoring. This savings, 41.35 months of effort per year due to the proposed refactorings, is shown in cell N12.
The project can expect to save 41.35 person months of effort per year due to the proposed refactorings. Given that these refactorings are estimated to cost just 14 months of effort, the investment in refactoring is paid off in less than 6 months, and the project experiences a net benefit thereafter. Or, to put it in financial terms, the project can expect a 295 percent return on investment in the first year alone.
Wrapping Up and Looking Ahead
Architecture problems lead to quality issues. The approach that we have devised allows us to actually quantify the costs of architectural flaws. Using the Titan tool and the DRSpace-based analysis approach we can accurately pinpoint important design flaws that degrade the maintainability of a project. These flaws include:
The identification of such flaws allows us to assess architectural debt and, significantly, its economic impact and predict the estimated impact of repair strategies (through refactoring).
We welcome your feedback on this approach in the comments section below.
The Titan software is being licensed by Drexel University, and is available for free for academics. For more information, please visit http://drexel.edu/commercialization/about/contact/.
Here is a list of our recent papers on DRSpaces. We will update links as the papers become available: