Recent research has demonstrated that in large scale software systems, bugs seldom exist in isolation. As detailed in a previous post in this series, bugs are often architecturally connected. These architectural connections are design flaws. Static analysis tools cannot find many of these flaws, so they are typically not addressed early in the software development lifecycle. Such flaws, if they are detected at all, are found after the software has been in use; at this point they are far more costly and time-consuming to address.

In our first post in this series, we presented a tool that supports a new architecture model that can identify structures in the design (based on an analysis of a project's code base) that have a high likelihood of containing bugs. Typically, investment in refactoring to remove such design flaws has been difficult for architects to justify or quantify to their managers. The costs of refactoring are immediate and up-front. The benefits have, in the past, been vague and long-term. And so managers typically refuse to invest in refactoring.

In this post, which was excerpted from a recently published paper that I coauthored with Yuanfang Cai, Ran Mo, Qiong Feng, Lu Xiao, Serge Haziyev, Volodymyr Fedak, and Andriy Shapochka, we present a case study of our approach with SoftServe Inc., a leading software outsourcing company. In this case study we show how we can represent architectural technical debt (hereinafter, architectural debt) concretely, in terms of its cost and schedule implications (issues of importance that are easily understood by project managers). In doing so, we can create business cases to justify refactoring to remove root causes of the architectural debt.

The Design Rule Space Analysis Approach

Our recent research has shown that defective files seldom exist alone in large-scale software systems. They are usually architecturally connected, and their architectural structures exhibit significant design flaws that propagate bugginess among files. We call these structures the architectural roots of bugginess. These architectural roots are the prime sources of technical debt in a project.

The practice of locating architectural debt, as it is practiced in industry, is typically an informal practice based on experience and intuition. For this reason it does not always successfully locate the architectural issues or design flaws that propagate bugs among multiple files.

This post presents an economic model for use of a design representation, known as Design Rule Space (DRSpace), and associated set of tools (Titan) that allow us to reverse engineer an existing code base, extract architectural design information, manipulate and organize that information, and then analyze that manipulated form of the information to discover architectural flaws. The approach we have developed is highly automated, which will save developers and researchers much-needed resources and time.

Economic Analysis: A Return on Investment for Refactoring

Removing the architecture roots of bugginess requires refactoring, but the benefits of refactoring have historically been difficult for architects to quantify or justify. Working with researchers from Drexel University's Computer Science Department, we recently applied our approach to identify and quantify such architecture debts in a large-scale industrial software project. Using the data collected from this case study we were able to provide convincing evidence of the economic benefits of refactoring.

As detailed in our earlier post, using the DRSpace approach and the Titan tool chain we have been able to analyze the important design structures in software projects. These structures are led by design rules: important interfaces or abstractions. Using this analysis, we can identify the buggiest areas of the project. Given this information, we can contrast parts of a project with zero design flaws and calculate the bug and change rates in these parts. Next, we can examine the bug and change rates in problematic areas of a project--those with design flaws. This analysis allows us to estimate the benefits that would accrue if these design flaws were removed. In much the same way that economic analyses extrapolate current economic data to estimate the future value of financial investments, we can estimate the future value of "investing" in software refactoring to remove architectural debt.

High-Cost and Mounting Problems with Technical Debt

The software development organization that we worked with in applying our approach was SoftServe Inc., a leading outsourcing company with more than 3,500 employees and 200 active projects in eight countries. The company, however, was facing high and mounting problems with technical debt in a project that we selected for our case study. Working with commercial tools such as SonarQube, Understand, and Structure 101, SoftServe developers and architects had been maintaining their system for nearly two years and were actively trying to improve the maintainability of the code base, remove dead and cloned code, and rationalize its architecture.

SoftServe architects had maintained a list of technical debt items in a web portal system that were of multiple types including the following:

• multiple code violations detected by SonarQube
• numbers of TODO and FIXME tags reported by Eclipse
• lack of reusability detected by code reusability scenarios

These technical debt items were not, however, providing value to SoftServe because they did not quantify the debts, nor did they indicate how to remove the debts or estimate the expected value of removing the debts.

Using the DRSpaces approach detailed in our earlier post, we identified instances of architectural flaws within these DRSpaces:

• Unstable interface. A leading file with a large number of dependents is buggy and has frequent changes.
• Modularity violation. Two or more files change together frequently in a project's revision history, but they have no structural relationship. This flaw is also called implicit cross-module dependency.
• Improper inheritance. A parent class depends on one of its subclasses, or a client of the inheritance hierarchy depends on both the parent class and one or more of the subclasses.
• Cyclic dependency. Two or more classes (or packages) depend on each other, and these dependencies form a cycle.

In our analysis we identified three DRSpaces that were particularly problematic, as shown in the figure below. We named these three DRSpaces after their leading files, Pear.java, Apple.java, and Bean.java (for the purposes of the paper all important file names were obfuscated). These three DRSpaces led clusters of 139, 158, and 165 files respectively.

The files in these DRSpaces were both highly buggy and their relations exhibited numerous design flaws. And no metrics or code analysis techniques could have found such flaws.

Estimating the Benefit from Refactoring

As detailed in our paper, as a basis for our estimates of the benefits of refactoring, we used existing project annual averages shown in cells B11-B13. An average file in this project experienced .33 defects annually (i.e., there were 265 defects affecting a total of 797 files) and 2.9 changes annually (i.e., 2,332 changes over 797 files) requiring an average of 169.95 lines of code to resolve (there were a total of 135,453 lines of code committed for the project's 797 files).

By contrast, the 291 distinct files in these three DRSpaces experienced about 238 bugs and 2,332 changes annually, a rate of .82 bugs per file and 8.01 changes per file.

For our economic analysis we made the assumption that the refactoring of the three DRSpaces would bring them down to project averages. This is, we feel, a conservative assumption for two reasons:

• The current project averages already include these flawed DRSpaces, which inflates the project-wide averages.
• It is likely that the refactoring could result in much better structure than the project average, since the average project file has not been refactored. Thus the refactoring could conceivably result in lower defects, changes, and committed lines of code for these three DR Spaces.

Thus we feel that using existing project averages as our target for improvement is very conservative. Our follow-on longitudinal study will allow us to test the validity of this assumption.

Based on this assumption, we calculated the expected benefit from refactoring these three DRSpaces to remove the design flaws. Cells L2-L4 list the expected number of defects that would affect each of the existing problematic DRSpaces--34, 34, and 10--assuming that the refactoring brought them down to project averages. Similarly cells M2-M4 and N2-N4 show the normalized expected numbers of changes and committed lines of code under the same assumption--that the refactored DRSpaces exhibit project average behaviors. The totals for the expected numbers of defects, changes, and lines of code are presented in cells L6, M6, and N6.

These calculations allow us to calculate the expected benefit from these refactorings. We calculate the expected benefit as the difference between the actual numbers of committed lines of code and the expected numbers of committed lines of code. These expected "savings" are given in cells L8, M8, and N8. We ignored the loss of time and reputation due to bugs that are avoided (L8) and focused purely on the lines of code that we expect the project will not have to commit, due to refactoring (N8). Our analysis revealed that the project can conservatively expect to save 24,808 lines of code annually by refactoring these three problematic spaces. Next we take the company average productivity numbers and use this to calculate the expected person-months of effort avoided by refactoring. This savings, 41.35 months of effort per year due to the proposed refactorings, is shown in cell N12.

The project can expect to save 41.35 person months of effort per year due to the proposed refactorings. Given that these refactorings are estimated to cost just 14 months of effort, the investment in refactoring is paid off in less than 6 months, and the project experiences a net benefit thereafter. Or, to put it in financial terms, the project can expect a 295 percent return on investment in the first year alone.

Wrapping Up and Looking Ahead

Architecture problems lead to quality issues. The approach that we have devised allows us to actually quantify the costs of architectural flaws. Using the Titan tool and the DRSpace-based analysis approach we can accurately pinpoint important design flaws that degrade the maintainability of a project. These flaws include:

• cyclic class dependencies
• cyclic package dependencies
• improper inheritance
• modularity violations
• unstable interfaces

The identification of such flaws allows us to assess architectural debt and, significantly, its economic impact and predict the estimated impact of repair strategies (through refactoring).

We welcome your feedback on this approach in the comments section below.

Additional Resources

The Titan software is being licensed by Drexel University, and is available for free for academics. For more information, please visit http://drexel.edu/commercialization/about/contact/.

Here is a list of our recent papers on DRSpaces. We will update links as the papers become available:

1. Q. Feng, R. Kazman, Y. Cai, R. Mo, L. Xiao, An Architecture-centric Approach to Security Analysis, Proceedings of the 13th Working IEEE/IFIP Conference on Software Architecture (WICSA 2016), (Venice, Italy), April 2016.
2. L. Xiao, Y. Cai, R. Kazman, R. Mo, Q. Feng, Identifying and Quantifying Architectural Debts, Proceedings of the International Conference on Software Engineering (ICSE) 2016, (Austin, TX), May 2016.
3. R. Mo, Y. Cai, R. Kazman, L. Xiao, Q. Feng, Decoupling Level: A New Metric for Architectural Maintenance Complexity, Proceedings of the International Conference on Software Engineering (ICSE) 2016, (Austin, TX), May 2016.
4. R. Mo, Y. Cai, R. Kazman, L. Xiao, Hotspot Patterns: The Formal Definition and Automatic Detection of Architecture Smells, Proceedings of the 12th Working IEEE/IFIP Conference on Software Architecture (WICSA 2015), (Montreal, Canada), May 2015.
5. R. Kazman, Y. Cai, R. Mo, Q. Feng, L. Xiao, S. Haziyev, V. Fedak, A. Shapochka, A Case Study in Locating the Architectural Roots of Technical Debt, Proceedings of the International Conference on Software Engineering (ICSE) 2015, (Florence, Italy), May 2015.
6. L. Xiao, Y. Cai, R. Kazman, Titan: A Toolset That Connects Software Architecture with Quality Analysis, Proceedings of the 22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE 2014), (Hong Kong), November 2014.
7. L. Xiao, Y. Cai, R. Kazman, Design Rule Spaces: A New Form of Architecture Insight, Proceedings of the International Conference on Software Engineering (ICSE) 2014, (Hyderabad, India), June 2014.