Software Engineering Institute

In an effort to determine how to make secure software development more cost effective, the SEI conducted a research study to empirically measure the effects that security tools—primarily automated static analysis tools—had on costs (measured by developer effort and schedule) and benefits (measured by defect and vulnerability reduction). The data used for this research came from 35 projects in three organizations that used both the Team Software Process and at least one automated static analysis (ASA) tool on source code or source code and binary. In every case quality levels improved when the tools were used, though modestly. In two organizations, use of the tools reduced total development effort. Effort increased in the third organization, but defect removal costs were reduced compared to the costs of fixes in system test. This study indicates that organizations should employ ASA tools to improve quality and reduce effort. There is some evidence, however, that using the tools could “crowd out” other defect removal activities, reducing the potential benefit. To avoid overreliance, the tools should be employed after other activities where practicable. When system test cycles require expensive equipment, ASA tools should pre-cede test; otherwise, there are advantages to applying them after system test.