Posted on by Emerging Technologiesin
In 2012, the White House released its federal digital strategy. What's noteworthy about this release is that the executive office distributed the strategy using Bootstrap, an open source software (OSS) tool developed by Twitter and made freely available to the public via the code hosting site GitHub. This is not the only evidence that we have seen of increased government interest in OSS adoption. Indeed, the 2013 report The Future of Open Source Software revealed that 34 percent of its respondents were government entities using OSS products.
The Carnegie Mellon University Software Engineering Institute (SEI) has seen increased interest and adoption of OSS products across the federal government, including the Department of Defense (DoD), the intelligence community (IC), and the Department of Homeland Security. The catalyst for this increase has been innovators in government seeking creative solutions to rapidly field urgently needed technologies. While the rise of OSS adoption signals a new approach for government acquirers, it is not without risks that that must be acknowledged and addressed, particularly given current certification and accreditation (C&A)techniques. This blog post will discuss research aimed at developing adoptable, evidence-based, data-driven approaches to evaluating (open source) software.
In this research, members of the technical staff in the SEI's Emerging Technology Center (ETC) explored the availability of data associated with OSS projects and developed semi-automated mechanisms to extract the values of pre-defined attributes. The challenges of applying data analytics to address real problems and of understanding OSS assurance align with the ETC's mission, which is to promote government awareness and knowledge of emerging technologies and their application, as well as to shape and leverage academic and industrial research.
Our research leveraged the "openness" of OSS to develop an evidence-based approach for assessing and assuring OSS. This approach, which focused on producing evidence in support of assurance claims, is based on generating artifacts and creating traceability links from assurance claims to those artifacts.
Beyond a Trust-Based Approach
If we think of traditional, "shrink-wrapped" software, we accept that the software is developed by and purchased from a vendor who delivers a product against specified requirements. The software comes with installation instructions, FAQs, and access to support via hotlines and websites. Generally speaking, there is a company or some kind of legal entity that stands behind the product.
With OSS development, however, multiple developers from different organizations (even independent developers) can contribute to the code base of a product, which may or may not be backed by a single legal entity. In some cases, developers include helpful information in the software repository; in other cases, users are on their own to get the software working in their environment. Specific functionality may be driven by the community of developers, or by a small core team.
Current methods to assess software (OSS or otherwise) are trust-based and rely heavily on expert opinion. For example, users may run experiments with the software in a controlled environment to determine whether or not it is safe to operate. When certifying and accrediting OSS or any software, however, the trust-based model is not valid for several reasons:
In this age of sequestration and other pressures on the expense of acquiring and sustaining software-reliant systems, government agencies can realize numerous benefits from a good OSS development and adoption strategy, including cost savings and increased flexibility in the acquisition and development of systems.
Foundations of Our Approach
In 1998, after Netscape published the source code for Netscape Communicator, Bruce Perens and Eric S. Raymond, founded the Open Source Initiative, an organization dedicated to promoting OSS. Since that time, a large number of OSS repositories have surfaced including Github, Launchpad, Sourceforge, and Ohloh.
In developing an approach, our team of researchers and software developers at the SEI wanted to create a tool that leveraged features of OSS, including the openness of code, development environment, documentation, and user community. Our aim was to design and develop an integrated, semi-automated software assessment capability that would allow an assessor to explore the evidence supporting an assurance claim.
The upside of the renewed interest in OSS adoption, both in government and industry, is that a wealth of data now exists within these repositories that provide insight into development of OSS as well as the code review and code committal process. Our aim with this research was to move beyond simple bug counts and static analysis and provide richer context for those charged with assessing software systems.
While no one measure or metric could provide an accurate assessment of software, we reasoned that several characteristics could provide acquirers with a more complete view of OSS assurance. During our study, we identified measurable characteristics that could be of interest, particularly if assessed in combination. For example, we examined complexities of the coding language used, test completion, and vitality or inertia of the project. Other characteristics that we evaluated included
Context is important. Using the data collected to help build an understanding of the development environment, developer activity, and user community commitment helps potential adopters get a better sense of the viability of the OSS project.
When we first began this research, we focused on identifying data that would allow us to make valid comparisons between identifiers of quality in different software repositories. We soon realized, however, that quality attributes really are context dependent. For example, OSS acquirers may place various levels of importance on whether software is updated during daytime hours by full-time employees or during evening hours by hobbyists. Instead of placing a value judgment on these variables, we altered our approach to identify characteristics such as the ones listed above that can be used by decision makers to determine relevancy and weighting.
As we progressed through the research, we also realized that OSS repositories were starting to explore ways to represent data relevant to the OSS projects in the repositories. For example, Github maintains a graphs section that highlights data, such as code stability, trends over time, and a separate punch card section that represents the volume of code commits over the span of a week. Another example involves Ohloh, which provides a side-by-side comparison along different parameters about the OSS projects.
Another challenge that we encountered surfaced after we began exploring the OSS repositories. We found that while there are many typical developer tools being used, they were all being used differently across different software projects. One example of this involved JIRA, a bug tracking software that offers users configurable fields. Another example can be found in the Apache Software Foundation project Derby, some bugs have fields for urgency, environment, issues, fix information, or bug behavior facts while others do not.
All indicators point to increased adoption of OSS. In November 2013, Federal Computer Week published an article detailing the adoption of OSS across the DoD. An article on OSS and government in Information Week earlier that month stated that "Federal agencies, looking for new ways to lower their IT costs, are exploiting open-source software tools in a wider range of applications, not only to reduce software costs, but also to tighten network security, streamline operations, and reduce expenses in vetting applications and services."
In the coming year, we will continue our work in data analytics and OSS assurance. We are interested in collaborating with organizations to
If you are interested in collaborating with us, please leave a comment below or send an email to firstname.lastname@example.org.
For more information about the SEI Emerging Technology Center, please visit
To read the article Has Open Source Officially Taken Off at DoD? by Amber Corrin, please visit
To read the article Agencies Widen Open-Source Use by Henry Kenyon, please visit
To read the article Army C4ISR portal uses open-source software for faster upgrades by William Welsh, please visit