As noted in the National Research Council's report Critical Code: Software Producibility for Defense, mission-critical Department of Defense (DoD) systems increasingly rely on software for their key capabilities. Ironically, it is increasingly hard to motivate investment in long-term software research for the DoD. This lack of investment stems, in part, from the difficulty that acquisitions programs have making a compelling case for the return on these investments in software research. This post explores how the SEI is using the Systems and Software Producibility Collaboration and Experimentation Environment (SPRUCE) to help address this problem.

Decades of public and private research investments--coupled with the inexorable growth of globalization and connectivity--have commoditized many information technology (IT) products and services. For example, commercial off-the-shelf (COTS) hardware and software is now produced faster, cheaper, and generally at a predictable pace. During the past two decades, users and developers of IT systems have benefitted from the commoditization of hardware and networking elements. More recently, the maturation and widespread adoption of object-oriented programming languages, operating environments, and middleware is helping commoditize many software components and end-system layers.

Due to this IT commoditization trend, acquisition professionals, senior leaders, politicians, and funding agencies often assume new software innovations will continue to appear at a predictable pace, and that the DoD can benefit from these innovations without significant investment in software research. While mainstream IT systems may not need this investment, mission-critical DoD systems--particularly at the tactical edge--cannot. Without sustained investment in software research, therefore, the DoD is in danger of "eating the seed corn" and reaching a complexity cap that will make it harder to succeed in an era of budget cuts and other austerity measures.

Challenges to Effective Software Research Impact

One challenge to motivating investment in software research is presenting a convincing pathway for how sponsored research finds its way into practice. The underlying problem for the DoD is the ad hoc and often serendipitous nature by which members of the software community (including academic researchers, defense contractor software architects and developers, DoD acquisition program and research sponsors, as well as commercial tool vendors) collaborate to identify, develop, test, and transition promising software technologies. This lack of systematic collaboration by various groups in the software community outlined above has created a dysfunctional-- yet all-too-common--situation whereby DoD programs cannot find software technologies that meet their needs, regardless of their inherent promise. As a result, across the DoD acquisition programs repeatedly encounter problems developing, validating, and sustaining software. To exacerbate the problem, the "landing path" for software technologies is typically not DoD program engineers, but organizations (such as commercial vendors or standards bodies) responsible for maintaining the technology. These organizations are often not structured or motivated to leverage the results of advanced research projects effectively.

For example, DoD software researchers have historically received funding for research programs of approximately three years in duration. These programs involve creating a project plan, building teams, working on technologies, generating and evaluating prototypes, and writing papers to publicize the work. Throughout this period, there is typically great enthusiasm for the project from the technical community. Once the program ends, however, the community often disbands and the project descends into the "valley of disappointment," a phenomenon in which researchers struggle to transition their prototypes to the DoD acquisition community, while the practitioners are equally frustrated with not being able to apply research results to practical problems.

Getting stuck in the "valley of disappointment" is a common problem in technology research and development projects, as evidenced by Geoffrey Moore's book Crossing the Chasm. Moore presents this problem from a venture capital perspective: a group of researchers develops a technology and identifies some early adopters, but struggles to transition from the early-adoption to majority-adoption phase. Some reasons for this valley are that researchers are often required to work on abstracted problems because that's all that they can access and acquisition professionals don't have the luxury of transitioning "science projects."

Crossing the "Valley of Disappointment" with SPRUCE

To address the challenges describe above, the Assistant Secretary of Defense Research & Engineering Enterprise (ASDR&E), through the Air Force Research Lab (AFRL), funded researchers at Lockheed Martin Advanced Technology Laboratories, in partnership with Booz Allen Hamilton, Vanderbilt University (where I worked on SPRUCE before joining the SEI), Drexel University, Virginia Tech University, Lockheed Martin Aeronautics, and Raytheon to create the Systems and Software Producibility Collaboration and Experimentation Environment (SPRUCE). SPRUCE is a collaborative set of web-based services that matches DoD challenge problems with the methods, algorithms, tools and techniques developed by researchers. One way to think about SPRUCE is as an "eHarmony" portal for researchers that unites domain experts from the DoD acquisition community who face concrete technical challenges with software researchers who can solve them.

For example, acquisitions professionals could be searching for an approach that will allow them to run legacy code on a multi-core platform or an algorithm that minimizes the amount of processors and network bandwidth in an avionics system. SPRUCE refers to these people as the "problem providers," who post challenge problems into the SPRUCE portal. Conversely, researchers are "solution providers" who use SPRUCE to post candidate solutions to available challenge problems.

SPRUCE allows problem providers to explain their needs in a structured way--along with representative data sets and reproducible experiments--so that solution providers from software researchers can decide if they have methods or technologies that would make an impact on the posted problem. If researchers operate in an open environment (which is typical at universities), they can post their solution on the SPRUCE portal. If researchers operate in a closed environment (which is typical at companies), they can contact the problem providers directly and discuss options for collaboration.

SPRUCE addresses many problems facing DoD software researchers:

1. It allows researchers access to real-world problems and realistic data sets. Even if the problem providers have anonymized their problem (for example, by removing proprietary information), it still represents an actual challenge faced by the DoD. Once researchers demonstrate that their solutions work on abstracted problems that are relevant to a particular domain and derived from real-world scenarios, it is easier to convince the original problem providers that the results are ready to be applied in practice.
2. SPRUCE facilitates healthy competition among research groups. For example, SEI researchers may believe they have the most effective techniques and tools for detecting similarity in malware, but SPRUCE allows them to compare their results against techniques and tools devised by other researchers for a common data set. In addition to the obvious competitive benefits, this approach also allows a better collaborative evolution of the solution by incorporating the best parts from each approach into a refined approach.
3. SRPUCE helps researchers locate sources of funding because it provides an immediate way for them to showcase their results in a forum that has an audience (the challenge problem providers) interested in solutions to real problems. Over time, researchers will populate the SPRUCE repository with their solutions, providing a way for them to find audiences for new funding and additional collaborations on real problems.

In its four years of funding, SPRUCE has focused primarily on capturing DoD challenge problems and helping DoD software researchers collaborate more effectively with other members of the DoD software community. SPRUCE has also influenced the National Science Foundation (NSF), which recently created the Cyber Physical Systems Virtual Organization (CPS-VO) community as a web portal for problem providers in cyber-physical systems. NSF-funded researchers use CPS-VO to post challenges and to work collaboratively to solve challenge problems with their colleagues around the world.

SPRUCE represents part of the trend towards more collaborative research and development among scientists and engineers. For example, UAVForge.net is attempting to use crowd sourcing to go from concept to fly-off of air vehicle designs in under six months. Likewise, a recent article from Wall Street Journal titled The New Einsteins Will Be Scientists Who Share proclaims "publicly funded science should be open science." Portals like SPRUCE help move researchers and practitioners from isolated pockets of collaboration to mainstream adoption.

Using SPRUCE to Guide SEI Research

At the SEI, we are using SPRUCE to showcase our solutions and, more importantly, to capture real-world challenge problems from our stakeholders. For example, the SEI hosted a workshop in August 2011 that brought together researchers and problem providers from Lockheed Martin, Boeing, AFRL, Carnegie Mellon University and Virginia Tech to elicit guidance for our work in real-time scheduling and currency analysis for cyber-physical systems. The workshop participants provided the SEI with challenge problems from avionics domain experts to ensure research we are doing addresses real DoD problems. As a result of this workshop, the problem providers populated the SPRUCE database with problems that SEI technologists will use to guide our future work. We are in the process of conducting challenge problem workshops for other software research projects at the SEI to ensure we continue to work on relevant problems that have high impact on DoD operational needs.

SPRUCE also allows us to continually improve our metrics and measures of success. As a federally funded research and development center, the SEI is often requested to substantiate data and success criteria. Problem providers--who by their nature have a close connection to real-world problems--help define the success criteria. This approach allows an external party, like a DoD contractor, to define the success criteria for SEI researchers who then work to achieve those criteria. At the same time, it showcases the solutions that SEI technologists have developed for various technologies, such as multi-core platforms.

In a commoditized IT environment, human resources are an increasingly strategic asset. In the future, therefore, premium value and competitive advantage will accrue to individuals, universities, companies, and agencies that continue to invest in software research and who master the principles, patterns, and protocols necessary to collaboratively integrate commoditized hardware and software to develop complex systems that cannot yet be bought off-the-shelf. Success in this endeavor requires close collaboration between academia, industry, and government. The SPRUCE portal described above helps to facilitate this collaboration by bringing key stakeholders to the table and ensuring that government investments in software research have greater impact on DoD acquisition programs.

Additional Resources:

For more information about the SPRUCE portal, please visit
www.sprucecommunity.org/default.aspx.

To read about the need to motivate greater DoD investment in software research, please see the National Research Council's Critical Code: Software Producibility for Defense report available at
www.nap.edu/openbook.php?record_id=12979&page=R1.