Acquisition executives in domains ranging from modernizing legacy business systems to developing real-time communications systems often face the following challenge:

Vendors claim that model-driven engineering (MDE) tools enable developers to generate software code automatically and achieve extremely high developer productivity.

Are these claims true? The simple answer might be, "Yes, the state of the practice can achieve productivity rates of thousands of function points and millions of lines of code per person-month using MDE tools for automatic code generation." The complicated reality is that MDE consists of more than code generation tools; it is a software engineering approach that can impact the entire lifecycle from requirements gathering through sustainment. While one can make broad generalizations about these methods and tools, it is more useful to consider them in the context of a particular system acquisition. Aligning MDE methods and tool capabilities with the system acquisition strategy can improve system quality, reduce time to field, and reduce sustainment cost. On the other hand, when MDE methods and tools do not align with the acquisition strategy, using them can result in increased risk and cost in development and sustainment. This blog post highlights the application of MDE tools for automatic code generation (in the context of the full system lifecycle, from concept development through sustainment) and also provides a template that acquirers can use to collect information from MDE tool vendors.

Foundations of Our Work: AADL

Researchers at the SEI have been doing work in MDE for several decades. In particular, Peter Feiler and Julien Delange have been working with the Architecture Analysis and Design Language (AADL) modeling notation for use in real-time embedded systems, safety-critical systems, and other high-assurance systems. Their work has focused on analysis and an up-front assurance that the system will function as intended, with less emphasis on code generation. This latest SEI MDE effort--in addition to me, the team included Harry Levinson and Jay Marchetti--focuses more on code generation, specifically in the context of business systems with code generation benefits realized by the developer.

As detailed in our technical note, Model-Driven Engineering: Automatic Code Generation and Beyond, while certain domains can achieve extremely high productivity using model-driven approaches, it is important to realize that code generation is just one small piece of the entire software lifecycle. In software engineering, there is a tight coupling between the system domain (e.g., business system, command and control system, or avionics system), the methods used throughout the system lifecycle, and the tools used to support the chosen methods. Furthermore, government acquirers have the challenge of selecting contractors to develop their systems. This selection process includes evaluating the development team, the development methodology, and the tools in the context of the system acquisition strategy. Or, to state it more simply, in MDE, if you develop code using one tool it can be expensive to switch to another tool later in the software development lifecycle. In addition to acquiring code, therefore, software acquirers should also consider the tools needed to sustain the software. When an organization adopts a model-driven approach, it is also adopting a particular toolset and technology. These additional adoptions are of particular concern in the Department of Defense (DoD), where the focus is on acquiring and maintaining longer-lived systems. In many commercial contexts, there is less hesitation to rebuild a system from scratch.

Code Generation in Business Systems

Firms such as Gartner, Forrester, and IDC have focused their analyses of MDE technology on commercial IT developers and software providers. As stated earlier, our analysis focused on the unique acquisition concerns of the DoD and other federal agencies in which systems are acquired and maintained for longer periods of time. Specifically, we examined business systems since this is an area where code generation tools are having significant impact. This analysis included existing technologies and approaches used for commercial off-the-shelf technologies (COTS), and it investigated how those same principles can be applied to the acquisition of MDE tools. We used the PECA Method (Plan, Establish Criteria, Collect Data, Analyze Results) to organize an acquirer's technology assessment, and used an established risk framework to identify criteria within the overall process.

Acquisition Strategy Implications

An acquisition strategy specifies which artifacts and data rights to acquire, as well as which artifacts to evaluate at each program decision point. The acquisition strategy also defines the approach to identifying, managing, and mitigating program risks. The use of MDE for automatic code generation has several implications for acquisition strategy, including the following:

• artifacts, data rights, and licenses. Development tooling is usually not a significant concern when acquiring software-intensive systems, but it can be a significant concern when acquiring a system developed using an MDE approach, particularly when using MDE for automatic code generation. In MDE-developed software, the models are the primary development artifacts, embodying the software architecture design and component designs, and ultimately driving the automatic code generation. Ideally, all software sustainment and evolution will also use the MDE approach, which requires data rights and necessary licensing for the tools, models, and generated code. When only the automatically generated source code is acquired (without the models used to generate the code), then sustainment and evolution are more difficult because the automatically generated code is not structured for human readability and comprehension.
• design review scope and timing. The acquirer must review and evaluate appropriate artifacts at the right time in the acquisition cycle. For example, in an approach using MDE for automatic code generation, the software architecture documentation may consist of a subset of the code-generation model, along with accompanying documentation, to provide context and design rational. The software architecture should be evaluated early in the design process (as discussed by Bergey and Jones). The evaluation scope and criteria, however, may need to be expanded to account for the use of the model not just to represent the architecture for communications among stakeholders but also to directly generate the executing software.
  
  

Finally, reviewers may need to use the MDE tools to view the models--exporting the model into a generic format, such as portable document format (PDF) files, may not provide adequate visual resolution and the ability to efficiently navigate through the model. Tool availability and access to the network where the model is stored become issues that the acquirer must address in planning the evaluation.

• impact on program risk. While an MDE approach promises automatic code generation, improvement of cost and schedule, reduction of technical risk by enabling early analysis, and the ability to demonstrate capabilities and validate requirements by using executable models or rapid prototypes, it also introduces new risks, including the following (see our technical note for a more detailed accounting of risks introduced by an MDE approach to automatic code generation.):
  
  - a development-time dependency on the tool chosen to support the process. The chosen tool is used to create and modify the model, which is then processed to generate the code. Unlike traditional source code, which can be created and modified by many different tools, the state of the market for MDE tools is that, in most cases, a model can be edited and modified only by the tool that created it, and changing tools may require rebuilding the entire model.
  
  - cybersecurity assurance. As noted earlier, development time and run-time dependencies have several implications for cyber assurance. For example, as cybersecurity policy and practices evolve, the tool may generate compliant code (e.g., code that is compatible with required authorization mechanisms, access control policies, or encryption practices).
  
  - run-time portability. Portability concerns manifest as a desire to execute the automatically generated code in several environments, each comprising different hardware and software infrastructure. These concerns may also manifest as a desire to change the system's hardware and software infrastructure over time.
  
  - runtime performance. The automatically generated code must satisfy the system's runtime throughput, latency, concurrent request processing, and other performance quality requirements. While the use of an MDE approach may provide early confidence that these requirements can be met, if one or more of these requirements change, there is a risk that the automatically generated code may not satisfy the new requirement.
  
  - usability of generated user interfaces. In some system domains, such as business systems, the MDE tools may generate user interfaces as part of the automatically generated code. The generated user interfaces may support functions such as system configuration and administration, system monitoring, and end-user activities.

A Template for Collecting Information from MDE Tool Vendors

To help business system acquirers select and evaluate MDE tools, we have created a questionnaire template to use during the "Collect Data" step in the process, which can be downloaded here. Our accompanying technical note provides detailed guidance on how to interpret responses during the "Analyze Results" step.

To develop the template, we started by reviewing guidance about how to develop criteria for developing a tool based on your program-specific acquisition issues. We wanted to understand particular risk areas that may or may not be relevant for a program. We also wanted to understand how the use of MDE tools could help mitigate some risks but also introduce or increase other risks.

We also relied on earlier SEI research that created a risk taxonomy. We used that taxonomy to examine how MDE approaches can either help mitigate some of those risks or may introduce risks to a program.

Wrapping Up and Looking Ahead

Our premise through all of our analysis of MDE methods and tools is that it is impossible to make broad-brush statements that are true for all programs. There are many mitigating factors, including

• the goals of the program
• the context that a program is working in
• high priority objectives
• existing risks

Our analysis and risk taxonomy can help programs decide whether MDE approaches can help or hurt an organization, specifically whether a particular approach will fit with a program's risk profile and goals.

MDE provides the opportunity to reduce development costs, improve the quality of the software developed, and possibly increase the agility of the development process. Programs can realize these benefits only if the concept fits with their acquisition strategies. MDE tools that specialize on a particular type of system provide high productivity but solve only a very narrow type of problem. Our analysis found that the narrower the scope of an MDE tool, the more that tool is tied to a vendor.

We welcome your feedback on our work.

Additional Resources

You can learn about our research on MDE by reading the technical note Model-Driven Engineering: Automatic Code Generation and Beyond. The template for the accompanying questionnaire can also be downloaded from this site.