A group of researchers recently set out to try to identify the human factors that play a role in software security. The researchers identified factors such as team size, level of focused attention, physical separation of developers, time of day when code is written, and hours worked per day. The researchers examined each factor and its relationships to vulnerabilities found in the software.

In a Dark Reading article detailing their findings, the authors noted the following:

Understanding human factors is especially important as we develop new models for remote work. Managers could use human factors research to shape a remote work environment with fewer sustained work hours and fewer concurrent projects that in turn fosters more secure development practices.

This is just one example of the growing recognition that people issues, once considered a soft topic, can greatly influence the outcome of a software engineering project. We are seeing these issues crop up in our own work with federal programs, specifically within the Department of Defense as organic software sustainment organizations are increasingly tasked with engineering and developing the software capabilities of a system. These issues and the resulting struggles that these teams face motivated us to share lessons through a series of blog posts to help other programs avoid repeating the same mistakes.

Our initial post gave an overview of the issues that the DoD should address to make this transition successful. A follow-up post examined process concerns, and this post focuses on people concerns and provides recommendations for supporting software teams transitioning from sustainment to engineering.

As we have outlined in previous posts, the cost of externally contracted software acquisition, as well as costs associated with the proprietary nature of end-state software without clear government-use rights, is driving the DoD to develop organic software engineering capabilities. These capabilities have been largely outside the scope of the DoD’s traditional software sustainment organizations. To avoid the concerns of externally contracted software, the DoD increasingly supports the creation of government-managed labs and centers, as well as the transformation of organic sustainment organizations into engineering organizations that expand traditional government software capabilities.

In this and related posts, we set the context, outline the problem, and then offer suggestions for addressing these issues.

Overcoming the “Just Works” Mindset

Context. In both the sustainment and engineering of long-lived systems, software teams often find a solution that “just works” and apply it repeatedly and blindly, without analyzing its fit with each new usage. For example, software developers often forgo the bridge pattern and the pointer to implementation (PIMPL) idiom in programming languages that lack garbage collection and instead forgo refactoring to use those patterns because an existing implementation already “just works.”

Problems. Software sustainment groups often churn out fixes that are narrowly scoped, without consideration for the global view. Effective software solutions require collaboration, leverage experience, and provide analysis of the architecture and design applied to meet the requirements.

The “just works” anti-pattern arises from multiple factors:

• Time pressures—Organizations and individuals naturally emphasize product functionality over other qualities needed to create a viable, long-lived solution. Organizations devote insufficient time for software developers to identify and implement the most effective solution(s) for a given problem. In the production of code for long-lived systems, the code’s usage and lifetime are not considered. This impatient approach is problematic for both sustainment and engineering organizations.
• Narrow mindset—When done naively, Agile forces developers to focus on the short term. When working on software, many developers focus on implementing a series of small solutions. While this approach is consistent with Agile practices, the result may not scale to support long-lived and maintainable products.
• Lack of experience—Inexperienced developers have not seen the benefits of incorporating more robust and reusable patterns in their daily work.

Solutions. Engineering the most effective solutions requires collaboration and interaction with others to create interfaces and set expectations. If any group is to scale their work to the system-of-systems scope, which requires a holistic perspective on global architecture and design considerations, developers need to reorient to a broader mindset beyond “just works.” We have applied several methods and techniques to help overcome the “just works” anti-pattern:

• Learn and apply relevant patterns and idioms—Although techniques like the bridge pattern and the PIMPL idiom can be tricky to implement, over time they often provide distinct advantages in maintainability, testability, and reusability. Leveraging these design patterns and idioms enables developers to create software that can evolve over the lifecycle of a system to meet product requirements. As Martin Reddy points out, use of the bridge pattern and, more specifically, PIMPL in implementation to avoid exposing private data and implementation details maintains a strong separation between an API’s interface and its implementation.
  
• Create a defined context for your code effort—Code lifetime is only one of many factors that creates a context for expectations on code quality. For example, SEI’s Quality Attribute Utility Tree (QAUT), from its Architecture Tradeoff and Analysis Method (ATAM), can be used to define and create the expected context for a software development effort. In defining a context, scenarios and the expected results are tied to the mission drivers of the effort. This creates a mutually agreed-upon environment and mission for all the stakeholders.
  
• Learn and apply modern quality assurance practices and tools—Going beyond the “just works” anti-pattern requires an investment in the future of a software effort, as well as a willingness to recognize and accept one’s own skills, mitigate one’s shortcomings, and build a better relationship with a team.

Organizations that recognize the lifetime of a software effort extends well into the future often use modern software development and QA practices, test-driven deployment (TDD), and continuous integration and continuous delivery, and they reduce technical debt by refactoring.

Collaboration is the key to accomplishing these activities. Architecture, design, and process infrastructure tasks are best accomplished in a coordinated team effort. As we will explore in the next section, team members accustomed to going it alone need to embrace collaboration practices to achieve success.

Moving from Individual Contributors to a Collaborative Mindset

Context. Engineering a new, complex software system requires collaboration. Software sustainment organizations are driven to shrink defect lists, an often-solitary endeavor that creates strong individual contributors focused on diagnosing, analyzing, and fixing defects in relative isolation.

Problem. Engineering large, complex systems and systems-of-systems cannot be done in isolation. Engineers must create interfaces among software components and subsystems and set expectations for how the systems may behave when interacting. Groups of strong individual contributors accustomed to working problems alone may not easily transition to this collaborative setting.

Solutions. Team-building exercises can be viewed as soft by some organizations. However, their value has been proved through research showing that people are more likely to share an idea and accept feedback from someone with whom they regularly engage on topics not directly related to work (movies, games, sports, etc.). Trust falls aren’t the only—or even the most—effective means to encourage collaboration. While not often discussed in software engineering literature, even simple activities such as going out to lunch with colleagues can make an impact.

COVID-19 has created social distancing protocols that mandate remote work for many employees. The extra effort needed to stay connected to team members is more widely understood. Savvy managers will have remote members at the top of their checklists when making decisions that affect their teams, demonstrating that priority to the entire team in everyday actions. Virtual brownbag lunch situations and happy hours have enabled connections and team building, and have generally positive effects on well-being.

In our experience, fellow software engineers are more likely to take constructive feedback from someone with whom they are on friendlier terms. A better relationship helps a team member understand that feedback from someone they respect and admire stems from a simple earnestness to improve. Feedback in such an environment allows engineers to recognize other people’s experience and expertise and leverage that in their own development.

DoD Code at Large vs. General Issues

Context. DoD systems are often extremely large and complex, and, in the interest of fostering innovation through competition, will have contributions from multiple contractors, subcontractors, and organic government teams.

Problem. The array of contributors that feed into a large software effort create a number of potential concerns. Software delivered from contributors will be of varying quality, style, and structure. The integration into a larger system creates potential for complex issues. Engineering organizations that feed into the DoD do not always deliver systems that include comprehensive architecture and design considerations.

Defense software engenders contention for access to simulation environments, buildouts of hardware, and so on for a particular platform. Because they exist as a single instance, environments and buildouts can be difficult to tie into shared, modern DevSecOps pipelines. Software efforts need to have simulators and emulators, and all come with trade-offs in fidelity. Ongoing research is attempting to enable higher fidelity simulations and hardware-in-the-loop (HWIL) into DevSecOps-style continuous integration (CI) efforts.

Solution. Accept that large systems of systems may require extra people power to integrate, test, and maintain. Over time, some engineers develop a deep, organic understanding of a system. That knowledge, combined with encouraging leadership, allows for new ideas to streamline and automate more of the process. This feedback loop is how iterative methodologies increase the velocity of development efforts. Iterative methodologies rely on incremental improvement to a product and the product development process to more rapidly deliver software. If an organization is too rigid about changes to the latter, the goal of a more streamlined, rapid, and innovative product lifecycle cannot be achieved.

Government Employment Complexities

Problem. In organizations where professional-development opportunities are limited, team members’ working problems alone will not develop and enhance their skills and may not advance their careers.

Context. Humans do what we are incentivized to do. Software-sustainment organizations often emphasize short term tasks and goals (e.g., bug fixing). A narrow focus on immediate needs creates barriers to long term viability and is the major contributor to the “just works” anti-pattern.

Problem. Incentives may be misaligned between maintenance and engineering organizations. One consideration is the pay-gap issues within the DoD. Government employees are often at a distinct disadvantage on the salary scale from their counterparts who work in industry. This is yet another factor that contributes to an environment where software systems with outdated, inelegant solutions require more work hours (and possibly overtime) throughout their lifecycles. These types of systems do not ultimately allow the government to field new software systems efficiently.

Solution. Incentivize teams through alignment with industry pay practices. Organizations can create rewards for teams and individuals who improve product iteration latency through use of reuse, COTS, open-source, and open standards.

The Importance of People Issues

Fortunately, an increasing number of researchers and practitioners are realizing the importance of addressing people issues within software engineering. A group of researchers at the University of Maryland’s Department of Information Systems recently published a paper proposing the inclusion of an area into the Software Engineering Body of Knowledge that would focus on people issues (i.e., organizational, individual, and methodology-related aspects). The authors wrote

It is important to know more about these issues because an increasing amount of empirical evidence points us in this direction. Also, we believe that there is a need to change what is made available to the practitioners today in terms of software engineering knowledge, and make them realize the importance of these issues and incorporate them into everyday project development and management practices.

As we have stated previously, our intent with these posts is not to place blame on any group, but to urge DoD leaders to address the underlying incentives and long-held foundations that create disadvantages for sustainment groups when transitioning to engineering, which is our future reality.

A future post will outline technical issues and offer recommendations for teams transitioning from sustainment to software engineering.