When you’re developing software that will send people into space, you need to make sure that it works as expected. In safety-critical systems like these, the process of independent verification and validation (IV&V) is intended to ensure that a product meets its requirements and functions as intended. While most IV&V methods have been associated with the waterfall model of project management, an adoption of an Agile mindset and principles allows IV&V teams to be more aligned with contemporary software development processes and produce better outcomes. In this post, I discuss how Agile principles can work with IV&V processes, examples of how we put Agile IV&V into practice at NASA, and advice for transitioning to Agile.

IV&V

IV&V is a common practice in the public and private sectors as a form of risk mitigation or as part of compliance requirements. Typically, the process of verification asks, Are we building the product right? In other words, Is the implementation of the product consistent with the specification? Validation asks, Are we building the right product? In other words, Does the product as specified align with the actual mission need?

Crucially, the independent part of IV&V means that verification and validation are performed by analysts who are not part of the development team. These processes were developed to serve as a second set of eyes that could provide greater assurance of mission success. IEEE 1012, the industry standard for verification and validation, sets forth three parameters for independence: technical, managerial, and financial. If a team achieves these areas of independence, there is less chance of outside influence over the analysis and findings, removing potential organizational conflicts of interest and allowing the team to focus on the work at hand.

In practice, this approach can cause tension. The mandatory nature of IV&V in many government projects can create an us versus them mentality. Moreover, IV&V practices were developed at a time when waterfall project management methodologies were standard. In waterfall models, software is developed sequentially, with requirements gathered first. Developers then create the design, implement it, and test the software. IV&V would be undertaken throughout that process with specific review gates serving as milestones for analysis to be complete. With more software teams moving to Agile processes, however, IV&V analysts may find themselves out of step with the development process. As a result, teams may find that they aren’t receiving feedback at necessary points in the development processes, resulting in wasted work and feelings of frustration.

Agile Principles and Frameworks

Agile processes, by contrast, emphasize iterative and incremental development cycles. Originally proposed by a group of software developers in 2001, the Manifesto for Agile Software Development includes four values and 12 principles that undergird Agile thinking. These principles emphasize customer satisfaction, transparency, and flexibility—important values for creating strong, collaborative working relationships between IV&V and development teams and a large part of why Agile approaches have much to offer IV&V processes.

Many variations on Agile frameworks have emerged since 2001. Most include the concept of a backlog: a prioritized list of work that need to be completed by the team. Teams refer to the backlog to plan out work and allocate resources. Unlike waterfall approaches, development teams using Agile don’t need to plan out their work from the start to finish. By working on smaller timescales, they can adjust more quickly to problems uncovered along the way. This includes challenges identified in the IV&V process. Below are examples of a few common Agile frameworks and elements that have been helpful in incorporating Agile methods into IV&V.

Scrum

Scrum is a common Agile framework used in a variety of industries. The framework emphasizes teams working in short sprints, typically for two to four weeks in duration. These sprints are accompanied by a number of planning and check-in rituals to ensure continuous communication and collaboration within the team. These rituals include an initial planning meeting where the team defines the goal of the upcoming sprint and identifies any backlog items that might be included. Additionally, many groups will hold regular (usually daily) stand-up meetings where team members share progress and identify obstacles. After a sprint is complete, teams hold retrospectives to assess the work done and find areas for improvement.

Scrum also emphasizes self-managed teams. These teams have a high level of autonomy to develop their own plans and approaches to completing work. The goal of a self-managed team is to give members a sense of ownership and collective responsibility for outcomes, without work plans being imposed from the outside.

Scaled Agile Framework (SAFe)

SAFe is a set of processes that aims to facilitate Agile practices in larger teams. There are many challenges that larger organizations face when implementing Agile workflows, and SAFe addresses more complex development processes, such as the need to plan for a longer timescale with a planning interval (PI). The PI is a timeboxed sequence of development sprints followed by a planning iteration. PIs are typically somewhere between two to three months in length, though they may be slightly longer in government contexts. It is our experience that, in the more general case of Agile at scale, architecture plays a crucial role in success.

Agile for IV&V

With this background in Agile frameworks in mind, what does Agile look like in the IV&V context?

The first several priorities in the Agile Manifesto are to “satisfy the customer through early and continuous delivery” and to “deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” If we think of the deliverable in IV&V as assurance (rather than software or a product), we can understand the value of Agile IV&V: fast, reliable assurance that works on the cadence that developers do. This is analogous to the continuous authorization to operate (ATO) that is used across the Department of Defense (DoD), enhancing the security posture of our DoD systems.

These principles, however, often require a culture and mindset shift in IV&V. Analysts will need to move away from looking at the entirety of the software to working through smaller pieces, perhaps at the level of an individual capability or algorithm. Working in these smaller batches is a change from waterfall approaches, but it also allows teams to identify errors and incorporate fixes much faster.

Beyond the practical transition from waterfall project management styles, Agile IV&V also calls for transparency and increased communication. Scrum rituals can be helpful in building this culture. For example, sprint planning and retrospectives give the whole team insight into progress and allow team members to speak about what is (and isn’t) going well. Standup meetings that include discussion of IV&V activities increase transparency into day-to-day work and provide opportunities for quick feedback and alignment.

Agile IV&V at NASA

When I worked in project management at NASA’s Katherine Johnson IV&V Facility, I began implementing Agile IV&V. At the time, NASA was developing Orion, a multi-purpose crew vehicle designed for the Artemis missions that will eventually return astronauts to the moon. Orion’s software is complex, and the software developers had moved to a SAFe model, with major releases every three months. The IV&V analysts assigned to Orion were used to more traditional development models and had difficulty keeping up with the pace of development, leading to V&V findings being delivered to the software developer sometimes months out of phase.

Our team recognized that we needed to take a different approach. The SEI’s Will Hayes helped us understand Agile principles and how they could be used in the IV&V context. Will helped us define our objectives and incorporate Agile methods into our assurance work. We adopted several practices, including making use of a backlog, daily stand-ups, and retrospectives.

We needed to represent our work to our stakeholders to foster good communication between our teams and help us plan more efficiently. To visualize our progress, we created a heat map that showed our progress, areas of risk, and the overall project status.

Each of the heat map’s hexagons represents a specific capability our team was assessing for the Artemis I mission. By breaking up the work into individual capabilities, we brought in the Agile concept of working in small increments, giving us the flexibility to reprioritize and iterate as needed. The analysts started by identifying the key, top-level capabilities that were necessary for mission success. From there they independently identified the capabilities that would be necessary to make sure those top-level capabilities were successful. These capabilities were then scored for risk using a tool developed by NASA’s IV&V program. The colors on the heat map are those risk scores on a traditional risk scale: red indicates that a capability is at the highest level of risk, yellow means that there is some risk, and green indicates that the capability has the lowest level of risk.

We used this heat map and risk scores to help us prioritize and manage our backlog during our PI sessions, held three times per year. In these sessions, we planned work for the following four months, focusing on the highest risk capabilities first.

Once we put these Agile principles into action, we saw remarkable results that all stakeholders could easily understand. Breaking up the work into capabilities like we did, we could speak to all levels of the program in way that made more sense than just speaking in terms of issues found. From a technical perspective, the IV&V team was able to focus our work on the most high-impact problems and riskiest areas rather than trivial defects. We were also able to cut our delivery cadence from months to weeks, a time frame much more in line with the developers’ work. Simply put, we were able to produce better, more useful work faster than ever.

Better Culture, Better Outcomes

At NASA, Agile IV&V gave analysts a deeper understanding of the systems they were working on, as well as better communication with the program and development team. Now as an Agile transformation leader working at the SEI with Will Hayes, I am continuing this work with our DoD customers to help transition IV&V practices.

Moving to an Agile mindset is a culture change. It requires trust, psychological safety, and a willingness from the team to try something new. The good news is that Agile practices can help foster those shifts and make changes along the way if something isn’t working. These concepts can work for small teams or large teams as well. From an IV&V perspective, the key thing for our team was the backlog of capabilities that we independently built. Another huge piece for anyone moving into Agile are some of the rituals highlighted. These rituals can help build trust between development teams and analysts. As trust increases, teams will be more likely to communicate difficult issues. When teams can speak about problems candidly and without fear of reprisal, they are more likely to take calculated risks, which can find deeply hidden issues and lead to innovations in the way work is done.