Over the past several years, operators (who are users of systems) have had increasing opportunities to provide direct, regular feedback to developers during the conceptual and developmental stages of the software-development lifecycle. This increased developer and operator communication has led to a more user-centered approach to solving problems. With the new DoD Software Acquisition Pathway (SAP), the use of operator-feedback sessions is likely to increase exponentially in government software-intensive acquisition programs. In this blog post, we introduce a design approach that considers operator feedback and leverages feedback sessions, leading to richer experiences for operators and developers who attend them.

The SAP explicitly calls out the need for a user agreement and value assessments. The user agreement captures the commitment made between the program office (or development organization), the sponsor (funding source), and the end-user operators to ensure proper resourcing of user involvement as frequently as possible throughout the development process. The value assessment is conducted at least annually to determine whether the operational value of the software being developed from the end users’ perspective is commensurate with the resources invested. These requirements will result in an increase in interactions between developers and operators. While the increased interactions are intended to increase the government’s ability to deliver capability at the speed of relevance, such interactions can be challenging and should be monitored to prevent waste and delay.

The use of operator-feedback sessions in industry and government is not a new practice. The adoption of this technique has been increasing as government programs embrace Agile and DevSecOps methodologies, practices, tools, and techniques, which provide opportunities for developers and operators to get and give feedback about development projects. While operator-feedback sessions are used in both industry and government, we have observed a government mindset that must be considered when planning and conducting feedback sessions in government settings.

Most civil servants and members of the U.S. Armed Forces have an enduring sense of duty to the people and country they serve that compels them to apply themselves diligently and behave with integrity. Although these admirable traits help them excel at their jobs, they can also lead to a mindset that discourages them from questioning the tools and processes presented to them.

While often attributed to the Marines, the adapt and overcome mindset is prevalent across the Department of Defense and other government departments and agencies. Over time, government employees—more often than in private industry—become reconciled to the tools they are given, which can make it hard for developers to solicit feedback from these operators about tools and processes.

When designing and executing software projects, developers rarely, if ever, have unlimited time and resources. Due to these limitations, it is critical that developers or software architects identify (1) what options are the best to invest in, and (2) what requirements are most critical to address. Not gathering this information can lead to development failures and systems that may not work as expected or be adopted broadly.

In designing systems, developers should take system operators (i.e., users) into consideration, immersing themselves in the operators’ experience and understanding their perspective. This user-centered approach is known as human-centered design. It provides critical insights and allows developers to design and create functional systems that meet user needs, saving time, rework, and resources.

It’s helpful for program offices to ensure that developers include operators during most stages of soft-ware development. In particular, including and securing their buy-in during early exploratory phases of the development lifecycle can affect the end product. That said, even though there are benefits to including operators during software development, there are also associated costs. For example, removing operators from day-to-day tasks to get their input can result in loss of operator productivity, making it potentially harder for their work units to absorb the loss of personnel availability.

For these reasons, the time operators spend on development processes must be well managed and appreciated. We have often observed operator fatigue caused by too many interactions that didn’t yield positive, actionable results.

During feedback sessions, operators frequently express the wish to share their knowledge and contribute to a better product. However, their contributions are often drowned out by voices louder than theirs or by people who outrank them, leading to the operators feeling disengaged and participating only begrudgingly in future efforts. Unfortunately, this disengagement can be a barrier to critical thinking when discussing the concepts or concerns that directly affect the software being designed and developed.

Given the government mindset and associated cost of the operators’ time, we have developed the following recommendations for getting the most out of operator-feedback sessions and otherwise gathering operator input for development projects.

Analyzing and documenting system workflows and building a set of functional requirements can provide perspective for operators attending a feedback session. Providing these artifacts also encourages discussion among operators for key issues in the limited time available, while enabling smaller system-interaction concerns to be raised.

Actions

• Create documentation that establishes the baseline functionality required for system success.
  • Establish initial draft of functional requirements before conducting feedback sessions. The documentation should
    • include flowcharts of interactions and data usage in a format centered around the operator’s needs
    • form the basis for defining the technical requirements of the system
  • Use functional requirements documentation and workflows to help focus and facilitate feedback discussions.
• Update baseline functionality documentation after each feedback session to reflect insights gained and support future feedback sessions.
• Set a consistent schedule for operator-feedback sessions in advance and try to use each feedback session effectively.

In larger programs where there are multiple teams and development efforts, all efforts should collaborate to create a one-sheet description of their software and distribute it to operators before the operator-feedback session. This description can establish a context for the operators and lead to better use of time.

Actions

• Create a template for a one-page document called something such as, “Information You Need to Know.” Developers should use this template to write about their project so that presenters at the feedback session can use their time more efficiently.
• Clearly state items like
  • objectives
    • How does this fit into the mission parameters?
    • Is there an element of service design (or a change in workflow) associated with the new application?
  • features and benefits
    • How are you supporting the component elements of the task?
    • How are you planning to reduce steps or consolidate relevant pieces of information?
  • challenges
    • How are you overcoming data-access issues?
    • What requests for functionality may be difficult or impossible based on the environment?
• Share the “Information You Need to Know” document with operators before the operator-feedback session. Having this one-page document as readahead material helps operators understand the basic objectives of the system when offering their input.

These action items enable operators to have in-depth discussions in a time-limited environment. This approach also lets the operators digest the content and organize their thoughts and opinions ahead of time.

To more accurately document the operators’ workflows and needs related to the system, we suggest conducting a contextual inquiry. A contextual inquiry is a one-on-one interaction where a researcher watches a user in the course of the user's normal activities and discusses those activities with them.

Observing an operator while they complete a task in a normal setting can help prioritize broader system requirements and organize functionality into logical groupings where applicable.

Action

• Review all available documentation on existing systems. This step is important but rarely comprehensive. This is why it’s so important to perform a contextual inquiry to get a deeper understanding of the needs of operators related to the system and offer overarching direction as well as fine-grained advice on system interactions. This will also allow comparison between the stated needs of the system and the reality, or ground truth, of the situation.

Clearly documenting the ground truth—information provided by direct observation as opposed to information provided by inference—of the steps and data points in all tasks performed by an operator who is actively interacting with a system can initiate discussion and identify possible breakdowns within the proposed system. Similar processes can be consolidated to reduce the complexity of the overall system. Documenting the ground truth can also identify rare or edge-case scenarios that, while accounted for, should be carefully integrated to avoid overcomplicating a system’s graphical interfaces.

Action

• Define and create visual workflow documentation that accounts for all steps and systems involved in as many tasks as possible that system operators must perform to complete the objective.

Use facilitation techniques, such as building an impact vs. effort matrix, to share with operators. Such a matrix provides operators with insight into how they should interpret the necessity of a function versus the amount of time and effort needed to create that function. Developers can use this information to prioritize efforts and functionality. New suggestions raised during future feedback sessions can be mapped to such a matrix to provide perspective.

Action

• Collect functional requirements and map them to tasks, working with operators to define (1) how long a task will take, and (2) its perceived impact on the functionality of the system. This activity can help ease conversations about development tasks that become backlogged and identify which efforts are required for the minimum viable product (MVP).

Collect and consolidate information about the system and its operators. The resulting reference material can be refined over time and provide a reference that can be used to gauge the success of a given solution.

Actions

• Consolidate all information previously mentioned into a system-documentation wiki.
• Use the functional requirements and their associated technical requirements to plan sprints based on each requirement’s perceived level of effort.
• Use the impact vs. effort matrix to determine the perceived value of these requirements.

Regular operator-feedback sessions are a great addition to a program office’s development lifecycle and lead to better, more functional software. Awareness of the benefits and possible drawbacks of operator-feedback sessions enables the improvement of associated processes, and provides returns on the time operators spend contributing in operator-feedback sessions and away from their everyday tasks.

Clear documentation can ensure that each operator’s time is spent efficiently and that developers clearly understand the needs of system operators throughout system development and across multiple development teams.

Our next blog post about operator-feedback sessions in government settings will examine effective and ineffective feedback-session practices in greater detail.