The new DoD Software Acquisition Pathway (SAP) recommends the use of operator-feedback sessions in government software-intensive acquisition programs. In a previous blog post, we discussed the increased use of operator-feedback sessions in government settings in response to the SAP and to increased use of Agile and DevSecOps methodologies, practices, tools, and techniques in government environments. In this post, we focus on the good and not-so-good practices and outcomes that we have observed while conducting operator-feedback sessions in government environments.

Feedback Sessions: The Good Parts

Many program offices regularly conduct operator-feedback sessions and recognize their value. When a program office hires a contractor or a government-led service provider to start a new software-development project, the hired organization often lacks expertise in the domain for which the software is being developed. This hired organization may understand system design or have a basic understanding of the objective, and may even have built similar products for other programs. However, they usually lack the tacit knowledge needed to fully understand the operators’ needs that they have been hired to fulfil.

It’s risky to assume that the processes and functions supported by the developed system—particularly those of sensitive missions—are clear to the members of the hired organization. It’s especially challenging when members of the hired organization assume roles that are new to them. These risky assumptions can lead to misunderstandings.

Operator-feedback sessions can challenge these assumptions and reduce misunderstandings. They can provide a casual environment where operators and developers can discuss the development work and contribute to better institutional knowledge by transitioning the operators’ tacit knowledge to the developers. The operator who directly and regularly completes a task has deeper insight into the program’s need for the system than even the best software developers could have. Factoring in that insight is an important part of developing a system that supports its objectives and requirements successfully. In this section, we summarize the benefits of holding these sessions.

The power of asking why—It’s valuable to have an opportunity to ask someone doing a thing, “Why do you do that?” That simple question can uncover information that an external team of developers hired by the program office would never think of.

This informal, direct approach is valuable, particularly considering that traditionally, most external groups working on military software interact only formally with their government customers. This formality often results in the program office telling the developers what it thinks is needed, the developer making that thing happen, and the developer executing all of the tasks that the program office defined for that request.

Some government systems grow organically from a need where the initial turnaround—from request to fielded system—is urgent; thus, decisions are sometimes hastily made to solve the problem expeditiously, with full intentions to revisit it in the next contract, cycle, or fiscal year, or when the system has been successfully used for some period. However, often the revisiting part of the plan doesn’t happen.

This short-term approach means that prototype software often becomes the system of record, and the process of successfully completing tasks is distorted by the tools that are available to complete them. An incomplete system leads to incomplete execution of tasks. As the system grows and expands, developers deprecate or add functions to meet the needs of current objectives. However, developers rarely update interfaces and interactions to reflect these changes, and onscreen buttons, key commands, and terminal functions proliferate out of control, negatively affecting usability.

These changes lead to a significantly higher cognitive load for the operator, dead or useless system functions, a higher risk of mistakes, and longer training times for new users. The ability to determine why a function is needed and how it relates to a documented requirement for system success can make the difference between a merely functional system and one that empowers operators.

Multiple people thinking about the same problem—We are all smarter together than alone. Studies about things such as jellybean-guessing contests show that the average number of jellybeans guessed by all participants is often almost exactly right. Having a large number of people think about a solution to a problem can lead to the discovery of a better solution.

Operator-feedback sessions expose multiple groups and many different operators to problems they would have otherwise not known about. During these feedback sessions, operators may learn about a solution that a vendor used to solve a similar problem. This type of cross-pollination of projects wouldn’t happen in a normal scenario, but happens freely during operator-feedback sessions.

Building empathy—An important part of operator-feedback sessions is that it provides an opportunity for participants to listen to each other’s tone of voice and gain group consensus about a topic. These elements educate external groups about the urgency and intensity of operators’ needs. Functions that may not have been considered essential can, in light of the interaction with operators, be identified and promoted as mission-critical functions. These types of functions may be related to quality-of-life or efficiency improvements, and they lead to a more usable system.

Before March 2020, when the COVID-19 pandemic surfaced, operator-feedback sessions were most often held in person and in large groups—a format that encourages and leverages face-to-face interactions. Now, during COVID-19, they are mostly held as video-based meetings or with a limited number of participants. While there is still value in this modified format, care must be taken to engage the available operators and avoid putting the responsibility on them to interact in a video conversation, which can be an intimidating experience.

Building a mental model—For external developers, part of the software-development process is understanding the operator and their domain knowledge to formulate solutions. This understanding can inform developer decisions and enable developers to consolidate functions. Building a mental model—understanding the operator’s technical expertise and the data with which they are interacting—yields potential requirements for data visualizations or features of the system that will ultimately help the operator using the system make decisions and reduce their cognitive load.

Breaking down barriers to adoption—There are situations where the lack of functionality, features, or informational data in a workflow leads to a system that is too cumbersome or not functionally complete enough for the operator to finish a task using the system. We often observe operators being forced to use multiple systems, sometimes on different workstations, to collect all the information they need to complete a task successfully. Worse yet, sometimes operators ignore the new system entirely in favor of using the old, more feature-complete system. At operator-feedback sessions, operators can share information, experiences, and recommendations, thereby breaking down the barriers often seen when new systems are developed.

Turning implicit knowledge to explicit knowledge—Program offices are excellent at using the tools available to them to complete tasks, even when those tools are less than ideal. Following this mode of use, operators develop tips and tricks within a team to help them operate as efficiently as possible.

Over time, these operators come to understand which systems provide the best (or at least better) data and when multiple sources can be combined to solve a problem. This understanding is rarely documented; it could be used to inform software developers about tactics to make the system more complete or more functional.

Benefits of informal discussion—People usually like to talk, and they like to be heard. In modern meeting administration, efficiency and hierarchy often dictate how discussions are managed and what topics are discussed. Concerns that are paramount to the success of a system can go unheeded. Whether this neglect happens due to the rank and position of the reporting observer or because of perceived time and resource constraints, these conditions draw focus from what could be the most important hurdles associated with a program.

Conducting informal discussions can reveal important observations. When a program office holds operator-feedback sessions, they are deciding to provide this informal channel. In between feedback-session presentations, developers can have casual discussions with operators while doing simple tasks, such as refilling their coffee or getting a donut. These hallway discussions often raise concerns that are valuable to the development project.

Differences among operators—Age, experience level, and technical understanding are all factors that influence how operators interact with a system. Often, an operator uses a single system for multiple, semi-related tasks, and certain operators take the lead on particular tasks due to their experience, preferences, or ability. Unless developers interact with operators firsthand, they might not be aware of the differences among operators. The format of operator-feedback sessions makes it easy for participants to discuss and identify these differences, leveraging them to build a common understanding or even devise approaches to support the software-development effort.

Feedback Sessions: The Not-So-Good Parts

As with any process, there are pitfalls that can arise when collecting information during operator-feedback sessions. In this section, we describe some of these pitfalls and their effects in the hope that readers will learn to recognize and avoid them.

Knowing what feedback to expect—Operators are often experts at using a system, but they’re not necessarily experts in understanding the data that the system consumes or produces. This observation is not a slight on operators; it’s easier to teach a person to use a microwave than it is to teach them the theory of how it works. When developers ask operators theoretical questions, the operators often don’t have the answers. Moreover, they may feel inadequate to answer these questions, yet may also feel obligated to say something. The open and sharing nature of operator-feedback sessions can intensify these issues.

Saying and doing may differ—An operator’s mental model affects their input about a system. In situations where an operator is completing a complicated task, they often mentally consolidate steps in that task to conform with their mental model. This consolidation can lead to a flow diagram, based on their input, that does not reflect all the actual steps required to complete the task successfully.

For example, if an operator sees a process as three easy steps, they might not account for other steps that they perform intuitively (e.g., logging into a different workstation to get information their system lacks). Using only this operator’s input increases the chance that the new system won’t account for the required steps, and the operator’s experience—such as getting information from another workstation—will perpetuate. Since operator-feedback sessions are typically conducted outside the operator’s normal work environment, it may be hard for them to explain what they do when their equipment is not in front of them.

Observational bias—The example of the operator’s mental model affecting their input is a form of bias that can go both ways. Developers can have the Maslow’s hammer mental bias. (“If all you own is a hammer, all your problems look like nails.”) Due to this confirmation bias, the input the developers notice most is input that reinforces their decisions, rather than objectively noticing all valid input.

Conversely, because some operator groups see certain workflows or tasks as more important or desirable than others, they discuss them repeatedly. This narrow focus of discussion can yield a system that focuses excessively on one workflow to the detriment of the overall system requirements. Discussions during operator-feedback sessions may reflect this bias, so conversations might be skewed.

Outspoken advocates—In any group setting, some participants will naturally be more verbal and make their needs known; since they’re the loudest, they tend to get their way the most often. In these settings, there is potential for members of the vocal minority to bend a program to fit their needs while members of the silent majority suffer.

When conducting operator-feedback sessions to get input about a system, some people might inadvertently stifle these discussions. Having a better understanding of the specific user groups involved can identify the blind spots created by this phenomenon that reduces the input of some users.

Scope creep or “too many cooks in the kitchen”—When gathering input, suggestions and comments are valuable, but actively soliciting them can be risky. For example, features and functions are great, but they are not all equal. Asking operators what features or functions they want runs the risk of their answering with the first thing that comes to mind, even if they don’t think that the feature or function is actually important.

This type of interaction can lead to a “Homer Car” situation. In an episode of The Simpsons, Homer Simpson’s long-lost half-brother gives Homer an opportunity to design a car. Homer creates a non-functional monstrosity of a car that bankrupts his brother’s car company. This type of situation happens in real-life systems as well. For example, interfaces can become unusable because they are twisted around edge-case functionality and noncritical information. Although it is encouraged and often valuable, unfiltered feedback in operator-feedback sessions might result in scope creep that can inhibit development efforts.

Feedback Burnout—Any given program office may have many different software-development projects in play at one time that affect the same operators. When these development projects are all separate, developers can overtax operators in multiple efforts to gather important (and sometimes redundant) input from them. These multiple efforts lead to less feedback overall and resistance by operators to participate.

Achieving Effective Operator-Feedback Sessions

While there are many advantages and benefits to conducting operator-feedback sessions in government environments, there can be pitfalls and unintended outcomes of conducting these sessions without proper planning, preparations, and facilitation. With thoughtful consideration and planning, an operator-feedback session can achieve all the good, while avoiding the not-so-good.