Why is it easier to verify the function of a software program rather than its qualities? For example, one can easily determine whether the method in a class allows for parameters or whether certain inputs are accepted or rejected. On the other hand, it is much harder to determine whether a program is secure or easy to use or that it is easily evolvable for changing business needs. The latter attributes are called the quality attributes of a software system, and their importance is often overlooked in software development. There are many kinds of quality attributes, and software engineers must be able to identify those appropriate to a system, construct software that supports them, and, often, provide evidence to evaluators that the system has the intended attributes. Furthermore, as systems change over time, their qualities may change as well.

In this post, I explore the essential elements that make up quality and present four engineering-centric techniques to creating quality software.

Functional Requirements and Quality Attributes

In software engineering, deciding what a system will do is specified by its functional requirements, while how the system does things (and the attributes that emerge from its operations) are described by its quality attributes. We use the term quality attribute instead of non-functional requirement because the latter carries the unfortunate additional connotation that this kind of attribute is not useful or pertinent to a system’s design.

These categories are based on the observation that some system properties that are local to a module, component, or function can be easily identified, compartmentalized, and tested. Other system properties, in contrast, are cross-cutting and apply to the system as a whole. For example, consider a quality attribute that describes a computation: The system shall be able to handle 1,000 concurrent users with the 99th percentile of response times under 3 seconds. This specifies the system’s capacity to handle a certain load, which is an aspect of performance. It does not define what the system does, such as whether it uses a cache or a specific transport protocol to achieve these speeds; instead, it describes how well it can handle a specific operational condition.

The Guide to the Software Engineering Body of Knowledge distinguishes quality attributes as constraints on a system, whereas functional requirements are features or capabilities of a system.

Quality attributes can be furthered categorized between qualities that describe how a computation should be done (such as its performance, scalability, efficiency and reliability) and qualities that describe how it should be structured or organized (modularity, extensibility, maintainability, or testability). Being able to differentiate between these qualities can be useful in a software project. For example, if performance is an important attribute for the system, critical code paths can be identified early in development that informs how the system’s modularity and maintainability will be affected.

In addition to specifying quality attributes, there needs to be an evaluation or test that can measurably determine to what degree this attribute exists in the system. Since the system is constantly changing as development continues, these tests become an important source of validation for its designers that the system continues to exhibit the desired attributes. Whereas tests for functional requirements can be performed at the unit or integration level (since it is specific to what the system does), tests for quality attributes may comprise multiple levels of unit or integration testing across components or even require end-to-end tests. Some quality attributes may be tested by directly translating the specification into an executable as provided by Cucumber or other Behavior-Driven Development tools, which allow for the running of a whole suite of tests behind the specification. Some quality attributes may be hard or even impossible to test, such as whether the system is maintainable. One possible solution would be to make this attribute more specific and testable to a degree that its stakeholders would think acceptable such as: The system architecture shall be organized to minimize coupling and isolate variabilities by having all modules be less than 1000 lines of code and have a cyclomatic complexity of less than 10 each.

Aren’t We a Software Factory?

Achieving a system’s desired quality attributes takes domain expertise, tradeoffs, and contextual decision-making. This requires skilled senior engineers and architects working in tandem to develop, achieve, and sustain the quality attribute. However, many organizations focus on making repeatable processes that they hope will create high-quality software. Problems start when people think that an assembly-line approach to the software methodology of the day will reliably produce quality software. After all, aren’t we a software factory? The conflation of software engineering as a discipline akin to manufacturing is an old but misguided idea. In his book Modern Software Engineering, Dave Farley describes software engineering as a design activity, not a manufacturing one:

Software development, unlike all physical production processes, is wholly an exercise in discovery, learning, and design. Our problem is one of exploration, and so we, even more than the spaceship designers, should be applying the techniques of exploration rather than the techniques of production engineering. Ours is solely a discipline of design engineering.

The consequences of developing software as a design engineering discipline, rather than a manufacturing process, are profound: the quality of the product cannot be baked in or checked once and handed off like a stage in a production line. The practice of accepting a user story once it meets its requirements and assuming that its quality remains constant ignores the fact that small changes in one part of the system can profoundly change the quality profile of the entire system (one of the goals of architecture design is to reduce the possibility of these kinds of interactions from taking place). In agile software development, constant change is the norm. A certain quality attribute may be present in the system after one code change but absent in the next. It is therefore important to understand what produces quality attributes in software, and how can its quality be verified?

An Engineering-Centric Approach: Four Techniques

Processes that create quality software require an engineering-centric approach. Software development should aim for the qualities of engineering: manageability, rationality, and science. If we assume a working environment that allows for iterative, incremental improvement and for experimentation and change, then the following techniques can be used: 1) create a model of the system that will solve the current problem, 2) invite everyone to continuously improve the system, 3) assert quality through rigorous testing and 4) include telemetry to diagnose problems before they occur.

This is not meant to be an exhaustive list, and I am not claiming anything new with this method. There are methods specifically for quality improvement such as the plan, do, check, act cycle (PDCA), Kaizen, and Scrum, and these apply well for the development of quality software. They provide values and principles that are important for any kind of iterative improvement. However, my hope here is to provide specific techniques that embody these values such that it makes software engineers’ daily practices more rational, scientific and evolvable.

first technique—Make a model of what you are trying to solve for in the current moment, not the problem for next week or next year but the problem they are facing now.

Suppose you are an engineer at Unicorn Corp tasked to create an application programming interface (API) for a payroll system that gets year-to-date earnings after taxes for a portion of employees. A future task will be to get earnings before taxes, and a backlog feature is to get earnings within a specified calendar range. One approach to make this API would be to anticipate these future changes by adding input parameters for future dates as well as a flag for whether or not earnings should be taxable or not. So, your API design may be a starting date, an ending date, and a Boolean flag. This seems like a good idea except you may not have realized that in the near future, management will also want employees from other divisions to use this API, and they may have additional deductions for benefits or contributions that need to be factored in separately. Additionally, future company growth requires that the API support multiple currencies and different fiscal years, depending, depending on the location and financial practices of employees. As a result, your simple flag and date range parameters might lead to a rigid design that cannot easily accommodate these variations without significant refactoring.

A model is a simplified view of the real system that eliminates details not relevant to the problem. In this case, this view is earnings for a specific region with specific fiscal dates. We can model the system using common software methods for managing complexity (i.e., modularization, cohesion, separation of concerns, abstraction/information hiding, and coupling). A model makes a complex system simple. It abstracts away the parts not relevant to the problem and highlights those that are. It would be overwhelming for an engineer to account for all the factors of an international payroll system. So, start by meeting a basic user need without optimizing it. Defer decision-making on the details through abstraction. Don’t do extra work now. Satisfy the user need of the moment, while making it easy to change or enhance in the future. In the case of our API, allow for a single input parameter that takes in an object with start/end date fields. As user requirements grow, additional fields can be added to the object along with validation methods for each.

This technique allows for making progress in an iterative fashion, not compromising on delivery. Defer or encapsulate the parts of a system you don’t understand yet, so they don’t distract or get in the way of your current work. Solving for the current problem while providing extensibility for future change is a key contributor to quality in the software product.

There are other benefits. Breaking changes down into smaller, more manageable chunks enables greater intellectual ownership of the codebase. This enhances the knowledge of everyone involved in system development of what the code is doing and prevents the creation of “dark corners” that no one understands. It also creates less technical debt, since fewer decisions have to be made about what each code section is doing. As functions, classes, and components grow, close architectural support should be provided to ensure the overall system architecture is maintained or even anticipates a need to change (yes, even architecture is subject to change, though ideally at a slower pace than system components).

second technique—Ensure a strong culture of collaboration. Ideally, beyond the engineers, every individual who interacts with the system (such as business analysts, designers, customer representatives) should have a mental model of the aspects of the system that are relevant to their work. In such an environment, if they notice something unusual or challenging, they can make changes as appropriate.

Let’s say there’s a business analyst in Unicorn Corp who assembles monthly payroll reports. During review, he discovers the reports often contain discrepancies that frequently lead to client complaints and additional support tickets. The analyst discovers that the current system does not consider mid-month changes in employee deductions, causing the data to be inaccurate. Recognizing the problem, the analyst meets with the development team. The developers acknowledge the importance of fixing this problem and mention that they had accepted as technical debt the ability for the system to make mid-month updates. The team changes their priorities for the current sprint and work to fix this problem. They test it along with the help of the analyst and deploy it, successfully fixing the issue.

We want to empower everyone on the team to drive a necessary change, noting that this can be done either directly or through communication with the team who can. If a certain feature has to be delayed because an engineer thinks a technical debt story requires attention, then the timeline would need to be adjusted to account for this work. In truly agile environments, changing the timeline is expected. Close communication with management enables the team to work together with an acceptable level of risk and revision. Appropriate communication with the client will ensure that everyone can agree on the changes and the quality of the final product will not be compromised.

third technique—Model and test the functional and quality intentions shared by the team. It is not enough to make a test to fulfill the user story requirement; tests exist to give confidence to the team that the feature works or fails as expected under varying conditions. Tests are especially valuable during refactoring, which is an inevitable part of agile development.

For instance, suppose the team at Unicorn Corp is working on refactoring a key component of their payroll processing system to improve its performance. The refactor involves changes to how deductions are applied and processed. During this refactor, the team relies on their existing suite of automated tests to confirm that the new implementation maintains accuracy and reliability. As the developers modify the code, some tests fail, providing immediate feedback on where functionality has diverged from the expected behavior. This feedback is crucial because it highlights potential issues early and allows the team to address them promptly. If no tests had failed during the refactor, it would suggest that the tests either weren’t comprehensive enough or weren’t properly aligned with the updated code. By using test-driven development (TDD) and similar practices that align the development of code with the development of unit tests, the team ensures that their code remains modular, easily changeable, and extendable. The iterative nature of TDD means that each refactor is accompanied by a series of tests that fail and then pass as the issues are resolved, thus minimizing the risk of introducing bugs and streamlining the refactoring process. Ideally, this results in a testing strategy that is aligned with high-quality code that is more modular, easier to change, and easier to extend.

fourth technique—Include instrumentation in executable code to facilitate diagnosis. How can we maintain resilience and availability when the application crashes or service degrades? A typical response is to replicate the problem in a development environment, adjusting parameters until the root cause is identified. This can be a challenge when errors are intermittent. Additionally, if diagnosis is expensive and time consuming, then the delay in repair could harm reputation. Instead, if telemetry had been collected and analyzed during production, potential issues could have been detected and addressed earlier, ideally before impacting users.

For example, at Unicorn Corp, the development team noticed that their payroll processing service occasionally experienced slowdowns during peak usage times. Rather than waiting for users to report performance issues, the team had implemented comprehensive instrumentation and monitoring. This included real-time metrics for CPU and memory usage, response times, and the number of active service instances. One day, the system’s telemetry alerted the team to an unusual increase in CPU utilization and a rise in response times just before a major payroll run. This early warning allowed the team to investigate and identify a memory leak in the system’s caching mechanism that could have caused significant slowdowns. By addressing this issue proactively, before it affected end users, the team was able to maintain the high quality of the service. Instrumentation provided real-time insights into the health of the system, enabling the team to resolve issues before they became problematic for users.

Engineering is about making accurate measurements to produce better solutions. Waiting around until a problem occurs is rarely good engineering practice. When applications are instrumented and measured, it becomes easier to provide real-time or near-real-time insights into the health of the system and its services.

Engineering Quality in Software Is an Iterative Process

The pursuit of high-quality software demands a focus on both functional requirements and cross-cutting, harder-to-define quality attributes. Functional specifications delineate clear actions and behaviors. Qualities, such as security, resilience, and ease of use, are less tangible yet profoundly impact a software system’s fitness for use and long-term success. Recognizing these attributes as integral to design and development processes ensures that software not only meets initial demands but also evolves with changing business needs. Achieving and maintaining such quality demands an engineering-centric approach that values iterative improvement, rigorous testing, and continuous refinement of mental models. By embracing these principles, software engineers can foster environments where robust, adaptable software systems thrive, fulfilling their purpose reliably as it evolves over extended lifetimes.