This second installment in the blog post series on implementing DevOps in highly regulated environments (HREs), which is excerpted from a recently published paper, discusses the first step in a DevOps assessment: setting expectations with the organization. This step is a critical task in an assessment because it sets the boundaries of what will be performed and delivered.

Expectation setting for a DevOps assessment in an HRE should focus on:

1. Specifying exactly what software development lifecycle (SDLC) process will be assessed
2. Specifying what aspect of an SDLC process will be addressed
3. Detailing the deliverable's contents
4. Creating awareness that all identified obstacles may not be solved
5. Declaring that multiple assessments may be required
6. Clarifying that DevOps is not the one solution for all obstacles

The above list may seem obvious, but our experience has shown that not setting expectations results in developers and operational staff feeling underwhelmed with a DevOps implementation. Most organizations may recognize they have an SDLC-related problem, and they assume a one-time DevOps assessment will solve all of it. The truth is most of these organizations only know the result of a problem, such as a missed deliverable, massive overspending, loss of clients, etc. They don't really know what the problem is and what caused it to begin with. Engineers, operators, and management often don't realize how deeply rooted in a culture the cause is. After they realize there is a problem--and learn that it's related to software development--they make a few calls, discuss it with a few people, read a few articles and find a potential solution.

In this case it's DevOps, so an organization is selected and told "we need DevOps". The assumption is that DevOps will solve their problems. In reality, DevOps can solve many--but not all--problems. When an assessment is completed and some of those problems still persist, they may become upset or confused as to why. For this very reason expectations must be set so the organization has an understanding of what to realistically expect from the assessment.

Let's discuss each of the six focus areas stated above:

1. Specifying exactly which SDLC process will be assessed. Within one organization, several groups can perform software development, which holds true even within one group that is composed of several smaller and specialized software development teams. Each software development team may have its own culture with an SDLC, project management, communication with outside groups, tooling, and methods of artifact delivery. An assessor should not assume that the SDLC and culture employed by one development group are used by all others.
   
   Assessors must also understand where personnel are positioned within an organizational and managerial chart. In some cases, an assessment involves dealing with managers of large groups that requested the assessment, even though the task is performed within a smaller subgroup. It is important to state and document the precise group that will be assessed and the specific SDLC that will be the focus of improvement. Make sure all engaged stakeholders clearly understand and agree to this. In some case a negotiation will occur before final agreement on what to assess, which is fine as long as all parties agree to it. Be specific when documenting the group and SDLC. Vague wording will work against you. Specificity in group name and SDLC will remove any chance of misinterpretation. In some cases, the SDLC process is at a higher organizational level that may involve several groups. Be sure to understand who these groups are and specify their names and Points of Contact (PoCs).
   
   
2. Specifying what aspect of an SDLC process will be addressed. The second most important expectation to set is stating the focus of the assessment. In many cases, the assessment will revolve around solving a single problem or set of problems. These problems, which must be precisely stated and agreed upon by all stakeholders, may be as diverse as missed deliverable dates, suboptimal communication between groups and with stakeholders, inability to gain project approvals, late stage requirements, and high employee attrition.
   
   Each stated problem creates a scope of issues that will be solved as a result of addressing the problem. It is imperative to detail those issues. The goal of detailing problems and associated issues is to remove the possibility of incorrect assumptions by the organization's personnel on what is being fixed. Avoiding these incorrect assumptions helps avoid comments at the end of the assessment such as Why wasn't this fixed? Or but I thought or I was told that you were going to and other similar statements. By articulating the problems and issues to fix there is no room for assumptions. At the end of the assessment, a checklist of these issues can be submitted and marked fixed or not fixed followed by a discussion of future steps.
   
   
3. Detailing the deliverable's contents. A DevOps assessment of an HRE always concludes with the standard method of a set of deliverables, which can be one or more of the following:
   
   
   • A report detailing observed obstacles impeding the effectiveness and/or efficiency of an SDLC.
   • A set of recommendations that are meant to resolve or minimize the identified obstacles
   • A list of preliminary work that must be carried out to mature an SDLC to a point where DevOps can be considered
   • A recommendation of repeated assessments to further the goal of a desired DevOps state

To avoid assumptions and underwhelmed reactions to the work it is important to capture what will be delivered. In some cases, the SDLC of the group is so sub-optimal that DevOps cannot be implemented. In these cases, deeper problems reside and they must be pointed out. A section of the report should be labeled Roadblocks to Full DevOps and used to detail these gaps and provide approaches to address them. These approaches will likely not involve a DevOps approach. In fact, these approaches highlight fundamental principles missing in the SDLC that must be present to function correctly. More often than not, this report will suggest that the group consider another round of assessments once these recommendations are in place due to the fact that a desired DevOps state is typically not achieved after just one assessment.

Awareness that all identified obstacles may not be solved. Many times, it will not be possible to resolve or lessen all identified obstacles. In other cases, some obstacles may require multiple steps to resolve or minimize them. There will be obstacles that cannot be dealt with due to imposed policies. The organization's personnel must be aware of this before any work starts. The assessment team cannot give the impression that all issues will be solved in one shot, which is not a realistic mindset when dealing with HREs.

The final report should be structured in three sections: (1) obstacles resolved with the stated recommendation, (2) obstacles minimized with the stated recommendation, and (3) obstacles that currently cannot be resolved or minimized. Obstacles that are minimized should include an explanation of what else must occur to minimize further or resolve. Obstacles that are currently unresolvable or unminimizable should include a detailed explanation justifying this conclusion. These obstacles are also opportunities for the assessment team to conceive out-of-the-box approaches that work around the obstacle, which effectively removes it from the SDLC.

1. Declaration that multiple assessments may be required. As discussed earlier in this blog post, it is highly likely (and a strong reality) that multiple assessments will be needed for a software development group to reach their desired DevOps posture. The gap between the current and desired DevOps posture is measured not only by successful implementation of DevOps components, but also by the SDLC's current levels of effectiveness and efficiency. Remember, the ultimate goal of DevOps and other similar approaches is it to improve the overall process of an SDLC. After an assessment is completed, the next step is to determine what can be addressed in subsequent assessments and so on. A critical point to make is that certain obstacles may never be resolved or minimized due to the imposed policy of the HRE choice, the obstacle may just persist as a bottleneck in the SDLC.
   
   
2. DevOps is not the one solution for all obstacles. An organization will often assume that DevOps will solve all ailments and that it a single implementation is the only requirement to achieve this utopian state. Do not assume that the individuals within an organization requesting a DevOps assessment fully understand its concept and purpose. As part of setting expectations, it is important to give a thorough description of DevOps. After this description has been provided, it is important to point out how the SDLC stands to benefit from implementing DevOps. Again, it is important to reiterate that DevOps alone will likely not resolve all the issues for which the assessment has been requested. Establish a mutual agreement that the assessment will be performed with the goal of improving the SDLC as much as possible implementing both DevOps and non-DevOps recommendations. Include in the agreement that there is no guarantee that all identified obstacles will be resolved.

Looking Ahead

This blog post covers how to set expectations before starting an assessment in an HRE It is important to be transparent with the organization about the potential outcome of the assessment. Being honest and upfront by detailing realistic expectations and precise articulation about what will be done, what will be the focus, and what will be delivered will lay the groundwork for a smooth transition to start the work.

The next post in this series will evaluate an HRE's pre-assessment DevOps posture.

Additional Resources:

Read the paper from which this this series of blog posts is excerpted, Implementing DevOps Practices in Highly Regulated Environments, which was presented at the 2018 International Workshop on Secure Software Engineering in DevOps and Agile Development.

Read other posts in the DevOps blog.

Listen to the SEI Podcast Roundtable, How Risk Management Fits into Agile and DevOps in Government.