search menu icon-carat-right cmu-wordmark

Formalizing DevOps Assessment Findings and Crafting Recommendations: Fifth in a Series

Headshot of Jose Morales

Reviewing DevOps assessment findings and formalizing them into a final list is critical to precisely identifying obstacles to the client. Drafting the appropriate recommendation is key to improving the organization's software development. We will dicuss both topics in this blog post.

This blog post series, based on a paper by me and my colleagues Hasan Yasar and Aaron Volkmann, discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the software development lifecycle (SDLC) within highly regulated environments (HREs).

  • The first post explored challenges and goals of implementing DevOps in HREs.
  • The second post discussed the first step in a DevOps assessment: setting expectations with the organization. This critical task sets the boundaries of what will be performed and delivered.
  • The third post discussed the second step in a DevOps assessment: establishing the pre-assessment DevOps posture of an HRE.
  • The fourth post in our series explored how to perform the DevOps assessment.

In this fifth installment of our series, we will discuss how to formalize assessment findings and provide appropriate recommendations.

Formalizing Recommendations

At this point, as a member of the assessment team, you should have completed the DevOps assessment, interviewed various personnel, and documented their observations of problems within the SDLC. A list of problems sits before you, jotted down hastily on paper. You now need to create a formal representation of these notes that clearly conveys each problem. After the formal representation is complete, you then need to provide recommendations to remediate these issues. Recall that the assessment was meant to find

  1. problems with the SDLC that can be solved using DevOps concepts
  2. problems negatively impacting the current DevOps implementation
  3. enhancements that will further improve the efficiency and effectiveness of the current DevOps implementation

Ideally, you would provide a DevOps solution to remediate each problem. In reality, DevOps will not solve them all. Some issues will require non-DevOps approaches, but their remediation will still positively impact the SDLC, which is the overall goal of the assessment. In HREs, security issues, as well as more typical issues (such as approval processes, resource acquisitions, and licensing) usually require non-DevOps solutions.

During the assessment, you will likely discover two types of issues:

  1. Stated: issues discussed with you by one or more interviewees
  2. Observed: issues discovered by you and (potentially) not mentioned by interviewees

In practice, there is no difference in impact between stated and observed issues. At a more detailed level, you may wonder why the interviewee did not perceive the observed to be a problem. There could be many reasons why interviewees did not state issues you observed. The interviewees may not have been aware of them or could have been insufficiently informed. Perhaps they misunderstood or failed to notice the issue's impact on the SDLC. The issues may have been superseded by more obvious or impactful ones. However, these reasons are beyond the scope of the assessment.

Take your list of issues and

  1. divide each entry into stated and observed
  2. create a single sentence and clearly state the problem for each
  3. review and validate the list, removing and adding issues as needed

The final list should capture every issue discovered during the assessment with a single sentence clearly stating the problem so that others can understand it. With this list in hand, formalize each issue using the following format: issue, reason, method, and recommendation.

  • Issue. Also known as the obstacle, the issue is a plain-language statement of a problem with a process in the SDLC. You can use the single sentence that you previously crafted or create a new one. The key is to assure that the issue is plainly understandable. Do not use language that hints, implies, or infers the problem. Do not use words or phrases with unclear meanings, which can lead to incorrect assumptions by the reader. Bottom line: state the issue in a direct manner using plain language.
  • Reason. Precisely state the reasons why the issue negatively impacts the SDLC. The reasoning should be clear and sound. Poorly supported issues may waste the time and money the entity spends on remediation, damage your credibility, and endanger future engagements.
  • Method. This section lists the sequence of steps that led to the issue. It is your responsibility to understand this sequence. The mechanics of following some process is likely the cause of every issue. The steps should support the reason given for why this is an issue. The sequence of steps can be enumerated as atomic actions stated in plain language. There is no limit on the number of steps, but if there are many, reconsider the starting point of the sequence and determine if a sub-sequence of steps will still produce the issue. In some cases, a long list of steps is needed to showcase how the issue is occurring.
  • Recommendation. Here you list the steps to eliminate or alleviate the issue. The depth and breadth of the recommendation is unbounded, but do not recommend actions so extreme and complicated that they will be ignored. Such a situation will not only derail issue remediation but also put your judgment into question and cause the entity to ignore all your recommendations. Remember that the recommendation can be DevOps or non-DevOps based. If the latter, do not feel compelled to explain why a DevOps solution is sub-optimal or simply not available or appropriate. You should have already set expectations by explaining that SDLC improvements will include some non-DevOps approaches.


When formalizing your findings and recommendations, apply the following considerations--when appropriate--to maximize positive impact on issue remediation.

  • Methods and Actions. The method is a formal component of an issue. In some cases, an issue occurs when executing a single step (or sub-step) in the required sequence. Label each step in a sequence as an action. Actions are sub-steps or sub-sub-steps of the sequence for some process that is causing the issue. Make sure to highlight the particular action or set of actions causing the obstacle. Listing issues and actions for a method allows the reader to track down where in the SDLC the obstacle occurs. If done well, the reader will appreciate the action(s) in question and, likely, be able to implement your remediation recommendations.
  • Avoid putting individuals at risk. When discussing the cause of an issue, do not mention specific individuals, role types, job titles, or anything else that can identify an individual or group of people. You may put their jobs at risk. In some cases, it may be unavoidable to mention terms such as administration, senior leadership, or management. In other cases, referring to hardware procurement, licensing issuance, or software upgrade may imply specific personnel. In these situations, try to find a way to convey the message without identifying an individual either directly or indirectly. Use the implicating word or phrase only as a last resort to ensure the message is expressed appropriately. You always want to maximally convey the message, which is the top priority. Referring to a group, such as a department or team, is often preferable to referring to an individual.
  • Negative vs. constructive criticism. Formalize the issues using the most positive language possible. Fundamentally, an assessment is a form of criticism toward the current status and usage of an SDLC by a group of people. Identifying failures, faults, inefficiencies and sub-optimal parts within the SDLC is a delicate process. This process can even result in ill feelings from those who were involved in creating, executing, maintaining supporting the part of the SDLC where issues were discovered. Frustration may also come from those who are aware of a particular issue and either are not able to do anything about it or are tasked to fix it and fail due to lack of skills or delays. When you point out an issue, you may unintentionally identify personnel. You should have previously advised management not to retaliate against individuals or groups as a result of the assessment findings. Make every attempt to phrase your formalized issues objectively and avoid, where possible, words and phrases with a negative undertone.

Looking Ahead

In this post we described how to formalize observed issues found in a DevOps assessment. This process is not straightforward. The guidance we provided should help you facilitate the creation of a list of easy-to-understand issues. The next step, which we'll discuss in our next blog post, is to review the formalized issues with peers and the client.

Additional Resources

Read the paper this series of blog posts is based on, Implementing DevOps Practices in Highly Regulated Environments, which was presented at the 2018 International Workshop on Secure Software Engineering in DevOps and Agile Development.

Read other posts in the DevOps blog.

Listen to the SEI Podcast Roundtable, How Risk Management Fits into Agile and DevOps in Government.

Get updates on our latest work.

Each week, our researchers write about the latest in software engineering, cybersecurity and artificial intelligence. Sign up to get the latest post sent to your inbox the day it's published.

Subscribe Get our RSS feed