The time has come for the final step of the DevOps Assessment: the final report. Now is your chance to document all your findings, recommendations, and related material. The report is the key artifact documenting every aspect of the entire DevOps assessment: persons (team members, customer, and all others involved in the assessment minus the actual interviewees), places (locations of interviews and other meetings related to this assessment, they can be physical or virtual locations, dates, etc. along with the identified issues that negatively impact the software development lifecycle (SDLC) and remediation recommendations. In this blog post, we discuss the process of writing and delivering the final DevOps assessment report to the client.

In general, the layout of the report should be as follows:

1. Executive summary
2. Introduction
3. Current SDLC
4. Current DevOps practices
5. Issues to implementing DevOps
6. Issues in the current SDLC
7. Issues in current DevOps
8. Non-SDLC issues
9. Conclusions

The above list is generalized, so not all portions may apply to every report. For example, you may have assessed an organization (i.e., a client) that wants to implement DevOps, thus the seventh point, Issues in current DevOps, may not apply. You can add other sections as desired. The remainder of the post will explore these 9 areas in depth.

Executive Summary. The summary states the findings in a direct manner with no background information. It should summarize the pre-assessment DevOps posture, each issue discovered, and a summary of remediation recommendations. It also includes the answers to all the key questions that the client would want to know, such as

• How much effort will it take to reach our desired DevOps posture?
• How many issues are there, and how hard is it to fix them?
• Can we adopt DevOps now or are changes needed to be better prepared?
• What is the next step for us to get DevOps implemented?
• How should we execute the recommendations?

A good executive summary will clearly answer key questions along with explaining the issues. Assume the summary is a miniature final report. Approach the summary as if it is a condensed, standalone final report that provides the client with a high-level view of issues identified during the assessment.

Introduction. This section is a standard professional introduction, similar to those used in a majority of reports. It should contain

1. The goal of the assessment.
2. Details of steps taken to execute the assessment. This includes time span when interviews were conducted and the number of interviews. Remember to sustain anonymity and not mention, hint, or imply any person or group.
3. Points of contact (POCs) and timeline of significant events during the engagement. PoCs are the ones that sponsor your entrance into a building, that assist you in getting access, verification, etc. Interviewees are non PoCs
4. Number of issues found, categorized into general areas such as culture, process, and pipeline.
5. The number of issues observed by personnel and inferred by the assessment team.
6. Number of both DevOps and non-DevOps recommendations.
7. An explanation of why non-DevOps recommendations are required.
8. An opinion of the overall current DevOps posture in the organization.
9. Explanation of time and resources needed to either: prepare for DevOps implementation or execute an initial DevOps implementation or enhance the current DevOps implementation.

The introduction is a subset of the executive summary and should set the expectation for the client of what will be presented in the balance of the report. Only briefly mention various topics since they will be explored more in depth later in the writeup.

Current SDLC and Current DevOps practices. This part of the report is the pre-assessment DevOps posture that we discussed in a previous blog post. Recall that before conducting an assessment, you and the client should establish their current, if any, DevOps posture for the SDLC under analysis. This assessment is done by capturing their ideal SDLC and comparing with their current SDLC implementation.

The issues or obstacles identified in this assessment contribute to a delta, such that minimizing or eliminating them will bring the client closer to the ideal SDLC. The established posture functions as a baseline of current status for the SDLC. Any changes made as a result of the assessment are viewed as a deviation from this baseline.

This section of your report should start by describing the ideal or desired SDLC. A visualization is important to capture this utopian state. Follow this description with a second visualization showing the current SDLC. Add text describing the various major components of the ideal SDLC and the desired characteristics. Compare those with the current version. Conclude this section by listing the major changes that need to occur based on the current implementation. You should explicitly state that addressing these were a key component of the assessment.

Issues to Implementing DevOps. This section is optional and may not be needed if the client already has an implementation, in some form, of DevOps. This section of the report addresses the issues that can make the initial implementation and adoption of DevOps practices hard. In some cases, the client's SDLC may be so rudimentary that it requires implementing multiple remediation steps before even considering DevOps practices. If this is the case, clearly state the reasons why implementing DevOps at this time is not an ideal choice. Follow this by listing the changes needed to adopt DevOps. The reasons can be cultural, process, or othe,r and you should group these into paragraphs for easier comprehension.

Issues in the Current SDLC. Having studied, in detail, every aspect of the current SDLC, this section of the report captures issues that make the SDLC either ineffective, inefficient, or both. For each, state one or the other so the type of negative impact is clear. Even though all issues are important, in this section highlight those that address the technical process of executing the SDLC. In other words, first list mechanical issues in the SDLC, followed by processes that personnel abide by and finally culture issues.

Current DevOps Issues. This section is optional because the client may not have any DevOps practices in place. If the client has DevOps practices in place, this section should focus on issues discovered in the current implementation of DevOps practices and include issues from the entire sphere of DevOps including pipeline, culture, and process.

One consideration is the level of technical detail provided. Since the audience is unclear, you should not delve deep into technical details regarding an issue. Since the final DevOps assessment report is accessible by a broad audience, avoid delving into technical issues. At most, explain in plain language and add a note that you are willing to discuss technical details with the appropriate personnel at a convenient time.

Non-SDLC issues. As previously discussed, issues may be discovered that are not directly related to the SDLC but impact it in some negative way. This section documents these issues and makes a strong case as to why they impact the SDLC. Remember that every issue added is extra work for the client. Make sure to convince them of the importance of fixing these issues in spite of not being directly involved with the SDLC.

Conclusions. Similar to the introduction, this section should reiterate several of the same key points. One additional set of statements to add here is "next steps," which indicate a willingness to guide the client in executing your recommendations. It is also important to convey a willingness to observe the recommendations in action and further assist with other changes in the report.

Delivery. Send the report only to the funders and requestors of the assessment as well as co-assessors and the assessment team's management chain.

Now that you have sent the final report, you may receive some comments or questions. Although the report should not be widely distributed, after it has been delivered it is possible that it will be shared internally. Discuss all incoming comments, questions, follow-ups, and concerns from the client with your management before responding. Working with management, come to a collective agreement on how to handle each comment, question.

Additional Resources

This blog posting is latest installment in our series DevOps and HREs, which is based on a paper by my colleagues Hasan Yasar and Aaron Volkmann and me that discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the SDLC within highly regulated environments (HREs).

Read the paper this series of blog posts is based on Implementing DevOps Practices in Highly Regulated Environments, which was presented at the 2018 International Workshop on Secure Software Engineering in DevOps and Agile Development.

Read other posts in the DevOps blog.

Listen to the SEI Podcast Roundtable, How Risk Management Fits into Agile and DevOps in Government.