In an ideal scenario, organizations that complete a DevOps assessment will implement all of the assessment's recommendations to improve their software development lifecycle (SDLC). Establishing the client's post-assessment DevOps posture allows the client to understand their progress since the pre-assessment posture establishment and how close they are to an ideal DevOps and SDLC environment. In this eighth installment of our blog post series on DevOps in highly regulated environments (HREs), we discuss the when, where, how, and why of establishing a post-assessment DevOps posture.

This blog post series, based on a paper by me and my colleagues Hasan Yasar and Aaron Volkmann, discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the SDLC within HREs.

Recap

At this point, you (the assessment team) have completed a DevOps assessment with an HRE organization for the purpose of improving their SDLC by implementing DevOps concepts. You have executed the following major milestones:

1. Established a pre-assessment DevOps posture, where you clarified the client's current SDLC process.
2. Executed a full DevOps assessment, where you interviewed several people, identified obstacles, and constructed recommendations to address them.
3. Delivered a report of obstacles and recommendations to the client, where you captured the observed and inferred issues that impede effectiveness and/or efficiency of the SDLC, along with approaches to fix or alleviate them.
4. (Optional) Assisted implementing recommendations, where you worked with the client to implement the actual recommendations, either by guiding them on what needed to be done or actually executing the needed tasks. This activity is an optional addition to the assessment. In some cases, it is left up to the client to follow your recommendations.
5. (Optional) Observed new projects executed after recommendations were implemented, where you observed a new software project being executed with some or all of the recommendations implemented. You also helped to work out any issues occurring as a result of the newly modified DevOps-driven SDLC.

The milestones listed above complete the DevOps assessment. Establishing the post-assessment posture is optional, and the clients may not desire it. Usually there is not enough funding or time to bring you back for this task. In spite of this, establishing the post-assessment posture is critical to measuring the success of the recommendations given in the final report. Whenever possible, therefore, you should establish the post-assessment posture at least once as part of the overall DevOps assessment engagement. Post-assessment posture establishment can recur regularly, as discussed later in this post.

When

The ideal time to establish the first post-assessment posture is based on how many software projects have been executed by the client after modifying the SDLC according to your recommendations. You could say 3 months, 6 months, 8 weeks, or 1 year, but the timeframe is meaningless if only a small number of projects have been performed by this point. Use the following two conditions to determine the right time to establish the post-assessment posture:

1. How many recommendations have been implemented.
2. How many projects have been completed since recommendations were implemented.

Considering these two conditions, the ideal scenario would be that all recommendations have been fully implemented and several projects have been completed since that time. The projects should be diverse and should touch every step of the SDLC multiple times.

To account for all issues in the SDLC that might arise, the client should have explored a wide array of projects. A DevOps-driven SDLC may work flawlessly until a unique, unexpected, or rare scenario results in never-before-seen issues. As rare as this scenario may be, its issues should be addressed and fixed.

The recommendations in the final report will include some about DevOps and others about different domains. Discuss with the client how many of recommendations have been implemented and mutually analyze the impact of the change.

Some recommendations may fix major obstacles that justify the assessment and the implementation of recommendations; the establishment of the post-assessment posture can seem minor in comparison. Other implementations may be small incremental changes that in isolation do not significantly impact how the SDLC functions, but grouped together can produce noticeable change. A discussion with the client is the best way to determine when to conduct this posture establishment. Of course, regardless of what has been implemented or not, the client can decide if they want the post-DevOps posture established or not. It is not beneficial to proceed with the post-assessment DevOps posture if

• no recommendations have been implemented
• only a small number of non-impactful recommendations have been implemented
• just a few projects have been completed
• all projects completed to date were very similar in execution and predictable in outcome, providing no opportunity for unique scenarios to arise

Explain to the client why this is not the right time and offer to work with them to select a more opportune moment.

Where

Ideally, the post-assessment DevOps posture should be done on site, in the same location and with the same personnel who were originally involved with the assessment. If these conditions are not possible, revert to video calls, leave phone calls for last, and don't bother with emails.

Before discussing how to establish the post-assessment DevOps posture, clearly identify the goal. Establish any movement closer to the ideal DevOps and SDLC environment that the client documented in the pre-assessment DevOps posture. To achieve the desired DevOps and SDLC scenarios, you will need:

• the pre-assessment posture, specifically the current state of the SDLC at that time, and the ideal state
• the DevOps assessment final report

With these two items in hand, meet with senior engineers and their immediate management. The meetings should focus on the recommendations that have been implemented and their impact to the overall SDLC and DevOps posture.

When you meet with the engineers and their management, review the pre-assessment DevOps posture. Review the ideal states of DevOps and the SDLC, which were identified during establishment of the pre-assessment posture. Go through the list of obstacles from the assessment's final report and identify which have been addressed.

For each obstacle addressed, document whether the assessment team's recommendation was implemented. For the obstacles that were remedied using an alternative solution, spend time to understand the solution and why it was chosen over what you provided. In fact, by learning the details of the solution's mechanics and the reason for its selection, you may be able to find a flaw in your reasoning and use the lesson learned to improve in future assessments.

For each addressed obstacle, validate that it has been eliminated or alleviated and make a note of each because it will impact the post-assessment posture establishment. Follow this by reviewing if any new obstacles have arisen and, if they have, determine if they are the result of implementing a recommendation for a previous obstacle. It is possible that the mechanics or process of an implemented recommendation may require tweaks to other portions of the SDLC. These tweaks can introduce new obstacles. Conversely, changes within the client organization--rules, laws, policies, requirements, admin or management changes, and so on--may become new obstacles. You should also review the new SDLC process for inferred obstacles, or obstacles not recognized as such by the client. At the completion of reviewing the new SDLC process, make a list of all the newly discovered obstacles.

Next, review the ideal posture and compare each step to the current SDLC. Recall that the ideal posture is what the client envisions as the most desirable SDLC and DevOps environments. Discuss with the client the improvements, if any, at that particular step. Follow up by enumerating the desirable properties of the ideal SDLC. If these properties were not captured during the pre-assessment posture, infer them from the ideal state itself.

For each property, ask if it has been fully, partially, or not at all satisfied. Mark fully satisfied properties. For partially satisfied properties, discuss with the client what was and was not resolved. Ask if the unresolved portion is the result of pending recommendation implementations. The client may have implemented recommendations meant to achieve a desired property, but the implementations failed to do so. In this case, you should review the recommendations and any notes capturing your reasoning that this would achieve the desired property. Try to identify faults in your thought process or, potentially, in the realized implementation.

After the above-mentioned tasks are completed, create a report of the post-assessment DevOps posture containing the following:

1. obstacles addressed by implementing the assessment team's recommendation
2. obstacles addressed by implementing another solution
3. obstacles waiting to be addressed
4. obstacles resolved by your recommendation
5. obstacles resolved by another solution that is not your recommendation
6. obstacles partially resolved by your recommendation and/or another solution
7. obstacles not resolved by your recommendation and/or another solution
8. the ideal state's desirable properties
9. properties achieved
10. properties partially achieved and reasons why
11. properties not achieved and reasons why
12. new obstacles resulting from recommendation implementation
13. new obstacles resulting from implementing another solution
14. new obstacles resulting from the client's environment
15. overall assessment of post-assessment DevOps posture
16. recommendations for next steps

Note an addressed obstacle underwent a resolution attempt, whereas a resolved obstacle was successfully removed as a result of the attempt. The overall post-assessment posture can be derived from numbers 1 - 14 above. There are several ways to establish the overall posture using various algorithms, logic, and so on. In the most basic case, use the following:

E = number of previously identified obstacles = 1. + 2. + 3.

N = number of newly identified obstacles = 12. + 13. + 14.

D = number of desirable properties = 8.

R = number of obstacles resolved from E = 4. + 5.

A = number of achieved desirable properties from D = 9.

O = E + N

T = D + O

U = R + A

P = T - U

O represents the total number of identified obstacles from E and N. T represents the total number of tasks that must be achieved to reach the ideal state for DevOps and the SDLC based on the established pre-assessment DevOps posture. U represents the total number of satisfied tasks. P represents the total number of pending tasks, which is the current posture. P includes obstacles not resolved and partially resolved as well as properties not resolved and partially resolved.

When P = 0, all tasks have been completed and the desired posture has been reached. Note that the values for E, N, and D can be 0, representing cases where there are no unresolved obstacles and no unachieved desirable properties. When this occurs, R and A should be 0 by default because their value reflects progress made on E, N, and D. Also note that N is assumed not to be addressed or resolved at this time because they have been newly discovered and are left as a future pending task. The value for P is what you report to the client. You can detail the values further by dividing them into fully and partially resolved, plus any other derived values of significance. The above approach can and should be re-calculated for every follow-up post-assessment posture establishment.

A key task on the part of the assessment team is to convince the client why a post-assessment posture establishment is a critical component of any DevOps assessment. The answer is simple: it provides a quantitative measurement of improvement. Without this posture establishment, there is no structured way to determine if goals were achieved or not. Only internal feedback may serve as a measure of success, but that could be highly subjective and imprecise.

Recurring Assessments

After the initial post-assessment DevOps posture is completed, repeat the process periodically to ensure the ideal state is sustained or to measure new progress or setbacks if further changes have occurred. Engineers may not take note of obstacles that appear. The DevOps assessment team serves as the objective party that can observe what the client fails to see.

Wrapping Up

In this post, we discussed the importance and mechanics of establishing, periodically if possible, a post-assessment DevOps posture. The critical benefit of this task is to demonstrate the value to the client in implementing DevOps to improve their SDLC. In the best scenarios, many identified obstacles have been eliminated, and the client's overall software and system production apparatus has improved.

In this series, we focused on the process of executing a DevOps assessment. On the surface, this task seems simple and straightforward. For members of the assessment team, a detailed understanding of the steps required reveals potential complications and how to overcome them. A successful assessment results in an improved DevOps process that achieves or approaches the desired state. The key takeaways of this series are the following:

1. Ensure that each member of the assessment team has a detailed understanding of the SDLC.
2. Recognize that not all obstacles can be resolved with DevOps techniques.
3. Set clear expectations at the beginning.
4. Conduct DevOps posture evaluations before and after the assessment.
5. Be transparent in sharing information with key stakeholders and the assessment team's management.
6. Guarantee and ensure the anonymity of people that you interview for the assessment.
7. Use a positive tone in the final report and reserve critical language for the obstacles.
8. Document everything.
9. Always conduct interviews in teams of at least two people.

Additional Resources

This blog posting is latest installment in our series DevOps and HREs, which is based on a paper by my colleagues Hasan Yasar and Aaron Volkmann and me that discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the SDLC within highly regulated environments (HREs).

Read the paper this series of blog posts is based on Implementing DevOps Practices in Highly Regulated Environments, which was presented at the 2018 International Workshop on Secure Software Engineering in DevOps and Agile Development.

Read other posts in the DevOps blog.

Listen to the SEI Podcast Roundtable, How Risk Management Fits into Agile and DevOps in Government.

Read the paper from which this series of blog posts is based upon, Implementing DevOps Practices in Highly Regulated Environments, which was presented at the 2018 International Workshop on Secure Software Engineering in DevOps and Agile Development.

Read other posts in the DevOps blog.

Listen to the SEI Podcast Roundtable, How Risk Management Fits into Agile and DevOps in Government.