Reviewing DevOps assessment findings and formalizing them into a final list is critical to precisely identifying obstacles to the client. Drafting the appropriate recommendation is key to improving the organization's software development capabilities. This blog post series, based on a paper by me and my colleagues Hasan Yasar and Aaron Volkmann, discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the software development lifecycle (SDLC) within highly regulated environments (HREs).

• The first post explored challenges and goals of implementing DevOps in HREs.
• The second post discussed the first step in a DevOps assessment: setting expectations with the organization. This critical task sets the boundaries of what will be performed and delivered.
• The third post discussed the second step in a DevOps assessment: establishing the pre-assessment DevOps posture of an HRE.
• The fourth post in our series explored how to perform the DevOps assessment.
• The fifth post in our series discussed how to formalize assessment findings and provide appropriate recommendations.

In this post, the sixth installment in our series on DevOps in HREs, we discuss the process of reviewing the formalized list of issues and drafted recommendations with your fellow DevOps assessment team members, your management, and the client.

The Internal Review

At this point, as discussed in the fifth post in this series, the issues arising from assessment interviews and studies of the SDLC process have now been formally described. Before completing the final report, however, it is a good idea to review the list of issues and their recommendations with your peers. This review is a straightforward process meant to validate the findings of the DevOps assessment and add/delete/modify anything that may have been overlooked.

Start the review by calling a meeting with team members who conducted the DevOps assessment, the team's immediate management, and other relevant stakeholders. Give these team members a presentation describing the engagement to date, including the number of days spent in interviews and the number of interviews conducted during that time. Provide the internal team with descriptions of why these folks were chosen for interviews and demonstrate, using an organizational chart of the client, that coverage of all relevant areas to the SDLC under analysis was achieved.

During this review, it is essential to convince the assessment team, internal management, and other stakeholders deemed necessary that you thoroughly covered all parts of the organization, via interviews, engaged directly with the SDLC. It's also important to demonstrate that the interviews were conducted primarily with technical staff with a smaller part composed of management. Recall from previous posts that the technical folks actually engaged with the SDLC will be keenly aware of issues and the most qualified to express those problems to others. Therefore, make sure to provide an overall summary of the main areas where issues arise, such as communication, administration, code testing, and deployment.

After these steps are done, state each issue, your reasoning for its existence, and the potential negative SDLC impact and recommendation. The critical step in reviewing issues with peers and management is convincing them that the issues are causing negative SDLC impact and are compelling to address. As the review progresses, encourage others to express concern or disagreement regarding any issue or recommendation. It is imperative to modify/remove issues that peers identify as not valid in the internal review. The blowback of presenting questionable issues to the client can be quite severe and should be avoided.

Repeat meetings with peers and management until majority consensus on the issues and recommendations is established. At this point you can now meet to review issues and recommendations with the client.

Review with the Client

Reviewing issues and recommendations with the client is different than with internal peers and management. During your meeting with the client, it is possible that the client may

1. disagree with you
2. mention names of employees to blame
3. provide reasons not to be held accountable
4. become deeply concerned about management blowback
5. hold you accountable for false claims
6. express frustration

The above is just a small list of potential outcomes. Remember, your findings are a formalization of criticism. Assessments typically focus on the negative, bad, failing, and faulty but also report the good and positive. With this in mind, selecting which client personnel to review the issues with is critical and highly important.

The first--and possibly only--person you want to review the issues and recommendations with is the one who is funding you. This individual can then determine the other members of the client organizations who should attend, including possibly those in the client's organization who tasked the funder with seeking a DevOps assessment. These folks are the ones interested in the results and steps to fix and/or mitigate. The number of people from the client's end at your review meeting should be small. You need to control unwanted leaks and rumors.

After starting the meeting with professional greetings, review the purpose of the meeting with the client. Remind the client that the purpose of the assessment is to improve the SDLC. State that the issues to be mentioned are impacting SDLC effectiveness or efficiency and should be addressed. Finally, ask if the clients have any questions before starting.

If there are no issues, begin by reading the issues discovered during the DevOps assessment and the subsequent recommendations. Simply state each issue, how it negatively impacts the SDLC, and the potential remedy to apply. Pause after completing each issue and ask for questions. Provide compelling evidence to assure the client of the veracity of each issue. After each issue has been reviewed, ask for final questions and inform the client that the final report will be delivered soon. It is important to give the client the opportunity to make requests and potential changes to the recommendations included in the final report, but do not add or remove issues in response to requests by the client, which could hurt SDLC improvement efforts.

It's also important to keep in mind that the final report should only include those issues that were observed by you and your team during assessment interviews. Conducting thorough research for each issue will ensure that the client leaves the meeting confident in your findings. If, for some reason, the client has cause to doubt any part of your findings, they may completely ignore your final report. As a result, no improvements would be made in the SDLC.

Risk of Meeting with the Client Before Submitting Final Report

Reviewing the issues with the client is optional and more of a courtesy. The update is a draft of your issues and recommendations that will compose the majority of the final report. As discussed previously, there are some risks to reviewing issues ahead of the final report. After the funders are aware of the issues, they can take immediate action even before the final report is submitted, including firings, re-organizations, pay halts, forced vacations, HR investigations, and more.

If the funders and/or client may be held accountable for the findings, they may mount counterarguments and prepare a defense. One way to mitigate this negative backlash is to make it clear to the funders that the findings are meant as a starting point to improve the SDLC with DevOps and non-DevOps concepts. In meeting with the client, stick to fact-based statements and avoid negative statements regarding the issues discovered during the DevOps assessment.

Assuring Negative SDLC Impact

A key goal of reviewing issues and recommendations with the client is to assert that the issues are real and negatively impact the SDLC. The client must be convinced that you have discovered actual problems with the SDLC. One potential way to do this is by describing your reasoning in determining a negative impact. The following can help you illustrate negative impact to the SDLC process:

1. Provide a brief on the general SDLC process, its goals, implementations, and methods.
2. For each issue, explain how it occurs and where it occurs in the SDLC.
3. Provide hard facts such as interviewee statements, written process details, flow charts of the SDLC, etc.

Convincing the client that these issues are a reality will encourage trust in your overall final results. Remember, the goal is to improve the SDLC. All efforts in the review must by navigated toward that goal. If the client questions why a recommendation is not based on DevOps practices, reiterate that DevOps will not be the best approach to solving all the issues. There will be issues outside the scope of DevOps the require mediation, which is normal and should be expected. Even though a critical reason for the goal is applying DevOps to the SDLC there will be issues that will be solved with non-DevOps practices.

To-Don'ts in the Review

1. In meeting with the client, stick to fact-based statements and avoid negative statements regarding the issues discovered during the DevOps assessment. Do not provide detailed dates and times where meetings occurred.
2. Do not identify the groups, teams, directorates, or divisions that were approached for interviews.
3. Do not express any of your or the client's or assessment team members' personal views on the demeanor of personnel regarding management, projects, or the organization.
4. Do not let the client dictate which issues appear in the final report.
5. Do not let the client have the final review/say on the report before widely distributing.
6. Do not state your personal feelings on the atmosphere of the client's organization. Save that for the final report.
7. Do not share anything else with the client. Stick to the issues and recommendations.
8. Do not solicit client input on the tone, content, or any other perspective regarding the final report.
9. Do not compromise on your findings.

Breach of Ethics or Policy

As we discussed, the client may want to take control of the final DevOps assessment report in various ways. Remember, you are meeting with the funders and requestors of the assessment. They might try to compromise or harass you to purposely modify some aspect of the issues and/or recommendations. Do not express any of your client's personal reviews on the demeanor of personnel regarding management, projects, or the organization. If you do not agree to the client's requests regarding the assessment findings, the client might become upset. Report these situations to your management and seek guidance. As you await instructions from management, cease all communication with the client. Move forward with creating the final report, and do not let these situations change the tone of the writing; keep it objective and professional.

Wrapping Up and Looking Ahead

In this blog post, we have discussed reviewing issues and recommendations with internal peers and management and with the client. The bottom line is, you need to give others confidence that

• the issues are real.
• the issues are negatively impacting the SDLC.
• the issues need to be fixed.
• the recommendations are an excellent remediation approach.

Do not compromise anything. Stick to your findings and remain objective, which will aid you in a successful review process. The next step is to write and deliver your final report, which we will discuss in our next blog post installment.

Additional Resources

Read the paper this series of blog posts is based on, Implementing DevOps Practices in Highly Regulated Environments, which was presented at the 2018 International Workshop on Secure Software Engineering in DevOps and Agile Development.

Read other posts in the DevOps blog.

Listen to the SEI Podcast Roundtable, How Risk Management Fits into Agile and DevOps in Government.