The overall purpose of a DevOps assessment is to help improve the software development lifecycle (SDLC). Applying DevOps in highly regulated environments (HREs), be they academic, government, or industrial, can be challenging. HREs are mandated by policies for various reasons, most often general security and protection of intellectual property. The restrictions of these policies make the sharing and open access principles of DevOps that much harder to apply.

This blog post series, based on a paper by me and my colleagues Hasan Yasar and Aaron Volkmann, discusses the process, challenges, approaches, and lessons learned in implementing DevOps in the SDLC within HREs.

• The first post explored challenges and goals of implementing DevOps in HREs.
• The second post discussed the first step in a DevOps assessment: setting expectations with the organization. This critical task sets the boundaries of what will be performed and delivered.
• The third post discussed the second step in a DevOps assessment: establishing the pre-assessment DevOps posture of an HRE.

In this fourth installment of our blog post series, we discuss how to perform the DevOps assessment.

Spend Time with the Right People

At this point in the process, you and the HRE organization have set expectations and agreed on the entity's current DevOps posture. Now the assessment can start.

Using the entity's organizational chart, schedule meetings with the software developers and peers of the group whose SDLC you are assessing. It is important to talk with the people actually developing the code and similar activities, along with their team leads. These people are the most keenly aware of the SDLC's obstacles and will likely be willing to tell you, with the hope that you can make their working lives easier.

Spend a smaller amount of time meeting with middle and senior management. The further removed a person is from performing work following the SDLC, the less they are aware of the problems. Management may actually give a very positive description of how things are going, perhaps because managers aren't always fully versed in all the granular details that can impact a software development project. Managers can also be held accountable, so they may not want others to know their problems.

Anonymity Is Critical

A critical task for the assessment team is to guarantee complete anonymity of participating personnel. Anonymity will encourage participants to speak frankly about the issues at hand. Without a guarantee of anonymity, some employees may not fully divulge due to fear of retribution from management and peers. Achieving anonymity can be done in the following manner:

1. Ensure that management will not question their personnel on details regarding interactions.
2. Create an agreement to refrain from listing names or titles of personnel in the final report and any other artifacts handed over to the organization.
3. In reports and other written artifacts, do not refer to any product, process, project, location, or anything else that can uniquely identify an individual.

The Importance of Confidentiality

Keeping conversations confidential will further make the participant feel at ease and free to speak frankly and openly. Meet with only one person at a time: just you, the rest of the assessment team, and the individual. Meeting with groups is not ideal because some people may not feel comfortable speaking openly in front of others. Meeting with singular participants also allows you to validate what others have said. The meeting room should be comfortable, for example, it should be large enough to fit everyone easily, equipped with executive chairs, and with comfortable temperature and soft lighting. You want the participant to feel they are interacting with you for their benefit, not being interrogated.

Make Introductions

Start the meeting with the usual professional greetings and introductions. Then clearly state the purpose of the meeting:

We have been tasked by (name of requestor) to perform a DevOps assessment of this SDLC (or the SDLC used by your group). The purpose of this meeting is for you to tell us existing problems that hinder the performance of the SDLC.

If you have a paper copy of the SDLC, show it at this point and ask if the person is familiar with it. If they are not, then provide a briefing and offer to email them a copy for use during the meeting. If they feel they want to review the SDLC more deeply, then reschedule. You should follow the purpose statement by informing the person of their anonymity and the confidentiality of their responses:

We guarantee anonymity and confidentiality. We will not reveal to others that you spoke with us and will not identify you directly or indirectly in our deliverables. We do this so you can speak openly and frankly.

You then ask the participant to verify their placement in the organizational chart. Also ask how long they have been in this current position. Follow up by asking if the participant has any questions and address them accordingly. After these steps are completed, begin discussing the SDLC.

Overcoming Obstacles

Ask the individual what their responsibilities are and what work they do during a typical day. They should tell you succinctly what they do and how their day is carried out. Begin by asking what their role in a software project is and how they fulfill it. The whole purpose of these questions is to ascertain what this person perceives to be the current process.

When showing or referencing the SDLC and referring to their position, ask them what problems delay or stop progress. As they state each problem, ask detailed questions. For example, when a participant states a problem such as, "I cannot upload code to a repo when I want," detailed follow-up questions could include the following:

• Explain to me the steps of uploading to the repo?
• Detail step 3 for me again.
• So, when you submit to the queue for uploading, you get rejected due to missing admin privileges. Where in the system are those privileges assigned?
• Why do you think the privileges keep erasing?

As you and the participant discuss the problem, take notes to formulate what the obstacle is. Remember that, at first, you may only get symptoms and need to ask follow-up questions to discover the root causes. When you feel you have a good idea of the obstacle, state it to the individual for verification. Repeat this process until the participant has no more problems to speak of.

Tooling and Processes

At this point you have a handful of obstacles as described by the participant. The next phase of questions are more generic and refer to the existence of tooling and processes. These questions will validate the existence of DevOps principles and implementations:

1. How is stakeholder communication accomplished, how often, and by what means?
2. Do all requirements come at the beginning of a project, or are some added after work has begun?
3. How is artifact delivery carried out?
4. How are HRE external stakeholders given access to artifact documents and project progress?
5. Is there a feedback loop between stakeholders and end users in the production environment?
6. Are staging environments used always, never, or only when the production environment is inaccessible?
7. Can changes be made to an artifact during and after initial deployment?
8. What is the project authorization process?
9. What other bottlenecks, not covered here, are causing delays to project commencement and completion?
10. What is the hardware acquisition process?
11. Are there any non-HRE development activities contributing to the project, and how are they managed?

At this point in the conversation, you should have a list of problems the person has stated and that you have inferred. Keep in mind that you will discover issues that the participant will not recognize as such. Ask if you can follow up with the person on issues discovered during this conversation. For each discovered issue, dig deeply into process details to ensure full understanding. Discuss these same issues with other participants to validate their existence, symptoms, and root causes.

For each identified issue, it is important to understand where and how it impedes the SDLC. If this is not clear, conduct further questioning until it becomes clear. After the conversation is finalized, you and the participant should cordially depart, keeping the door open for future engagement. The participant should leave feeling good and fulfilled for discussing the issues with you.

Cautions

The entire interview portion of the assessment is basically a conversation between yourself and individuals working for the organization that hired you. There is no exact formula for executing these conversations; they are both an art and a science. The conversations are dynamic and will change in response to the answers participants provide. Be flexible enough to change the course of questions in real time. Remember, the purpose of this entire conversation is to gather more detailed understanding for a discovered issue. It is a mistake to enter these conversations with a strict agenda of questions and no willingness to depart from it. The conversation must flow dynamically to maximize information gathering.

Beware of these conversations slipping into a therapy session, where the participant starts describing issues that fall outside the scope of the SDLC. Issues affecting the SDLC can intersect with personnel issues, so expect this type of feedback. When it occurs, ask questions to determine if there is an SDLC component to the problem. If there is, note it and provide solutions. If not, document it and present it as a non-DevOps issue that is affecting the SDLC in some way. Refrain from saying it's just one person's problem, that could be viewed as a personal perspective and could get ignored. Instead, frame the issue as affecting personnel who are part of executing some aspect of the SDLC.

Wrapping Up and Looking Ahead

Now that the assessment is complete, the next step is to formalize the findings and propose solutions. This can be harder than it looks and will require conversing again with personnel. We will explore this step in our next blog post.

Additional Resources

Read the paper from which this series of blog posts is based upon, Implementing DevOps Practices in Highly Regulated Environments, which was presented at the 2018 International Workshop on Secure Software Engineering in DevOps and Agile Development.

Read other posts in the DevOps blog.

Listen to the SEI Podcast Roundtable, How Risk Management Fits into Agile and DevOps in Government.