On July 14, 2016, the House Ways and Means Subcommittee on Social Security convened a hearing on the Social Security Administration's (SSA) information technology modernization plan. The hearing focused on the current state of the Social Security Administration's (SSA) Information Technology (IT) modernization plan and best practices for IT modernization, including oversight of agile software development. Agile development approaches, relatively new in government settings, create opportunities for rapid deployment of new capabilities but also pose challenges to traditional government oversight and management practices. A team of researchers from the SEI's Agile in Government (AIG) team were part of an expert panel brought in to testify before the members of the subcommittee. The team, comprised of me, AIG principal engineer Will Hayes, and AIG program manager Eileen Wrubel, developed written testimony that was submitted to the committee in conjunction with verbal testimony delivered by Hayes. This blog post, the first in a series, presents the written testimony as submitted to Congress, drawing upon seven years of research the SEI has conducted on the use of Agile in government settings. Specifically, this post provides a summary of challenges observed by the SEI in overseeing Agile programs in government such as progress measurements, IT transformations beyond Agile, and workforce development of government staff working in Agile settings.

In the following testimony, the Software Engineering Institute Agile in Government team addresses these points:

• Agile is not a "silver bullet" that solves all the complex management and engineering problems of government IT Transformation efforts, but it has contributed to successful IT Transformation efforts.
• The benefits achievable when development organizations use Agile methods only manifest when the development (government or contractor developer) and oversight (government acquisition) efforts are aligned.
• The approach and mindset to government obligations in oversight must change when Agile is the focus of development and SEI has observed negative consequences in organizations that do not address these changes.
• Changing the oversight approach in Agile settings means asking different questions on a new cadence than oversight organizations have in the past. This leads to different measurement and reporting approaches as well.
• A focused government workforce development effort is required to enable the knowledge, skills, and abilities needed for effective oversight and interaction in Agile settings.

2 Software Engineering Institute's Agile in Government Team Perspective on Agile in IT Transformation

2.1 Introduction

The Software Engineering Institute's (SEI) Agile in Government team is pleased to provide testimony to the House Ways and Means Committee on the topic of Agile opportunities and challenges in IT transformation oversight. For over three decades, the Software Engineering Institute (SEI) has been helping government and industry organizations to acquire, develop, operate, and sustain software systems that are innovative, affordable, enduring, and trustworthy. We serve the nation as a Federally Funded Research and Development Center (FFRDC) sponsored by the U.S. Department of Defense (DoD) and are based at Carnegie Mellon University, a global research university annually rated among the best for its programs in computer science and engineering.

Overseeing the acquisition of large software intensive systems in government settings brings many challenges; however, we believe our observations and experiences with government programs using Agile methods in IT transformation can elucidate the issue. Keeping our focus to senior oversight aspects of these projects, there are practical application and policy implications of our observations.

2.2 Major Observations

Agile is an iterative approach to software delivery that builds and delivers software incrementally from the start of the project, instead of trying to deliver it all at once near the end (for more detail, see the Appendix). In the case of an Agile life cycle, requirements and solutions evolve through collaboration among self-organizing teams and project sponsors to encourage rapid and flexible response to change. The following observations reflect over seven years of SEI involvement in research and field support of the application of Agile methods and principles in government settings.

The SEI's Agile in Government team was chartered in 2009 to initially answer the question posed by a senior US Air Force official: "Can Agile methods actually be used in government settings?" That question led to researching multiple dimensions of the application of Agile methods and principles in a wide variety of government scopes and settings. [Ward & Miller 2016]

Our first observation is one that may seem obvious -- Agile can't solve all your IT Transformation problems. IT Transformation is a multi-dimensional challenge. Agile approaches, although not a silver bullet, can contribute to successful transformations when appropriate conditions are met. In a recent article titled "Embracing Agile," the Harvard Business Review asserts several conditions that reflect an appropriate setting for Agile success [Rigby 2016]:

• The problem to be solved is complex.
• Solutions are initially unknown.
• Product requirements will most likely change.
• Work can be modularized.
• Close collaboration with end users (and rapid feedback from them) is feasible.

We have worked in several government settings that meet all of these criteria and have often found Agile to provide benefits that are valued by the sponsors of the projects as well as the end users of the software products, albeit not without challenges. The SEI is familiar with many of the particular adoption hurdles of Agile development and management, especially in a contracted setting. [Lapham 2011]

However, the benefits are only achievable when oversight and development are aligned in several areas to include both development execution and mindset changes around the topics of: [Miller 2014]

Lastly, particular challenges relevant to oversight are catalogued in Table 3.

Agile, as an alternative to traditional project management, can be a response to common acquisition failures. Large technology projects are fundamentally uncertain - time and technological advancement ensures that what you want today, is almost never what you want tomorrow or what you will get in a week. Agile's inherent flexibility can reduce some of the adaptability risks ingrained into large software projects.

2.2.1 Agile in Software Acquisition

For example, requirements creep is a universally acknowledged problem, and dealing with changes in requirement is considered the bane of program managers; yet, regrettably "Previous experience shows that changes within an SIS [software-intensive system] are inevitable, whether or not there are changes in requirements or technology" [Kennedy 2011].

A strength of Agile approaches is to explicitly anticipate and factor in the effects of inevitable change; however, oversight of systems development is often not consistent with Agile principles and methods. (See the Appendix)

2.2.2 Oversight for Agile has Different Questions

Organizations that are successful ask different questions -- not the ones typically found in oversight guidance. We observe the following themes in effective organizations:

• emphasis on mission needs priority vs cost
• continual focus on product quality
• evolving systems based on learning vs big bang
• oversight cadence aligned with delivery of small batches of work

3 Overseeing Agile Settings in a Government Context

In most cases, government organizations aren't DOING Agile development; more often they engage with Agile development contractors. Even in the case where organic development is occurring within a government organization, senior level oversight of Agile should take on a different cadence and focus than in traditional, large batch development settings.

3.1 Focus Areas of Oversight Approach in Agile Settings

The responsibility for oversight and due diligence doesn't change, but the approach to oversight in an Agile setting does. Topics relevant for senior government oversight personnel include, but are not limited to:

• level of understanding government personnel have for the complex tradeoff space involved when making replace, evolve, or fix decisions with an aging legacy system
• level of understanding government personnel have of Agile concepts and their relevance to program performance; examples to discuss might include:
  • identifying clashes between Agile timelines and traditional long-lead time expectations
  • managing technical debt and appropriately investing in modernizing legacy system components
  • assuring investment in technical infrastructure required to enable rapid delivery of quality products (e.g., test environments)
  • level of understanding of government program about how to partition system deliveries to gain incremental value on a regular, short term basis while maintaining alignment with enterprise architecture vision
  • level of willingness of government program office to engage in small batch interactions on a frequent (as much as every two weeks) cadence with the developer organization(s).

Although these areas of focus are not all directly amenable to traditional measurement, the discussion of the topic with the government program office is likely to lead the senior oversight staff to greater or lesser confidence in the government program's ability and commitment to execute in an Agile fashion.

3.2 Progress Measurements that are Relevant for Oversight of Agile Settings in Government

The following suggestions are offered with the caveat that relevant leading indicators of progress are highly specific to a particular context. The adage "what gets measured, gets done" is both fortunately and unfortunately true. Measures often lead to unintended consequences that are in opposition to the goals of those who established the measures.

Many of the typical measures you will see, with Agile methods in the literature are focused on development team measures, however, those are not relevant for senior oversight and their use by senior level oversight is likely to skew the focus of the work being done. [Austin 1996] [Hayes 2013]

We offer two types of suggestions with regard to measures:

• Categories of measures that should be discussed with the government program in terms of how things relevant to the particular effort are being measured. Agreement on measures to send forward to the senior oversight group would be a result of this discussion
• Measurement examples and their connection to the categories discussed above.

The following are examples of measures we have seen used in executive level dashboards for services similar to those provided by government agencies.

Agile approaches in major IT transformation efforts must successfully account for key success/failure drivers that persist regardless of the development cadence and methodology chosen.

5 Call to Action

If asked for recommendations, the SEI Agile in Government Team would emphasize two areas:

• Ask different questions of those using Agile as part of their IT Transformation strategy
• Support development of Agile oversight knowledge, skills, and abilities for the government workforce

5.1 Ask Different Questions

Any successful IT transformation involves more than the dedication of talented development staff, you need to ask about alignment of people, processes (Agile or otherwise), technology support, or business context. There are two particular contexts for which different questions could be relevant--evaluation of an agency's IT transformation strategy, and oversight of an agency's ongoing IT transformation effort.

About the (Agile) IT Transformation Strategy

• If iterative and incremental approaches to modernization are intended, how does the cadence of work done by the contributing elements of the organization align to the modernization roadmap? How will leadership keep the effort synchronized, when the pace of deployment is expected to increase dramatically?
• Depending on the scope of workflow activities served by the IT transformation effort, how large and diverse is the cohort of service delivery staff who will see an IT-driven change? How deep into the flow of normal work activities will this IT-driven change permeate?
• Is there evidence of consideration for resolving high-uncertainty or high-risk areas at appropriate times in the timeline - rather than deferring undue amounts of risk?
• With a roadmap for incremental deployment of these IT-driven changes in hand, stakeholders can ask focused questions about the value returned to those served by the organization. Questions can focus on the timeframe and/or scope of deploying certain new technologies (or the retirement of old subsystems) in light of the scope of anticipated impact.

About the ongoing (Agile) IT Transformation Effort

• How successful is the organization in defining small batch approaches to the work? Are the development teams able to sustain a short iteration length (measured in weeks, not months)?
• What is the trend in the time it takes to mature a concept all the way through development activities to the point where it is an accepted element of a service or maintenance workflow?
• How consistently does the organization deploy capabilities in accordance with the transformation roadmap?
• Are planned major technology updates/introductions occurring according to the roadmap?
• How well is the balance of routine system maintenance, defect repair, and new deployments being managed? Does the pattern of activity match the plan, and is the organization's capacity adequate?
• What patterns of feedback are seen from the users of IT systems involved in the transformation? User experiences associated with the IT systems as well as the Agile approach should be sought.

No matter what specific measures are chosen to provide oversight, it is vitally important to avoid limiting visibility to measures expressed in units of resource consumed (be it time or money). Most Agile methodologies call for steady resource loading and fixed iteration lengths. Schedule and cost overruns are lagging indicators, which can be anticipated by examining the flow of delivered capabilities in Agile development.

5.2 Workforce Development of Government Staff Working in Agile Settings

Increasing access to Agile-related training for government staff has already started in some areas. OMB Digital Service has sponsored development of customized learning paths and related courses to help contracting officers and others involved in managing Agile efforts in government to understand the concepts and to offer guidance in fulfilling agency needs with Agile projects. [OMB 2016] The US Digital Service has also sponsored several outreach and awareness building communication activities targeted at government agencies. [USDS 2016]

Several federal agencies, including Department of Homeland Security and the Internal Revenue Service, have written guidance or policies related to enabling Agile IT projects to productively occur.

In the Department of Defense arena, the Defense Acquisition University is in the process of enhancing its IT acquisition curriculum to include content related to Agile. A Continuous Learning course on Agile considerations in government acquisition is in work, for which the SEI Agile in Government team is an author.

One caution: There are many certifications available in the commercial community related to Agile, and almost all of them focus on roles directly involved in software development. Although many have useful content for government staff to know, a certification program that focuses on the commonly available Agile methodologies would not serve the knowledge and skill needs of the staff who write the contracts and do the oversight of government IT projects.

Appendix A: Agile Tenets and Principles

The basis for all Agile methods are a set of four tenets and twelve principles that were developed by a group of software methodologists in 2000. They were concerned that more and more software projects in the commercial space (their primary area of work) were becoming larger, more unwieldy, and less successful than smaller, more "agile" projects they had observed.

Throughout two days of collaboration, they established the Manifesto that is reproduced here. Any method that claims to be Agile should be able to discuss how its practices and methods ex-press these tenets and principles. With each principle, we have included a short statement of adaptations that we see being needed to express the principle in government settings.

Manifesto for Agile Software Development
http://agilemanifesto.org/

"We are uncovering better ways of developing software by doing it and helping others do it."

Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Wrapping Up and Looking Ahead

Future posts in this series will take a deeper dive into the areas that need to be considered beyond Agile including the following:

• managing technical debt
• effectively dealing with cybersecurity
• understanding the quality issues
• architecting for evolution and incremental delivery
• communicating needs effectively to system developers

Additional Resources

To learn more about the SEI's research in agile adoption in government settings, visit https://www.sei.cmu.edu/research-capabilities/all-work/display.cfm?customel_datapageid_4050=21716.

References

[Austin 1996]

Austin, R. Measuring and Managing Performance in Organizations. New York: Dorset House Publishing. 1996.

In addition to providing excellent guidance, this book offers coherent advice on avoiding the misuse of measurement.

[Hayes 2013]

Hayes, W. et al. Agile Metrics: Progress Monitoring of Agile Contractors. CMU/SEI-2013-TN-029. Software Engineering Institute, Carnegie Mellon University. 2014. https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=77747

This SEI report provides initial guidance on issues to address when designing a program of Agile progress metrics in a government setting.

[Kennedy 2011]

Kennedy, M. "Achieving Better Acquisition Outcomes in Austere Times." Defense Acquisition University Symposium. April 2011.

This presentation from (at the time) DAU professor Matthew Kennedy, reviews relevant aspects of Agile development for government program offices.

[Lapham 2011]

Lapham, M.A. et al. Agile Methods: Selected DoD Acquisition and Management Concerns. CMU/SEI-2011-TN-002. Software Engineering Institute, Carnegie Mellon University. 2011. https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=9769

This SEI report addresses several topics of concern to acquisition professionals considering or already taking on adoption of Agile methods and principles. This report includes a chapter on cultural issues that commonly must be addressed in Agile adoption.

[Miller 2014]
Miller, S.M. "The Readiness & Fit Analysis: Is Your Organization Ready for Agile?" Software Engineering Institute, Carnegie Mellon University. 2014. https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=90977

This white paper summarizes a method that the SEI has developed to help government and other regulated organizations understand the adoption risks they may face when they undertake adoption of Agile methods and principles.

[Nielsen 2009]

Nielsen, P. Challenges to Effective Acquisition and Management of Information Technology Systems. Invited testimony for the U.S. Senate Armed Services Committee. 2009.

Dr. Nielsen's testimony outlines the major challenges that the SEI has seen in supporting major software acquisitions over the previous twenty years.

[NIST 2010]

Various authors. "Guide for Applying the Risk Management Framework to Federal Information Systems." SP-800-37. National Institute of Standards and Technology. 2010.
https://csrc.nist.gov/projects/risk-management/risk-management-framework-(rmf)-overview

This guide provides organizations in the federal space with guidance on applying NIST's Risk Management Framework, designed to provide a risk-based approach to cybersecurity threat management.

[OMB 2016]

Various authors. Announcement of Agile Contracting Curriculum. 2016. https://www.actiac.org/groups/proposed-digital-service-contracting-professional-training-and-development-program-challenge

This program is meant to help acquisition professionals understand various aspects of Agile and lean concepts so as to better prepare contract materials in Agile settings.

[Rigby 2016]

Rigby, D. et al. "Embracing Agile." Harvard Business Review. May 2016.

https://hbr.org/2016/05/embracing-agile

This article provides a high-level summary of issues in leadership of an organization moving towards Agile. Although the examples go beyond IT transformation settings, they are still relevant to IT transformation.

[USDS 2016]

Various authors. United States Digital Service blog.
https://www.usds.gov/blog

This site provides ongoing information about the work that U.S. Digital Service sponsors and engages in.

[Ward & Miller 2016]

Ward, D. and Miller, S. Update 2016: Considerations Using Agile in DoD Acquisition, 2016-TN-001, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA 15213

This is an update to the original technical note that started the SEI Agile Adoption research. It adds in a case study and references 2015 version of DoD 5000.02 and its changes that better support Agile use in DoD settings.