For more than 10 years, scientists, researchers, and engineers used the TeraGrid supercomputer network funded by the National Science Foundation (NSF) to conduct advanced computational science. The SEI has joined a partnership of 17 organizations and helped develop the successor to the TeraGrid called the Extreme Science and Engineering Discovery Environment (XSEDE). This posting, which is the first in a multi-part series, describes our work on XSEDE that allows researchers open access--directly from their desktops--to the suite of advanced computational tools and digital resources and services provided via XSEDE. This series is not so much concerned with supercomputers and supercomputing middleware, but rather with the nature of software engineering practice at the scale of the socio-technical ecosystem.

Background: Bringing Disciplined Engineering to TeraGrid

From 2001 to 2011 the NSF's network of supercomputers, services, and middleware--known as the TeraGrid--played a key role in establishing and supporting the emerging discipline of computational science. Computational scientists used the TeraGrid to study the atomic-level structure of protein needed for sound perception, investigate how gas giant formations like Jupiter and Saturn are formed, and gain new insights into the early stages of HIV-1 infection to prevent the virus' ability to "hijack" other healthy cells.

By 2008, however, it was becoming clear to NSF's Office of Cyberinfrastructure (OCI) that TeraGrid's success had generated new demands from computational scientists that could not be satisfied by the existing infrastructure. While TeraGrid had provided technologies and services to integrate the nation's most advanced supercomputing resources, by 2008 there was a growing perception among computational scientists that not only did the national cyberinfrastructure ecosystem need to provide access to supercomputing resources, there needed to be a easier way to integrate a much broader array of digital assets and services in conjunction with much greater stability and other quality attributes in order to support top-tier research programs.

In 2008, NSF/OCI issued a call for proposals for TeraGrid Phase III: eXtreme Digital (XD) Resources for Science and Engineering. As expected, the Phase III solicitation called for a continuation of and substantial improvements to core TeraGrid technologies, as well as to its outreach, education, consulting, and infrastructure-management services. The solicitation also included something surprising--a strong emphasis on software and system engineering in Phase III. NSF reinforced this new emphasis in a series of town hall meetings for potential submitters.

Among the takeaways from those meetings, one thing was certain: The winning proposal would need disciplined engineering processes. But which processes? It was this question that Pittsburgh Supercomputing Center director Michael Levine posed to the SEI. The SEI's foundational role in identifying "process" as a lever for improving practice--and in achieving industrial-scale adoption of software engineering process through the Capability Maturity Model Integration (CMMI) and its many derivatives--made it a natural choice for Levine and his team.

Levine, a key leader in TeraGrid, also understood that NSF's culture and the TeraGrid community in particular required something different than the conventional notion of "process." It was this understanding that ultimately led Levine to the SEI's Research, Technology, and Systems Solution (RTSS) program. After a few rounds of discussion, RTSS joined a team that included the National Center for Supercomputing Applications (NCSA), Pittsburgh Supercomputing Center (PSC), Texas Advanced Computing Center (TACC), and the National Institute for Computational Sciences (NICS). The team assembled by Towns at the NCSA ultimately developed the winning proposal. The successor to TeraGrid would now be known as the eXtreme Science and Engineering Discovery Environment (XSEDE).

Our Challenge: Creating an Engineering Culture

NSF decided to award one-year planning grants to the two most competitive preliminary proposals: XSEDE and XROADS.

NSF had awarded the planning grant with the stipulation that the results were to be shovel-ready implementation plans, ready on Day 1. That is, the winning proposal would need to go operational with no disruption of service to any current customer of TeraGrid. The XSEDE team recognized the following challenges of developing a credible management and technical plan:

• For the management plan, this meant completely working out all the details about project structure and governance.
• For the technical plan, this meant defining the software and system architecture needed to support immediate implementation.

The SEI team believed that the challenge was of a crosscutting nature: Project governance, software and system architecture, and engineering practice must be mutually reinforcing. Our first challenge was to persuade the XSEDE team to abandon the belief (as it turns out, a belief that was widely held by both XSEDE and XROADS teams) that "doing" software engineering was nothing more than following a process, and, by extension, the SEI could define "The Process."

To overturn this mistaken belief would require persistence. We undertook two efforts to do so:

• We developed a traditional System Engineering Management Plan (SEMP) that described top-level engineering processes and organizations.
• We piloted architecture-based engineering practices to help the architects engage internal stakeholders to decide what mattered most in the architecture.

As expected, the XSEDE team was mostly indifferent to the content of the SEMP, except to note that it lacked narrative "pizazz." They were surprised, however, by the effectiveness of techniques--such as the Mission Thread Workshops led by the SEI's Mike Gagliardi--at shedding light on the organizational implications of engineering decisions.

Most importantly, the team began to appreciate that the steps of "The Process" do not define most of what we regard as software engineering. Instead, real software engineering practices reside within these steps--the parts that often aren't written down.

Our New Challenge: Doing it Again, In Operation

With the one-year planning grant complete, in 2011 XSEDE and XROADS teams submitted and briefed their plans to the NSF Review Panel. When asked what worried him most about the review process, John Towns (XSEDE PI) replied he was concerned that the panel would decide to fund part of each of the XSEDE and XROADS efforts. That funding model would violate the by-now integrated management, technical, engineering and governance structures that XSEDE had developed in its plan.

As it turned out, Towns' fears were partly realized when NSF announced its intent to award XD to XSEDE with the proviso that it incorporate the most meritorious elements of the XROADS effort and team. We were now confronted with the need to revisit planning assumptions and decisions made with a full year of deliberations--in effect to "re-plan" the effort in the span of a few short weeks.

In hindsight it is clear that NSF made a wise decision in merging the XSEDE and XROADS proposals, both in terms of avoiding the schism in the community that might have arisen from a "winner-take-all" outcome, and in terms of incorporating several exciting architectural ideas introduced by XROADS, such as delivering cloud resources as an alternative to leadership class "big iron" computers, and adopting "software as a service" as a business model and product delivery channel.

On the other hand, much of what we had accomplished in terms of introducing the seeds of an engineering culture in XSEDE had been thoroughly disrupted. After a year of engaging the XSEDE team on the nuances of software engineering methods, these methods had begun to pay dividends. Of course this progress relied on the skills and perspectives of just one of the two teams with which we had engaged.

Now our engineering teams would be reconstituted with people from both teams. Not only had we potentially lost a "common sense" of software engineering, but we now had engineering teams with "competing views" of what to accomplish. We had to reconstitute an engineering culture while simultaneously realigning our technical objectives. Moreover, because the TeraGrid effort was coming to its contractual end, we needed to do this "live," which is like having to redesign and rebuild an aircraft while in flight.

To be fair, although we knew that the merger was going to be challenging, we also knew that we were substantially further along than we had been one year earlier. Both teams had spent their planning efforts wisely, and their overall approaches were more similar than different, and where different, they were mostly complementary. There was also something at work that is difficult to objectively quantify: The members of the XSEDE and XROADS teams were mostly well known to one another, and there was a universal desire to move beyond the competition of the past years, roll up our sleeves, and get to work on the real mission of advancing the nation's computational science.

Results after One Year

What did we manage to accomplish? Did we build a redesigned aircraft while in mid-flight without crashing and burning? After one year, we can safely claim a good measure of success, the results of which are summarized in the figure below.

Figure 1 shows a portion of the XSEDE program that is engaged in engineering work on a day-to-day basis, as described below:

1. The SEI helped introduce an explicit and rigorous treatment of software architecture as a keystone of the XSEDE engineering approach. We reinforced concrete practices for documenting software architecture and for understanding, documenting, and using quality attributes to design and evaluate how well an architecture serves its strategic aims.
2. The SEI helped introduce a range of concrete software development practices, beginning with unit, integration, and acceptance testing, which had been completely lacking in TeraGrid. We augmented these practices with a rudimentary software lifecycle that included basic development phases and verification gates for detailed design and various test phases.
3. The SEI also helped XSEDE leadership evolve to a more realistic and nuanced understanding of "process." XSEDE had evolved from its early quest for The Process to a program that appreciated the importance of establishing an engineering culture, as well as the need for that culture to define and evolve its own engineering processes.

It is fair to say that on Day 1 of the XSEDE/XROADS union, none of the practice areas reflected in the graphic and outlined above were in place, and indeed, few if any had any legacy in TeraGrid. This substantial progress is due not to the SEI but to XSEDE leadership and staff.

Our plans for Year 2 combine two main thrusts

1. Consolidation by documenting and applying practices and making small improvements as needed.
2. Planning for new and potentially groundbreaking practices to scale architectural design to allow contributions from hundreds to thousands of members of the XSEDE community and from the larger computational science ecosystem of science grids, communities, and technology providers.

Upcoming Topics in the XSEDE Thread

My next post will explore establishing an engineering culture within an emerging social-technical system. Other topics that I plan to post about are:

• What are the quality attributes of a socio-technical system that will produce not just one intended system but many systems?
• Can the web of cultural norms and incentives that make up a socio-technical system be described? Can their mutual alignments be measured or influenced? Can these measures predict the quality of engineering decisions?
• What is the role of an integrating technology platform in the formation and maintenance of a socio-technical ecosystem? How can a socio-technical ecosystem be established to design the platform for another ecosystem? (In fact, this is the XSEDE case.)
• Are persistent institutions required to sustain engineering practice in a socio-technical ecosystem? Can a practice be sustained without such institutions?

In addition to the above posts, I'd like to enlist the help of other contributors to the XSEDE effort, as well as researchers of engineering practices in and for socio-technical systems. Please feel free to leave your thoughts in the comments section below.

Due Credit Before Closing

While most of the credit belongs to XSEDE leadership and staff, I will take a few moments here to give credit to SEI colleagues who have contributed to this result:

• Mike Gagliardi introduced XSEDE to large-scale architecture thinking though Mission Thread Workshops.
• Joe Batman demonstrated mature systems engineering practices by showing the connectedness of engineering decision-making and project governance.
• Paul Clements, and later Felix Bachmann, introduced disciplined approaches for documenting architecture in a way that serves stakeholder interests.
• Scott Hissam defined the Software Development and Integration (SDI) approach (see Figure 1) to integration testing, acceptance testing, and version management, and invaluable assistance in environment integration.
• Rhonda Brown and Mike Konrad drew on their experience in CMMI, TSP and more recently, the combination of TSP with software architecture practices, to improve the agility of SDI's overall approach.

Additional Resources

For more information about the XSEDE project, please visit https://www.xsede.org/.
For more information about the SEI's work in architecture centric engineering (ACE), please visit
www.sei.cmu.edu/about/organization/rtss/ace.cfm
For more information about the SEI's work in system-of-systems engineering, please visit www.sei.cmu.edu/sos/.
For more information about the SEI's work in ultra-large-scale systems, please visit www.sei.cmu.edu/uls/.