This post is the second installment in a two-part series describing our recent engagement with Bursatec to create a reliable and fast new trading system for Groupo Bolsa Mexicana de Valores (BMV, the Mexican Stock Exchange). This project combined elements of the SEI's Architecture Centric Engineering (ACE) method, which requires effective use of software architecture to guide system development, with its Team Software Process (TSP), which is a team-centric approach to developing software that enables organizations to better plan and measure their work and improve software development productivity to gain greater confidence in quality and cost estimates. The first postexamined how ACE was applied within the context of TSP. This posting focuses on the development of the system architecture for Bursatec within the TSP framework.

Challenges
From a TSP perspective, the project faced several challenges. First, the few developers who had worked on the existing system had either moved into management or possessed technical skills that were out-of-date with modern development technologies. Second, the remaining developers, while competent, did not have experience in building the type of system that Bursatec needed. Another challenge was that several executives within the organization were in favor of outsourcing the work.

Our Approach
In the Bursatec project, we initially followed the standard TSP implementation approach, which emphasizes the importance of initially securing senior management commitment. This commitment is typically established via a TSP Executive Strategy Seminar, which covers the key practices and principles of TSP from a senior management perspective. Although Bursatec is a large organization, with several layers of checks and balances befitting a national stock market, the organization itself was very open, which allowed for streamlined communication between senior managers and the engineering team.

In this open environment, the director at Bursatec, as well as his boss--who was president of the Mexican Stock Exchange--participated in the executive training. The executive training included an overview of rational management, the idea that management decisions should be made based on objective facts and data, and why this type of management is required to maintain successful TSP teams. We then trained the team leader of the project, as well as several other peers and senior developers at Bursatec in the basics of day-to-day management of TSP teams.

We next trained the entire Bursatec development team--including the architects and team leader--in the fundamentals of the Personal Software Process (PSP), which teaches individual software engineers how to plan and manage high-quality software development work. The team would go on to apply the PSP concepts in a project, team-based environment. In an unusual development, the Bursatec director attended this class and also authored several programs using PSP methods, which he did as well as any of the developers. Having such a senior manager there--not just in the class but using the methods--sent the strong message that this was how "we" would be working going forward.

After completing the PSP training, we conducted a Quality Attribute Workshop. This workshop is an architecture activity where Bursatec stakeholders defined the five most important quality attribute requirements (also known as quality attribute scenarios) that their new trading system had to fulfill. Not surprisingly, given the importance of speed for the new system, the stakeholders identified runtime performance as chief among the most important quality attribute scenarios.

For the Bursatec developers, one benefit of defining quality attributes is that the practice placed significant emphasis on ensuring that the attributes be measurable. For example, the performance attribute was measured in two ways: the time for individual transactions (how fast each one was processed) and the throughput (how many transactions per second on an ongoing basis). In this context there is perfect harmony between what the ACE approach asks architects to do and what the TSP approach demands of developers. TSP teams receive fairly general direction for eliciting and capturing such quality attributes, the understanding of which often drives a project's structure in addition to the structure of the developed product. With the Bursatec project, the ACE methods provided clear, specific direction on the early lifecycle issues that TSP normally leaves to local practice. Later in the project, TSP drove a disciplined implementation of the architecture that might otherwise have eluded developers.

The TSP Launch
Immediately after the conclusion of the Quality Attribute Workshop, we conducted the TSP launch, which is a series of nine meetings held during the course of four days in which the team reaches a common understanding of the work and the approach that it will take and produces a detailed plan to guide its work. The TSP launch includes producing the necessary planning artifacts (such as goals, roles, estimates, task plan, milestones, quality plan, and risk mitigation plan) that brought together a team of 14 members, including the team leader. Our goal was to plan the architecture activities in the context of supporting Bursatec and their existing time and budget constraints.

During the launch, about half of the team focused on the architecture, including several people who were brought in as domain experts. These individuals were experts at interpreting the functional requirements and ensuring that the developers met them. For example, one individual had expertise in the Mexican Stock Exchange while another domain expert had extensive experience in the options and futures markets, specifically how those instruments are traded in Mexican markets.

The other half of the team, seven developers, focused on two important needs for the system: high- speed communication and a testing framework. To successfully develop the system in the timeframe needed for Bursatec, it was critical that the system be tested automatically rather than manually. Testing (including regression testing to ensure that no other aspects of the system are compromised) of new functionality on the current system takes as much as a month and is performed manually. This testing motivates a quality attribute scenario for rapid testability of most new functions within a day, which leads naturally to an architecture that supports automated testing.

The Bursatec developers then implemented the system's underlying infrastructure based on an early version of the system architecture, while the architects elaborated their work based in part on the early developer work that supported a decision to purchase a particular commercial package for high-speed communication. This version of the architecture was subject to an Architecture Tradeoff Analysis Method (ATAM) review that ensured the quality attribute scenarios captured in the QAW were still the right ones, and that the proposed architecture addressed those scenarios.

After the initial architecture iterations and the ATAM, the architects and other developers worked as a single, integrated team, removing the potential issues that sometimes arise when software architects throw their artifacts "over the wall" to developers. The architects dealt with issues and revised the architecture as necessary while shouldering a normal development workload. The team named role managers--a TSP concept--to focus on issues surrounding performance and garbage collection, two implementation issues critical to the success of the new trading system.

Measurable Results
While TSP can be used to manage all aspects of the software development phase, from requirements elicitation to implementation and testing, this is the first time that the approach has been applied to ACE technologies. The combination of these approaches offered Bursatec architects and developers a disciplined method for developing the software for their new trading engine. Through 6 major development cycles including 14 or so iterations over 21 months, the overall team developed over 200,000 lines of code, spending about 12 percent of their effort after the Quality Attribute Workshop on architecture and approximately 14.5 percent of effort in unit testing, performance testing, and integration testing.

In contrast, the SEI would normally expect almost twice as much testing effort at this point in development, with potentially much more in system testing to push the overall total close to or beyond the 50-percent mark--an unfortunately realistic expectation in our industry. As of October 2011, system testing at Bursatec proceeds on schedule with a very low defect count (unusual in our experience), and the system is on target for deployment beginning in early 2012. Due to the early investment in architecture and a detailed, data-driven approach to managing both their schedule and their quality, less testing was required throughout system development.

Another benefit of combining TSP with ACE is that the team of Bursatec developers was prepared for inevitable changes in the architecture requirements, indeed in changes of any sort over the 21 months of development. When the team received new requirements, it could evaluate them quickly for technical impact and implementation cost in terms of time and effort. With the quality attributes formally captured, the architecture in place, and detailed development plans at every step, a project with enormous risk potential in both technical and business terms ran on-time, within budget, and generally without the drama that large development efforts often exhibit.

Additional Resources:

To read the SEI technical report, Team Software Process (TSP) Body of Knowledge (BOK), please visit
www.sei.cmu.edu/library/abstracts/reports/10tr020.cfm

For more information about the SEI's work in Architecture Centric Engineering (ACE), please visit
www.sei.cmu.edu/about/organization/rtss/ace.cfm

For more information about the SEI's work in the Team Software Process (TSP), please visit
www.sei.cmu.edu/tsp/

To read the SEI technical report, Combining Architecture-Centric Engineering with the Team Software Process, please visit
www.sei.cmu.edu/library/abstracts/reports/10tr031.cfm