|
|
CASE STUDY
Extreme Availability with IBM WebSphere and DB2
SIAC, the technological hub of the securities industry, chose IBM to provide the hardware and software infrastructure for a new trading floor application used by the New York Stock Exchange. This case study describes this unique challenge and monumental project.
By David W. Coleman, IBM, and Charles F. Bowman, SIAC
Availability has always been a key measure for IT infrastructures. Unfortunately, achieving high availability is quite difficult. Complicating any effort to attain high availability is that there is usually no simple stopping point -- in almost all cases there are ways to increase availability, regardless of the starting level. This paper discusses in detail the design and implementation of a large, complicated system. This system is of particular interest because it was developed under the strictest requirements for availability.
Many IT professionals spend considerable effort creating designs and methodologies to improve system availability. There is also a significant body of technical literature on this topic. Within the general area of availability, there are a number of special topics, including high availability (typically meaning availability of 99.9 percent or better, although the precise definition varies considerably) and continuous availability (usually meaning that no scheduled downtime can be tolerated, although once again the definition varies somewhat). In today's on-demand world, large enterprises have increasing availability requirements and increasing opportunities to meet those requirements.
All enterprises must consider seriously the tradeoffs associated with the cost of high availability. A few enterprises are addressing a requirement for extreme availability (as close to 100 percent as possible), perhaps none as seriously as the New York Stock Exchange (NYSE). The NYSE is the world's largest stock exchange, handling the trade of over 300 billion shares per year, worth more than US$10 trillion. The NYSE has a long tradition of developing extreme availability trading systems and cannot tolerate the affects of system outages.
To ensure its extraordinary requirements for IT systems availability, the NYSE relies on the Securities Industry Automation Corporation (SIAC). SIAC is the technological hub of the securities industry, currently providing key systems support to:
- New York and American Stock Exchanges (NYSE and Amex)
- Depository Trust & Clearing Corporation (DTCC) and its affiliates
- National Market System
Starting in 2001, NYSE/SIAC began the process of replacing the order management system that supports brokers on the trading floor. It was looking for an "availability partner" and chose IBM after deciding that IBM was the only vendor "willing to make it all work."
The approach chosen was to design and develop one application, deployed in multiple phases, from simple through full functionality. The new application, NYSE TradeWorks, would become the primary Broker Order Management application that would manage the complete lifecycle of orders entered by member firms of the NYSE. TradeWorks will provide Firms with an essential tool to facilitate the quick and efficient communication of order instructions, execution reports, and market intelligence between NYSE Floor Brokers and off-Floor Trading Desks. TradeWorks enables Trading Floor staff to effectively manage an extremely high volume of orders, by providing automated tools to intelligently route order flow to the appropriate Broker or Trading Post and the ability to monitor and control the order through to execution. TradeWorks is fully integrated with the NYSE's e-Broker wireless handheld computer, used by more than 650 Brokers to trade in excess of 550 million shares of stock each day.
SIAC has completed the development and rollout of the initial phase, which uses IBM's WebSphere Application Server and DB2 to handle what will be a high-volume transaction oriented system. WebSphere Application Server serves as the two-phase commit transaction manager and DB2 (residing on a mainframe) functions as the primary operational data store. The application environment requires rigorous failover capabilities, absolute data integrity, extensive peer-to-peer communication, and careful load balancing across sites, while processing very high transaction volumes and meeting stringent performance and response time constraints.
This case study summarizes the requirements of the new trading-floor application and provides a detailed technical overview of the features of the newly created solution. It also includes a description of the rigorous testing requirements for the new design, which include tests that addressed multiple simultaneous failures, secondary failures, and worst moment failures, which are failures timed to cause as much difficulty as possible. These last three categories are frequently omitted in test design, and worst moment failures in particular can require development of special tools.
Defining the problem
Due to the critical nature of its business, the NYSE cannot tolerate exposure to any downtime for its enterprise-critical IT infrastructure. Thus, the Exchange is willing to underwrite the extraordinary measures required to achieve extreme availability, that is, very close to 100 percent availability.
Achieving extreme availability is always challenging. To support the effort, SIAC relies on a set of guiding principles for its IT infrastructure. These principles set forth requirements in the areas of site and building architecture, network design and operation, system hardware and software, and of course procedures/guidelines for design, development, deployment, and operation.
Chief among these requirements is that there be no single points of failure, which implies multiple levels of redundancy and replication, including full site duplication. Thorough testing is an imperative (see Test, test, and test). And a key operational requirement is a deployment process characterized by a gradual, well managed rollout of application upgrades and the ability to immediately roll back an upgrade and restore the previous version of the application if a problem is discovered in the upgraded version.
The deployment requirement is actually even more specific than suggested above. Major production changes are handled by staged rollouts on a user-by-user basis. The new and old systems run concurrently. A rollout involves introducing the new system to users (or sometimes small groups of users) gradually, with the objective of identifying possible problems while affecting the fewest possible users. Individual users are added to the new system gradually, with careful attention paid to possible correctness issues. Every transaction in the new system includes a synchronization sub-transaction, which enters the information in the old system as well. (The converse is true as well: every transaction initiated in the old system has a corresponding sub-transaction applied to the new system.) Regular audits are performed to ensure that the data are equivalent in the old and new systems. Thus, if at any point it is necessary to roll back one or more users to the old system, no information is lost. This comprehensive methodology naturally places additional constraints on performance because good performance is required not only during normal operation but also during rollout, when twice as much processing is taking place.
Historically, SIAC and the NYSE developed their applications from scratch, in house. When they started looking at the scale, scope, and cost of changes needed for several existing applications, they decided to investigate the possibility of using outside middleware to speed future development and decrease the costs of implementation and maintenance. To that end, SIAC invited several vendors to participate in an exhaustive set of proof-of-concept tests. SIAC demanded an availability partnership and ultimately determined that IBM middleware would be the best choice to meet their stringent performance and reliability requirements.
Preparing for change
SIAC/NYSE selected IBM to meet their requirements for extreme availability. A major reason was the strength of IBM's software, of course. However, there was another significant factor in the choice. Experience convinced SIAC to seek a strong strategic partnership with their primary vendors, knowing it would require significant changes to the way they had been doing business.
IBM's ability (and willingness) to accept responsibility for problem determination, even when the problem was not caused by IBM products, was also a key decision factor in choosing IBM. In this kind of environment, a partner who is willing and able to accept overall responsibility for resolution, even if it involves third-party hardware or software, is necessary.
Traditionally, SIAC's culture was highly technical and relied on in-house expertise. This change would broaden the project focus from highly technical development to one that included the integration of technical and nontechnical (as well as non-SIAC) teams, increase the significance of training, and possibly increase the difficulty of problem determination. To mitigate these challenges, SIAC demanded design and operational simplicity. In addition, they insisted on phased deployment, with a significant portion of the infrastructure delivered in the earliest phases.
Overview of the solution
The basic requirements that any trading floor application for NYSE needs to meet are fairly easy to define. The application needs to be extremely reliable, highly available, very fast, and highly scalable. Of course, a fair amount of detail is hidden by such broad terms. In this environment, extremely reliable means that "bugs" in production systems should be few and far between. Component reliability is of major importance when so much money is at stake. Very fast has multiple meanings but, in the context of a system interacting with humans, it typically translates into a response time requirement of one second or less, from the end user's perspective.
Highly available in the context of the NYSE has been discussed previously; here it means no outages. This requirement has significant implications in the overall system design. For example, it means there must be two geographically separated sites working as hot backups for each other. This has implications for both the hardware design and the way the software must function. Further, the high availability requirement has significant implications for how hardware and software upgrades must take place. And finally, because the NYSE maintains high trading volumes, with significant peaks within an overall trend of increasing volume, any solution must be highly scalable.
At a high level, as shown in figure 1, the NYSE TradeWorks design is a three-tier architecture, involving Java-based clients running the WebSphere Application Server atop a Linux workstation, a RISC-based middle tier running the WebSphere Application Server and DB2 Connect, and a mainframe-based DB2 database store, all interconnected using a RISC-based messaging backbone for communication among the various clients.
The Linux-based Java client uses the Swing user interface API to create a highly optimized graphical user interface specially designed to facilitate broker operations. This client also contains special logic to handle both failover and upgrades; this means that there is special code for both selecting the appropriate server, as well as recovering from failures of that server.
The WebSphere-based middle tier acts as a transaction manager for two-phase commit transactions. For reliability reasons, it uses a quadraplex storage approach for important files, such as the transaction logs. (Quadraplex because, in the final design, data are duplicated in two separate sites, and within each site data are duplicated on two separate storage arrays. This is all done by the storage hardware for performance reasons.) From an application standpoint, the middle tier is stateless, though for performance reasons, a distributed cache is supported for read-only and read-mostly data. As a result, any middle-tier server can respond to any transaction request from any client. Though improving overall scalability, availably, and reliability, this architecture does place additional reliability and availably constraints on the database. Specifically, the database is an operational data store that must always be available for the system to function.
The middle tier application software contains special logic to allow multiple versions to run concurrently (in a coordinated fashion) to allow for a gradual rollout of upgrades. The software is designed so that every transaction can cross instance boundaries, thus simplifying the issues associated with coordinating transactions (and hence databases) in both the old and the new instances. See "Implementation details" for more information.
The DB2 database, used as an operational data store, runs on a mainframe -- largely because a mainframe has well developed processes to deal with a variety of reliability, availability, and live serviceability issues. Further, there is a very standard way of maintaining consistency of the database across geographically separated hot sites. And of course DB2 on the mainframe has a long history of handling high volumes of transaction processing.
The messaging backbone (labeled JMS Servers in figure 1) serves as both a general publication medium and a "back channel" to update interested parties when transactions occur. It is SIAC-hardened communication infrastructure based on Open-JMS, and fully compliant with the Java Message Service (JMS) specification. It supports two-phase commit transactions as well as publish/subscribe topics and round-robin queues. Thus, for the kind of transaction described in Figure 2, when a broker-user initiates a business transaction, the middle tier queries the database, validates the request, performs the appropriate business processing and, as part of a two-phase commit transaction, commits states changes to the database and publishes results to other interested users and back-office systems through the messaging backbone.
Implementation details
Of the many design details of interest, three areas in particular distinguish the NYSE TradeWorks system:
- Use of true two-phase commit for transaction management
- Use of dual stack for production release management
- Use of active/active failover
Two-phase commit
Two-phase commit is a well-established method for providing integrity in transaction processing systems. It is well suited for synchronizing multiple interacting resources while a transaction is being processed. Unfortunately, it is also complex to implement properly and, because it is both general purpose and absolutely rigorous, it can be slower than some specialized alternatives. One common alternative to two-phase commit when using redundant systems is to use parallel transactions with synchronization only after the transaction completes (or fails). This second approach, with its heuristic completion rules, tends to be both application-specific and complex to implement (when all success/failures cases are considered). Indeed, this second alternative has been used in the past at SIAC, and it was the continually growing complexity of the heuristic rules that caused SIAC to abandon this approach in NYSE TradeWorks in favor of true two-phase commit.
Dual stack
In order to achieve extreme availability, SIAC cannot deploy new versions of enterprise-critical applications using a traditional "big bang" approach, where all systems are upgraded simultaneously. Rather, each new version of an application must support an incremental rollout methodology and interoperate with current production software. This is true even when the new version consists solely of middleware or operating system changes. In addition, in the event of an anomaly, the new version must be able to "fall back" with little or no impact on users. Providing this kind of capability for every change is necessary for the highest levels of availability, but it tends to be both difficult and costly. Further, if a custom solution is used for each different rollout, the probability of error is quite high. A standardized approach to this problem is therefore quite important.
Stated simply, the problem is how to reliably upgrade an application (or any of its pieces), given that there is always the possibility that the upgrade will fail in its initial rollout and that no downtime is acceptable when handling such upgrade failures. Dual stack, as designed and implemented by SIAC, is a general-purpose solution to this common problem. It is an approach that fully duplicates the system to be upgraded so that both the old and new versions execute simultaneously.
Figure 3 illustrates the concept of dual stack. It shows a user of the existing version initiating a transaction on the "current" stack (Version 1.0 in figure 3). Upon receipt, the middle tier propagates this transaction to the "new" stack (Version 2.0 in figure 3). The transaction must complete successfully on both stacks for the request to succeed. Similarly, a client initiating a request on the "new" version of the application would cause the middle tier to propagate the transaction to the old stack (if required -- services introduced with the new version would have no analogue in the old system and thus would not require propagation).
As part of the upgrade deliverable to operations, developers provide the operations staff with a database verification routine that executes at the end of each trading day; this processing ensures that the current and new databases are equivalent. Note that this guaranteed synchronization of the permanent databases, coupled with the full duplication of each transaction during the trading day, greatly simplifies the rollout/fallback methodology for the operations staff. To provide even greater assurance for the process, a number of other special approaches are also regularly used, such as running the dual stack for a period of time with no users on the new system, just to check that the dual stack rollout itself does not cause instability in the current system.
While simple in concept, the dual stack design involves many low level details and requires careful design and coding to work smoothly. This is particularly true in a stateful system, such as a transaction processing system. For NYSE TradeWorks, dual stack requires a special arbitration component (implemented in the client) to handle switching between the old and new versions, as well as special code to handle the issues involved in having each transaction simultaneously run in the old and the new environment. As one might expect, the dual stack design also places additional constraints on transaction performance.
Active/active failover
There are three common failover patterns used to provide redundancy for high availability: active/active, active/standby (in several flavors, detailed below), and active/backup (used mostly for disaster recovery).
Active/active failover is a pattern where every key system in use is backed up by another system that is also in use. The major advantage of active/active is that because the failover system is always in use, there are fewer chances for mistakes in the setup of the failover system. Such setup mistakes, which are frequently the result of a change to the active system that is not properly replicated on the failover system, are a common enough occurrence to be a major source of failure.
Active/standby failover is a pattern where every key system in use is backed up by another system which, while not in use, can be brought online fairly quickly. Active/standby is sometimes broken into active/warm standby and active/cold standby, with the warm/cold distinction being made on the basis of how long it takes to get the standby system working in replacement of the original active system. Typically active/standby is simpler to implement than active/active, since there is the simplifying assumption that only one of the systems is active at any given time. Further, it can be cheaper to implement also, if a single system is used as a standby for more than one active system. (This is called n+1 redundancy in some of the literature on this topic.) Note that n+1 redundancy increases the overall risk of an outage slightly, since it cannot handle two or more simultaneous active system failures.
Active/backup failover is a pattern where every key system in use can be restored from backups. Restoring from backups is typically fairly slow, which is why this approach is most commonly used in disaster recovery scenarios, as opposed to high availability scenarios. However, if a business is not highly sensitive to outage duration, active/backup can work well. Note that it is sometimes possible to arrange an option to rent the hardware that would be needed in case of an outage of the active system. This can make active/backup failover significantly less expensive than other options.
From the above descriptions, it should not be surprising that SIAC chose active/active backup, despite the additional development expense associated with such an approach, because it is the most robust of the various failover alternatives.
Taking the first step
Because the changes involved with using vendor middleware potentially have such broad impact, SIAC took an even more gradual approach than usual in introducing NYSE TradeWorks to the trading floor. The multiyear development project was broken into several increments, with the early deliverables specifically designed to implement and exercise as much of the new architecture and infrastructure as possible, while avoiding uses where an outage would cause trading disruptions to the Exchange.
As noted above, to achieve this goal SIAC conducted an extensive proof-of-concept test that modeled significant portions of the NYSE TradeWorks application. As part of this exercise, SIAC performed extensive functional, performance, and failure/recovery testing. This led to the final selection of middleware products and allowed SIAC designers to test the viability of the new architecture.
Following that, SIAC, in conjunction with IBM, began development of the first production increment of NYSE TradeWorks. Though comparatively small in feature set, this first version of the system allowed SIAC to gain production experience with the designs and exercise the new infrastructure with minimal business risk to users. Subsequent versions containing features more broadly used by the broker community will be overlaid on the existing, now proven infrastructure.
Test, test, and test
The NYSE TradeWorks system is built using a number of different commercial quality components. However, that is not sufficient to ensure the extreme availability that the NYSE requires. Commercial hardware and software varies in reliability, so considerable care must be taken in the initial selection. Further, even the best components must be tested in the type of environment they are going to be used to validate the expected reliability and uncover any compatibility issues. SIAC is well aware of this need and thus planned accordingly.
A significant component of SIAC's development process is their highly systematic approach to testing, which includes:
- Qualification testing, where outside products are subjected to a specially-designed suite of tests to determine their suitability in the SIAC environment
- Functionality testing, where each deliverable is subjected to a well-defined suite of test cases to ensure it will perform properly in production
- Performance testing, where overall system performance is carefully measured both for capacity planning, and to ensure that end users will experience acceptable response times
- Stress testing, where the overall application is subjected to substantially oversized loads to help discover bugs and to determine where the ultimate failure points of the system are likely to be
Failure/recovery testing for extreme availability is a major undertaking that involves designs/plans for recovery from:
- Multiple simultaneous failures
- Secondary failures while recovering from initial failures
- Injection of failures at the "worst possible moment"
In general, the SIAC testing methodology involves causing individual components to fail in various ways, and observing the recovery behavior. After the individual failures seem to be handled properly, multiple failure scenarios considered likely to occur are tested. Given the handful of key components involved in NYSE TradeWorks, this leads to a large number of failure scenarios that must be tested. The key individual software components tested were the client application code, the client WebSphere code, the middle-tier application code, the middle-tier WebSphere code, the middle-tier database communication layer, the final-tier messaging application, the final-tier WebSphere for the messaging application, the mainframe database stored procedures, and the mainframe database itself.
In addition to software failures, various kinds of hardware failures must also be tested. This includes failure of entire machines (client, middle-tier, and final tier), as well as failure of key infrastructure components such as the disk subsystem and network interfaces.
The rigorous qualification and testing process for NYSE resulted in enhancements and planned enhancements to the function and performance of WebSphere Application Server, DB2, and DB2 Connect.
Conclusion
The NYSE and SIAC found with IBM what they were looking for in an availability partnership. Their requirements for availability, reliability, scalability, and performance are, by any measure, extreme and well beyond what most commercial off-the-shelf software provides. IBM demonstrated not only the suitability of its middleware to the challenging trading floor environment but also its commitment to enhance the middleware as needed to meet such extreme requirements.
The proven ability of IBM's software to support extreme availability is of great importance in a world where on-demand computing is becoming the norm. Further, as SIAC discovered in the process of developing NYSE TradeWorks, the value of partnering with a company with expertise in all pertinent areas of IT can be significant when creating business critical distributed applications.
The extensive attention to system design, and much more rigorous than usual testing, has paid off. The initial deployment of the NYSE TradeWorks application has been very eagerly accepted by the NYSE Floor users, with the most common feedback being a request to speed up deployment so more of the brokers in each firm can use it.
ARTICLE INFO
Web Edition: 2004 Week 51, Doc #15076
 FREE ACCESS
Keyword Tags: IBM, IBM DB2, IBM Lotus, IBM Solutions, IBM WebSphere, IBM WebSphere Application Server, IBM WebSphere Solutions, IT Networking, Java, Linux, Messaging, Training, Wireless
|
|