How Automated API Testing Saved a NZ Bank $2.1 million

To reduce the risks associated with their business-critical transactions, a leading New Zealand bank and financial-services provider wanted to extend their existing GUI-based testing to more extensively exercise the application logic within internal systems.

This logic resides within their ESB: a message broker component that features 80+ services and 750+ operations. The ESB represents a single point of failure within the infrastructure—but the company was unable to test it directly or automatically.

With the expertise of Parasoft partner IntegrationQA and Parasoft’s market-leading API Testing solution, the company gained a service testing methodology supported by automated, reusable tests that performed a broad scope of testing directly from the service/API layer.

The result: they saved over $2.1 million NZD in testing and development costs over an 18 month period.

The Challenge: Ensuring that Application Logic Continuously Satisfies Expectations for High Reliability

The company’s existing functional testing process provided some degree of black-box testing for the integration points between front-end and back-end systems. However, the company feared that this approach of relying on the front-end as the sole entry point for testing new functionality, technology updates, and large-scale changes to the existing infrastructure was not deep and rigorous enough to ensure that the application logic continuously satisfied the company’s high reliability standards.

To reduce their risk exposure, they wanted to extend their existing end-to-end test scenarios to include direct and extensive testing of the ESB and associated services. However, they lacked access to the domain expertise and technology required to take their existing testing process to this level.

The Solution: Automated Testing at the Message Layer

To help the company achieve their quality objectives, IntegrationQA worked with them to establish a dedicated test team who would specialise in testing the infrastructure at the service level using Parasoft API Testing solution. Recognizing the importance of implementing new technologies within an effective process, IntegrationQA also helped them develop a new testing methodology which included:

• Service Diagnostics/Health Check: Establishes a suite of service-level smoke tests for the ESB, providing a health check for an ESB deployment to any test environment. This is critical because any deployment for a single service also impacts existing unchanged services (since the ESB was designed to have all services within the same container).
• Static Analysis: Analyses the service contract for compliance with industry-accepted standards.
• Service Testing: IntegrationQA established an approach to test the ESB at the service level. The Service Test team developed the following suite of tests for each service:
  • Smoke Testing: Via the service layer, ensures interoperability between the service consumer and service provider.
  • Transformation: Ensures that data transformation is accurate and satisfies the service consumer’s expectations.
  • Data Variance: Ensures that the service properly handles all possible mandatory and optional data scenarios.
  • Error Handling: Ensures that the service accurately transmits an error occurrence to the service consumer or to logging mechanisms.
  • Scenario/End-to-End Testing: A context-driven culmination of the above tests; verifies that the services properly implement the expected business workflow.

Results: Risk Reduction, Cost Savings, and Additional Time for Validating Business Requirements

Using the new technologies and methodologies, the team was able to “shift left” their functional testing from the System Testing phase to the Unit and Integration Testing phases. By exposing more defects when it’s easier, faster, and less expensive to fix them, they achieved significant time and cost savings.

Across all projects, 65% – 100% of the defects uncovered by the Service Test team were identified during the Unit and Integration Testing phases. Approximately 75% of the defects originated at the ESB level, which received limited test coverage with their previous testing methods (as underscored by the fact that one major ESB defect was identified for every 10,000 records processed). Across all projects, between 55% and 100% of discovered defects were classified as Severity 2 and could not be deferred.

Cost Reduction per Test Environment

Once confidence in the Service Test team grew and the new service testing methodology added proven value to the company’s quality efforts, it became an integral part of all pipeline project test planning and estimations. To accommodate the increased workload, the team grew in size from 3 initial members to 10 full-time technical test analysts. IntegrationQA took on the role of team lead, consultant, and facilitator for multiple internal projects at various lifecycle stages.

As the new processes and technologies became more ingrained and mature, the team expanded the scope of work to include performance testing of the ESB, mobile application message testing, ETL testing of the data warehouse, as well as application testing of internal monitoring engines for customer accounts and transactions.

The company has experienced the following benefits from using Parasoft API Testing to implement the recommended service testing methodology:

• The ability to begin testing all service-related requirements—plus many functional requirements —during the Unit/Integration Test phases enabled earlier detection of technical issues and significant cost savings by eliminating the need for redesign and rework.
• Directly testing the application logic reduced functional testing time and lowered overall testing costs. Once the team adopted service testing, the scope for functional testing was narrowed to focus solely on the front-end logic.
• The move to test automation meant that testing could easily be repeated to ensure that application changes did not negatively impact previously-tested functionality.
• Continuous execution of ESB-oriented test suites gave stakeholders daily health-check updates on whether the central infrastructure component was meeting their expectations.
• With increased visibility into (and understanding of) the ESB and its service repository, the team could more efficiently plan—and more accurately estimate—the testing activities required to mitigate business risks.

Over 18 months, the company has saved over $2.1 million NZD in testing and development costs (based on the company’s internal defect cost valuations).

Learn More About this Bank’s Automated API Testing Framework

Read the 12-page API Testing technical paper to get the nitty-gritty details about how this company established a service testing methodology supported by automated, reusable tests that perform a broad scope of testing directly from the service/API layer.

You’ll get a close look at how they were able to create reusable, easily-maintainable tests that could robustly exercise any of the available ESB services via 3 main test patterns:

• Confirmation – Smoke test
• Evaluation – Transformation, data variance, and error handling
• Replication – Business workflow scenarios

Forrester API Testing Tool Comparison

If you’re considering bringing an API Testing tool into your own organization, don’t miss Forrester’s new Functional Test Automation Wave report, which compares the top 9 API testing tools.

The process for developing the current Wave included generating a detailed taxonomy of 40 product attributes, gathering and validating data for nine vendors through lab-based evaluations and client interviews, then placing the evaluated vendors in one of four categories: Leaders, Strong Contenders, Contenders, and Risky Bets.

The report emphasizes that API testing is a critical component for modern application delivery: “In order to support continuous quality and faster development, especially for those on an Agile and DevOps journey, testing must shift from a UI-only approach to a UI-and-API approach.”

What is API Testing

Want to learn about the fundamentals of API Testing? Take a look at the Parasoft API testing page: https://www.parasoft.com/solutions/api-testing/