Evaluating Service Virtualization Tools?

Move Beyond the Marketing Hype by Asking These Questions

Since service virtualization tools become part of your core architecture, selecting the vendor and product that’s the best fit for your organization’s process and culture is crucial to your long-term success and return on investment.

The vendor that’s most prominent in the marketplace may or may not deliver the best results in your environment, but how can you determine this before you’ve already committed considerable resources to the initiative?

As you begin your service virtualization tool comparison, be sure to ask each service virtualization tool vendor a broad array of questions that will help you gain a clear understanding of their approach to service virtualization—and how that will impact your organization’s productivity, costs, and success if you ultimately commit to working with this vendor.

With diligent probing, you’ll be able to zero in on important evaluation criteria related to:

• Innovation: Where is the product heading and what is the road map?
• Enterprise scalability: Does the solution scale to your project but also does it deploy across your organization? If the organization and project grow, how well suited is the solution to meet this growth?
• Process fit: Does the proposed solution fit in your current process. How well does it evolve as your processes evolve?
• Support: What is the support model? Is there premium support or onsite deployment mentoring, training and consulting?
• Productivity: Can the vendor articulate the return on investment for the solution? Do you see a clear time and cost savings plus the promised improvement in quality?
• Total cost of ownership: Can the vendor illustrate the total cost of ownership? The goes beyond the price of the tools but also time and effort to deploy and use the tools on daily basis.

Want some guidance on what questions to ask? Here are a few questions to get you started:

• Describe the organization’s journey towards having an service virtualization offering—how/why?
• Can you access the vendor’s thought leaders? Can the thought leaders participate in your deployment?
• Can the vendor share any general metrics regarding the performance of their service virtualization offerings?
• What do I need to know in order to build a basic virtual asset from scratch and how much time would it take?
• How easy is it to reconfigure the environment (e.g., switching between different data sets, performance profiles, etc.)?
• How does the solution deal with test data?
• How easy is it for testers and other team members not intimately knowledgeable of APIs to use and adopt?
• Does the solution support a Center of Excellence model as well as team-lead adoption?

One of the most critical aspects of service virtualization solutions is the hidden costs associated with professional services from the vendor. In many cases, the ratio of professional service to technology costs can be quite high and should be considered as part of the total cost of ownership. Although professional services might be required to jump-start your process, in the long run, the technology’s architecture, flexibility, and suitability to your evolving needs will dictate costs—whether direct OpEx associated with your staff, or indirect OpEx associated with a system integrator.

Before you begin examining service virtualization tool vendors, we strongly recommend that you ask yourself some questions about your implementation strategy. Once you know what plan of action makes sense for your needs, you’re in a much better position to determine if a particular vendor will help you get there.

Service virtualization plays a key role in API testing, so when considering  an API test strategy, an important step is to evaluate how virtual services are being consumed within the team and how it suits both developers and testers. Developers may adopt virtualization initially as they grow out the implementation then later to be adopted by testers. At this point it becomes critical to consider the possible knowledge gap between the test and development teams since APIs might be new territory for testers.

Parasoft’s How to Adopt & Scale Service Virtualization paper outlines a number of key considerations, ranging from general decision criteria (your risk tolerance, access to dependent systems, etc.), to the expected implementation focus (environment-based, project-based, demand-based, or hybrid), to the anticipated team structure (distributed, center of excellence, or hybrid). You can also visit our service virtualization tutorial to learn more about the technology.