If you haven’t already heard, a couple of major changes have happened in the load testing tool market

1. Microsoft is deprecating its load testing feature out of Visual Studio after 2019.
   
2. HPE has sold its load testing tools (LoadRunner, Performance Center, StormRunner) to Micro Focus. Since Micro Focus already has its own load testing tool (SilkPerformer), there is some speculation that Micro Focus might discontinue one of these toolsets to focus on the other.

For organizations using the above tools, it may be time to consider other options. Even if your organization doesn’t use these tools, this is a good opportunity to take a step back, reassess what tools are out there, and which are the best fit for your unique situation.

Features of a load test tool

Before we start looking at what tools are available, let’s look at the key features typically offered by load testing tools. Some of these are core features (needs) and others are add-ons (wants). Establishing a good understanding of which features matter the most to your team/organization will help inform your decision-making.

Core-features:

• A script recorder to capture business scenarios into a basic test script
• Script enhancement such as test data set up and dynamic parameter handling
• Test executor which lets you organize the scripts, allocate virtual users and generate user loads
• Result reporting which lets you view your load test results during/after the test is conducted

Although these are the common features across all load testing tools, some do these better than others. Specifically, commercial tools offer enhanced versions of the core features to speed up the test suite development and execution activities.

In addition to these core features, tool companies try to offer extended features to help organizations make most of their load testing solution.

Considerations for choosing a tool

Protocol support

Protocol is the way a client and server communicate in a software system. The load testing tool must support the network protocol your application is using. Almost all load test tools (open-source as well as commercial) support web-based communication protocol. If the application under test uses a proprietary protocol (such as SAP, Citrix or mainframe) then only a few commercial tools support this. It is generally a good idea to conduct a proof-of-concept using the load test tool before proceeding any further.

When it comes to protocol support, there is another aspect to it; some organizations prefer to standardize testing tools across teams, while others do not mind mixing tools provided the job gets done. Depending on your organization’s preference, it is better to ask yourself “Does this tool need to meet the needs of my entire organization or just an individual team?” If the former, you’ll need to choose a tool that meets the requirements of the majority of applications in your technical landscape.

IT goals

The tool you select needs to align with your organizational IT goals. It will be an added benefit to choose a tool that can be easily integrated with your DevOps pipeline or the APM tools already in place. If your organization is moving at a rapid pace, adopting the most recent technologies and industry trends then you might benefit from having a load test tool that is being actively developed and maintained to keep up with market trends.

ROI

Essentially, most of the decisions in an organization come down to this. Some costs like licenses are more obvious, while others like manual effort may not be as visible.

It’s a good idea to consider a holistic list of factors while conducting your cost-benefit analysis. These include licensing cost, licensing model, test asset creation and maintenance effort, time to market etc. If you already have most of your test assets built in a different tool, some load testing tools provide the option to migrate the scripts across to new tool, which offsets the overall cost.

Technical capability of your staff is another key factor to consider when calculating the cost. Some tools have a steep learning curve or need expensive technical resources to develop the test assets while others are easier to learn.

Online support

Whether a paid or free tool, this is definitely an important factor. More often than not, you will encounter a number of issues when using any load testing tool for the first time. Having a wide user community in the case of open-source, or responsive technical support in the case of paid tools can reduce delays in performance testing.

Popular choices

There are options for organizations choosing to migrate off LoadRunner or Visual Studio, albeit less than you may think. I have listed a few key tools based on my experience, but a similar approach can be undertaken for other load testing tools.

My objective here is not to compare and contrast these tools. Instead, I want to outline some popular options and their features, to help you find the tool that suits your needs.

JMeter

As a tool of choice within the performance testing community, JMeter’s popularity is ever-increasing. It is open-source and is reasonably stable. It has a larger online community than other open-source tools such as Gatling or Locust.io.

JMeter offers really good core functionality. It supports a wide range of protocols (HTTP, JMS, WebDriver, SMTP etc) and comes with several free plugins for server monitoring and CI/CD integration.

Some may view JMeter as not overly user-friendly as it requires writing code to achieve non-trivial tasks. JMeter load tests need to be conducted without using the Graphical User interface, which provides limited visibility into the results while the test is running. It does not support many commercial protocols such as SAP or Citrix and lacks the features like built-in automated script enhancement. Having said that, JMeter source code is freely available for extending its core functionality.

JMeter scripts can be easily run on cloud load test platforms such as Flood.IO or BlazeMeter (more on this later).

Neoload

Neoload is a commercial tool developed by Neotys. It is one of the most user-friendly load testing tools in market. NeoLoad promotes the idea of code-less load test scripting making (although this may always not be possible in the real world), which makes it easier for less experienced testers to build the test suite.

Neoload comes with automated script enhancement frameworks specially for packaged applications (supported list here). Being a purely performance tool company, Neoload has an undivided focus on enhancing the product features to suit the market. It is actively developed in terms of protocol support, adding SAP GUI to the bundle of protocols recently. Neoload also has cloud load testing services, which allows you to simulate load from various geographical locations.

Neoload is a reasonable alternative for those looking to migrate from LoadRunner as it allows you to repurpose existing test assets built in LoadRunner.

You can find the full set of features on Neotys website.

Tricentis Flood

Tricentis Flood is a cloud load testing platform at its core. It lets you run test scripts created in open-source load testing tools like JMeter or Gatling and run them on the cloud platform to scale thousands of users on a subscription basis.

Flood has a scripting generation component called “Flood Element”. Traditional load testing tools work at a network protocol level i.e. they make direct HTTP calls to the webserver instead of launching real browsers. This makes performance test scripting reasonably complex, especially if the system under test has really complex network traffic. Using Flood Element, the test scripts can be built similar to Test Automation scripts with lower effort. Flood can be integrated with Tricentis Tosca to re-use functional automation scripts for load testing.

You can find more information on Tricentis Flood website.

BlazeMeter

BlazeMeter is another popular load testing solution available in the cloud space. Similar to Flood.IO, this is a load test infrastructure solution. However, when combined with JMeter core load testing functionality, BlazeMeter offers strong end-to-end load testing capabilities.

Blazemeter has extended the core load testing features offered by JMeter. Some of its key features include:

- Hassle-free JMeter generation using Chrome plugin- Automated script enhancement (using SmartJMX)- Allows live monitoring of JMeter test results- LoadRunner to JMeter script converter

You can find the full set of features here.

However, being cloud platforms, Flood and BlazeMeter have two key limitations:

1. Cloud platforms are primarily designed to load test web applications that are accessible over the internet. Therefore, they may not be a suitable choice for load testing fat client systems.

2. For web-applications that are not readily accessible over the internet, additional network configuration may be needed to allow the communication between system under test and the cloud.

In Summary

Whether you use Visual Studio/LoadRunner or not, now is a great time to step back and assess your current performance test tooling. Take a moment to consider not just whether it is meeting the needs of your organization now, but whether it will continue to do so in the next five years as your organization changes and evolves.