Comparing Performance Testing Tools

Leave a Comment / PerformanceTesting / By

Hi Guys,

This is a continuation of a previous post that I had done for comparing different performance testing tools.

An honest review 🙂

Performance Test Tools SCOPE

  1. JMeter / Blazemeter
  2. Load Complete (Smart Bear)
  3. Web LOAD (Rad View)
  4. Loadrunner (HPE)
  5. Neoload

Test Case SCOPE

The test cases chosen for evaluating different tools are

  1. Creating an application in the a very complex banking application from Step 1 to Step 6 which basically involves interacting with 6 different pages. In each step the application is interacting with server and there are innumerable session ID’s and random ID’s generated. I had almost identified at least 15 unique id’s generated which are used in different sync and async calls to it’s own server and other 3rd party servers for different validations, checks, analytics, appointment booking etc.
  2. Back end – Login to the back end application. Randomly select an application in the ‘Assess’ status. Then Click the evaluate button in ‘Edit’ Application
  3. API + The performance in creating the key for each operation performed by the user + to check the performance when 20 users access the uri in a step up load style and retrieving application data.
  4. Database + Check the performance when 10 users get the access to DB + Each user perform retrieving data 10,000 time to know the load time. Brief evaluation report :

JMeter / Blazemeter

About

The Apache JMeterâ„¢ application is open source software, a 100% pure Java application designed to load test functional behaviour and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

Technologies Supported

  1. Ability to load and performance test many different applications/server/protocol types
    • Web – HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, …)
    • SOAP / REST Webservices
    • FTP
    • Database via JDBC
    • LDAP
    • Message-oriented middleware (MOM) via JMS
    • Mail – SMTP(S), POP3(S) and IMAP(S)
    • Native commands or shell scripts
    • TCP
    • Java Objects
  2. Full featured Test IDE that allows fast Test Plan recording (from Browsers or native applications), building and debugging.
  3. Command-line mode (Non GUI / headless mode) to load test from any Java compatible OS (Linux, Windows, Mac OSX, …)
  4. A complete and ready to present dynamic HTML report
  5. Easy correlation through ability to extract data from most popular response formats, HTML, JSON , XML or any textual format
  6. Complete portability and 100% Java purity.
  7. Full multi-threading framework allows concurrent sampling by many threads and simultaneous sampling of different functions by separate thread groups.
  8. Caching and offline analysis/replaying of test results.
  9. Highly Extensible core
  • Pluggable Samplers allow unlimited testing capabilities.
  • Scriptable Samplers (JSR223-compatible languages like Groovy and BeanShell)
  • Several load statistics may be chosen with pluggable timers.
  • Data analysis and visualization plugins allow great extensibility as well as personalization.
  • Functions can be used to provide dynamic input to a test or provide data manipulation.
  • Easy Continuous Integration through 3rd party Open Source libraries for Maven, Graddle and Jenkins

(Source : (jmeter.apache.org))

Support

Jmeter is the most prominent open source tool available in the current market(2017). Since it’s in the market for long period, huge community support is present. Most questions are already answered in different forums

Platform compatibility

Windows, Mac and Linux Systems

Reporting

Many limitations in reporting as compared to other performance testing tools in this scope.

Price

Jmeter is free.But need Blazemeter for more realistic analysis.

$499/mo – 3000 Concurrent Users. Total p.a – $6000

Companies using Jmeter

Atlassian, Pfizer, movieticets.com,intel! Etcc

Review

  1. I had first started with Jmeter evaluation. Chosen the Test Case 1 for evaluating this tool. J-meter is basically a record and playback tool. Then we put load on top of the ‘test plan’. The test case 1 involves 6 different pages , there are lot of async call for E-mail, phone and Address validation. On top of that there are different unique ID’s generated due to security reasons. On the day 1 of evaluation itself, it’s found that it will be time consuming to correlate* manually.
  2. We already have a lot of automation test cases which are in use and can check the performance in those automated test cases. Test cases are automated using C#, Selenium & N-unit. I had recorded by pointing the Jmeter proxy to the automated scripts proxy. The recording was successful, still correlation found to be a big issue.

As the famous saying depicts – Time is Money. So… It didn’t qualified for my purpose.

Test case 2,3 & 4 are not evaluated as the test case 1 itself didn’t meet the evaluation criteria.

Load Complete (Smart Bear)

About

LoadComplete is a desktop tool for load, stress, scalability testing of websites and web applications. Designed from the ground-up to help you focus on the tasks at hand, it does not require advanced programming skills. It works with both traditional HTML web sites and with Rich Internet Applications using modern web technologies like AJAX, ASP.NET, Java, Flash, Flex and Silverlight.

Technologies Supported

  1. Technology Platforms
    • .NET
    • Java
    • Php
    • Python
    • Ruby on rails
  2. Protocols
    • HTTP and HTTPs
    • Websockets
    • Flex
    • Java Serialization
    • JMS
    • MS Silverlight
  3. Frameworks
    • HTML5
    • Adobe Flash
    • Adobe Flex
    • MS Silverlight
    • Angular JS
    • AJAX

(Source : (https://smartbear.com/product/loadcomplete/specifications/))

Support

I like the support provided by Smartbear in the evaluation period. The person who contacted me from smart bear for helping with evaluation was really good. He along with his team did a good demo.

During the demo I raised a concern regarding the difficulty in replaying the recorded scripts. I got a hot response from their engineer but their solution didn’t work and further responses seems cold.

Platform compatibility

Windows

Reporting

Report generation is easy and pretty readable as compared to Jmeter.

Price

Need two products to automate Web and API unlike other performance testing tools.

  1. Web – Load Complete $500 – 500 VU pm. $850 – 1000 VU pm
  2. API – Load UI Ng Pro $11,999 – 1000 Requests / Tests $17,999 – unlimited

Total p.a – $23,999

Companies using Load Complete

Adobe, American Red crosss, Arc Light

Review

I tried to evaluate using the test case 1 as mentioned in the Jmeter. Correlation seems easy as compared to Jmeter and the reporting is too easy. But as the application is complex, it’s seems difficult in terms of customized coding as compared to other testing tools. During the demo I had raised a concern that, I was not able to verify my recorded script. The reply from community and the contact person at Load Complete seems cold. So to save time in evaluating the tool, I moved to Web LOAD.

Test case 2,3 & 4 are not evaluated as the test case 1 itself didn’t meet the evaluation criteria.

Web LOAD (Rad View)

About

WebLOAD is load testing tool, performance testing, stress test web applications. This web and mobile load testing and analysis tool is from RadView Software. Load testing tool WebLOAD combines performance, scalability, and integrity as a single process for the verification of web and mobile applications.

Technologies Supported

The list is too big to copy here. Most of the technologies are supported.

(Source : http://www.radview.com/technologies-supported/ )

Support

The support is super awesome. 10/10 for the person who supported me in the Rad View during the evaluation period.

I had raised the same concern that I encountered during evaluating the Load Complete with Web LOAD guys and they had spent a lot of time to solve the issue. Even though it took 3 hours to solve, I am extremely happy that the job was done!.

Test case 1 was recorded and replayed successfully with the support of Rad View guys. There are three different softwares inside the Web LOAD. They are

  1. Web LOAD IDE – this is a test creation tool
  2. Web LOAD Console
  3. Web LOAD Analystics

Platform compatibility

Windows

Reporting

Report generation is easy. Report seems more readable and valuable as compared to Load Complete.

Price

Basically there are two different pricing available with Rad View.

  1. Annual subscription Model –
    • Annual Cost + 300 VU;’s = $11,700.
  2. Purchasing Software
    • One Time software cost + 300 VU’s = $21,882.
    • Annual Maintenance cost – $3,824.

Companies using Load Complete

GoDaddy, Citrix, Symantec, Xerox

Review

There are three different softwares inside the Web LOAD. They are

  1. Web LOAD IDE – this is a test creation tool
  2. Web LOAD Console – For putting the load on test case.
  3. Web LOAD Analystics – for generating reports from the output from Console. Test case 1 was recorded and replayed successfully with the support of Rad View guys. And I had done some stress testing using and found two bottlenecks and reported the same to Mike.

Test case 2 was recorded for the back end. But while replaying the test case there seems an ‘connection lost’ error. The rad view support seems cold to this enquiry.

Neoload

About

NeoLoad is the fastest, most automated performance testing tool on the market that helps you eliminate bottlenecks in web and mobile applications.

Technologies Supported (Source : https://www.neotys.com/technologies )

Support

I didn’t get any reply for demo. So basically I don’t find the company support useful. The tool seems easy without any support.

Platform compatibility

Windows

Reporting

Report generation is easy. Report seems more readable and insightful.

Price

Pricing seems to be that 1/3rd of HP Loadrunner.

Companies using Load Complete

Accenture, Cognizant, HCL, Mindtree etc..

Review

Neoload seems to be very easy to learn and all the main features (like test case creation, load and analysis) are integrated into a single software unlike other popular tools like Web LOAD and HP Loadrunner.

I had used the test case 1 for evaluating this tool. the test creation was pretty straight forward but lost some time in creating a random E-mail id. Correlation and maintainability is something that I encountered while evaluating the tool.

Loadrunner (HPE)

About

HPE LoadRunner is a software testing tool from Hewlett Packard Enterprise. It is used to test applications, measuring system behaviour and performance under load.

Technologies Supported (Source : https://saas.hpe.com/en-us/software/loadrunner )

Support

I got a person assigned to me for asking questions. I requested demo, but didn’t get a satisfactory reply. The communit support seems good. Got answers for the most issues while developing the test cases.

Platform compatibility

Windows

Reporting

Report generation is intuitive, readable and insightful.

Price

Pricing $0.65 per Virtual user / per day.

So for 300 VU for 100 days – $16,800.

Companies using Load Complete

Couldn’t get any official information on the companies using HP Load Runner. But claimed to be the best performance testing tool.

Review

The main three parts of Loadrunner are

  1. Virtual user generator – For creating the test case
  2. Controller – for creating the load / stress plan
  3. Analysis – for analysing the test results. I had started with the test case 1 as I had done in other test tools evaluation. Initially I started the Web HTTP/HTML. I had recorded the test case 1 and couldn’t replay the script I recorded. I had seeked the support from HPE, but the response was cold. Then I did the same test case using their TruClient, which is a patented technology from HP for creating the test cases. While automating Web application, I found that it’s comparatively easier to maintain the script.

Most importantly truClient is simulating more or less the real world scenario unlike other tools which use direct interaction with the server. Another reason is – in the back end – when I click the back button, it’s throwing me error (using other applications) due to the browser sessions which need to be done manually. In Loadrunner, truClient takes care of such things.

Test case 1,2,3, and 4 was automated sucesfully using the HP Loadrunner.

Reasons to Select HP Load Runner

FrontEnd Performance Testing

  • Focus on user perceived performance (as opposed to server performance and resource utilisation)
    • Rendering times
    • Loading times / network timing
  • Load injection used to simulate real usage scenarios Note :
  1. Correlation : When a user logs into an application, the application may assign a session id to a particular user. The session id is managed by the server and the client so that the user does not need to know anything about the session id. When the server receives a message, it uses the session id to identify which user it has received a message from. Knowing who the user is can be used for many activities (Source http://www.testingperformance.org/performance-articles/correlation)

Leave a Comment

Your email address will not be published. Required fields are marked *