We reviewed 10 of the most queried test automation tools for the web on the market. We reviewed key features, price, prerequisites to use, pros and cons.
The candidates selected for review include:
- 2 new trending players – testRigor, Testim
- 4 open source – Selenide, Webdriver.io, Cypress, Robot Framework
- 4 enterprise tools – Ranorex, Tricentis Tosca, UFT, TestComplete
Open source tools require you to have knowledge of some level of web development, programming languages, and be tech savvy to install and set up the framework.
Meanwhile, enterprise tools require less or no coding. However, all of them are Windows based standalone platforms with very complex UI, which requires additional training for users.
Setup requirements – all frameworks required setup and installation except testRigor and Testim
testRigor
testRigor is an increasingly popular new cloud-based end-to-end testing system. It allows you to build tests from the end-user’s point of view avoiding using selectors/locators. It doesn’t require any installation, infrastructure setup or coding knowledge. It uses plain English, so writing test steps is as intuitive as writing any statement. The main advantage of testRigor is easy test maintenance. It groups failures automatically and provides steps to reproduce. Also, it can generate tests based on how your end users are using your application.
Key Features
- Web, mobile and API testing
- Plain english no-code test case creation
- Chrome extension test recorder
- Scheduled test execution and monitoring tools
- Parallel and parameterized testing
- CI/CD integration
- Generation of test cases based on analyzing user behavior
Code Sample
Price
- Free Public Plan
- $1,800.00/ month
Prerequisites
- No prior coding or technical experience
- Understanding of basic testing
- Fast and easy setup and start using without any installation or downloads
- Test case creation is 15 times faster than writing Selenium tests
- Adapting new functionality is as easy as find and replace; testRigor’s quote “99.5% less test case maintenance time”
- Runs tests in parallel
- Cross mobile and web workflows
- Easy to create modular functions
- Integration with Jira, TestRail, Azure DevOps, Twilio, Zephyr, etc.
- Unlimited number of test cases and test runs
- Queued test suites can take time to execute if you are not buying more infrastructure from them
- Test recorder scripts usually require review and tweaking
Testim
Testim.io is an end-to-end Agile software testing tool augmented with AI features. The AI allows you to speed up automation of web-based applications. This web-based platform allows teams to create good software quality practices while creating test suites. Testim allows for “smart locators” which solve the problem of rigid element identification within HTML code.
Key Features
- Smart Locators – assist to identify best locators
- To create tests user can use codeless visual editor or edit code in JavaScript IDE
- Ability to export any tests into JS code
Price
- Free Community Plan
- Enterprise price N/A
Prerequisites
- No prior coding or technical experience
- Understanding of basic testing
- Web browser support
- Available integrations: CI/CD, Jenkins, Jira, Slack, Applitools, Browserstack, etc.
- Run tests in Parallel
- Infrastructure hosted in the cloud
- Minimal setup, no installation needed
- No native mobile testing
- Creation of test cases limited to using a recorder. Otherwise, user has to write code in JS
WebDriverIO(JS)
WebDriverIO is a custom extension of Selenium’s Webriver, written in Javascript, and requires Node.js to run. A testrunner client is included which has the ability to create and execute both web and mobile test automation. WebDriverIO is an operating system and language neutral tool that allows introspection and control of user interfaces. With WebDriverIO the tester can accomplish more with fewer lines of code than they could in Selenium.
Key Features
- Automates browser and native mobile app testing
- Wdio setup wizard with support for Cucumber, Mocha and Jasmine
- Extendable helper functions
- Integrates with native browser tools such as Chrome Devtools and Google Lighthouse
- Integrated testrunner
Code Sample
Prerequisites
- Knowledge of JavaScript, Mocha/Jasmine, XPath, CSS
- Browser and native mobile support
- Large variety of community plugins available
- Third party integrations make debugging easier
- No explicit API testing support
- Slower than some other testing tools and frameworks
- It requires deep knowledge of coding and web development
Cypress JS
A front end browser testing framework that can test anything that runs in a web browser. Their mission is to create an enjoyable experience that generates developer happiness while championing test practices that actually work. Cypress is a technical testing framework where the typical users are developers and QA engineers building web apps with Javascript frameworks. This system uses Domain Specific Language for end to end tests.
Key Features
- Testrunner interface
- Snapshots as your tests run
- Waits for your commands to run without adding explicit waits and sleeps
- Integrates with developer tools
- Unit test functionality of spies, stubs and clocks available
- Network Traffic Control emulation
Code Sample
Prerequisites
- Knowledge of JavaScript, Mocha, XPath, CSS
- Tests are more stable and way faster than with Selenium
- Native access and operates within the application
- No explicit, implicit waits in tests. Cypress handles waits elements automatically
- Ability to send http calls directly
- Great documentation and growing community support
- Doesn’t support switching tabs or multiple browsers
- Supports only Chrome, Firefox and Edge
- Tests run inside the browser, supports only JavaScript
Robot Framework
Initially developed by nokia networks and open sourced in 2008, Robot Framework is a generic Python automation framework used for acceptance test-driven development. The unique foundation allows for multiple test case styles, spanning from behaviour-driven to data-driven test cases. This system has robust features such as human readable keywords, variables, test tagging and integration with external third party libraries. The system provides the Robot Framework Development Environment(RIDE) editor as a download or you can use any text editor. Test results are provided in the form of XML or HTML reports and logs.
Key Features
- Extensive collection of external libraries that support web, mobile, desktop, API, DB testing, etc.
- Operating system and application independent
- Test tagging of test cases
- HTML and XML test reports and log files
Code Sample
Prerequisites
- No requirements to start using this framework. However, Python knowledge will help to write complex tests
- Human-readable keywords, easy to start writing tests without coding knowledge
- Capabilities can be extended with Python and Java libraries
- Besides Selenium it supports numerous libraries
- Compatible with any operating system
- In comparison to other frameworks it has the shortest learning curve
- Challenging maintenance of the tests and the framework
- Lack of parallel test execution
- Slower compared to other open source frameworks
Selenide
Selenide is a browser testing framework that acts as a wrapper library around Selenium webdriver. It allows for short succinct code lines in an easy, readable format. It features the Selenide webdriver manager which controls the varied webdrivers required by selenium for cross browser testing (Chrome, Edge, Firefox, … ) Additionally, it allows testers to create free tests with open source tools.
Key Features
- Simplifies Selenium based testing
- It takes screenshots automatically on every test failure.
- Selenide API inspired by jQuery – working with locators similar to jQuery
- Automatic handling of most problems with Ajax, waiting and timeouts
Prerequisites
- Basic knowledge of Java, XPath, CSS, TestNG or any other testing frameworks
- Allows tests to be created with fewer commands than selenium. Easier and simpler syntax compared to selenium
- Selenide offers less code set up than selenium
- Selenide WebDriverManager automatically downloads latest webdriver for you
- Language support only for Java family languages
- Relatively new and small community around this tool
Ranorex
A Microsoft Windows based testing tool that allows you to create multiple test suites for different platform types. This installable .NET tool can create automated tests for hard-to-test desktop clients and browser based apps. In addition, there is support for testing mobile end points and native applications on real iOS and Android devices. You are able to create data driven tests and regression test suites using both keywords and code. Selenium is built into the system which allows you to run legacy tests you may have from Selenium.
Key Features
- Keyword driven testing
- Studio IDE
- Capture and replay functionality
- Execute tests on desktop, web, mobile endpoints and native iOS, Android testing on real devices
Prerequisites
- Tool training is required
Price
- Starts from $3,590 per license annually
- Free 30-Day Trial
- Selenium webdriver is built into the studio
- Fits for windows desktop application testing
- Good training and support
- Takes time to understand the UI and requires training to use this tool
- Supports only C# and VB.net
- Requires software framework installation
- Windows support only
Tricentis Tosca
Tricentis is an AI driven testing platform that allows for end-to-end continuous testing. They provide a fully automated system that encompasses the entire lifecycle of testing and works with both enterprise and small cloud based apps. Additionally, they offer a suite of tools that range from test automation to test management. Tricentis Tosca currently boasts many fortune 500 companies as customers.
Key Features
- Codeless test creation
- QA consulting services
- Service virtualization
Price
- N/A
- Users quote:
- “$11,000 for one license.”
- ” We bought in bulk for $7,000/license”
Prerequisites
- Tool training is required
- Applicable to Agile groups and complex enterprise apps
- Accessible for QAs without coding knowledge
- Supports native app and browser testing
- Mature product with enterprise customers
- Supports parallel testing
- Even though it is a codeless tool, there is a steep learning curve to use it
- Not the greatest API documentation
- Hard to reuse modules which leads to maintenance challenges
- Expensive
- Windows support only
Unified Functional Testing (UFT)
Formerly known as HP QuickTest Professional(QTP), UFT is a functional and regression test automation tool for applications. UFT QAs and developers are able to test the UI, the service layer, and the database layer from a single console. UFT supports desktop, web, and mobile product testing and is geared toward creating automation scripts for a less experienced coder. Additionally, the tools are suited for continuous integration development.
Key Features
- Multi-layer test scenarios include GUI and API testing
- Visual user-experience tool sets
- Insight object recognition tool for testing emerging technologies
Price
- $3,200 per license annually
- Free 30-Day Trial
Prerequisites
- Tool training and knowledge of VBScript are required.
- Multiple browser support
- Tech support during business hours
- Large product adoption with 5000+ customers
- Support for integration with third party tools
- Complicated and outdated UI/UX
- It requires additional training to start using this tool
- Maintenance of test cases is pretty challenging
- Windows installable client only
- For API, load or performance testing you must purchase additional products
TestComplete(SmartBear)
A windows installable application that serves as the single tool for all UI test automation. TestComplete allows for multi-browser, native iOS, and Android testing. The test tool is created so team members of different skill levels can use it effectively. TestComplete allows you to scale tests over multiple physical and cloud based test systems. With “Intelligent quality add on” you are able to detect and test application components which may not have been previously noticed.
Key Features
- Codeless test case creation
- AI powered hybrid object-recognition engine
- Automated Test Reporting & Analysis
- Free training & support
- Test automation for BDD
- Service virtualization
Price
- Start from $2,256 per license
- Free 30-Day Trial
Prerequisites
- Tool training is required. Additional knowledge of JavaScript or VBScript will help
- Multiple browser support
- Tech support during business hours
- Large product adoption with 5000+ customers
- Support for integration with third party tools
- Complicated and outdated UI/UX
- It requires additional training to start using this tool
- Maintenance of test cases is pretty challenging
- Windows installable client only
- For API, load or performance testing you must purchase additional products
For open-source tools, the main advantage is that they are all free. The main disadvantage is that they require technical and coding knowledge. For open-source tools, we chose Cypress and Robot Framework. Cypress is a better fit for Developers and QA Automation Engineers, while Robot Framework is for QA Engineers who want to start learning automation.
On the other hand, legacy enterprise tools are supposed to be easier to use. However, with complex user interfaces and requirements for coding knowledge, QA engineers still need to go through additional training. Besides, all of them run only on Windows machines, and they are a better fit for windows application tools than web applications.
We would go with testRigor as the first choice and Testim as a second for web applications testing. Each offers a modern, intuitive, user-friendly interface, robust tests and a free version. We chose testRigor over the Testim for better handling of test creation and easier maintenance. Most importantly, they both resolve the technical barrier to writing automated tests without diving into the coding, making these tools accessible to any team member and getting the job done by anyone.
