WebdriverIO is a progressive automation framework built to automate modern web and mobile applications. It simplifies the interaction with your app and provides a set of plugins that help you create a scalable, robust, and stable test suite.
- Extendable – Adding helper functions or more complicated sets and combinations of existing commands is simple and really useful
- Compatible – WebdriverIO can be run on the WebDriver Protocol for true cross-browser testing as well as Chrome DevTools Protocol for Chromium-based automation using Puppeteer.
- Feature Rich – The huge variety of built-in and community plugins allows you to easily integrate and extend your setup to fulfill your requirements.
Cost is often cited as the limiting factor in choosing a test automation tool. WebdriverIO is a good alternative since it is open source and has a large user base on their Gitter Support Channel to help users. The team also has Open Office Hours. This allows users to get started contributing to WebdriverIO easily with the help of people that know the code base well.
Here are some of the highlights of the pros and cons of WebdriverIO, with input directly from reviews found around the software community.
Overview of WebdriverIO
WebdriverIO has been an ongoing open-source JavaScript / TypeScript project headed up by Christian Bromann on GitHub since 2016. It is a Node.js automation framework that supports a browser, mobile and local desktop application test script development in Visual Studio Code.
In recent years, the product has continued to increase support with several other full-stack application integrations. It can test applications written in Angular Vue, React, Clojure, and many others. In version 8.0, it has moved from CommonJS modules to ECMAScript at its core.
In terms of pricing, the tool is free. A great deal for GUI, Mobile and API tool with a wide range of full stack testing capabilities. It can be implemented in JavaScript or TypesScript. It supports two popular frameworks out of box: Mocha, Jasmine and Cucumber. Several boilerplate projects are available to get jump started.
Top features in WebdriverIO
This tool is relatively easy to get started for beginners. The initial installation provides a full file structure for Spec files, or tests, command line shortcuts, page object model support and sample tests to get started.
The framework can be customized through one config file. This file is fully documented with default values and options. This allows extensive configurations of functionality and implementation of advanced features.
Installation requires Node.js and Node Program Manager (npm) to install base configuration of WDIO. While Webdriver 7.0 can support Node.js as low as 12.0 it is recommended that version 8.0 use the latest Node versions 18.0.
Full stack technology automation
WebdriverIO can be used to test web applications by interacting with elements on a browser webpages and verifying their behavior. This includes actions such as clicking, typing, and selecting from dropdown menus. It can also be used to test desktop applications by simulating user interactions with the application, such as clicks, key presses, and mouse movements. It tests mobile web applications on devices such as smartphones and tablets as well as sending HTTP requests to the API and verifying the response.
Supporting Services
- Appium for testing native, mobile web and hybrid application on iOS and Android platforms
- Applitools for AI Visual testing
- Browserstack, Sauce Labs and Experitest for cloud based cross browser testing on web and mobile devices
CI/CD Tool integration
WebdriverIO provides support to run tests on containerized instances in Docker as well as continuous integration pipelines like CircleCI, Jenkins and GitHub Actions.
WebdriverIO provides a single configuration file for adding functionality.
Command line shortcuts provide a quick way to tests scripts to execute in multiple environment configurations. For example tests run in a Docker instance will benefit in speed by running headless, without a browser. But debugging is far simpler if the browser is visible to the Test Engineer. It also allows switching from Staging to Production urls with User Variables.
Integration with Node.js community-faker to support data generation. Since WDIO is a Node.js based framework, it can take advantage of popular Node.js packages from the npm registry. This API creates fabricated data from no less than 22 categories and 100+ subcategories.
While support for the popular faker package has been since discontinued by the author after version 5.5.2, the npmjs community has continued with community-faker version 5.5.5
Mock Web Resources and Spies
WebdriverIO catches up with other popular testing tools like Puppeteer, Playwright or Cypress by simplifying REST API mocking.
WebdriverIO continues to provide more functionality to its automation interface by shipping new network primitives to its API. Users can easily mock web resources in their tests with custom responses that allow reduced testing time of individual scenarios.
WDIO ships with a network Throttle command that allows modifying the network throughput of the browser.
// Throttle to Regular 3G
browser.throttle('Regular 3G')
// diable network completely
browser.throttle('Offline')
// set custom network throughput
browser.throttle({
'offline': false,
'downloadThroughput': 200 * 1024 / 8,
'uploadThroughput': 200 * 1024 / 8,
'latency': 20
});
Full stack support for Reporting.tf
- Dot
- Spec
- Allure
- Junit
- Sumo Logic
- Concise
- ReportPortal
Reporting with Dot and Spec Reporter
This style of reporting is best for the SDET to review for debugging and quick run sessions. The Spec reporter outputs if the test opasses or fails. The Dot reporter gives the overall count of tests Passing, Failed or Skipped.
Reporting in Allure
Allure has become the defacto choice for result reporting. It produces an HTML file with resources to build a nice interactive dashboard interface. This is best for giving overall results to stakeholders. It includes screen captures and can be configured to show end to end videos of test runs.
Support for Mocha, Cucumber and Jasmine frameworks
WebdriverIO provides integrations with the most common development frameworks. This includes Mocha, Cucumber and Jasmine.
Serenity integration
Recently users have been integrating WebdriverIO with John Ferguson Smart’s Serenity framework making a combination of two frameworks that enables users to have a complete test automation solution.
The WebdriverIO has created a two integrations for John Ferguson Smart’s Serenity/JS projects in both Cucumber and Mocha, which enables either Serenity’s Console Reporter
or it’s BDD HTML Reporter
New features recently added to WebdriverIO
Locator strategies
WDIO supports all the common locator strategies that Selenium supports including: CSS Query, xPath, Tag Name, Name Attribute, Link Text, Partial Link Text, Element with a certain text or partial text. However the new Accessibility Selector strategy goes well beyond these approaches.
Prior to that WebdriverIO version 7.24.0 introduced the Accessibility Selector. This means elements can now be identified by aria-label strings. For example this close button named “X”.
<button aria-label="close button">X</button>
can now be accessed by referencing with aria/ notation:
await $('aria/close button').click()
This extends to label > input elements by strings such as “Username”
<label for="username">Username</label>
<input id="username" type="text" />
await $('aria/Username').setValue('foobar')
And placeholder strings:
<input placeholder="Your Username" type="text" />
await $('aria/Your Username').setValue('foobar')
As well as alt text tags for images
<img alt="A warm summer night" src="..." />
await $('aria/A warm summer night').getTagName() // outputs "img"
Support for Puppeteer and extended Image Validation
WebdriverIO version 8.0.x recently added support for Puppeteer with pixel-by-pixel image validation and customizable options.
Switch from CommonJS modules to ESM
Another new feature is a browser plugin (@wdio/browser-runner) that allows users to run unit and component tests in the browser. It also changed from CommonJS to ECMAScript Modules (ESM).
Elimination of redundant TypeScript types
In prior releases, WebdriverIO auto-generated all type definitions, which caused the creation of a lot of duplicate types and inconsistency. WDIO has rewritten the complete code base to add type safety. With this overhaul, all types are directly taken from the code itself and centralized in a single new helper package called @wdio/types. If you have been using TypeScript, you will now have much better type support for various commands and the configuration file.
Enhanced performance metrics with Google Lighthouse
WebdriverIO can run on the WebDriver protocol for true cross browser automation, but also automate specific browsers using browser APIs such as Chrome DevTools. This allows for integrations into tools that allow broader testing capabilities such as Google Lighthouse on a mobile environment. With the @wdio/devtools-service, WebdriverIO can use Google Lighthouse to run performance tests. WDIO updated Google Lighthouse to the latest version to enable new performance metrics such as Cumulative Layout Shifts or First Input Delay.
Proxy Objects
The loss of the @sync package option in 2021 to provide global synchronous execution and promise unwrapping resulted in extensive code refactoring adding repetitive await statements when chaining locators.
await (await (await $('#foo')).$$('.bar'))[42].click()
Proxy objects have been leveraged to reduce much of the redundant await statements:
await $('#foo').$$('.bar')[42].click()
New PWA Check Command
WDIO has deepened integration to the Google Lighthouse tool with audits for capturing the quality of your progressive web apps (PWA). These applications are built with modern web APIs to deliver enhanced capabilities, reliability, and installability while reaching anyone, anywhere, on any device with a single codebase. To test if your application fulfills the PWA requirements WDIO introduced a checkPWA command that runs a variety of audits, validating the set up of your app.
const result = browser.checkPWA()
expect(result.passed).toBe(true)
Pros of WebdriverIO
In terms of market reception, here are some of the key aspects of WDIO that have received praise:
Support of multiple automation design approaches with plain JavaScript, TypeScript or Cucumber.
Flexible report generation including Allure.
WebdriverIO can generate reports in various formats, including HTML, PDF, and XML in customized layouts. These reports provide a wealth of information about the test execution, including test status, execution time, attached files (e.g. screenshots), and environment details. Allure allows users to track the status of tests over time, allowing the ability to identify trends and problems with the test suite.
Chromedriver service
The Chromedriver service constantly keeps the Chromedriver version in sync with the installed version of Chrome.
Support for concurrent browser execution
Tests can be run on up to five Chromium browsers concurrently. Failed tests can be set to rerun 2 or 3 times. They can be rerun immediately or delayed after the entire suite has completed.
Cross Browser support
Webdriver provides support of Chrome, Edge Firefox and Safari. These browsers are supported on Windows or Mac.
Integration with Applitools for visual validation
WebdriverIO can be used in conjunction with Applitools Eyes to perform visual testing of the application front end interface. This is accomplished by installing the Applitools Eyes WebdriverIO SDK. The SDK takes a screenshot and compares it to a baseline image, highlighting any differences.
Cons of WebdriverIO
In terms of drawbacks, here are some of the key points from critical reviews out in the market:
Loss of @wdio/sync support
This is due to Marcel Laverdet sunsetting the Node-Fibers project. This means a lot of refactoring of older projects adding numerous async keywords to functions and await statements to unwrap returned promise objects. This increases the code footprint significantly. If an await is missing it makes it likely code will be executed in concurrent threads and out of order.
The alternate option is to keep the version of Node.js at v14 or lower. This in turn blocks users from taking advantage of improvements in later versions of JavaScript in Node.js. It also prevents projects from using new features in WebdriverIO v.8.0 and higher.
Users are responsible for keeping the versions of the node packages up to date.
However this can be accomplished with the node-check-updates (ncu) package.
Concurrent test case execution is limited to 5 browsers.
This limitation can be overcome with Jenkins and Docker.
Few of the supported functions are fully unit tested in TypeScript
For example a Jasmine typescript boilerplate project was released with “jasmineNodeOpts” long after the configuration name had been changed to “jasmineOpts”. Sample tests provided complete within seconds but never pick up if they extend beyond 10 seconds, a Jasmine timeout will close the browser. WebdriverIO will throw a “Couldn’t find page handle” error for any test longer than 10 seconds.
Not well supported in JetBrains WebStorm
Much of the day to day debugging runs are executed from the command line to support System Variables. JetBrains WebStorm is a preferred IDE among professional JavaScript developers. However its lack of breakpoint activation when runs launched from the internal command line reduces its viability as a code editor for WebdriverIO. This makes Visual Studio Code the preferred option.
Some of the error messages can be misleading
In some cases change requests are closed with statements such as “Technically the error message is correct.” It’s up to the user to have a list of potential error messages, their actual root causes and solutions.
The Gitter support channel is not on par with paid tools
While the community tries to be helpful and responsive, users are just as likely to get “It works on my machine.” in response to more technical queries.
Conclusion
WebdriverIO is good with speed and technology. While JavaScript is exceptionally speedy, keeping track of async and missing await statements can cause greater delay in debugging. While the WDIO config file has a significant amount of functionality, debugging issues can be significantly difficult for new users. If ease of use is a priority for your team, you may want to consider other options that offer bespoke options like Katalon or Tosca. If more concurrent execution to get results faster is a priority then tools like testRigor should be considered.