End-to-End Testing Frameworks: Playwright vs Puppeteer
While quality assurance is a fundamental and indispensable part of software development, end-to-end testing is one of the central QA processes.
When to use end-to-end testing frameworks?
Depending on the project development stage, the main purpose of end-to-end testing is to make sure that
- the new functionality added does not affect the app, and all its features work as intended
- the whole app works as expected and is ready for release.
Consequently, it makes sense to start e2e testing when a part of approved app functionality is already developed, and is planned to remain unchanged in the future (at least till the app release). E2E testing automation is definitely reasonable for long-term projects.
There may be no point in writing E2E tests for very small applications, e.g. one-pagers.
Testing email sending and receiving functionality is a tricky task as well. You cannot use the same email address as it may soon become recognized as a source of spam. In that case the test will fail, meaning you will have to deal with a flaky test. It’s questionable and quite project-specific if automated email testing is reasonable.
Gain clarity with your project!
Introduction to E2E testing with Playwright
While Playwright is excellent for web app testing, it is totally unsuitable for mobile app QA. Yet, you can definitely use it for mobile web testing to check how web applications will work on mobile phones or tablets.
Playwright E2E testing simulates a user's interaction with a web browser. In other words, it imitates human actions within a web app to double check if everything works as intended. And if it doesn’t, the QA engineer will know that something went wrong from the automatically generated report.
Playwright is a complete framework that includes not only a library, but also a test runner, reports, and all other tools necessary for the QA process. It takes down numerous limitations you may face when working with other testing platforms, allowing to
- Work with a few browser tabs simultaneously, i.e. parallel testing
- Add any external NodeJS modules, e.g. libraries, or additional HTML reporters with extended functionality like advanced graphs
- Access all the development tools, e.g. console, etc.
- And much more.
3 main layers of Playwright
NodeJS layer is implemented as a NodeJS library. It provides the main functional infrastructure and enables
- launching the browsers
- interactions with web pages
- managing web page components.
Browser automation protocols ( Chrome DevTools, Microsoft Edge DevTools, and others) allow Playwright to interact with browsers that work on different platforms and operating systems.
Playwright works on Windows, Mac, and Linux. It has an in-builtsupportto launch and manage browsers on any of these operating systems. Thus you can write automation scenarios that do not depend on the platform they will run on. Moreover, Playwright enables running the tests in a Headless mode when there’s no UI (you don’t see the browser). It is needed for running the tests in the continuous integration (CI) environment.
Playwright vs Puppeteer: why we have never regretted switching to Playwright
As a part of quality assurance on Poly facility management app, we’d used E2E testing with Puppeteer since 2019. Two years ago, it became the first project on which we switched to Playwright. And here are our reasons.
Ease of migration. The test syntax in Puppeteer is almost identical to the one in Playwright, which has made the transition rather fast and easy. With over 500 tests to migrate, it took one QA engineer less than one week to do the full test code refactoring.
Time and cost efficiency. Besides the adjustment to Playwright, test code refactoring has also provided test optimization. It decreased the test run time by 25-30%. Partly it was because
- the capabilities provided by Playwright allowed us to decrease the number of code lines
- the built-in reporting tool in Playwright offers video recording which does not extend the test run time, unlike the solution which we had integrated with Puppeteer.
Playwright turned out to be a cost-efficient solution not only by saving QA engineer’s time, but also by decreasing the time of using paid continuous integration tools and services like CircleCI, where every minute counts.
Framework configuration. Playwright provides a complete out-of-the-box solution which includes a library, a test runner, reporting functionality, API, methods, and all other components necessary for automated testing. Besides, it is flexible when it comes to adding any external NodeJS modules. For instance, we use it with Jest test runner, which is compatible with numerous testing frameworks, offers extended reporting functionality, and has a large active users community. Besides, Jest was convenient for our team, as we’ve been using it on Poly before switching to Playwright.
On the other hand, Puppeteer is just a NodeJS library, which means you need to find and configure external compatible tools like the test runner and reporting solution.
Test automation capabilities. Puppeteer offers limited automation capabilities and a lot must be written by the QA engineer by hand. It requires writing very detailed scripts while Playwright provides out-of-the-box solutions.
WebSocket API. There’s a WebSocket connection between the tests and Playwright to work directly with the browsers. It eliminates the need for an additional layer consisting of a server, like Selenium, granting higher reliability, stability, and speed.
All in all, Playwright has certainly outgrown Puppeteer, and seems to be its next evolution step. However, if you want to use it just as a library, avoid the new features, and write your own test scripts by hand, you can easily do it in Playwright as well.
Playwright features overview
Playwright has a well-written documentation with a detailed description of different features and configuration options. Its explicitness makes it easy to decide whether you need a particular functionality or not, and which parameters to choose for your exact requirements.
After the installation there’s access to a large library with test examples, demos and snippets. You can pick NodeJS, Python, Java or .NET environment to view the featured code snippets. It helps to explore how Playwright works, and is really useful for beginners.
Although there’s multi-platform support, the new features and updates are primarily released for Node.js (JavaScript, Typescript), and for the other platforms they are added later on.
Let’s briefly go through the most frequently used functionality.
Running tests in UI mode
You can use a convenient dashboard to access and run the tests with just a click instead of a command. It also allows you to pick a locator just by clicking on a page element you want to test, so the code snippet is configured automatically. Isn’t that cool?
The UI mode dashboard provides you with very useful functionality for tests monitoring and management. With just a mouse click you can
- select the browser
- access tests code, console, log, and network
- filter the tests
- view Playwright trace
- view the timeline, and more.
Reporting
Playwright has its own in-built HTML reporter which allows you to attach screenshots and record a video. You can also pick the video file format, and the directory it will be saved to. This functionality makes it easier to debug the tests, particularly when they are running within the CI pipeline. Right at the stage of reviewing the test report you can
- faster understand if there’s a bug within the test or the app
- find out what exactly it is.
We’ve experienced the reports being so informative, there was no need to open the application after running the tests. It was possible to directly add a bug fix request. And what’s important: video recording in Playwright does not increase the test run time.
To extend the reporting functionality even more, you can use third-party libraries, e.g. Allure reporter. This out-of-the-box ready-to-use solution provides nice diagrams illustrating test status, duration, etc. Screenshots and video recording are enabled here automatically by default.
Parallel testing
Parallel testing is the ability to run the tests simultaneously. In Playwright, all the tests are isolated on separate pages by default. They don’t overlap in any possible way with the other tests used (no cache, cookies, etc.), and run on separate tabs at the same time.
When you need to reuse the context of the previous test for the next test, e.g. when you log in at test No1, and need to proceed at test No2, it’s also possible to use a single test page (reuse a single page between the tests). You can achieve this through the setup, so you won’t log out from the first test’s page, and will proceed to the second test there.
Playwright API testing
API testing is possible both with inbuilt functionality like requests, and external libraries, like axios, node-fetch, apollo/client.
Page object models
You can create logic to test a certain page object, save it, and then reuse it each time when this object is dealt with. Thus you can reuse such page object models in multiple tests. If you change an object model in one place, it will be automatically updated across all the tests that use it. It makes fixing the tests much easier.
Locators
Playwright locator is a good way to find a selected group of components. It provides excellent code readability and clarity.
await page.getByLabel('User Name').fill('John');
await page.getByLabel('Password').fill('secret-password');
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page.getByText('Welcome, John!')).toBeVisible();Source: Playwright documentation
However, you cannot fully rely on the provided locators, as they may not cover all the components you need. So it’s a good idea to master traditional approaches, e.g. using CSS or XPath, as well.
await page.locator('button').click();
await page.locator('//button').click();
Annotations
Playwright annotations significantly extend your testing capabilities and may be used in a multitude of configurations:
- Skip tests if they are not your current focus or if you know they cannot be passed, so that they don’t stop the whole sequence. You’ll see a list of skipped tests in the report, so you can return to them later
- Add a fail tag to run the test and check if it really fails
- Avoid running a test by adding a fixme tag, e.g. when the test needs to be improved or fixed
- Increase test timeout with a slow tag, and more.
For instance, the annotations may be useful when you are developing or debugging tests. You can skip a group of the tests to focus on the particular one you are currently working with. And vice versa, if there’s a sequence with one test that doesn’t work as intended, or when the feature it checks is broken, the runner can skip such a test so it doesn’t fail.
Retries
You can run a code snippet as many times as requested with the indicated timeouts or until it passes the test. This feature may be useful for so-called flaky tests - the tests that one time get passed, the other time can fail. Our advice is do not overindulge with Playwright retries. It’s highly likely that there is a way to improve the test and make it stable instead. However, in some cases such a feature can be useful.
This brief overview of Playwright’s functionality only provides you with a first impression of the framework, and does not cover its every aspect. But if it catches your attention, you definitely will find more details in the documentation.
Playwright AI capabilities
Playwright does not have its own built-in AI functionality, but you still can benefit from all the advanced features offered by AI testing tools.
How about Auto Playwright? - you may ask. Despite its name, it’s a separate solution. It is easy to integrate into Playwright. This AI tool assists with generating test scripts based on descriptions or specifications provided. Its ability to understand natural language instructions is invaluable for nontechnical users. It enables them to actually run basic quality assurance tests, as well as speeds up their learning curve.
Besides, Auto Playwright can adapt to changes in the application by analyzing the test results and user feedback, helping to keep the tests up-to-date with minimal manual intervention.
You can integrate many popular AI testing tools like Testim and Applitools Eyes with Playwright to enhance the test automation process. They enable such advanced functionality as cross-browser visual testing, parallel tests execution, offer automated QA reports generation and convenient dashboards, and more.
AI coding assistants like GitHub Copilot can suggest code snippets, help with boilerplate code, and even generate test scripts based on the context of the code being written.
The use of AI-powered functionality certainly allows achieving higher time and cost efficiency. It helps to automate and streamline the process of writing Playwright tests, making it easier to create comprehensive test suites with less manual effort.
Final tips
Playwright is not the only QA automation solution we use at Apiko, but it’s certainly a useful tool worth considering. It contributes greatly to time and cost efficiency of quality assurance. For instance, if we compare Playwright vs Selenium, its significant advantage is no need for additional drivers to run on CI, e.g. with Docker.
If you decide to give Playwright a try, our three tips will be
- Use video recording in reports
- Benefit from parallel testing dividing the tests between multiple tabs
- It often may be enough to run tests only for one internet browser, as choosing a few browsers would increase the test run time considerably.
Feel free to reach out if you’d like to share your thoughts or need a hand with QA.