Playwright vs Selenium: The Ultimate Showdown for Automating D365 | TTC Global

Playwright vs Selenium: The Ultimate Showdown for Automating D365

Explore the strengths and weaknesses of TTC Global's Selenium-Java and Playwright-Typescript frameworks for effective test automation.

Mei Reyes2b 2
  • Mei Reyes-Tsai
  • 25 September 2024

Most organisations are sensitive to the licensing costs involved in scaling up proprietary off-the-shelf test automation tools, so they look to Open-Source tools. While these are free of licensing costs, they require significant engineering resources to use and maintain. These days, the choice for Open-Source test automation frameworks often comes down to Selenium vs Playwright.

In this blog, we provide a comprehensive comparison between TTC Global's Selenium-Java and Playwright-Typescript frameworks, specifically in the context of automating Dynamics 365. We evaluated the strengths and weaknesses of each framework, providing insights into their capabilities, performance, and suitability for various project needs.

What is Dynamics 365?

Microsoft Dynamics 365 is a portfolio of business applications designed to enhance operational efficiency and reduce complexity of business operations. Many of our clients are required to test their D365 implementation, and we are often asked for our recommended Open-Source test automation tool.

In response, TTC Global has evaluated the relative advantages and disadvantages of our Selenium-Java Framework against our Playwright-Typescript Framework in driving automated test scenarios against Dynamics 365. The evaluation project encompassed an end-to-end order to cash workflow in Microsoft Dynamics 365 Business Central. We chose this specific product and workflow to reflect recent interest from clients and to cover all aspects of the user interface of the system under test, such as tables, input fields, dialogs, and PDF files. We also automated the workflow separately using each of the two frameworks.

TTC Global's Selenium-Java Accelerator Framework

Selenium is a popular Open-Source automation testing suite widely used for the automation testing of web applications. It automates browsers and interacts with UI elements to replicate user actions to test whether a web application is functioning as expected. Through its single interface, the Selenium framework allows the tester to write test scripts in multiple languages, including Java, Ruby, Perl, C#, NodeJS, Python, and PHP. Its extensive community support and documentation make it a go-to choice for testers looking for a reliable and versatile testing solution.

Selenium Client Libraries


TTC Global has developed an in-house framework based on Java and Selenium. The TTC Selenium Accelerator framework is fully featured and supports all the relevant action types required to automate any applications and any workflows. As an Open-Source Java-based framework, it requires the automation engineer to be familiar with basic programming concepts, as well as the Java language, in addition to the framework itself—thus it has a steeper learning curve compared to no-code test automation tools. That said, the framework provides convenient methods and structure, well suited to end-to-end testing, and is easy to use after the initial onboarding.

The TTC Java framework is built with Open-Source components and leverages Selenium WebDriver to drive user interactions with real branded browsers. This is one of the most popular and time-tested automation framework architectures. It is well suited for end-to-end test automation and delivers trustworthy test results, but it also has notable drawbacks related to the framework architecture. Selenium methods can sometimes be susceptible to intermittent runtime failures caused by dynamic page content updates during test execution. To eliminate these, TTC has had to implement a relatively heavy Java framework with custom user interface interaction methods over raw Selenium to resolve these issues. As such, TTC Selenium-Java Accelertor framework provides sophisticated improvements over raw Selenium, such as the ability to dynamically identify the relevant iFrame based on the element locator. However, it may require substantial maintenance to ensure it keeps working as expected with new browser versions -

One benefit of the open source TTC Java framework over no-code test automation tools is its extensibility. If the specific automation project requires non-standard actions, such as integrating with a mainframe or a proprietary tool, the TTC Java framework can be extended in-house by leveraging the Java language capabilities and third-party Open-Source components.

The TTC Playwright-Typescript Framework

Microsoft’s Playwright is a popular and more recent addition to headless browser testing frameworks. Built by the same team that created Puppeteer (headless browser testing framework for Google Chrome), Playwright is an Open-Source NodeJS-based framework.

It’s designed for testing from a browser’s perspective, offering features like auto-wait, network interception, and emulation capabilities for mobile browsers. Playwright integrates WebSocket technology, which is essential for real-time web communication among clients and servers. This enables tests to mimic and verify complex, interactive web applications efficiently. This capability is important for apps relying on instant data exchange, enhancing the framework’s utility in creating robust, reliable test scripts for a wide range of web applications.

Web Socket Commands


TTC Global has also developed a Playwright-Typescript accelerator framework that is lightweight and fit for purpose, easily covering all automation requirements for any web application, including the Dynamics 365 automation workflow.

The framework is essentially a collection of useful utilities coupled with raw Playwright in a modular project structure that facilitates the design and implementation of reusable business logic abstractions in the test project. As such, it is easy to use for anyone familiar with Playwright, with almost no learning curve. Playwright itself, though, requires some automation engineering experience to work with, as well as solid Typescript knowledge.

All the extensive Playwright features are supported, along with additional methods useful in test automation. The framework is unlikely to require any extensions in a test project, unless there are unusual requirements.

The TTC Playwright-Typescript accelerator framework has adequate lightweight documentation in a readme file, and extensive Playwright documentation can be accessed online, along with the large and active Open-Source community.

TTC Playwright framework has extensive features and produces trustworthy test results. The only drawback to note is its ability to handle iFrames, which, unlike the Java framework, requires the test automation engineer to identify the relevant iFrame for each user interface component. The typical advantages and disadvantages of an Open-Source solution apply here, including superior extensibility, but also a higher requirement for technical skills to use the framework.

Selenium vs Playwright

Like the Java framework, the Typescript solution is based on Open-Source tools, with the same pros and cons as the Java framework. However, there are significant differences as well. Unlike the Java solution, the Typescript framework is based on Playwright, rather than Selenium. This gives it distinct advantages in test automation. The underlying Playwright framework uses its own browsers, with different APIs from branded browsers like Chrome or Firefox, but the same rendering engines. This allows automated tests based on Playwright to use Playwright-specific methods for setting up preconditions and driving robust user interface interactions without the need for additional custom code. It also enables advanced functionality not possible with Selenium, such as interfering with the network requests the browser makes during the test.

The TTC Playwright-Typescript framework is lightweight and low maintenance compared to the Java framework. It enhances raw Playwright with some additional useful methods and business logic abstractions. Despite Playwright’s reliance on custom browsers, these browsers use the same rendering engines as branded ones like Chromium or WebKit. This ensures the system under test behaves the same as it would for users, while offering powerful functionality for certain edge case scenarios.

One notable disadvantage of the Typescript framework compared to Java is its inferior ability to handle dynamic iFrames. Although a method is provided for iFrame selection, it is not fully integrated into the element locator strategy as it is in the Java framework, due to some architectural limitations of Playwright. However, it is still possible to adequately handle multiple iFrames. Other than this, Playwright does not have any significant drawbacks compared to the Selenium framework.

Overall, the Playwright framework has more features than Java and is easier to maintain, making it a more favorable choice unless the team has a strong preference for and expertise in Selenium. Both Open-Source frameworks, however, lack support for desktop application automation. This can be a significant drawback for projects requiring such capabilities. Nevertheless, we have recently demonstrated that desktop applications can be driven on Playwright using FlaUI.

Here is a short comparison of the features of Selenium VS Playwright:

FeatureSeleniumPlaywright
Cross-browser supportRequires driver installationsBuilt-in for Chrome, Firefox, WebKit
Language SupportJava, Python, C#, Ruby, and moreJavaScript, Python, Java, C#/.NET
PerformanceCan be slower, prone to flakiness (this is addressed by TTC Global's accelerator framework)High due to efficient architecture
Debugging capabilitiesRequires plugins for advanced debuggingAdvanced with screenshots, videos
CI/CD IntegrationExtensive integration capabilitiesBuilt-in support
Community SupportLarge, well-establishedGrowing with strong backing

Here is as summary of the pros and cons of each framework:

Selenium-JavaPlaywright-Typescript
Fully-fledged custom framework with full flexibilityLightweight set of utilities over standard Playwright
Good for multiple iFramesLimited support for iFrame switching
Limited browser state reuse capabilityFull support for sophisticated browser integration with advanced features
Limited ability to support advanced browser integration: page events, network mocks, etc.Extensible locators and fixtures

A side-by-side comparison of the pros and cons of each of our evaluation criteria on automating D365 using either frameworks:

CriteriaPros of PlaywrightCons of PlaywrightPros of SeleniumCons of Selenium
Ease of UseHighly flexible and powerful scripting capabilities. Supports complex automation scenarios.Requires programming knowledge in TypeScript. Writing tests can be complex for beginners.Highly flexible and powerful scripting capabilities. Supports complex automation scenarios.Requires programming knowledge in Java. Setting up and writing tests can be complex for beginners.
Setup and ConfigurationMinimal Setup and a few commands to make Playwright work.Requires installation of Node.js, VS Code, and various extensions.TTC Global's Selenium accelerator framework supports easy setup by cloning a skeleton test project structure.Requires setup of Java, Selenium libraries, and browser drivers. Configuration can be time-consuming.
Scripting and Automation CapabilitiesHigh flexibility in automating browser interactions and adept at managing both traditional and complex applications. Our Playwright Accelerator Framework enhances the capabilities of Playwright's comprehensive feature set.Robust handling for complex web applications, including iFrame handling, robust wait, and customized web element interactions
Integration with Other Tools

Has language support for multiple programming languages. It offers cross-browser compatibility, enabling consistent testing across various platforms.

Some JavaScript libraries are hard to integrate with TypeScript.Extensive integration support through various libraries and plugins, file/PDF parser, API integration, BrowserStack for cross-browser testing, comprehensive test reporting through Allure report, capable of integration with various CI/CD tools (AzureDevOps, Bitbucket, Github).
Execution Speed and Performance

Playwright has their own browser which makes the execution faster. Playwright also has the ability to Mock APIs.

Performance on ARM Chips not tested yet.

TTC’s Selenium Accelerator framework designed to improve some performance issues due to SUT limitations, supports parallel testing, reuse of web application's session.

Significant performance issue with ChromeDriver on Mac M2 chip.

Debugging and Error Handling

Powerful debugging capabilities using IDE Tools. Playwright has built in debugging features. It also has a trace viewer feature.

Powerful debugging capabilities using IDE tools and Java debugging features. Comprehensive logging with Apache Log4J, retry methods for long responsiveness, soft assertions for accumulated failures.

Reporting and Analysis

Playwright features built-in reporters for trace viewing and allows integration with Allure Reports for enhanced test reporting.

Comprehensive test reporting through Allure report with historical trends preserved. Reports can be published on AzureDevOps or Github page.

Community and SupportActive Discord channel and monthly updates on their YouTube ChannelPlaywright is relatively new to the market

Large, established community with extensive resources and support.


Choosing the Right Open-Source Framework for Your Project

In conclusion, both frameworks are fit for purpose and can be used effectively to automate D365. The Selenium-Java framework has the best support for dynamic iFrame, and this framework is also the heaviest of the two. The Typescript-Playwright framework is the most flexible and lightweight with few downsides.

The choice of the automation framework is ultimately down to the project constraints, such as technical resource availability, as well as the size and time frame of the project. For a large, long-term project, however, we recommend a well-architected Playwright-Typescript-based solution due to its superior maintainability and scalability, which will result in a lower Total Cost of Ownership.           

For more detailed analysis and comparison, please reach out to us and we can share the comparison study in details.

--

Contributing Authors

Mei Reyes-Tsai, GM of Technology & Delivery at TTC Global

Pavel Marunin, Senior Consultant at TTC Global

Lan Nguyen, Principal Consultant at TTC Global

Brian Rivadulla, Consultant at TTC Global

Colvin Cooper, Consultant of TTC Global