Most organisations are sensitive to the licensing costs involved in scaling up proprietary off-the-shelf test automation tools, so they look to Open-Source tools. While these are free of licensing costs, they require significant engineering resources to use and maintain. These days, the choice for Open-Source test automation frameworks often comes down to Selenium vs Playwright.

In this blog, we provide a comprehensive comparison between TTC Global's Selenium-Java and Playwright-Typescript frameworks, specifically in the context of automating Dynamics 365. We evaluated the strengths and weaknesses of each framework, providing insights into their capabilities, performance, and suitability for various project needs.

What is Dynamics 365?

Microsoft Dynamics 365 is a portfolio of business applications designed to enhance operational efficiency and reduce complexity of business operations. Many of our clients are required to test their D365 implementation, and we are often asked for our recommended Open-Source test automation tool.

In response, TTC Global has evaluated the relative advantages and disadvantages of our Selenium-Java Framework against our Playwright-Typescript Framework in driving automated test scenarios against Dynamics 365. The evaluation project encompassed an end-to-end order to cash workflow in Microsoft Dynamics 365 Business Central. We chose this specific product and workflow to reflect recent interest from clients and to cover all aspects of the user interface of the system under test, such as tables, input fields, dialogs, and PDF files. We also automated the workflow separately using each of the two frameworks.

TTC Global's Selenium-Java Accelerator Framework

Selenium is a popular Open-Source automation testing suite widely used for the automation testing of web applications. It automates browsers and interacts with UI elements to replicate user actions to test whether a web application is functioning as expected. Through its single interface, the Selenium framework allows the tester to write test scripts in multiple languages, including Java, Ruby, Perl, C#, NodeJS, Python, and PHP. Its extensive community support and documentation make it a go-to choice for testers looking for a reliable and versatile testing solution.

TTC Global has developed an in-house framework based on Java and Selenium. The TTC Selenium Accelerator framework is fully featured and supports all the relevant action types required to automate any applications and any workflows. As an Open-Source Java-based framework, it requires the automation engineer to be familiar with basic programming concepts, as well as the Java language, in addition to the framework itself—thus it has a steeper learning curve compared to no-code test automation tools. That said, the framework provides convenient methods and structure, well suited to end-to-end testing, and is easy to use after the initial onboarding.

The TTC Java framework is built with Open-Source components and leverages Selenium WebDriver to drive user interactions with real branded browsers. This is one of the most popular and time-tested automation framework architectures. It is well suited for end-to-end test automation and delivers trustworthy test results, but it also has notable drawbacks related to the framework architecture. Selenium methods can sometimes be susceptible to intermittent runtime failures caused by dynamic page content updates during test execution. To eliminate these, TTC has had to implement a relatively heavy Java framework with custom user interface interaction methods over raw Selenium to resolve these issues. As such, TTC Selenium-Java Accelertor framework provides sophisticated improvements over raw Selenium, such as the ability to dynamically identify the relevant iFrame based on the element locator. However, it may require substantial maintenance to ensure it keeps working as expected with new browser versions -

One benefit of the open source TTC Java framework over no-code test automation tools is its extensibility. If the specific automation project requires non-standard actions, such as integrating with a mainframe or a proprietary tool, the TTC Java framework can be extended in-house by leveraging the Java language capabilities and third-party Open-Source components.

The TTC Playwright-Typescript Framework

Microsoft’s Playwright is a popular and more recent addition to headless browser testing frameworks. Built by the same team that created Puppeteer (headless browser testing framework for Google Chrome), Playwright is an Open-Source NodeJS-based framework.

It’s designed for testing from a browser’s perspective, offering features like auto-wait, network interception, and emulation capabilities for mobile browsers. Playwright integrates WebSocket technology, which is essential for real-time web communication among clients and servers. This enables tests to mimic and verify complex, interactive web applications efficiently. This capability is important for apps relying on instant data exchange, enhancing the framework’s utility in creating robust, reliable test scripts for a wide range of web applications.

TTC Global has also developed a Playwright-Typescript accelerator framework that is lightweight and fit for purpose, easily covering all automation requirements for any web application, including the Dynamics 365 automation workflow.

The framework is essentially a collection of useful utilities coupled with raw Playwright in a modular project structure that facilitates the design and implementation of reusable business logic abstractions in the test project. As such, it is easy to use for anyone familiar with Playwright, with almost no learning curve. Playwright itself, though, requires some automation engineering experience to work with, as well as solid Typescript knowledge.

All the extensive Playwright features are supported, along with additional methods useful in test automation. The framework is unlikely to require any extensions in a test project, unless there are unusual requirements.

The TTC Playwright-Typescript accelerator framework has adequate lightweight documentation in a readme file, and extensive Playwright documentation can be accessed online, along with the large and active Open-Source community.

TTC Playwright framework has extensive features and produces trustworthy test results. The only drawback to note is its ability to handle iFrames, which, unlike the Java framework, requires the test automation engineer to identify the relevant iFrame for each user interface component. The typical advantages and disadvantages of an Open-Source solution apply here, including superior extensibility, but also a higher requirement for technical skills to use the framework.

Selenium vs Playwright

Like the Java framework, the Typescript solution is based on Open-Source tools, with the same pros and cons as the Java framework. However, there are significant differences as well. Unlike the Java solution, the Typescript framework is based on Playwright, rather than Selenium. This gives it distinct advantages in test automation. The underlying Playwright framework uses its own browsers, with different APIs from branded browsers like Chrome or Firefox, but the same rendering engines. This allows automated tests based on Playwright to use Playwright-specific methods for setting up preconditions and driving robust user interface interactions without the need for additional custom code. It also enables advanced functionality not possible with Selenium, such as interfering with the network requests the browser makes during the test.

The TTC Playwright-Typescript framework is lightweight and low maintenance compared to the Java framework. It enhances raw Playwright with some additional useful methods and business logic abstractions. Despite Playwright’s reliance on custom browsers, these browsers use the same rendering engines as branded ones like Chromium or WebKit. This ensures the system under test behaves the same as it would for users, while offering powerful functionality for certain edge case scenarios.

One notable disadvantage of the Typescript framework compared to Java is its inferior ability to handle dynamic iFrames. Although a method is provided for iFrame selection, it is not fully integrated into the element locator strategy as it is in the Java framework, due to some architectural limitations of Playwright. However, it is still possible to adequately handle multiple iFrames. Other than this, Playwright does not have any significant drawbacks compared to the Selenium framework.

Overall, the Playwright framework has more features than Java and is easier to maintain, making it a more favorable choice unless the team has a strong preference for and expertise in Selenium. Both Open-Source frameworks, however, lack support for desktop application automation. This can be a significant drawback for projects requiring such capabilities. Nevertheless, we have recently demonstrated that desktop applications can be driven on Playwright using FlaUI.

Here is a short comparison of the features of Selenium VS Playwright:

Here is as summary of the pros and cons of each framework:

A side-by-side comparison of the pros and cons of each of our evaluation criteria on automating D365 using either frameworks:

Choosing the Right Open-Source Framework for Your Project

In conclusion, both frameworks are fit for purpose and can be used effectively to automate D365. The Selenium-Java framework has the best support for dynamic iFrame, and this framework is also the heaviest of the two. The Typescript-Playwright framework is the most flexible and lightweight with few downsides.

The choice of the automation framework is ultimately down to the project constraints, such as technical resource availability, as well as the size and time frame of the project. For a large, long-term project, however, we recommend a well-architected Playwright-Typescript-based solution due to its superior maintainability and scalability, which will result in a lower Total Cost of Ownership.           

For more detailed analysis and comparison, please reach out to us and we can share the comparison study in details.

--

Contributing Authors

Mei Reyes-Tsai, GM of Technology & Delivery at TTC Global

Pavel Marunin, Senior Consultant at TTC Global

Lan Nguyen, Principal Consultant at TTC Global

Brian Rivadulla, Consultant at TTC Global

Colvin Cooper, Consultant of TTC Global