In Part 2 of TTC Global's AI-First Test Automation Series, we compare AI model performance across the same Workday HR test pool and Playwright Accelerator framework. The experiment tracks how much manual rework each model left behind, how much guidance it needed on complex scenarios, and where lower-cost models stopped delivering meaningful efficiency.
The results point to a clear practical issue for teams adopting AI in test automation: saving money on AI usage does little good if engineers spend that saving correcting incomplete or poorly structured output. In TTC Global's experiment, manual rework ranged from roughly 70 to 80% for the free low-tier model to around 10% with the strongest model tested.
About the Report
For QA leaders and test automation teams integrating AI into their workflows, choosing the right AI model for the task needs to be done carefully. Token price, while a major consideration, is only one part of the cost. Engineering effort, output quality, maintainability and the model's ability to work within an established automation framework all affect whether AI creates a meaningful efficiency gain.
To see how much model difference matters in practice, our engineers worked across four phases of an AI-First test automation experiment. We used the same pool of Workday HR test cases and our Playwright Accelerator framework to see how model capability affected usable output, manual rework and the level of guidance needed for more complex test scenarios.
The findings are directional, and the framework itself matured as the experiment progressed. That context is important for teams considering how the results apply to their own test automation environment.
Ultimately, the experiment supports a practical approach: use premium models where test complexity and engineering effort justify them, and match the model to the task rather than treating every AI-assisted testing activity the same way.
Frequently Asked Questions
In TTC Global's experiment, Claude Opus 4.8 produced the strongest results of the models tested, with approximately 10% manual rework and the best performance on accuracy and token discipline. The experiment does not prove that one model will be best for every test automation environment. Framework design, test complexity, input quality and AI configuration also influence the result.
They did in this experiment. Premium models required substantially less manual correction than Auto and the free low-tier model, particularly on complex enterprise workflows. From Opus 4.6 upward, the tested models were considered fit for purpose for the workflow evaluated in the report.
They may be able to assist with some simpler work, but TTC Global's free-model test produced too much rework to support the AI-First workflow evaluated. GPT-5.4 mini required approximately 70 to 80% manual rework on complex scenarios and could not complete the hardest flow unaided.
GitHub Copilot was used during the first three phases of the experiment with several underlying model options. Results varied considerably depending on the model selected. We suggest evaluating Copilot in combination with the model and workflow being used rather than judging the tool independently. Part 3 of the series goes further into the comparison between GitHub Copilot and Claude Code.
Yes. The final phase of TTC Global's experiment used Claude Code with Claude Opus 4.8 against the same Playwright Accelerator framework and Workday HR test pool. That phase produced the lowest manual rework of the models tested.
During the experiment, common issues included departing from documented test steps, failing to reuse existing framework methods, ignoring established utilities, hardcoding data, under-parameterising similar flows and hallucinating steps when requirements or test data were incomplete. The findings reinforce the importance of well-specified inputs, reusable components, configuration standards, guardrails and human review.