When Vibe Testing Won't Scale: Your Questions about AI-First Playwright Automation Answered

Getting Started & Discovery

Do you always need a detailed test case to start with? Why not let AI come up with those as well, based on the user story?

While you do not need a detailed test case to start, it's recommended. Test-case generation is deliberately decoupled from automation, so any AI agent or toolkit can feed it. Our AI-First Functional Testing toolkit, for example, does exactly that. Without spec steps, we skip the traceability review and let the explore agent discover the workflow from the application itself.

AI can, of course, come up with the test cases based on user stories, but test-case generation is a workflow with its own complexity. We keep it separate rather than bolting it onto automation. Where steps are missing or vague, the explore agent fills the gaps by discovering the actual behaviour in the application. And TTC Global does have that test case generation capability, it just lives in a different toolkit with its own workflow and rules.

Are you using Playwright MCP or Playwright CLI for the initial discovery, and why?

We use Playwright-cli for the automated explore phase. It works across every AI host, is session-scoped by role, and shares the test config, so discovery conditions match runtime. MCP is used where it fits better: Playwright MCP for interactive debugging against an already-running Chrome, Appium MCP for mobile inspection, and so on.

Evidence & Traceability

Is there an audit trail available showing what was actually tested, like a transcript or summary of testing?

Yes. On the execution side, we capture API traces, execution logs, JUnit XML, Allure results and failure screenshots, all published through the ADO pipeline into Test Plans and Test Runs. On the AI authoring side, each automation run produces a metadata file and an audit trail recording the implementation plan and the decisions behind it.

How does it deal with traceability? How could an agent integrate the generated tests back into Azure DevOps or similar, with traceability to user stories and test cases? Is this traced automatically, or does it need human review?

Traceability back into Azure DevOps isn't done by AI at all. We use a library that handles the integration deterministically. The results, test cases and runs are linked by the work item ID as a mechanical operation, so there's nothing to review and nothing to get wrong. The AI does the judgement-heavy work: understanding the requirement, exploring the application, writing the test. Pushing a result to a work item isn't judgement work, so we don't use a probabilistic system to do it.

Quality & Governance

Do we have an evaluation mechanism, preferably automated, for accuracy, relevancy, and other relevant metrics for the quality of results and output?

We have hard gates at every phase. The output can't ship unless it compiles, lints, passes the test, and traces back to the spec. The metrics come from artifacts we already produce: traceability coverage, pass rate, and so on. What we don't yet have is scored evaluation.

How does your agent framework handle self-correction? Specifically, what evaluations detect mistakes, how does the agent know it got something wrong, and can it decide on its own when to escalate to a human?

What we have is hard gates and rule-based escalation. They are deterministic and predictable. The escalation triggers on defined conditions, not on self-assessed confidence. The gates are deterministic rather than model-judged.

Model Choice & Cost

Why does your team use Claude? Have you tried different AI agents or models?

The framework is tool- and model-agnostic by design. We picked Claude based on measured results, and we can re-run that measurement whenever the market moves (we've done it four times already). It can work with any coding agent in the market: Codex, Gemini Code Assist, Cursor, and others.

Do we have a measurement of how much token cost we spend per change or requirement, and could this be embedded in our normal process?

We've measured it in a controlled study: same test pool, two tools, real bills. So we have cost-per-test figures. What we don't yet have is per-requirement cost tracking in business-as-usual.

Regulated Industries

How does agentic testing fit in a regulated industry such as pharmaceutical or medical devices?

In a regulated context, the agent is an authoring accelerator, not the last line of defence. What you validate is what you've always validated: the test script, the framework, and the evidence. What changes is that you get more coverage and better evidence for the same effort, and you can add whatever gates the regulation requires, because the pipeline is already built from gates. That's the advantage of a mature framework with deterministic, consistent output over vibe-coding your test automation.

Where to Go From Here

These questions reveal the real, practical concerns teams have once they move past the demo stage of AI and start thinking about what agentic automation looks like inside their own environment, with their own constraints.

If you want to go deeper, our ongoing research through the TTC Global Test Lab explores similar queries in more detail, lookinf into cost, model comparisons, failure modes, and what actually holds up at scale. Subscribe here to get that research as it's published, and keep an eye out for future webinars like this one.

And if you missed the live session, the full recording is available here.