Generating Playwright Tests with AI: What Works, What Breaks, and How to Review the Output

A pull request landed in our review queue in March with forty new Playwright specs in it. All green. The author — a good engineer, not a lazy one — had pointed an agent at a checkout flow the night before and let it run. The diff was clean, the naming was tidy, the locators were role-based. It looked like a week of careful work compressed into an evening.

Then we opened one of the tests. It clicked through the cart, reached the payment step, and asserted that the page heading read “Order Confirmed.” That was the whole check. It never looked at the total. Two of the forty specs were the same journey with slightly different wording. One logged in, did nothing meaningful, and asserted the URL had changed. The suite was green because the tests were easy, not because the app was correct.

Here is the payoff up front, because it is the whole point of this piece. You can now generate Playwright tests with AI in minutes, and the generation half of the problem is basically solved. The value — and the risk — has moved to the review: an AI-generated suite passes because the tests ran, not because the software is right, and almost nobody is talking about how to check that well. This post covers what AI test generation genuinely does well, where it breaks, how it stacks up against hand-written tests, and the exact review gate our team runs before any of it merges.

How do you generate Playwright tests with AI?

Let’s ground this, because the hype makes it hard to see. The short version: you install Playwright’s built-in agents, describe the flows you want in plain English, and the tooling drives a real browser to plan, write, and repair the tests for you.

Playwright Test Agents: planner, generator, healer

Microsoft shipped Playwright Test Agents in v1.56, and they’re a first-class part of the framework now — documented in the official docs. You run npx playwright init-agents –loop=[vscode|claude|codex|opencode], point it at an assistant, and you get three agents:

• a planner that explores your app and writes a Markdown test plan;
• a generator that turns that plan into executable Playwright test files, verifying selectors and assertions live as it goes;
• a healer that replays failing steps, inspects the current UI to find moved elements, patches the test, and reruns — or skips it if the feature is genuinely broken.

That pipeline is what modern Playwright automation looks like: the agents don’t guess from a static screenshot, they interact with a live browser session and make decisions from the real page state.

Why AI test generation works as well as it does

The reason this works is that Playwright’s agents read the page through its accessibility tree, not the raw DOM. They see Role: button, Name: “Checkout” instead of div.checkout-btn-v3. Recent releases lean further into that: v1.61 ships locator.normalize(), which rewrites a locator to follow best practice — test IDs and ARIA roles.

The tooling is genuinely good, and improving fast. So the framework isn’t the problem. The problem is what comes out the other end, and whether anyone checks it.

What works and what breaks?

AI test generation is strong at drafting and weak at judgment. It gets you a broad, tidy first pass fast, then quietly leaves the hard, correctness-defining decisions unmade. Both halves matter, so let’s take them in turn.

What AI test generation does well

We want to be fair here, because our team has seen the same tools save real time on real projects.

First drafts are close to free. Describe a flow in plain English and you get working TypeScript in a couple of minutes. For boilerplate — page objects, fixtures, the repetitive setup that everyone hates writing — this is a straight win. There’s no craft being lost when a machine writes your login fixture for the fortieth time.

Locators are better than what most humans write in a hurry. Because the agent works off the accessibility tree, its default reach is getByRole and getByLabel, not a hand-copied CSS path that dies the moment a designer renames a class. That’s the exact convention senior engineers spend months nagging teams into. The agent does it by default.

Selector refactors after a UI change are where the healer earns its place. A restyle used to mean an afternoon of locator archaeology. Now the healer re-finds the moved elements and patches them, and you review a small diff instead of rewriting twenty files. One practitioner field guide describes the same pattern with a hard constraint worth stealing: when you ask an agent to fix a failing test, tell it “don’t change the assertion” — otherwise it’ll happily weaken the check until the test goes green and proves nothing.

API tests from a spec. Point it at an OpenAPI file and it’ll scaffold a suite without touching a browser. Boring, mechanical, easy to verify — exactly the kind of work you want to hand off.

T.J. Maher, an SDET who put the planner and generator through their paces on his Adventures in Automation blog in early 2026, summed the good case up honestly: the planner “did well, getting me 80% there,” and produced “a detailed test plan, one you could review and edit.” Eighty percent, from a standing start, in minutes. That’s the pitch, and it’s real.

It’s the other twenty percent that ships bugs.

Where AI-generated tests break

The agent writes the draft. A human still has to decide what it proves.

The failures don’t look like failures. That’s what makes them dangerous. A flaky test screams at you. A bad AI-generated test sits quietly in the green.

Happy-path bias is the big one. Agents gravitate to the flow that’s easiest to reach — the one that’s most documented, most obvious, closest to what they’ve seen a thousand times. But the happy path is the flow least likely to be broken. The bugs that reach customers live in the expired session, the half-filled form, the payment provider that times out, the coupon that shouldn’t stack. An agent left to its own devices will test the thing that already works and skip the thing that doesn’t.

Weak assertions. Writing a good assertion means knowing what correct looks like, which means understanding the business rule the software is supposed to enforce. The agent doesn’t have that context. So you get our forty-spec PR: tests that execute flawlessly and validate nothing that matters. “Page loaded” is not a test. “Heading says Order Confirmed” is not a test if the order total is wrong. It’s the single most frequent issue reviewers flag in AI-generated test output — the model pattern-matches on the code in front of it and asserts on whatever the code happens to return.

It asserts the plausible, not the actual. Maher hit this cleanly: the generator decided a heading was findable with getByRole(‘heading’, { name: ‘Secure Area’ }) — a perfectly reasonable guess — when the real markup used an <h2> and he had to correct it by hand to page.locator(‘h2’). The agent wrote what should be true. Nobody checked whether it was.

Long flows drift. Practitioners report that AI-authored tests start losing state past roughly 15–20 steps, with the reliably-generated ones sitting around eight. Dynamic data is worse: the agent will cheerfully hardcode the email or order ID from the run it observed during authoring, so the test passes once and then rots. And OAuth/SSO is the single most-reported breakage — the agent walks into a third-party login wall, a captcha, or a 2FA prompt it can’t get through. The fix is to inject session state with storageState rather than let it click the login button, but the agent won’t know to do that unless you tell it.

Duplication. Because each generation run starts fresh, you end up with the same journey tested three slightly different ways across six files, and a test count that climbs faster than anyone’s understanding of what those tests actually cover. That’s not coverage. It’s a maintenance bill with a delay on it.

None of this means the agents are bad. It means they’re brilliant drafters and unreliable owners. When Xue and colleagues built a realistic web-agent benchmark instead of a cherry-picked one, they found “over-optimism in previously reported results” — the agents were markedly weaker on real sites than the headlines suggested. In one 2026 industry roundup, around 74% of practitioners said AI-driven testing still needs a human validating the output. Someone experienced has to stand between the draft and production.

AI-generated vs hand-written Playwright tests

The question we get most is whether AI-generated tests are simply better than the ones a person writes by hand. They’re not better or worse — they fail differently, and knowing how is what lets you use both well. A reviewed AI suite is typically faster to produce and broader than a hand-written one; a human, by contrast, usually has a behaviour in mind first and writes the assertion to pin it down.

Our verdict: this isn’t a contest one side wins. Use AI test generation for the volume work — first drafts, fixtures, API scaffolds, and post-UI-change selector repair, where it’s genuinely faster and broader than a person. Keep humans on the assertions and the edge cases, because that’s where “green” has to actually mean “correct,” and it’s exactly where the agents are weakest. The strongest suites we ship are AI-drafted and human-owned; neither approach alone is enough for production test automation.

How should you review AI-generated tests?

So here’s the part the tutorials skip. If you’re merging AI-written tests, you need a review gate, and it needs to be tighter than the code review you’d run on human-written code — because the failure mode is silent green, not red.

Nothing an agent generates should merge without passing through this.

The review checklist we hand teams

It’s short enough to actually use:

• Does the assertion prove behaviour? Not “the page loaded” — the right thing happened. Check the total, the state change, the record in the database, not just a heading.
• Is this a duplicate? Before you accept a new spec, ask whether an existing test already covers that path. Prune ruthlessly.
• Where are the edge cases? If every generated test is a happy path, the agent did the easy half. The valuable tests are the ones it didn’t write.
• Are the locators role- or label-based? Reject brittle CSS and nth-child. Run locator.normalize() if you want a second opinion.
• Is the data fresh per run? Hunt for hardcoded emails, IDs, and timestamps lifted from the authoring session.
• Is auth handled with storageState, not a live OAuth click-through? If the test logs in through a third party on every run, it will flake.

One more habit worth building in: the newer releases can produce agentic video receipts — an annotated walkthrough of what the agent did, which is genuinely faster to review than reading through a spec line by line. Use them. Reviewing the intent is often quicker than reverse-engineering the code.

Know when to stop generating

Know when to stop generating, too. The same field guides that praise the agents also warn that maintenance overhead starts to overtake authoring speed at around fifty tests. Past that point, conventions and architecture matter more than raw output — which is exactly when a lot of teams call us in for an automation review to set the guardrails before the suite calcifies. If you’d rather have someone drive that from the inside, an embedded automation engineer is often cheaper than unpicking a thousand generated tests later; our case studies show how that usually plays out.

The honest bottom line

Use the agents. For first drafts, boilerplate, selector repair, and API scaffolding they’re a real productivity win, and refusing them is just pride. But treat everything they produce as a draft under review, never a deliverable. The scarce skill in 2026 isn’t writing tests — it’s knowing which of the machine’s tests are lying to you, and having the gate in place to catch them before a customer does.

Green used to mean “it works.” Now it means “the test ran.” Getting green to mean something again is the job.

Key takeaways

• You can generate Playwright tests with AI in minutes — Playwright’s planner/generator/healer agents (v1.56+) make the generation half of test automation basically solved.
• AI test generation is a brilliant drafter and an unreliable owner. It nails first drafts, locators, fixtures, and selector repair, but ships happy-path bias, weak assertions, and quiet duplication.
• AI-generated vs hand-written isn’t a contest — AI wins on speed and breadth, humans win on correct assertions and edge cases. The best suites are AI-drafted and human-owned.
• The review gate matters more than the generation. Run a checklist tighter than a normal code review: prove behaviour, kill duplicates, demand edge cases, check locators, fresh data, and storageState auth.
• Know when to stop. Maintenance overtakes authoring speed around ~50 tests; past that, architecture and conventions beat raw output.

If your suite has quietly filled up with AI-generated tests and you’re no longer sure what they prove, that’s the moment to get a second set of eyes on it — an hour of review with the Testvox automation team usually tells you whether you’ve got a quick prune or a rebuild on your hands.

Frequently asked questions about generating Playwright tests with AI

How do you generate Playwright tests with AI?

Install Playwright’s Test Agents (npx playwright init-agents –loop=[vscode|claude|codex|opencode]), then let the planner explore your app and write a Markdown plan, the generator turn that plan into executable spec files, and the healer repair failing tests after UI changes. You describe flows in plain English; the agents drive a real browser through its accessibility tree and produce committable TypeScript.

Can I trust a Playwright suite the agents wrote on their own?

Trust the parts you’ve reviewed; treat the rest as unverified. The agents produce plausible, well-structured tests — but plausible isn’t correct. Until a human has checked that the assertions prove real behaviour, a green run only tells you the tests executed.

Are AI-generated Playwright tests better than hand-written ones?

They’re faster to author and broader in coverage, but weaker on assertions and edge cases. Hand-written tests start from a known behaviour, so their checks tend to mean more. In practice you want both: AI for the volume, humans for the correctness. Neither alone is enough for production.

What’s the single biggest mistake teams make with AI-generated tests?

Skipping the planner review. Everyone wants to jump straight to generated code, but a sloppy plan produces brittle tests at scale. Spend ten minutes editing the Markdown plan — resolving ambiguity like “click the submit button” when there are three — and the generated output is dramatically better.

Do the agents write flaky tests?

They can, and it’s usually self-inflicted: hardcoded data from the authoring run, live OAuth logins, or long flows that lose state past fifteen-odd steps. Injecting auth via storageState, generating fresh data per run, and keeping tests short fixes most of it. The healer helps with moved locators, but it won’t fix a test that was asserting the wrong thing to begin with.

Do you still need automation engineers if AI writes the tests?

Yes — arguably more than before. When anyone can generate a hundred tests before lunch, the value moves from typing them out to deciding which hundred to keep. As Maher put it after his own trials, the tools “cannot replace SDETs” — they save typing, not judgment.

Is AI test generation production-ready in 2026?

The generation tooling is production-ready; the output isn’t, until it’s reviewed. Playwright’s agents are a first-class, stable part of the framework, and teams ship real suites built with them. But roughly 74% of practitioners still put a human validation gate between the draft and the merge — treat the agents as trusted drafters, not trusted authors.

RELATED ARTICLES

Playwright Test Agents Explained: How Planner, Generator & Healer Kill Flaky Tests

A spec our team had written eight months earlier lit up a Slack alert...

17 July 2026

Playwright features every QA team needs to use now

Discover the Top Features of Playwright Every QA Team Should Know to enhance automation,...

25 April 2026

Playwright is the future of test automation in 2026

Discover why Playwright is the future of test automation. Learn how it overcomes Selenium's...

7 May 2026

Harshit Gupta

AI Test Architect & Lead SDET with 11+ years of expertise in Playwright, API testing, automation, and AI/LLM evaluation. He focuses on delivering high-quality, user-centric software through AI-assisted testing, efficient testing strategies, and continuous innovation. Connect with him on