agentic ai · qe leadership
Building a QA Agent Pipeline: Early Lessons from a POC
What it looks like to route a requirement through a requirements-reviewer, test-planner, executor, bug-analyser, and reporter agent, and where the pipeline still breaks on real projects.
For most of my QA career, the job has looked the same no matter where I've worked: someone writes a requirement, someone builds it, and QA has to work out what "done" actually means, usually by chasing that requirement across four or five different tools.
That's the part I wanted to fix first.
The problem we were solving
A requirement rarely lives in one place. Some of it sits in a Google Doc. Some of it was clarified in a Slack thread weeks ago. Some of it only exists as a Figma prototype nobody annotated. And often there's a one-line spec bolted onto a Linear ticket as an afterthought.
That's not a knock on how teams work. It's just what happens when things move fast. But it means QA's first job on a ticket isn't testing. It's tracking down what was actually asked for. That step alone eats real time, and it doesn't scale well once you're also trying to bring AI-assisted testing into the process.
So this POC wasn't really about getting AI to write test cases faster. It was about fixing the step before that: turning requirement-gathering into something automatic, instead of something QA does by hand on every ticket.
How it's set up
Everything is triggered by how a Linear ticket gets labeled. No new tool for engineers or PMs to learn, just the labels they already use.
When a ticket is labeled ready for QA, a hook picks it up and hands it to a requirements-reviewer agent. Its job is to go collect everything relevant (the linked Google Doc, the Slack discussion, the Figma prototype, the spec on the ticket) and pull it into one requirement.
That goes to a test-planner agent, which writes the test plan and test cases against it.
When the ticket gets relabeled ready for QA execution, an executor agent runs that plan across our lower environments. A separate bug-analyser agent runs alongside it, watching for anything that doesn't match the requirement (UI issues, unexpected behaviour) instead of waiting until the whole run finishes.
Once execution hits 100%, a status-reporter agent takes the results, maps any bugs back to the specific part of the requirement they violate, comments the summary on the ticket, tags the owner, and posts an update to the feature's Slack channel.
By the time a person looks at the ticket again, there's already a test plan, an execution result, and a mapped bug list waiting for them.
Where it actually got hard
The setup above sounds tidy. It wasn't, in practice, and that's the part worth writing down.
Requirements are often just incomplete, not scattered. Early on, the requirements-reviewer would stitch together a confident-sounding requirement out of pieces that were missing a decision entirely, because no one had made that decision yet. We had to build in a way for the agent to flag "this part is still unclear" instead of quietly filling the gap itself.
Running the bug-analyser in parallel with execution created noise before it created value. It's the right call for speed, but early on it flagged a lot of environment flakiness as if it were a real product bug. It took a few rounds of tuning before the difference between "the app is wrong" and "the environment hiccuped" was reliable enough for anyone to trust the output.
Mapping a bug back to the right line of the requirement is harder than it sounds. Saying "this test failed" isn't useful on its own. Saying "this failed because of what's in paragraph three, not paragraph one" is what actually makes the report worth reading. That mapping took more iteration than the test execution logic did.
The team didn't lean on the automated report right away, and that was fair. Early reports had real gaps. It only became something people acted on directly once it had built up a track record of being right.
Where it stands
This is a POC, not a finished framework, and I don't want to oversell it. But the shift is already real: QA's first move on a ticket is no longer "let me go find out what this means." That work increasingly happens before anyone's looking at the ticket.
The next stretch of work is less about adding agents and more about tightening what's already there: getting the bug-analyser more precise about what counts as a real defect, and getting the requirements-reviewer better at flagging gaps early instead of downstream.
More once there's more to report.