Building a QA engineer
Building a QA engineer
Software development is changing. The hype is deafening (I caught my grandma retweeting about vibe coding yesterday), but the changes are profound.
We’ve been busy building at Heal.dev, focusing on a mostly unloved part of software: quality assurance. We’re building a QA engineer.
This is one of the most needed things for code AI to be successful. Sure, we can ship faster with AI, and sometimes, well-written code. But it’s worthless if the end product doesn’t do what it’s supposed to do.
That’s true, by the way, whatever tooling you choose to use. QA is painful, codegen or not.
“I don’t have time for code review” is something I hear quite often from CTOs. Can we live without code review? Maybe? If the rate of code change is high, and code is commoditized, and most importantly, if It Works™, you can maybe skip code review altogether.
If the end product works.
“I have no time for QA, and I need it badly” is something I hear equally often. And that is going to get worse and worse as coding agents improve and spread. Coding agents make mistakes and are going to continue making mistakes for a long time. And when they don’t, humans will: how many bugs caught by QA teams are due to config and mishaps, not code?
Writing code faster is great. But if the software doesn’t work, is it worth shipping faster?
That’s why we’re building a QA engineer.
Why
Codegen is awesome. Yet, the more they’re left alone, the more coding agents introduce bugs, especially for stateful/edge case-heavy tasks. 46% of developers report an increase in downstream work like QA.
Of course, human QAs can’t keep up. In the words of a CTO I spoke to, “If I ship 5x more features because coding is so fast, I’m not gonna hire 5x more QAs. But I don’t want quality to go down”. Similarly, I heard from several Lovable users that they were loving the experience, but struggling to keep the generated app working as designed when iterating on it.
The timeline depends on who you ask. Anthropic’s CEO predicts AI will write 90% of code in a few months. Zuckerberg that “in the next 5 to 10 years, a significant amount of coding will be done by AIs”.
What’s certain is that app and code generation need a partner to build products that work for the end user. In the old world, that partner was a human QA engineer.
The trick is that a human QA engineer can do only a few things at a time. Automating scenarios takes time, so let’s stick to the happy path. Investigating hundreds of test failures takes time, and it’s so easy just to disable them. And now AI is spitting out PRs, and there’s an order of magnitude more tests to review.
We need an AI QA engineer.
What
Before we build it, let’s understand what it does. “Write tests” is not enough. What does that mean?
A QA engineer understands how features should work based on their knowledge of the product, documentation, and information from those who defined it (engineers, product managers, customers).
A QA engineer figures out how to build a test plan to ensure that the features work as they should.
A QA engineer automates that test plan, taking into consideration edge cases and potholes, like the app's statefulness. She knows what can’t be tested by automation, can flag it, and works to make it testable.
A QA engineer runs tests, investigates them when they fail, figures out why, and helps human and agent developers reproduce and fix them.
In other words, a QA engineer needs to plan, automate, run, and investigate.
Tools
That role wouldn’t be possible without tooling. And connecting to tooling is critical to succeed here.
To plan, read specs in Google Docs tickets, issues, and Figma to understand how the product should work. Ask questions to PMs in Slack to clarify details.
To automate, use playwright (or some other framework). Understand when open bug tickets block test automation, and wait until they’re closed. Open tickets when she finds bugs on the way to automating.
To run, set up a clean test environment, and run tests in CI
And to investigate failed tests, she can open up logs, APM and tracing, previous test runs, and previous bugs that look the same.
Heal
What we’re building at Heal is a system that can do all of that. We’re not quite done yet, but our agent can understand a test case, implement the test, investigate why it fails, and write a bug report with the root cause. Soon it’ll be able to do much more. It’s not perfect. And we’ve taken a pragmatic approach: while we’re perfecting the system, we have our own QA engineers in the loop who help refine test code and stamp bug reports. But every week that passes, we make our agent better, until it can talk to Lovable and tell it what should be fixed and why.
We’ll keep writing about how we’re building Heal soon. Meanwhile, if you need a QA engineer, hit me/Heal up!