Test environments as a product.

Our end-to-end tests run against real deployed environments — dozens of microservices, message queues, third-party sandboxes. That part is right: the point of an E2E suite is that nothing is faked. The part that had grown wrong was everything around the tests: setup and teardown had accreted, over years, into a sprawl of test controllers reaching directly into a couple of dozen databases owned by other teams. Every new test scenario added another direct-SQL shortcut, because the shortcut was always locally cheaper than the right thing.

The costs were the usual ones, just compounding quietly. Test data leaked between runs, so failures stopped meaning anything — a red test was as likely stale state as a real bug, and the suite's credibility eroded until a green badge told you almost nothing. Cleanup scripts knew the schemas of systems they didn't own, so unrelated migrations broke the test suite weekly. And nobody could answer the simplest operational question: what did this test run create, and did it get cleaned up?

The realization that reframed the work: this isn't a testing problem, it's an ownership problem. State creation and destruction across twenty systems is a product surface — and we were maintaining it as nobody's scripts.

• Real environments, real integrations. No mocking the world — the suite's whole value is that dev and stage are exercised as deployed, third parties included.
• Multiple consumers. The same setup/teardown operations were needed by the automated suite, by internal QA tooling, and by humans debugging — three callers, three auth stories, one truth.
• No central key to everything. Secrets and database access are granted per service; a test client that needs twenty connection strings is a security smell and an operational liability.
• Teardown must survive chaos. Test runners die mid-run, CI jobs get cancelled, networks flake. Cleanup that only happens when the test finishes politely is cleanup that doesn't happen.

An API, not a library

The core decision: a standalone test-environment service — a small REST API that is the only thing allowed to touch other systems' state for test purposes. Consumers say what they need ("a signed-up user in market X", "this order archived"); the service owns how — and every how is a service contract: the public product API where a real user journey exists, an internal admin API where one doesn't. No consumer holds a database connection string anymore; the schema knowledge that was scattered across the suite now sits behind one deployable with one owner, one audit trail, and one place to fix when a contract moves. A library could have deduplicated the code; only a service could deduplicate the access.

Run tokens: state gets an owner

Every test run starts by acquiring a run token, and every resource the service creates is stamped with it. That one convention turns three impossible questions into queries: what does this run own, what did it leave behind, and what is safe to destroy. Teardown stops being "each test remembers to undo itself" and becomes "destroy everything stamped with this token" — a set operation against an inventory, not a prayer.

Asynchronous teardown, with recovery

Teardown is a background job the service owes, not a phase the test performs. The runner can vanish at any point; the token's cleanup still runs, retries on transient failure, and — the part that took the most iterations — recovers: a teardown interrupted halfway (deploy, crash, timeout) resumes from its recorded state instead of leaving a half-deleted fixture that no later pass understands. Cleanup state is explicit and persisted, because "we'll just run the deletes again" is exactly how half-deleted state gets created in the first place. And when a cleanup genuinely fails, the fixture doesn't vanish into silence — it surfaces as a first-class orphan, a list someone can act on. A failure you can see beats a clean-looking lie.

Boring, deliberately

Minimal API, one small bookkeeping database of its own for run tokens and teardown state — and not a single connection string to anyone else's database: cleanup travels the same service contracts as setup. The interesting engineering is in the contract and the failure modes, not the stack — a test tool that needs babysitting has failed at its one job.

Where it went next

The API's second act was becoming an agent surface: its own OpenAPI document now generates an MCP toolset, so AI agents drive the exact operations the test suite uses — create a user, add credit, observe a ledger until a condition holds, tear a run down — through the same contracts, under the same run-token accounting. Observation endpoints grew wait-for semantics so a test (or an agent) awaits state convergence instead of sleeping, and the QA dashboard moved into the same process so humans, CI and agents share one surface. None of this was in the original design; all of it fell out of the decision to build an API instead of a script pile.

• Red means something again. With state scoped per run, a failing test points at the code, not at last Tuesday's leftovers.
• Schema changes stopped breaking the suite from a distance — the blast radius of a migration is now one service that owns the affected operation, found by its owner in one place.
• The access story became defensible: one audited service holds the sensitive reach into other systems, instead of every test runner holding a keyring.
• Humans got a tool for free. Because it's an API, QA tooling and one-off debugging use the same operations as CI — the "three callers" problem dissolved into one surface.

More notes in this series as the work ships — the skills platform and a rebuild of an accounting core are next. Questions or war stories of your own: email me.