Our highest-traffic, most business-critical endpoints were the last ones we wanted to touch. They were also the ones that needed type safety the most, and the ones where our existing contracts were most likely to be wrong, since we hadn't enabled response validation at all.

Strongly typed API contracts backed by Zod is something we had adopted relatively recently. For new endpoints, this was the easy path forward: create an endpoint, define the types, export the contract, import it into the frontend and you are set. However, pre-existing endpoints were not represented in the same way. Some containing years of partial data migrations, short-lived experiments, features long-replaced, and even some polluting underlying database internals all the way up the stack. A single "convert this to strict types, make no mistakes" prompt wasn't going to cut it, and neither was a multi-thousand-line PR no one wanted to review.

The Feedback Loop

In order to not disrupt production traffic, we created a NestJS middleware which would check against a defined contract. In our target strict mode, we would validate that the contract response properly parsed before returning the result from our API controller, returning an error code and logging if it did not. On the other hand in our initial default dry-run mode, we would continue returning the controller result as-is, but produce warning logs for mismatches:

• For missing fields: ts-rest response validation detected fields not in contract schema

{
	...,
  "path": "/api/v1/shifts/getUnverifiedShifts",
  "method": "GET",
  "level": "warn",
  "logContext": "TsRestResponseValidationInterceptor",
  "strippedFields": [
    "response[0].attendance",
  ],
  "status": "200",
  ...
}

• For validation failures: ts-rest response contract validation failed

{
	...,
  "path": "/api/vendor/dnrbot/exclusions",
  "method": "GET",
  "level": "warn",
  "logContext": "TsRestResponseValidationInterceptor",
  "errors": {
    "data": {
      "5": {
        "attributes": {
          "deleteNote": {
            "_errors": [
              "Expected string, received null"
            ]
          }
        }
      }
    }
  },
  "status": "200",
  ...
}

With our dry-run mode, we would be able to begin validating our best-approximation contracts for all endpoints in a safe manner in production. As logs came in, we would periodically poll them with a custom skill referencing how to triage the logs and where to look within our code. This allowed us to get alignment through self-healing from the shape of our actual production traffic without interruptions.

Once an endpoint stopped producing logs, we still needed to correlate that the contract was matching successfully and the absence of logs doesn't necessarily prove that without validating that the endpoint has received actual traffic. Utilizing our existing metrics for API hits, we would correlate endpoints which had served many requests over a time span and not logging any warnings. Then, we would promote the endpoint which enabled the strict validation.

Execution

Despite using the latest state-of-the-art models for coding, attempting to solve the problem by either spawning a single agent session for each endpoint or cramming all the endpoints into a single agent are unlikely to produce fruitful results. On the one hand, endpoints are likely synergistic with overlapping schemas for endpoints within a domain area and individual change sets could conflict, duplicate or produce different outputs. On the other hand, our final contract package tokenizes to approximately 450,000 tokens, which eats into context windows quite a bit!

To balance the two approaches, we grouped our endpoints into larger batches, pushing simple CRUD controllers into shared batches while splitting our largest controllers into manageable chunks. We started with a simple skill that contained information about our internal best practices, contract helpers and context for the migration, incorporating additional information and common issues as we went.

With our first wave breaking our 865 endpoints into ~40 separate tickets, we now needed to orchestrate the migration. This led us to build groundcrew, our internal tool for local agent orchestration, inspired by the OpenAI Symphony spec. More on that in a separate post.

The First Wave

After the initial migrations, warning logs began pouring in:

• Typescript interface defined as optional or T | undefined returning null
• Leaking Mongo internal fields __v, $doc and more on the edges
• Fields defined as Date in various date-like representations

Approximately ~300 endpoints were logging some sort of contract mismatch despite the fact that the endpoints were passing typechecking. After identifying the patterns in the errors, we iterated on our contract skill to incorporate the common failure modes. We ran triage against the logs, expanding .optional() fields to .nullish() or explicitly coercing null to undefined, adding mapper functions to remove any internal datastore pollution or enforcing Mongoose .lean() returns, and coercing, narrowing, backfilling and migrating Date fields to properly represent a Date object.

We triggered the triage in a number of ways periodically throughout the day(s), Devin Scheduled Sessions, Claude Code /loop, and custom scripts via systemd or cron. We began making a dent, but it wasn't fast enough. The feedback loop existed, but it painted a very narrow picture.

The progress through end of day Thursday, before looking deeper on Friday.

Looking Deeper

When looking at why we weren't making progress, the feedback that we were giving to the agents was a small slice and the skill surrounding it was focused on reviewing only the information it had directly.

For example, we have a workplace entity within our codebase. In order to colocate information, many endpoints that return a workplace identifier were also returning a full workplace object. The workplace entity being the oldest in our codebase contains many years of technical archeology. Workplace 1 might have had experiments X and feature alpha while workplace 2 had experiment Z and feature delta. Needless to say a high cardinality of fields which exist on some set of workplaces but not others.

The issue: getting an error on one specific endpoint path in a nested object property wasn't getting caught by the agents triaging as also needing to go check other usages of the same object. Our feedback loop said to fix this specific issue, not to understand, evaluate and remediate the root cause.

By updating our skill and giving examples of the re-use across the codebase, things turned around and began improving much more quickly!

The Long Tail

By expanding how we were interpreting our feedback loop, things quickly approached an asymptote. Despite this, there continued to be a trickle of endpoints with warnings.

As mentioned before, many of these were small pockets of data, long since forgotten about. A migration that missed accounts that were inactive at a time to convert from epoch seconds to ISO dates, internal workplaces where features were tested and iterated on before shipping a slightly different data pattern to users, and fields removed from their Mongoose models but never dropped from the datastore directly.

After nearly 800 pull requests over four weeks, all 865 endpoints now parse strictly against well-defined contracts, with a handful of known field gaps tracked and scheduled for cleanup.

The pattern generalizes: any migration where ground truth is ambiguous benefits from a dry-run mode that produces structured, agent-readable signal. The agents don't need to be right the first time, they need a way to be told when they're wrong.