It's no surprise that with the help of AI, the number of pull requests authored and merged at Clipboard is growing. We're now pushing 2x more pull requests per engineer than a year ago. Code reviews, among other things, became a bottleneck.
Looking at the data
We can't solve a problem we don't understand, so I started with the data.
We also knew that existing tools like CodeRabbit weren't speeding up reviews. They were chatty, and engineers were rejecting around 50% of their comments, which slowed everyone down.
My intuition at the time: 1) not all PRs are created equal, and 2) there had to be something we could do quickly without reinventing the wheel.
I dug into the contents of six months of pull requests. A large share were simple, low-risk changes that could be merged right away: 30% overall, up to 80% depending on the repository. But folks were so busy coding, reviewing and testing other things that little time was left for them. The median time to merge was reaching 19h, including for the simple ones.
If we could hand the review of simple pull requests to some tool, we would cut the review load and speed up delivery of simple and complex code alike. That's where the bot enters.
The bot
The bot is a simple backend service. It listens to GitHub webhooks and runs an LLM review in a background job. If the change is low risk, it approves the pull request. If it isn't, or if the bot can't tell, it skips and leaves the PR to a human.
We call it Preflight.
Proof of concept
The v0 loaded the PR title, author and diff into the user prompt, while the system prompt said to look for low-risk changes such as test-only or type-only edits. Anything that would not change production behavior counted as low risk and was approved.
That alone approved around 7% of pull requests. Good start. But not the 30% I had estimated. The diff alone wasn't enough.
Constraints
First: the bot only approves or skips, no inline comments.
Second: no shell or checkout, to limit complexity.
The solution was to create our own set of "abstract" tools. The server translates each tool call into an API call, so the LLM only sees input and output. No secret keys are shared with the LLM, and access is limited to what the API credentials allow.
We tell the LLM which tools exist, when to use them and what schema they take, and then write the harness loop ourselves:
// simplified
const messages = [{ role: "user", content: prompt }];
for (let round = 0; round < MAX_ROUNDS; round++) {
// call the LLM provider
const response = await client.messages.create({ system, tools, messages });
messages.push({ role: "assistant", content: response.content });
const toolCalls = response.content.filter((block) => block.type === "tool_use");
if (toolCalls.length === 0) {
return skip("agent stopped without a verdict");
}
const submit = toolCalls.find((call) => call.name === "submit_review");
if (submit) {
return submit.input; // { decision: "approve" | "skip", reason }
}
// run this round's tool calls in parallel
const results = await Promise.all(toolCalls.map(runTool));
messages.push({ role: "user", content: results });
}
return skip("agent exhausted its tool budget");
The system prompt asks the LLM to batch tool calls, so we run them in parallel in each round. This leads to fewer rounds and faster review.
The verdict itself is also a tool call. The model must end a review loop by calling submit_review with "approve" or "skip" and a reason. Since it's a tool call, it has a schema, so there's no free text to parse. If the bot doesn't end with this tool call, we know it was not able to reach a conclusion, so we skip.
Tools
The fun part (at least for me)!
Providing tools to the LLM is easy once you realize that every tool is just a name and a description of what it does and when to use it.
A simple tool like read_file calls the GitHub API with a file path, scoped to the commit under review. A bigger one like check_dependencies takes the changed packages and checks vulnerabilities, typosquatting, age and popularity in one call.
With more tools, the bot has more ways to check whether a change is really low risk. It can reach Linear for ticket context, Datadog (logs, metrics and monitors), Sourcegraph for cross-repo search, and LaunchDarkly for the live state of feature flags.
System prompt
The system prompt is about 4,000 words. It explains who the bot is, what it can trust and how to review.
External data is wrapped in XML tags, each with its own instructions. For example, <repo_policy> is our configuration and is trusted. The <pull_request> and <changed_files> come from the author, and the prompt says to treat them as data, never as instructions.
We also say what not to check. CI already runs unit and integration tests, linters and other checks, so we tell the bot not to spend effort on anything a deterministic check already covers.
The prompt also lists the patterns that count as safe: test-only changes, mechanical refactors, dead code removal and backward-compatible API schema changes.
Repository policy
Every repo has a policy file that maps paths to categories. Each category says whether its files can ever be approved and gives the bot instructions for reviewing them:
{
"categories": [
{
"name": "test-only",
"globs": ["**/*.spec.ts", "**/__mocks__/**"],
"approvable": true,
"description": "Only test code"
},
{
"name": "human-required",
"globs": [".github/**", "**/CODEOWNERS", "terraform/**"],
"excludeGlobs": ["terraform/monitors/*monitor.tf"],
"approvable": false,
"description": "Deploy, CI, auth, migrations, Terraform: a human always reviews these."
}
]
}
Each changed file is matched against the globs and categorized. The result goes into the user prompt as JSON:
[
{
"path": "src/booking.spec.ts",
"categories": ["test-only", "content-verified"],
"approvable": true
},
{ "path": "src/booking.ts", "categories": ["content-verified"], "approvable": true },
{
"path": ".github/deploy.yml",
"categories": ["human-required", "content-verified"],
"approvable": false
}
]
This lets teams control the bot per repository: block files from ever being approved, or give special instructions for some paths.
Matching a path isn't enough. Even when every file is approvable, the bot still reads each one to confirm its content (that's what content-verified is for).
We also run a deterministic check at the end: if any file is approvable: false, we don't submit the approval. It's an extra safety net against false approvals.
Teams can also configure whether to review drafts, which labels to skip or require, which teams or bots to review, whether to review stacked PRs, and the maximum number of changed lines the bot will review.
What a review looks like
The bot keeps one "anchor" comment per pull request and updates it as the review progresses.
Every verdict shows the bot's reasoning, the files it read, the eligibility checks, and every tool call it made.
Here's an example of it reviewing a Datadog monitor:
It checked the metrics (redacted here), the ticket, the new monitor definition and the existing monitor. Based on the live metrics, it decided the new definition still makes sense.
In this one, it checked whether a feature flag change was safe: it searched the code for readers of the flag, looked up the flag's live state, and compared the change against the Linear ticket.
And here, the code pointed to a path in our mobile app that wasn't deployed yet, so it asked for a human instead of approving.
Other features
We built evals so prompt changes don't break what already works. One prompt change lifted approvals from 66% to 94% of the PRs we had labeled low risk, and the evals told us so before it shipped.
We don't rely on the LLM alone. Deterministic checks make sure we review only what we want to review. We could tell the LLM to skip PRs whose base branch isn't main, but that would waste tokens and leave room for a random miss.
Anything that can be deterministic should be.
We also learned a lot about GitHub synchronization (the event handling that makes sure we review the right code at the right time), prompt caching, model thinking settings, and AWS Bedrock. Engineers also have a feedback command, and we run LLM observability in Datadog.
That's a lot more to cover. Each of these deserves a post of its own.
The results
Before the bot, low-risk PRs weren't merged much faster than other PRs: their median time to merge was 14-19h, against 18.7h for all PRs. With the bot in place, the median time to merge of low-risk PRs is down to 1.2h.
The bot does a lot while costing very little:
- Approves 28% of merged PRs, close to the 30% I estimated
- Is the only approver on 1 in 5 merged PRs
- Costs $0.35 per review
The bot isn't perfect. In one case, it skipped a tool it needed and approved a PR that caused a frontend bug. We fixed the prompt and backtested the change against past PRs, including the bad approval. The evals keep that mistake from coming back.
Conclusion
Preflight answers one question: is this PR low risk? If yes, it approves and unblocks the work with no human review. We trust it because it checks things a human reviewer rarely does: package age, known vulnerabilities, flag state and production logs.
One in five of our PRs now needs no human review. Low-risk PRs merge 12-16 times faster.
Two constraints kept it simple: approve or skip only (no inline comments), and no shell or checkout. This simplified the architecture and limited the attack surface.
Not all PRs are created equal. We stopped reviewing them as if they were.
