How to Build Sandbox-Verified AI Code Reviews with OpenRouter

Why Diff-Only AI Code Reviews Fail in Production

Most development teams evaluating AI code reviewers test tools that only read raw Git diffs. This approach has a fundamental flaw: the model has to guess what the rest of the file looks like, whether referenced helpers exist, and whether a suspected bug actually triggers a runtime failure.

When an AI reviewer cannot execute code, two problems occur: 1. High false-positive rates: The model flags speculative edge cases that working runtime environments handle cleanly. 2. Manual overhead: Fixes return as plain text snippets inside Markdown code fences. Developers must manually interpret, copy, paste, and test these snippets.

Real engineering efficiency does not come from asking a large language model to guess harder. It requires giving the model an isolated execution environment where it can reproduce bugs, verify failures with shell commands, and generate validated patches.

OpenRouter demonstrated this shift with a 233-line Python script (review.py). By relying solely on the Python standard library and OpenRouter's Responses API, the script uploads modified code into a hosted Linux container, executes tests, proves bugs, and exports patches directly into your local Git tree.

---

Inside the Sandbox Architecture: Responses API and Server Tools

The OpenRouter implementation departs from typical Chat Completions endpoints. Instead, it uses the Responses API (/api/v1/responses) paired with three server tools and one function tool to establish an automated reproduction loop.

+--------------------------------------------------------------------------+
|                            Local Environment                             |
|  Git Diff + Changed Files + Context Files                                |
+------------------------------------+-------------------------------------+
                                     |
                                     v
+--------------------------------------------------------------------------+
|                     OpenRouter Global Infrastructure                     |
|                                                                          |
|   1. Files API (Beta)                                                    |
|      - Stores up to 19 context/source files + git diff                   |
|                                                                          |
|   2. Shell Tool (openrouter:shell, Beta)                                 |
|      - Hosted Linux Sandbox (Python 3.11.14, SQLite 3.50.4)              |
|      - Model executes test checks & captures stdout/stderr               |
|                                                                          |
|   3. Apply Patch Tool (openrouter:apply_patch, Beta)                     |
|      - Generates V4A diffs; validates syntax automatically               |
|                                                                          |
|   4. submit_review Function Tool                                         |
|      - Returns strict JSON report (.git/ai-review.json)                  |
+------------------------------------+-------------------------------------+
                                     |
                                     v
+--------------------------------------------------------------------------+
|                         Local Working Tree / CI                          |
|  - Confirmed bug evidence logged                                         |
|  - Optional --apply applies verified patch                               |
|  - Exit code 0 (Pass), 1 (Block), or 2 (Error)                           |
+--------------------------------------------------------------------------+

The Component Stack

• Files API (Beta): Uploads the entire content of every changed file, test files, and the Git diff. Supplying full files prevents the model from speculating about imports, global variables, or class interfaces declared outside diff hunks. Up to 19 files plus the diff can be uploaded, fitting within the 20-file request limit.
• Shell Server Tool (openrouter:shell): Runs inside a hosted Linux container provisioned with Python 3.11.14 and SQLite 3.50.4. Before running checks, the environment copies uploaded files to the user's home directory. The AI runs shell commands, writes reproduction scripts, and logs raw output to prove whether an issue triggers a real exception.
• Apply Patch Server Tool (openrouter:apply_patch): Allows models to emit structured V4A diffs (create_file, update_file, delete_file). OpenRouter automatically verifies diff syntax. If the model generates an invalid patch, the API returns the syntax error to the model so it can self-correct.
• submit_review Function Tool: Emits a final, structured JSON payload. Using a dedicated function tool prevents issues where models like Claude Haiku 5.5 default to plain text or Markdown fences when JSON schema parameters compete with external tools.

Benchmark Performance and Economics

Running sandboxed reviews adds compute steps, but the operational cost remains low. In tests across a five-bug code change, reviews took between 1 and 4 minutes:

Model	Average Cost per Review	Runtime	Bug Fix Rate (5-Bug Benchmark)
Anthropic Claude Haiku 5.5	$0.0153	~96 seconds	Identified bugs, verified 4 in runtime
Anthropic Claude Sonnet 5.5	$0.0447	~60 seconds	Fixed all 5 bugs across 3 separate runs

Every finding in the generated report is classified as either: * confirmed: Accompanied by the exact shell command and terminal output that caused the code to break. * unconfirmed: Retained for issues that cannot be replicated with a lightweight script (such as database connection cleanups managed implicitly by CPython).

---

Four-Step Implementation Checklist for Engineering Teams

At Starbox AI, we recommend teams move away from manual prompt interfaces and integrate verified execution pipelines into their standard developer toolchains. Here is the four-step checklist to deploy this workflow:

Step 1: Set Up Credentials and Local Review Script

The review script requires Python 3.11 or newer, Git, and an OpenRouter API key.

Because the beta Files API, Shell tool, and Apply Patch tool require global infrastructure, requests must point to https://openrouter.ai/api/v1. Regional endpoints will return a 403 error.

export OPENROUTER_API_KEY="your-api-key"

Save review.py in your repository root. The script uses Python's standard urllib and json libraries, removing the need to manage third-party virtual environments or heavy framework dependencies.

Step 2: Validate Bug Reproduction on a Staging Branch

Before applying the reviewer to production pull requests, verify its behavior against a known edge case:

1. Create a branch containing passing unit tests that miss a critical regression.
2. Run the reviewer locally while passing the test suite as context: bash python3 review.py --context tests/test_orders.py
3. Inspect .git/ai-review.json. The output outlines each finding, its severity level (LOW, MEDIUM, HIGH), and the execution evidence.
4. Apply proposed fixes automatically: bash python3 review.py --context tests/test_orders.py --apply The --apply flag writes the model's V4A diffs directly to your working tree, allowing you to re-run local tests before staging changes.

Step 3: Enforce Local Guardrails via Git Pre-Push Hooks

Prevent breaking bugs from reaching remote repositories by catching them before push events.

Add a .git/hooks/pre-push script:

#!/usr/bin/env bash
set -e

# Run AI review against changes compared to origin/main
python3 review.py --block-at high --context tests/test_orders.py

Make the hook executable (chmod +x .git/hooks/pre-push). If a change contains a confirmed HIGH severity bug, review.py exits with status 1. Git will abort the push operation:

error: failed to push some refs to 'github.com:org/repo.git'

Developers can address the findings or review the patch without leaving their terminal.

Step 4: Gating Pull Requests in CI/CD Workflows

Local hooks can be bypassed using git push --no-verify. To ensure organizational compliance, run the verification script inside your central build pipeline, such as GitHub Actions.

Key configurations for .github/workflows/ai-review.yml: * Script immutability: Pull the review script from the target branch using git show "$BASE:review.py". This prevents a pull request author from altering the review rules to pass checks. * Isolated execution: Execute via python -I to isolate the script from the repository's internal import paths. * Execution boundaries: Set timeout-minutes: 15 to accommodate multi-turn model interactions while preventing hanging runs. * Exit handling: The script exits 0 when code passes or falls below the --block-at threshold, 1 on blocking bugs, and 2 if turns exceed eight iterations or file limits (>19 files) are breached.

Configure the CI job as a required status check in your branch protection rules.

---

Defensive Engineering: Handling Sandbox Edge Cases

Allowing an external model to run code and submit file modifications requires strict safeguards. The OpenRouter implementation resolves several edge cases that teams frequently encounter when deploying agentic workflows:

Common Technical Failure               Built-in Defensive Pattern
------------------------               --------------------------
API drops during long runs      --->   Stream requests ("stream": True) with 5s retry
HTTP 403 on Files API           --->   Set custom User-Agent header (bypass default urllib)
Arbitrary path writes           --->   inside() resolves paths, rejecting escapes
Repeated identical patches      --->   Apply patch to scratch copy; return status immediately
Hunk mismatch / duplicate lines --->   Match @@ lines sequentially; reject ambiguities
Accidental token leaks in git   --->   Write review reports strictly inside .git/

Preventing Directory Traversal

Because patch paths originate from the model, a malicious or confused model could attempt to overwrite root files or external system files. The script uses an inside() function to resolve every target path against the project repository root. Any path resolving outside that boundary is rejected immediately.

Sequential Hunk Application

When code snippets appear multiple times within the same file, unified diff parsers can misapply hunks. The engine matches each hunk by its @@ line, evaluating lines sequentially so earlier hunks cannot be overwritten. If a line reference is ambiguous or missing, the patch fails, and the script requests a corrected diff from the model.

Managing Local State and Sandbox Cleanliness

While the model runs checks in the hosted Linux sandbox, patches are not applied inside that remote container during the review turns. The model tests its reproduction scripts against the original base files.

When the model calls openrouter:apply_patch, the script applies the change to a local scratch directory on your machine, tests its syntax, and tells the model whether it succeeded. Once submit_review completes, all remote temporary uploads are deleted, keeping OpenRouter workspace storage clean.

---

The Shift to Execution-First AI Engineering

AI-assisted software engineering is shifting away from generic chat prompts and towards automated execution loops. Asking a developer to copy and paste code suggestions from a web interface creates context switching and leaves verification to manual effort.

By combining the OpenRouter Responses API with hosted Linux sandboxes, teams can introduce code reviews that verify issues before raising alarms. A reviewer that confirms a bug with runtime output, verifies its own fix, and presents a ready-to-merge patch for less than $0.05 transforms code review from a speculative chore into an automated engineering safeguard.