Review the agent,
not just the diff.

diff0 compares how your agent behaves before and after a code change—so silent drift shows up before you merge it.

Run the $0 demo View the source

diff0 / simplify-pipeline

comparison complete

source change

agent/instructions.md

− Delegate calculations to the reporter.

− Use its answer in the final response.

Existing eval suite4 / 4 passed

behavioral comparison

Drift detected

review

reporter subagent5 / 5 runs0 / 5 runs

eval pass rate20 / 2017 / 20

cost / session$0.0141$0.0095

Fisher exact + Holm adjustment·N=5 per ref

One real prompt edit · 10 runs · $0.1188 total

No SDK

Wraps the eval suite you have

Two refs

Base behavior vs. head behavior

N runs

Evidence, not anecdotes

One comment

The result lives in your PR

The gap after evals

Your eval passed.
Your agent still changed.

A scorer tells you whether the answer was acceptable. diff0 tells you how the agent got there—and whether that behavior moved with your code.

A tiny edit

Two prompt lines disappear. The source diff looks harmless.

instructions.md

− Delegate to reporter

− Use its final answer

Green evals

The showcased single run clears the suite. Repetition reveals what one green run misses.

reply-formatpass

uses-sql-toolpass

total-revenuepass

Visible drift

Repeated runs reveal that delegation vanished completely.

reporter subagent

5 / 50 / 5

How it works

Same repo.
Both realities.

diff0 checks out both git refs, runs the same eval suite on each, and compares what the agent actually did.

01
Checkout
Base and head run in isolated git worktrees. Your working tree stays untouched.
02
Run
Existing Eve evals run N times per ref, interleaved to reduce provider drift.
03
Collect
Tool calls, subagents, skills, tokens, cost, and duration come from Eve's own artifacts.
04
Report
Statistical changes and stable trace drift land in the terminal and one sticky PR comment.

Recorded, not imagined

Watch the behavior change land.

Real output from Claude Haiku 4.5. 5 runs per ref. The full comparison cost $0.1188.

~/diff0-demo (simplify-pipeline)

$ diff0 run --base main --head simplify-pipeline --runs 5
diff0 main...simplify-pipeline  YELLOW
  — No confirmed eval regressions across 5 runs per ref — 2 eval changes inconclusive,
    behavioral drift detected
eve 0.29.5 | model anthropic/claude-haiku-4.5 | 5 runs per ref | sandbox docker (inferred)
  comparison cost $0.1188 (gateway)
EVALS
  revenue/reply-format       base 5/5   head 3/5   lower pass rate (inconclusive)
    [score 1 -> 1 (+0); Fisher raw p=0.2222; Holm p=0.8889]
  revenue/uses-sql-tool      base 5/5   head 4/5   lower pass rate (inconclusive)
    [score 1 -> 1 (+0); Fisher raw p=0.5; Holm p=1]
  revenue/no-failed-actions  base 5/5   head 5/5   pass
    [score 1 -> 1 (+0)]
  revenue/total-revenue      base 5/5   head 5/5   pass
    [score 1 -> 1 (+0)]
BEHAVIORAL DRIFT
  subagents:
    reporter (unattributed): used in 5 of 5 base runs -> 0 of 5 head runs
      [statistically-confirmed; Fisher raw p=0.0079; Holm p=0.0159]
COST & PERFORMANCE
  cost/session        base $0.0141 ($0.0136–$0.0142)  head $0.0095 ($0.0077–$0.0126)  -33%
  tokens in           base 82,370 (77,251–82,468)     head 55,282 (45,353–71,597)     -33%
  tokens out          base 1,195 (1,177–1,225)        head 775 (642–1,113)            -35%
  cache-read tokens   base unavailable                head unavailable                n/a
  cache-write tokens  base unavailable                head unavailable                n/a
  duration            base 37.0s (34.2s–37.3s)        head 22.1s (17.5s–26.5s)        -40%
CHANGED FILES
  agent/instructions.md (+0 -2)
  1 file changed, 2 deletions(-). File attribution is correlational, not causal.
show full output
PER-RUN RAW SUMMARIES
  base main @ b8fb7f2
    run 1: 4/4 evals passed, 8 tool calls, skills: revenue-definitions, $0.0142, 37.0s
    run 2: 4/4 evals passed, 8 tool calls, skills: revenue-definitions, $0.0142, 34.2s
    run 3: 4/4 evals passed, 8 tool calls, skills: revenue-definitions, $0.0141, 35.4s
    run 4: 4/4 evals passed, 8 tool calls, skills: revenue-definitions, $0.0136, 37.2s
    run 5: 4/4 evals passed, 8 tool calls, skills: revenue-definitions, $0.0140, 37.3s
  head simplify-pipeline @ 77d4334
    run 1: 4/4 evals passed, 8 tool calls, skills: revenue-definitions, $0.0098, 22.1s
    run 2: 3/4 evals passed, 7 tool calls, skills: revenue-definitions, $0.0092, 21.0s
    run 3: 3/4 evals passed, 7 tool calls, skills: revenue-definitions, $0.0095, 23.1s
    run 4: 3/4 evals passed, 6 tool calls, skills: revenue-definitions, $0.0077, 17.5s
    run 5: 4/4 evals passed, 10 tool calls, skills: revenue-definitions, $0.0126, 26.5s
Statistical comparison across 5 runs per ref —
LLM runs are nondeterministic; treat proportions, not absolutes.

Evidence, with limits

Honest about what the runs can prove.

diff0 separates regression, inconclusive movement, and behavioral drift. It shows uncertainty instead of painting every change red.

N=5

runs per ref

$0.1188

showcase cost

baseline files

sticky PR comment

Pass-rate evidence

One-sided Fisher exact tests compare differing proportions across runs.

Multiple comparisons

Holm adjustment keeps a suite full of evals from manufacturing confidence.

Validity caps

Changed model, sandbox, run count, or scorer validity caps the verdict at yellow.

Drift is not automatically bad. Sometimes it is the change you wanted—like removing delegation and cutting median session cost by 33%. diff0 makes the tradeoff legible; review makes the call.

In your pull request

The report goes where the decision happens.

One comment updates in place on every push. No dashboard to remember, no second review loop.

github-actionsbotcommented

diff0: main...simplify-pipeline 🟡

No confirmed eval regressions across 5 runs per ref — 2 eval changes inconclusive, behavioral drift detected.

eve 0.29.5 · model anthropic/claude-haiku-4.5 · 5 runs per ref · sandbox docker (inferred) · comparison cost $0.1188 (gateway)

Evals

Eval	Base	Head	Status
`revenue/reply-format`	5/5	3/5	🟡 lower pass rate (inconclusive)
`revenue/uses-sql-tool`	5/5	4/5	🟡 lower pass rate (inconclusive)
`revenue/no-failed-actions`	5/5	5/5	✅ pass
`revenue/total-revenue`	5/5	5/5	✅ pass

Behavioral drift

Subagents

reporter (unattributed): used in 5 of 5 base runs → 0 of 5 head runs — statistically-confirmed (raw Fisher p=0.0079; Holm-adjusted p=0.0159)

Cost & performance

Metric	Base (median)	Head (median)	Δ
Cost / session	$0.0141 ($0.0136–$0.0142)	$0.0095 ($0.0077–$0.0126)	-33%
Tokens in	82,370 (77,251–82,468)	55,282 (45,353–71,597)	-33%
Tokens out	1,195 (1,177–1,225)	775 (642–1,113)	-35%
Duration	37.0s (34.2s–37.3s)	22.1s (17.5s–26.5s)	-40%

Changed files

agent/instructions.md (+0 −2)

1 file changed, 2 deletions(-). File attribution is correlational, not causal.

Per-run raw summaries

base — main @ b8fb7f2

Run	Evals passed	Tool calls	Skills loaded	Cost	Duration
1	4/4	8	revenue-definitions	$0.0142	37.0s
2	4/4	8	revenue-definitions	$0.0142	34.2s
3	4/4	8	revenue-definitions	$0.0141	35.4s
4	4/4	8	revenue-definitions	$0.0136	37.2s
5	4/4	8	revenue-definitions	$0.0140	37.3s

head — simplify-pipeline @ 77d4334

Run	Evals passed	Tool calls	Skills loaded	Cost	Duration
1	4/4	8	revenue-definitions	$0.0098	22.1s
2	3/4	7	revenue-definitions	$0.0092	21.0s
3	3/4	7	revenue-definitions	$0.0095	23.1s
4	3/4	6	revenue-definitions	$0.0077	17.5s
5	4/4	10	revenue-definitions	$0.0126	26.5s

Statistical comparison across 5 runs per ref — LLM runs are nondeterministic; treat proportions, not absolutes.

Start with the mock

See the drift.
Spend $0.00.

No Eve repo and no model key required. Clone the project and replay the entire comparison on the deterministic mock.

$ git clone https://github.com/knowbody/diff0 && cd diff0

$ bash demo/mock-demo.sh

Already have an Eve eval suite?

$ node /path/to/diff0/dist/cli.js run --base main --head HEAD --runs 5

Add it to CI

Run the comparison on pull requests and post the result as one self-updating comment. Fork PRs stay refused by default.

name: diff0

on: pull_request

permissions:
  contents: read
  pull-requests: write # required for the sticky PR comment

jobs:
  diff0:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # required: diff0 checks the base ref out locally
          persist-credentials: false # do not expose the job token to evaluated head code
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      # Pin an immutable release.
      - uses: knowbody/diff0/action@v0.1.0
        with:
          working-directory: .
          runs: "3"
          install-mode: scripts-off
          fail-on: regression

diff0

Code review for the part of your agent you cannot see.

Run the demo Explore on GitHub

Review the agent,not just the diff.

Your eval passed.Your agent still changed.