Should we switch models? A tool that answers Ship or Don't ship

A new model comes out. It's cheaper, the benchmarks look good, and someone asks whether your app should switch. Usually the answer comes from reading a dozen outputs side by side, or from running the eval suite on both models and comparing two averages: 0.84 before, 0.83 after.

Neither tells you what you need to know. A dozen outputs is an anecdote. Two averages don't say whether a 0.01 gap is real, or whether you'd see that much movement just from running the same model twice. They also don't say where the difference comes from, or what switching will cost each month.

I built upgrade-report to answer that properly. It runs a project's existing eval suite against a baseline (what runs today) and a candidate (the new model or prompt), compares them example by example with paired statistics, and writes a report that ends in one of three verdicts:

Verdict	Meaning	Exit code
Ship	Not meaningfully worse on the primary metric, no guardrail regression, nothing flagged	0
Ship with caveats	Nothing blocks, but a behavior, cost, slice or dataset-size flag needs a human look	10
Don't ship	The primary metric may have dropped by more than the margin, or a guardrail clearly regressed	20

The question it answers is: is the candidate at least as good as what we run today, where does it differ, and what will it cost?

This post covers how it works, the statistical choices behind the verdict, and how I tested a tool whose job is to tell people whether to ship.

The one function you write

I wanted adoption to be cheap. Teams already have eval datasets in LangSmith and evaluators they trust, so the tool uses those instead of asking anyone to rebuild them. The only new code is one function:

def run(inputs: dict, model: str, prompt: str) -> dict:
    """Return at least {"answer": str}."""

It's called once per example, per arm, per repeat. The best implementation is a thin wrapper around your real entry point, so the eval exercises production code rather than a copy of it. Optional keys unlock more of the report: doc_ids for recall@k, contexts for a faithfulness judge, citations and tool_calls for behavior diffs, and usage (token counts) for cost.

Getting a first report takes four commands:

pip install "upgrade-report @ git+https://github.com/atharvapingale/upgrade-report"
upgrade-report init           # config, pricing, target stub, sample data, CI workflows
upgrade-report run --dry-run  # validate the config and estimate what the run will cost
upgrade-report run            # writes reports/<timestamp>/report.{json,md}

init scaffolds a project that runs as-is on a small sample dataset, so you see a real report before wiring anything up. Then you describe the comparison in config.yaml:

baseline:  { model: model-a-2025-06, prompt: prompts/answer@v12 }
candidate: { model: model-b-2026-03, prompt: prompts/answer@v12 }

metrics:
  primary: faithfulness                                          # tested for non-inferiority
  guardrails: [refusal_correctness, format_compliance, recall@5] # must not clearly regress
  tracked: [answer_relevance]                                    # reported, never decisive

decision:
  primary_margin: -0.02

traffic:
  monthly_queries: 120000

The three metric roles matter. One primary metric carries the main decision. Guardrails are things that must not get clearly worse. Tracked metrics appear in the report but never affect the verdict, however much they move.

How it decides

Compare pairs, not averages

Both arms answer the same examples, so every comparison uses per-example differences: the candidate's score minus the baseline's score on the same question. Some questions are hard for any model and some are easy; comparing on the same question cancels that out. A paired comparison is much more sensitive than comparing two independent means, so you need fewer examples to see a real effect.

Confidence intervals come from a paired percentile bootstrap: 10,000 resamples of the per-example differences, with a fixed seed so the same inputs always produce the same report.

Measure the noise before judging the signal

LLM outputs vary between identical runs. If your metric moves by 0.03 when you run the same model twice, a 0.02 drop from the candidate means nothing.

So each arm runs several times (three by default), and the baseline's repeats double as a free A/A test. For each pair of repeats (1 vs 2, 1 vs 3, 2 vs 3), the tool bootstraps the mean per-example difference. The 95th percentile of those absolute means is the noise floor. A candidate difference inside the noise floor is reported as "within noise" and never counts as a regression.

My original design had a bug here, and it's an easy one to miss. That procedure measures the noise of comparing one repeat with another. But the candidate's delta compares averages of three repeats, and averaging three runs shrinks run-to-run noise by √3. With three repeats, the unscaled floor was about 1.7× too large. On a well-powered dataset, a real regression the size of the margin could hide inside it: a −0.03 drop with a CI of [−0.043, −0.017], which is clearly worse, would have shipped without a caveat. The fix is to divide the floor by √R, where R is the number of repeats.

Ask "not worse", not "better"

The primary metric is tested for non-inferiority. You rarely upgrade because the new model is strictly better on your eval. More often it's cheaper or faster, and you need to know it isn't meaningfully worse. So the question is whether the lower bound of the confidence interval stays above a margin (−0.02 by default), not whether the candidate wins.

The primary metric blocks a release when its delta is below the negative noise floor and its CI lower bound is below the margin. The first condition means an observed improvement can never block, which keeps the verdict monotonic: making the candidate better never makes the verdict worse.

Guardrails block only when their whole CI sits below the negative noise floor, or below an absolute tolerance if you configure one.

Pass/fail metrics get flip counts

For boolean metrics (was the answer correct, was the format valid), the tool runs McNemar's test on the discordant pairs: examples that passed in one arm and failed in the other. Below 25 discordant pairs it uses the exact binomial version. It always reports both counts, because "6 improved, 7 regressed" tells you more than a p-value. The report then lists the flipped cases, pass→fail first, with both answers and trace links.

Say when the dataset is too small

This is the check I'd most like every eval to have. The minimum detectable effect (MDE) is the smallest true difference the dataset could detect with 80% power at 5% significance, given its size and the spread of the paired differences:

MDE = (z₀.₉₇₅ + z₀.₈) · sd(differences) / √n

If the MDE is larger than your margin, the report says plainly that the dataset is too small to rule out a regression of the size you care about, and the verdict drops to Ship with caveats. In the sample report further down, 80 examples give an MDE of 0.106 against a margin of 0.05. That dataset can't rule out a 5-point drop, and a report that said "Ship" anyway would be giving false confidence.

Slices, carefully

Examples carry tags (billing, returns, legal and so on), and the report breaks the primary metric down per tag. Slices under 20 examples are marked directional only and never block. Many slices are compared, so roughly one in twenty will look significant by chance. To keep false alarms rare, a slice is flagged only when its whole CI clears its own noise floor, and even then it's a caveat rather than a block. The report states this next to the table.

What averages hide

Two models can get the same score and behave very differently. Every run computes behavior diffs alongside the metrics:

Answer length, median and p90 (flagged by default above a 25% change)
Refusal rate, from configurable patterns or a classifier (flagged above 3 points)
Format failure rate
Citations per answer, plus tool-call counts and mix for agentic apps
Target error rate

The error rate needs its own flag. Failed calls are excluded from metric scores, since you can't grade an answer that doesn't exist. Without a separate flag, a candidate that crashes on hard questions would look better, because its hardest examples would drop out of the average.

There's also an optional pairwise judge. On a sample of examples (75 by default), the judge sees both answers in a seeded random order and picks one. The report gives the candidate's preference rate with a bootstrap CI, and if the whole CI is below 50%, that's a caveat.

Cost and latency

Per-query cost comes from the token usage of each call, priced from a pricing.yaml checked into the repo. Provider prices change, and a report should be reproducible from its commit. Multiplying by monthly_queries gives the line that often settles the discussion: "$442 → $367 per month".

Latency is reported as p50 and p95 per arm. To keep those comparable, the runner interleaves calls: for each example, the baseline and candidate requests go out side by side, alternating which arm goes first. If the provider slows down for ten minutes, both arms feel it.

The eval run's own cost also goes in the report (target calls actually made, plus judge calls), and --dry-run estimates it before you spend anything.

The judge is part of the measuring equipment

Many metrics come from an LLM judge. If the judge changes between the baseline run and the candidate run, the comparison is meaningless. So the judge is treated as a pinned instrument:

The judge model must be an exact version. Aliases like -latest are rejected.
The judge configuration is part of the score cache key and is recorded in the report.
Each response's reported model id is checked against the pin. A mismatch raises JudgeDriftError and the run exits with an error.
The built-in Anthropic adapter doesn't enable server-side model fallbacks, because a fallback would quietly swap the judge.
Refusals and truncated judge replies raise an error instead of becoming a score.

Testing a tool that tells you whether to ship

A tool that makes ship decisions needs its own evidence. The suite has 92 tests that run in about 20 seconds with no network access. Alongside unit and snapshot tests, it includes simulations that check the statistics and the full decision against known ground truth:

Property	Result
Bootstrap CI coverage at true effects 0 / 0.05 / 0.3 (n = 200)	92–97.5% (target 95%)
McNemar type I error at α = 0.05	≤ 6%
Detection rate when the true effect equals the MDE	72–88% (target 80%)
A/A verdict is Ship (300 examples, 3 repeats, 5 slices)	97.2% of 500 runs (target ≥ 95%)
Primary drop equal to the margin	Don't ship in 90% of runs
Primary drop of 1.5× or 2× the margin	Don't ship in 100% of runs
Fake candidate failing a fixed 10% of examples	Don't ship in every seed

The A/A result is the one I care about most. If you compare the baseline with itself and the tool doesn't say Ship at least 95% of the time, it's too noisy to trust, and teams will learn to ignore it. Users can run the same check on their own setup: upgrade-report aa compares the baseline with itself on your dataset and config. If that doesn't come back Ship, you need more repeats or more examples before trusting any verdict.

Living in CI

Because the verdict maps to exit codes, CI integration is simple. A reusable GitHub Actions workflow runs on pull requests that touch the model config, prompts or retrieval code. It:

Restores a SQLite cache of previous runs.
Runs the comparison. The baseline usually comes from cache, so only the candidate runs.
Posts report.md as a PR comment, updating the same comment on re-runs.
Gates the merge. Don't ship fails the check. Ship with caveats passes the main check but fails a second acknowledge check until a reviewer adds a label, so someone has to read the caveats.

Caching is what keeps this affordable. Each arm run is keyed by (dataset_version, model, prompt_hash, target_version, repeat_index):

dataset_version is a content hash of the examples, so editing a single example invalidates it.
prompt_hash covers the prompt file's contents, not just its name.
target_version hashes the application code as git blob SHAs. Because the same digest can be computed from any git ref, --baseline-ref origin/main finds the baseline that main's CI run cached, even when the PR changes code.

Records are stored per example, so an interrupted run resumes where it stopped, and failed calls are never cached. If LangSmith is down, the run continues without tracing and the report says so; an outage in the tracing backend shouldn't block a merge.

Watching for new models

upgrade-report scheduled lists the models a provider offers and runs a full comparison for each one it hasn't seen before, with that model as the candidate. The first run only records what exists today. After that, a scheduled weekday workflow turns each new model release into a report in the job summary, without anyone having to remember to check. It only notifies; it never opens PRs or changes config.

What a report looks like

Here's a trimmed excerpt from the fixture report used in the snapshot tests. The models are fakes, so the numbers are synthetic, but the shape is real. This run had two candidate arms: a new model on the existing prompt, and a new model with an adapted prompt. The second arm exists because a prompt tuned for the old model can make a new one look worse than it is.

⚠️ Ship with caveats MDE 0.106 for correctness is larger than the margin 0.050: this dataset is too small to rule out a regression of the size you care about (+1 more)

Arm	Model	Prompt	Verdict	Δ correctness	Cost / query
candidate	`fake-verbose`	`prompts/answer@v1`	⚠️ Ship with caveats	+0.025	$0.0047
adapted_candidate (recommended)	`fake-good-b`	`prompts/answer@v2`	⚠️ Ship with caveats	−0.013	$0.0031

Metric	Role	Baseline	Candidate	Δ	95% CI	Noise floor	Status
correctness	primary · pass/fail	0.842	0.829	−0.013	[−0.083, +0.058]	0.087	≈ within noise
format_compliance	guardrail · pass/fail	1.000	1.000	+0.000	[+0.000, +0.000]	0.000	≈ within noise

	Baseline	Candidate	Change
Cost per query	$0.0037	$0.0031	−17%
Monthly (120,000 queries)	$442	$367	−$75.32
Latency p50	1.20 s	0.90 s	−25%
Latency p95	3.40 s	2.60 s	−24%

The first candidate has the better correctness delta, and if you only compared averages it would look like the winner. The report doesn't recommend it, though. Its answers were about 28 times as long as the baseline's (+2750%), and its cost per query rose 28%, so it tripped both the length flag and the cost flag. The adapted arm raised fewer concerns, costs 17% less per query than the baseline, and is 25% faster at p50.

Both arms still end at Ship with caveats, for the most useful kind of reason: 80 examples can't rule out a regression the size of the margin. The tool recommends an arm, but it doesn't claim the quality question is settled.

There's also a client-facing HTML version for explaining an upgrade to people who don't read CI logs. It keeps the verdict, the metric table, the cost impact and a few example comparisons, adds plain-language explanations, and removes trace links and internal IDs. Inputs can optionally be redacted.

What "Ship" doesn't mean

A Ship verdict describes this dataset, these metrics and this judge. It says nothing about traffic the dataset doesn't represent, and the judge's blind spots are the report's blind spots. It doesn't replace production monitoring. The caveats section is rendered on every report, including a clean Ship, to keep that visible.

Open questions

Some design questions are still open:

Multi-step agents. Should they be scored only on the final trajectory, or step by step too?
Stricter gates. Should Ship with caveats block merges for some projects, such as regulated clients?
Automation. Should scheduled mode open a PR when a new model passes, or keep only notifying?
Other backends. Is it worth supporting Langfuse and other tracing tools through an adapter, or should it stay LangSmith-only?

Try it

The code is MIT-licensed on GitHub. The getting-started guide takes you from install to a first report in about 30 minutes, and the methodology doc has the full decision rules and calibration results.

If you run it against a real upgrade, I'd like to hear whether the verdict matched your team's judgment, and especially about any case where it didn't.