A score you cannot reproduce is not a measurement.

Python 3.10 or newer. The core has zero required dependencies — PyYAML and the vendor SDKs are optional extras, and nothing is imported at module scope.

Three commitments shape the design. They explain the choices that would otherwise look pedantic — why a score carries a flag nobody asked for, why a gate fails on the first run, why the benchmark prints a number that makes the library look slow.

The category is crowded, and the distinction is the point.

What is left is the part that belongs in your repository: a suite that runs in CI, fails when the system gets worse, and names the cases that broke.

The most important thing EvalKit tracks about a score is whether it means the same thing twice. An evaluator that returns 1.00 today and 0.00 tomorrow on byte-identical input is not measuring your system; it is measuring the weather.

So every evaluator declares a flag, and the flag travels with the score.

A run using any non-deterministic evaluator carries a warning, and the run itself reports deterministic: false:

Being reproducible is not the same as being right. The offline backends are proxies, and the documentation lists what they miss rather than implying coverage they do not have.

The second commitment in practice. evalkit calibrate runs a judge over human-labelled examples and reports accuracy, the full confusion matrix, Cohen’s kappa and per-label F1.

Kappa is the one that matters, because accuracy flatters a judge that has learned nothing. Consider a judge that answers “looks good” to everything, run against a dataset that is nine passes and one fail — the shape of most real eval sets:

Ninety percent accurate, zero skill. It clears --min-accuracy 0.85 and fails --min-kappa 0.4, exiting nonzero. It never caught the one real failure — the column of zeros under fail is the whole story, which is why the matrix is printed rather than summarised away.

It measures agreement with your labellers on your dataset. It does not make the judge deterministic, and it does not transfer to a different distribution of answers. Re-run it whenever the judge prompt, the model, or the dataset changes — all three invalidate the measurement.

The third commitment. A suite declares thresholds; the CLI exits nonzero when one breaks.

A suite that failed to load must never look like a suite that ran and passed. Hence the third code.

Absolute gates — pass rate, mean score, error rate, latency percentiles, cost, per-evaluator floors — hold regardless of history. Relative gates compare against a baseline run: max_regression, max_score_regression, max_new_failures.

A failing run names the cases rather than moving an aggregate:

The bundled regression example makes the point sharply: the broken version is also faster and cheaper. A dashboard averaging latency and cost would have reported an improvement.

Eval suites are small. A pass rate of 0.84 on 50 cases is not meaningfully different from 0.80, and reporting the point estimate alone invites teams to chase noise.

Two runs of the same suite are paired data, so the test is chosen from the data: exact McNemar for binary pass/fail, where only the discordant pairs carry information; a paired sign-flip permutation test for continuous scores. Bootstrap intervals are seeded, so an interval never changes underneath a code review.

Read that carefully: two cases regressed and p = 0.50. Both are true. Six cases cannot statistically separate a real regression from noise, and the two named cases are still real bugs. The statistics tell you what the aggregate supports; the case list tells you what broke. Gate on the case list.

Roughly, resolving a pass-rate difference of d needs on the order of 1/d² cases. Which is why max_regression: 0.01 on a 100-case suite is not a quality gate — it is a coin flip with a threshold on it.

646 tests, all passing. 96% statement coverage (3,683 statements, 152 uncovered). The distribution follows the risk: the modules where a silent error would quietly bless a broken system carry the most tests.

The six bundled examples are executable documentation, and the twenty tests in test_examples run every one of them — asserting the exact numbers their READMEs quote. An example cannot drift from the code without turning the suite red.

The framework’s own cost is measured separately from model and network latency, because a real suite is dominated by the model by three or four orders of magnitude and would hide a framework regression completely. The system under test here is a no-op mock.

Roughly 80,000 cases per second, single-threaded, on the three-evaluator suite.

The Python API is primary; suites can equally be declared in YAML for CI.

Anything model-graded must say so, and the flag is a claim you are making:

In CI, the JUnit report emits two suites: one testcase per eval case, and one testcase per declared threshold. That second suite is what makes a failure legible — max_cost_usd $0.0412 > $0.0300 shows up as a named failing test rather than a bare nonzero exit.