CASCADE Routing audit, KYC extraction Seal 1151f5a1cfaae0c0 · measured, then frozen
Appendix A Method and reproducibility ← Back to the findings

One method, no secrets. Run it twice, compare.

A matte aluminium balance with two pans at exactly the same height; the right pan is the site's deep green.

This page shows the whole method: what is measured, on which slice of the corpus, how the routing is chosen, and how the record is frozen. Every claim lives in the source at a path your reviewers can open — and every step reruns on your machine, so the comparison is yours to make.

Measured on a slice the tuning never saw

The corpus is cut into three parts — training, dev, and held‑out — and the published measurement reads only the held‑out slice, forced in code, not by discipline. The middle slice exists because of a real mistake: a generative prompt was once tuned while reading held‑out scores. The leak was caught, repaired, and the split hardened so it cannot recur silently: a test fails the suite if two slices share a document or a phrasing.

One honest limit, stated by the repository itself: the corpus is authored by us. The held‑out slice defends against grading our own copy — it does not defend against unrealism. That is what measuring your own cases is for.

Verify the split

Run the suite. The split test walks the corpus and fails if any document or phrasing appears in two slices.

run it yourself
npm testfails if two slices share a document or a phrasing

Where it livessrc/corpus.ts:164 · src/cascade.test.ts:47

Every routing, enumerated — with one honest asterisk

Five fields, seven tiers per field: the optimiser walks all 16 807 assignments exhaustively — a recursive enumeration, no heuristic, no sampling. The count on this page is recalculated from the sources on every check, never typed in.

The asterisk: the seventh tier, human review, is an assumption, not a measurement — and the tool prints exactly that on every pass rather than letting the figure blend in with the measured six.

Verify the enumeration

Run the optimiser. It enumerates from the sealed record and announces the human tier as an assumption, every time.

run it yourself
npm run optimiseprints the optimal routing, human tier flagged as assumed

Where it livessrc/optimise.ts:379 · src/readme.ts:827

The record is frozen by a fingerprint

The published record carries a content fingerprint — a SHA‑256 taken over the record in canonical form and truncated to sixteen hex characters, and the reader refuses a record whose fingerprint is absent or no longer matches. Refuses, not warns: a hand‑edited measurement fails loudly. The stamp in this site's masthead is that fingerprint.

You do not have to take our word for it, and you should not hash the file by hand — a plain checksum of the file will not match, because the fingerprint is taken over a canonical form of the record, not over its bytes. The tool recomputes it in front of you instead, and prints what it finds.

Two limits the sealing tool states about itself: the fingerprint covers the whole record except fingerprint keys themselves, and it proves nothing about what happened before sealing — the only gesture that produces a measurement is the measurement command. And one repair published rather than hidden: the git commit the reference record was measured at was later rewritten away by a history purge; the repository records the replacement commit and labels the mapping “inferred, not recorded”.

Recompute the seal

Point the sealing command at the sealed record shipped with the repository. It recomputes the fingerprint and prints it; on an untouched record it changes nothing.

run it yourself
npm run sceller -- profiles-2026-08-20-coeur-rendu.jsonalready sealed, and the seal matches: 1151f5a1cfaae0c0

Where it livessrc/measure.ts:298 · src/sceller.ts:12

Intervals that refuse to flatter

Every rate ships with a Wilson interval at 95%. The repository admits, in writing, that the level is an inherited default rather than a weighed choice. The interval function throws on impossible input instead of returning a quiet NaN, and for two tiers graded on the same cases the repository prefers a paired McNemar test to comparing Wilson intervals, which it documents as too conservative for that job.

Verify the arithmetic

The interval code is thirty lines and its guard is tested; the declared method is in the published record.

run it yourself
npm testcovers the interval guard and the declared method

Where it livessrc/interval.ts:32 · src/inventory.ts:63

The figures that were retired

Not everything measured got published. On the constrained‑output bench, every duration moved by 16 to 60 per cent between two passes on a memory‑starved machine — so the durations were withdrawn, and in the data they were renamed so the defect travels with the number: the field itself is called msMedianeNonTransportableFamineMemoire — non-transportable, memory starvation. You cannot cite it without its qualifier.

The token counts from the same bench reproduce to the digit, and those are published. That is the line: what reproduces is shown, what does not is named and withheld.

Verify the retirement

Rerun the bench. Below the memory floor it marks its own durations non-transportable rather than publishing them.

run it yourself
npm run contrainteregenerates the bench record, durations qualified in the field name

Where it livesrapports/2026-08-22-contrainte-de-sortie.md:17 · contrainte.json:34

Run it twice, compare

The title is an instruction. The full measurement reruns with one command — it insists on an explicit flag because it downloads 1.3 GB of pinned weights on first run — the measurement itself took 32 minutes on the published pass — and freezes a fresh record you can set beside ours. Your own files run through the same grader and the same intervals. And two sealed passes compare case by case, because a rate that rose may simply have lost cases.

Run it yourself

The three commands below are the whole loop: remeasure, measure yours, compare.

run it yourself
MESURE_VOULUE=1 npm run measurereruns the full measurement and freezes a sealed recordnpm run measure:yours -- --cases=your-file.csvyour cases, same grader, same intervals — nothing leaves your machinenpm run diffcompares two sealed passes case by case

Where it livessrc/measure.ts:960 · src/your-cases.ts:1185

On your records, on your machine. Nothing leaves the network.