Why Binary Acceptance Breaks on a Probabilistic System
A Werkvertrag owes a result. § 631 BGB has the contractor (Unternehmer) produce the promised work; § 640 BGB has the customer (Besteller) accept it once it is contractually produced. Both sentences quietly assume there is a state of the world you can point at and call finished — which is why the service contract (Dienstvertrag), where effort rather than outcome is owed, is the usual escape hatch when nobody can define finished. Classical software lives comfortably inside the Werkvertrag assumption: the invoice import either parses the file or it does not, and you can demonstrate that in a room, twice, and get the same answer.
An extraction step built on an LLM does not have that property. Run the same 300 invoices through it twice and you get two numbers. Neither is the truth; both are samples from the same system. Nothing is broken — that is the system. So the sentence “the work is contractually produced” loses its referent, and what fills the vacuum is an argument. The customer points at three wrong answers and says it does not work. The vendor says it works and that three out of 300 is inside normal variance. Both statements are defensible, neither is provable, and the next step is a court-appointed expert (Gutachter), a year, and a bill neither side budgeted.
This reads, at first, like a vendor wriggling out of a promise. It is closer to the opposite — and the reason is § 640(1) S. 2 BGB, which we will come back to at the end. The short version: if you do not put a number in the contract, you have not avoided a threshold. You have delegated the choice of threshold to a court.
The Acceptance Data Set Belongs in the Contract
If the measurement is what is owed, then the thing being measured is part of the specification of work. It belongs in the contract as a signed annex (Anlage) — not in a Jira ticket, not in an email thread, and certainly not in the vendor's repository alone. That sounds bureaucratic until you watch a dispute where the two sides are measuring on different files and have both been honest the entire time.
Freeze it before implementation, not after
A test set assembled after the system exists is not a test. It is a mirror: it will contain the cases someone thought to check while building, which are exactly the cases that pass. Everyone involved knows this and nobody says it, because building the set is real work and postponing it feels free. So the date on the annex should precede the first line of production code. This is the single highest-leverage change in the whole construction, and the one customers resist most.
Then hash the file. A SHA-256 of the .jsonl, written into the annex, with both parties holding an identical copy. It costs one command and it means that six months later, in a room with a lawyer, the question “which 300?” has an answer instead of a conversation. Version it too — abnahme_set_v1, and if a v2 is ever agreed, it is a change request with a price, not a quiet swap.
On size, I will not give you a magic number, because there isn't one. What governs it is whether the set can distinguish the threshold from the number just below it: a 30-item set cannot tell 0.85 from 0.80 in any meaningful sense, so a threshold measured on 30 items is decoration. For a document-extraction task I usually land between 200 and 400 items, stratified — the routine majority, a deliberate block of the awkward cases, and a small block of inputs that are genuinely bad and where the correct behaviour is to refuse. The binding constraint is never statistics. It is that a human has to write down the right answer for each one.
Labelling is the customer's duty to cooperate — put it in writing
Only the customer knows what correct means for their invoices, their contracts, their tickets. A vendor cannot invent ground truth for your business and should be viewed with suspicion if they offer to. That makes the labels a duty to cooperate (Mitwirkungspflicht), and it should say so in the contract, with a date — because when the annex is late, acceptance is blocked, and both sides should be able to see whose clock is running.
Be honest about what you are asking for: labelling 300 documents is a few days of a domain expert who has other things to do, and it is the item most reliably left out of the plan. It is not overhead, though. It is the same work as defining the acceptance criterion, and you were going to pay for it either way — in the budget now or in the dispute later. On how to actually assemble and label a set like this, I have written that up separately in evaluating LLM systems; this article assumes you have one and asks what to do with it contractually.
{
"annex": "3 — acceptance data set and measurement protocol",
"dataset": {
"file": "abnahme_set_v1.jsonl",
"sha256": "<full hash — Annex 3a>",
"items": 300,
"frozen": "2026-03-02",
"labelled_by": "customer",
"strata": { "routine": 210, "edge_cases": 60, "must_refuse": 30 }
},
"metric": { "name": "macro_f1", "threshold": 0.85 },
"protocol": {
"runs": 5,
"temperature": 0.0,
"seed": 42,
"aggregate": "mean_over_runs",
"variance_band": { "max_stddev": 0.02 },
"model": "<vendor>/<model>@<pinned version string>",
"harness": "eval-harness @ git 1a2b3c4",
"executed_by": "vendor",
"witnessed_by": "customer"
},
"decision_rule": {
"stddev > 0.02": "no valid measurement — repeat once",
"mean >= 0.85": "ready for acceptance",
"mean < 0.85": "defect — remedial performance owed"
}
}
The Protocol: What Makes a Number Binding
“0.85 macro-F1” on its own is still not testable. Measured how? How many times? On which model version, on whose machine, computed by whose code, with whom in the room? Every one of those questions is a place where a dispute can live, and each one is cheap to close in advance and expensive to close afterwards. The protocol needs to pin, at minimum:
- Metric and threshold — one number that decides, computed by a harness both parties have a copy of. Not three metrics and a discussion.
- Temperature and seed — stated explicitly, because “we ran it” is not a setting.
- Run count —
n = 5, because a single run is a sample and you are trying to characterise a distribution. - Aggregation — mean over the runs. Say which. “It passed once” and “it passed on average” are different promises.
- Variance band — a maximum standard deviation across the runs, above which there is no valid measurement.
- Environment — model version string, harness commit, hardware class. Pinned, in the annex.
- Who executes and who witnesses — usually the vendor runs it with the customer present. Screen-shared is fine. Unwitnessed is not.
Now the honest part, because you will read confident claims to the contrary: temperature 0 and a fixed seed reduce variance, they do not buy determinism. Batched inference, non-deterministic GPU kernels and vendor-side routing all mean identical inputs can still produce different tokens, and a hosted model can change underneath a stable-looking name. On-premise you get closer, never all the way. That is precisely why the band exists instead of a promise of reproducibility. The thing I would not do, under any pressure, is contract “identical output for identical input.” It is not deliverable, and a clause you cannot honour is worse than no clause — it hands the other side a defect on a plate.
A re-run at 0.79: defect or measurement noise?
This is the question that actually shows up, and the answer is genuinely uncomfortable: it depends entirely on whether you contracted a protocol, and on nothing else about the system. With the annex above, a mean of 0.79 with a standard deviation of 0.01 is a defect (Mangel) — no argument, no severity debate, and the vendor owes remedial performance (Nacherfüllung). A mean of 0.79 with a standard deviation of 0.06 is not a measurement at all; it gets repeated, and then somebody should ask why the system is that unstable, because instability of that size is itself a finding worth having before acceptance rather than after. Without the protocol, 0.79 is just two people with different intuitions about what counts as close enough.
Which is also why the threshold should never be a number someone picked in a meeting. It should be a number you have already measured once, on real data, in a timeboxed pilot, before it goes anywhere near a signature. I will not sign a fixed-price Werkvertrag against a threshold nobody has ever hit — not because it is unfair, but because it is not information. It is a wish with a penalty attached.
The Clause, and the Dispute It Prevents
Here is the shape of it. Not the wording — the shape. Everything that decides anything is a reference to the annex. The clause below is in German, because that is the language the Werkvertrag it belongs to is written in; the English walk-through follows underneath. Anlage is the annex, Auftragnehmer the vendor, Auftraggeber the customer.
(1) Das Werk ist abnahmereif, wenn die Abnahmemessung nach Anlage 3
einen Mittelwert des Macro-F1 von >= 0,85 ueber 5 Laeufe erreicht.
(2) Verbindlich ist ein Lauf auf abnahme_set_v1.jsonl (SHA-256 gemaess
Anlage 3a) bei Temperature 0.0 und Seed 42 in der in Anlage 3b
festgeschriebenen Umgebung, durchgefuehrt vom Auftragnehmer im
Beisein des Auftraggebers.
(3) Eine Messung, deren Standardabweichung ueber die 5 Laeufe 0,02
ueberschreitet, ist keine gueltige Messung und wird einmal
wiederholt.
(4) Einzelne fehlerhafte Ausgaben stellen fuer sich genommen keinen
Mangel dar. Das Unterschreiten des Schwellwerts ist ein Mangel;
dem Auftragnehmer steht Nacherfuellung auf demselben
Abnahmedatensatz zu.
(5) Nach der Abnahme ist eine Aenderung der Modellversion oder der
Umgebung nach Anlage 3b eine Aenderung des Werks. Sie erfordert
eine neue Messung und ist gesondert zu vereinbaren.
What each paragraph does:
- (1) fixes what is owed — a mean macro-F1 of at least 0.85 over five runs on the set named in Annex 3. A number, not an adjective.
- (2) fixes how it is measured — which file, which hash, which settings, which pinned environment, who runs it and who watches.
- (3) fixes the noisy case — a standard deviation above 0.02 means there is no valid measurement, and it is repeated once.
- (4) fixes what counts as a defect — the threshold does, an individual wrong answer does not; and the vendor keeps the right to remedial performance on the same frozen set.
- (5) fixes what happens afterwards — changing the model version or the environment is a change to the work, and needs a new measurement, separately agreed.
Paragraph (4) is where the real argument lives, and § 640(1) S. 2 BGB is why. That sentence says acceptance may not be refused over immaterial defects — unwesentliche Mängel. Read it from the customer's chair. Your vendor delivers, three of 300 outputs are wrong, you refuse acceptance and withhold payment. The vendor points at § 640(1) S. 2 and says one percent is immaterial. You say it plainly is not. Now: who decides whether a 1% error rate on invoice extraction is material? If the contract is silent, the answer is a court, advised by a court-appointed expert, at some distance in the future, with cost risk on both sides and an outcome nobody can predict from here. That is not a worst case. It is the default case for a contract with no number in it.
So the threshold clause protects the customer at least as much as the vendor. It is the customer converting “material” from a legal opinion into an arithmetic fact that they chose, at the one moment they have maximum leverage: before signature. If your process genuinely needs 95%, say 0.95 — and pay for 0.95, because it is a different system. If you would in truth accept 0.85, saying so buys you something valuable: an unambiguous right to refuse acceptance at 0.84 without arguing about severity with anyone.
Two more things belong in that annex. First, remedial performance happens on the same frozen set — a vendor handed a fresh set at each attempt is being handed the exam. The flip side, which I would rather say out loud than have you discover: a vendor iterating against a visible frozen set will, eventually, overfit to it. If the stakes justify the extra labelling, the mitigation is a second sealed set of the same provenance, not disclosed, measured once at acceptance; the contracted threshold still applies to the visible set, but a large gap between the two is a signal both sides agreed in advance to take seriously.
Second, drift. Once accepted, the system is accepted at that model version in that environment. When the model vendor updates it under the same name and the number moves, that is not a defect in the delivered work — nothing about the work changed — but it is also not nothing, and someone owns it. Either you pin the version and accept that it will eventually be deprecated, or you buy a maintenance contract that re-runs the acceptance data set on a schedule and says what happens when the number drops. That second thing is a Dienstvertrag sitting next to the Werkvertrag, and that is the honest structure for it. The third option — nobody owns it — is the default, and it is how a system quietly degrades for a year. Decide before acceptance: once the work is accepted, it is generally the customer who has to demonstrate the defect, which is one more reason to have the measurement on paper.
Where Tippel Fits
Most of the work in this article happens before anyone signs anything: deciding what to measure, building the set that measures it, and finding out whether the threshold you want to contract is a threshold this problem can reach. That is what the AI Readiness Check is — a paid, timeboxed pilot on your real data that ends with a defensible number rather than a good feeling, which is exactly the input a Werkvertrag needs. If the number is not there, you have learned that for the price of a week instead of the price of a build.
When I then build the thing, the fixed price is quoted against that measurement, and the harness ships with the system, because I would rather be held to a number I helped define than argue about severity later. If you are staring at a draft contract with “die KI muss zuverlässig funktionieren” (“the AI must work reliably”) in it and wondering what to write instead, send it over.