Why Binary Acceptance Fails on a Probabilistic System

A Werkvertrag owes a result. § 631 BGB has the contractor (Unternehmer) produce the promised work; § 640 BGB has the customer (Besteller) accept it once it is contractually produced. Both provisions tacitly assume a state you can point at and call finished. The line to the service contract (Dienstvertrag) runs there too: it owes effort and not outcome, and parties usually retreat into it when nobody can define finished. Classical software fits the Werkvertrag: the invoice import either parses the file or it does not, and you can demonstrate that twice in a row with the same result.

An extraction step built on an LLM does not have that property. Run the same 300 invoices through it twice and you get two numbers, and both are samples from the same system. Nothing is broken; this is how the system behaves. The sentence “the work is contractually produced” thereby loses its referent, and the parties argue. The customer points at three wrong answers and says it does not work. The vendor says it works and that three out of 300 is within normal variance. Both statements are defensible, neither is provable, and what follows is a court-appointed expert (Gutachter), a year, and a bill neither side budgeted.

The approach. Do not force a probabilistic system into a binary acceptance. Move the owed result from the output to the measurement: what is owed is “≥ threshold on a frozen, hash-versioned acceptance data set, measured under a named protocol.” The eval harness thereby turns from a QA tool into the specification of work (Leistungsbeschreibung).

This can sound like a vendor watering down a promise. The construction protects the customer at least as much, though, and the reason is § 640(1) S. 2 BGB, which I come back to at the end. Whoever leaves the number out of the contract does not avoid a threshold but leaves its choice to a court.

Acceptance data set 300 cases · frozen · labelled SHA-256 named in the annex signed before implementation Protocol 5 runs · temperature 0 · seed pinned model + harness commit named executor and witness Threshold mean macro-F1 ≥ 0.85 std-dev ≤ 0.02 or re-measure below = defect, not opinion What is owed is this chain — not any single answer the system gives.
The parties agree all three objects in the contract. Drop any one of them and “it works” is an opinion again.

The Acceptance Data Set Belongs in the Contract

If the measurement is owed, the thing being measured is part of the specification of work. It belongs in the contract as a signed annex (Anlage) and not in a Jira ticket, an email thread, or the vendor's repository alone. That sounds bureaucratic until two sides in a dispute have measured on different files and both been honest throughout.

Freeze it before implementation, not after

A test set that someone assembles after the system exists tests little. It contains the cases someone thought of while building, and those cases pass. Everyone involved knows this and nobody says it, because assembling the set costs work and postponing it appears to cost nothing. The date on the annex should therefore precede the first line of production code. No other single change in this construction has more effect, and customers resist none more.

Then hash the file: write a SHA-256 of the .jsonl into the annex, and have both parties hold an identical copy. It costs one command, and six months later, in a room with a lawyer, the hash answers the question “which 300?”. Version the set too, as abnahme_set_v1. If the parties later agree a v2, it is a change request with a price, not a quiet swap.

I will not give a fixed number for the size, because there isn't one. The set has to distinguish the threshold from the number just below it: a 30-item set cannot tell 0.85 from 0.80 in any meaningful sense, so a threshold measured on 30 items tells you nothing. For a document-extraction task I usually land between 200 and 400 items, stratified into the routine majority, a deliberate block of awkward cases, and a small block of bad inputs where the correct behaviour is to refuse. Statistics never limits the size; the human who has to write down the right answer for each item does.

Labelling is the customer's duty to cooperate

Only the customer knows what correct means for their invoices, their contracts, their tickets. A vendor cannot invent ground truth for your business, and I would be wary if one offers to. The labels are therefore a duty to cooperate (Mitwirkungspflicht), and the contract should say so, with a date. A late annex blocks acceptance, and both sides should be able to see on whose side the delay lies.

Labelling 300 documents costs a few days of a domain expert who has other things to do, and this item is the likeliest to drop out of the plan. It is still not overhead: whoever labels defines the acceptance criterion, and you pay for that work either way, in the budget now or in the dispute later. This article assumes you have such a set and deals with what to do with it contractually.

anlage-3-abnahme.json
{
  "annex": "3 — acceptance data set and measurement protocol",
  "dataset": {
    "file":    "abnahme_set_v1.jsonl",
    "sha256":  "<full hash — Annex 3a>",
    "items":   300,
    "frozen":  "2026-03-02",
    "labelled_by": "customer",
    "strata": { "routine": 210, "edge_cases": 60, "must_refuse": 30 }
  },
  "metric": { "name": "macro_f1", "threshold": 0.85 },
  "protocol": {
    "runs":        5,
    "temperature": 0.0,
    "seed":        42,
    "aggregate":   "mean_over_runs",
    "variance_band": { "max_stddev": 0.02 },
    "model":       "<vendor>/<model>@<pinned version string>",
    "harness":     "eval-harness @ git 1a2b3c4",
    "executed_by": "vendor",
    "witnessed_by": "customer"
  },
  "decision_rule": {
    "stddev > 0.02":  "no valid measurement — repeat once",
    "mean >= 0.85":   "ready for acceptance",
    "mean < 0.85":    "defect — remedial performance owed"
  }
}

The Protocol: What Makes a Number Binding

“0.85 macro-F1” on its own is still not testable. It leaves open how and how often it is measured, on which model version, on whose machine, with whose code and in whose presence. The parties can argue over every one of those questions, and each is cheap to settle in advance and expensive to settle afterwards. The protocol must fix, at minimum:

  • Metric and threshold — a single number decides, and a harness both parties have a copy of computes it. Three metrics that people discuss afterwards decide nothing.
  • Temperature and seed — stated explicitly, because “we ran it” is not a setting.
  • Run count — n = 5, because a single run is a sample and you are trying to characterise a distribution.
  • Aggregation — mean over the runs. Name the aggregation, because “it passed once” and “it passes on average” are different promises.
  • Variance band — a maximum standard deviation across the runs, above which there is no valid measurement.
  • Environment — model version string, harness commit and hardware class, pinned in the annex.
  • Who executes and who witnesses — usually the vendor runs it with the customer present. A screen share is enough for that, an unwitnessed measurement is not.

Temperature 0 and a fixed seed reduce variance but do not remove it, even though you will read more confident claims. Batched inference, non-deterministic GPU kernels and vendor-side routing can still turn identical inputs into different tokens, and a hosted model can change underneath a stable-looking name. On-premise you get closer to determinism, never all the way. So agree a variance band, not a promise of reproducibility. Under no pressure would I contract “identical output for identical input.” It is not deliverable, and a clause you cannot honour is worse than none, because it hands the other side the defect.

A re-run at 0.79: defect or measurement noise?

The answer depends entirely on whether the parties agreed a protocol, and on nothing else about the system. With the annex above, a mean of 0.79 with a standard deviation of 0.01 is a defect (Mangel) that the contract itself names as material, and the vendor owes remedial performance (Nacherfüllung). A mean of 0.79 with a standard deviation of 0.06, by contrast, is not a valid measurement. The parties repeat it, and then somebody should ask why the system is that unstable, because variance of that size is itself a finding you would want before acceptance and not after. Without the protocol, 0.79 comes down to two people with different ideas of what counts as close enough.

The threshold must therefore never be a number someone picked in a meeting. Measure it once on real data, in a timeboxed pilot, before you sign it. I will not sign a fixed-price Werkvertrag against a threshold nobody has ever hit. That would not be unfair, but such a threshold carries no information.

The Clause, and the Dispute It Prevents

The clause below shows the shape and not the wording. Everything that decides anything refers to the annex. The clause is in German, because that is the language the Werkvertrag it belongs to is written in; the English walk-through follows underneath. Anlage is the annex, Auftragnehmer the vendor, Auftraggeber the customer.

§ 7 Abnahme — German contract clause (illustration, not a template)
(1) Das Werk ist abnahmereif, wenn die Abnahmemessung nach Anlage 3
    einen Mittelwert des Macro-F1 von >= 0,85 ueber 5 Laeufe erreicht.

(2) Verbindlich ist eine Messung aus 5 Laeufen auf abnahme_set_v1.jsonl
    (SHA-256 gemaess Anlage 3a) bei Temperature 0.0 und Seed 42 in der
    in Anlage 3b festgeschriebenen Umgebung, durchgefuehrt vom
    Auftragnehmer im Beisein des Auftraggebers.

(3) Eine Messung, deren Standardabweichung ueber die 5 Laeufe 0,02
    ueberschreitet, ist keine gueltige Messung und wird einmal
    wiederholt.

(4) Einzelne fehlerhafte Ausgaben stellen fuer sich genommen keinen
    Mangel dar. Das Unterschreiten des Schwellwerts ist ein
    wesentlicher Mangel und berechtigt zur Verweigerung der Abnahme;
    dem Auftragnehmer steht Nacherfuellung auf demselben
    Abnahmedatensatz zu.

(5) Nach der Abnahme ist eine Aenderung der Modellversion oder der
    Umgebung nach Anlage 3b eine Aenderung des Werks. Sie erfordert
    eine neue Messung und ist gesondert zu vereinbaren.

The paragraphs in turn:

  • (1) fixes what is owed — a mean macro-F1 of at least 0.85 over five runs on the set named in Annex 3, a number and not an adjective.
  • (2) fixes how it is measured — five runs, which file, which hash, which settings, which pinned environment, who runs it and who watches.
  • (3) fixes the noisy case — a standard deviation above 0.02 means there is no valid measurement, and it is repeated once.
  • (4) fixes what counts as a defect — falling below the threshold does, and the clause names it a material defect that entitles the customer to refuse acceptance; an individual wrong answer does not; and the vendor keeps the right to remedial performance on the same frozen set.
  • (5) fixes what happens afterwards — changing the model version or the environment is a change to the work, and needs a new measurement, separately agreed.

The dispute hangs on paragraph (4), and § 640(1) S. 2 BGB is why. That sentence says acceptance may not be refused over immaterial defects — unwesentliche Mängel. Read it as the customer. Your vendor delivers, three of 300 outputs are wrong, you refuse acceptance and withhold payment. The vendor invokes § 640(1) S. 2 and calls one percent immaterial, and you disagree. If the contract is silent on whether a 1% error rate on invoice extraction is material, a court decides later, advised by a court-appointed expert, with cost risk on both sides and an outcome nobody can predict today. That is how a contract without a number normally plays out, not the worst case.

So the threshold clause protects the customer at least as much as the vendor. With it the customer turns “material” from a legal opinion into an arithmetic quantity that they chose, and does so before signature, when their leverage is greatest. If your process needs 95%, write 0.95 and pay for 0.95, because it is a different system. If you would in truth accept 0.85, writing it down gives you a better-founded position to refuse acceptance at 0.84, because the contract itself names the threshold as material and severity does not have to be settled in a dispute first.

Two more things belong in the annex. First, remedial performance happens on the same frozen set, because with a fresh set at each attempt the measurements would not be comparable, and whoever measures often enough eventually hits a favourable sample. This has a flip side: a vendor iterating against a visible frozen set will eventually overfit to it. If the stakes justify the extra labelling, a second sealed set of the same provenance helps against that; nobody discloses it, and the parties measure it once at acceptance. The contracted threshold still applies to the visible set, but a large gap between the two is a signal both sides agreed in advance to take seriously.

Second, drift belongs in it. Once accepted, the system is accepted at that model version in that environment. If the model vendor updates the model under the same name and the number moves, that is not a defect in the delivered work, because nothing about the work changed. It is not without consequence either, and someone bears it. Either you pin the version and accept that the vendor will eventually deprecate it, or you buy a maintenance contract that re-runs the acceptance data set on a schedule and says what happens when the number drops. Such a maintenance contract is a Dienstvertrag next to the Werkvertrag, and that structure fits the task. If you settle nothing, nobody bears it; that is the default, and it is how a system degrades unnoticed for a year. Decide before acceptance, because afterwards it is generally the customer who has to demonstrate the defect, which is one more reason to have the measurement in writing.

I am an engineer, not a lawyer. The wording above shows what makes a clause testable and is not a template to sign. Nothing here predicts how a given court would rule on a given contract. Have a lawyer draft the clause. My part is narrower, and a lawyer cannot do it for you: I make sure that the draft has a number in it and that a harness computes that number on demand.