Initial commit

spec/08-fidelity.md

@@ -0,0 +1,124 @@
+# 8. Fidelity, receipts and spot-checks
+
+## 8.1 Levels of fidelity
+
+| Level | Meaning |
+| --- | --- |
+| F0 converted | One reading, not corroborated. |
+| F1 resolved | Every suspect in the node was answered against the page image and passed the gate. |
+| F2 corroborated | An independent witness agrees, by `match_hash` and complex keys, or the text is born digital, as a Typst or LaTeX source is. |
+| F3 proofread | A named person compared the node with the page image, and a valid receipt they signed lists the node at its current `node_hash`. |
+
+Levels apply to leaves.
+The pipeline records F0, F1 or F2 in a leaf's `fidelity`; F3 comes only from receipts.
+A correction that changes a leaf drops it back: the gate's crop evidence makes it F1 at most, until it is corroborated or proofread again.
+
+## 8.2 Levels
+
+A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`.
+A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does.
+A document reports, in `document.json`, the lowest level among its leaves and the count at each level.
+
+## 8.3 Receipts
+
+A paper has thousands of nodes, and nobody signs thousands of times, so people sign receipts: one signature over a Merkle root of everything reviewed in a sitting.
+
+| Property | Required | Holds |
+| --- | --- | --- |
+| `format` | Yes | `scienting-receipt/1`. |
+| `id` | Yes | The record ID ([4.6](04-identity.md#46-record-ids)). |
+| `doc` | Yes | The document key. |
+| `rev` | Yes | The revision reviewed. |
+| `level` | Yes | `F1`, `F2` or `F3`. |
+| `scope` | Yes | `all`, `section:<id>`, `page:<n>`, `targeted:<name>` (such as `targeted:lode`), or `qa:<receipt id>` for a spot-check. |
+| `nodes` | Yes | `[[id, node_hash], …]`, sorted by ID without repeats. |
+| `records` | No | `[[kind, id, hash], …]` for other things signed in the sitting: `annotation`, `digitization`, `correction`, `layer` or `checklist`. |
+| `root` | Yes | The Merkle root ([8.5](#85-merkle-roots)). |
+| `evidence` | Yes | What the reviewer looked at: `{kind, sha256, node?, page_fp?, box?, note?}` with `kind` `crop`, `witness` or `source-file`, or `{kind: "person", by}`. A regenerated crop is different evidence. |
+| `by` | Yes | The signer's roster key. |
+| `at` | Yes | When it was signed. |
+| `qa` | For `qa:` scopes | The spot-check ([8.8](#88-spot-checks)). |
+| `export_of` | No | The receipt this one re-issues under a keyed scheme. |
+| `keyed` | No | The scheme its values are keyed under, when it leaves the team. |
+| `signature` | Yes | `{scheme, key, value}` ([8.4](#84-record-hashes-and-signatures)). |
+
+A receipt lists nodes, records, or both.
+A correction invalidates a receipt for the corrected node only, because the node's `node_hash` no longer matches; the rest still counts.
+
+A receipt shared outside the team cannot carry plain node hashes, which would leak short texts, and its signature covers them.
+So an export re-issues it: a new receipt with `export_of`, values under the export's key, signed by the original reviewer or a team key with the `export` purpose.
+
+## 8.4 Record hashes and signatures
+
+A signed record's record hash is SHA-256 of the canonical JSON of the record without `id` and `signature`.
+Its `id` is the 13-character letter-first encoding of that hash.
+Its signing payload is the canonical JSON of `{"format": <the record's format>, "hash": <the record hash>}`.
+
+| Scheme | `value` |
+| --- | --- |
+| `ed25519` | The RFC 8032 signature of the payload, in standard Base64 with padding. |
+| `ssh` | An SSH signature of the payload in the namespace `scienting`, armored. |
+| `gpg` | A detached, armored OpenPGP signature of the payload. |
+| `sigstore` | A Sigstore bundle for the payload, as JSON text. |
+
+`signature.key` is the signer's roster key and equals `by`.
+Self-hosted deployments sign with team keys or a private Sigstore deployment; the public Sigstore instance, whose log permanently records each signer's identity, is for public layers only.
+`signatures.jsonl` verifies one receipt and the RFC 8032 test vector.
+
+## 8.5 Merkle roots
+
+A receipt's items are `["node", id, node_hash]` for each listed node, in order, followed by its `records`.
+The root follows RFC 6962: a leaf hash is SHA-256 of the byte 0x00 and the item's canonical JSON; an interior hash is SHA-256 of 0x01 and the two child hashes; a list of *n* > 1 items splits after the largest power of two less than *n*.
+A root over one item is that item's leaf hash; a receipt with no items is invalid.
+The tree allows a proof that one node was covered without revealing the others; `merkle.jsonl` gives roots for one to seven items.
+
+## 8.6 Receipt status
+
+A receipt is:
+
+1. **invalid** when its root, ID or signature does not verify, or its signer's key was not valid when it was signed (`registered_at` ≤ `at` < `revoked_at`);
+2. otherwise **revoked** when a valid revocation names it, or names its signer with a period that contains its `at`;
+3. otherwise **suspended** when a valid spot-check of it found an error;
+4. otherwise **valid**.
+
+Only valid receipts set levels.
+A suspended receipt stays suspended; its reviewer reviews the sitting again and signs a new receipt, which needs its own spot-check.
+
+## 8.7 Revocations and the roster
+
+A revocation, `scienting-revocation/1`, names one `receipt`, or a `signer` and a `period` `{from, to}`, with a `reason`, and is signed like a receipt.
+It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or an owner of the project.
+
+The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
+A package MAY carry the roster its receipts need as `roster.json`, so its receipts verify without the store.
+The roster is meant to match the one Lode already verifies; aligning the two is recorded in [DECISIONS.md](../DECISIONS.md).
+
+## 8.8 Spot-checks
+
+For each F3 receipt, a second reviewer checks a sample of its nodes and signs a `qa:` receipt.
+A seed the reviewer controls could be ground, by varying the receipt until the sample falls on careful work, so the seed is a signature nobody can steer:
+
+1. The seed is the checker's Ed25519 signature over the canonical JSON of `{"format": "scienting-spot-check/1", "receipt": <the original's record hash>}`, made with a key registered in the roster with the `spot-check` purpose before the original was signed.
+   Ed25519 signatures are deterministic, so the checker cannot choose it, and the reviewer cannot predict it.
+2. The sample size is the smaller of *n* and the larger of `minimum` and ⌈`rate` × *n*⌉, for the original's *n* nodes; the defaults are a rate of `0.05` and a minimum of 10.
+3. Rank the original's node IDs by the hexadecimal SHA-256 of the seed's 64 bytes followed by the ID's UTF-8, then by ID, and take the first ones.
+
+The `qa` object records `of`, `key`, `seed` (Base64), `rate`, `minimum`, the `sample` sorted by ID, and the `errors` found, each `{node, kind?, note?}`.
+The `qa:` receipt's `nodes` are the sampled nodes at the hashes the original lists.
+Anyone can recompute the sample; `spotcheck.jsonl` does.
+
+## 8.9 Reviewer error bounds
+
+One sample proves little; what a buyer wants is a bound on each reviewer's error rate across all their samples.
+With *k* errors in *n* sampled nodes, the bound is the one-sided Clopper–Pearson upper confidence bound at confidence *c*, by default 0.95: the rate *p* at which the probability of at most *k* errors in *n* is 1 − *c*, or 1 when *k* = *n*.
+It is reported to six decimal places, rounded up.
+With no errors the bound is about 3/*n*, so zero errors in 300 sampled nodes bounds a rate below 1%; `bounds.jsonl` gives cases.
+
+A QA report, `scienting-qa-report/1`, states for each reviewer who signed a delivery the nodes sampled, the errors found, the confidence, the method `clopper-pearson` and the bound.
+
+## 8.10 Targeted F3
+
+The target set of a document is every node that a Lode extract, a dataset value, a confirmed `claim-link`, a draft quotation or a pinned citation anchors, plus every display equation, plus every sentence with a digit in a `text` or `math` run.
+Digits in `cite`, `xref` and `fnmark` runs do not count.
+A project sets the digit rule to `all`, `anchored-sections` (only sentences in a section that holds an anchored node) or `none`.
+Neighbors shown for context are not in the set, and nothing new comes to rest on a node below F3 without the node joining the set.