paperspec/spec/08-fidelity.md

# 8. Fidelity, receipts and spot-checks

## 8.1 Levels of fidelity

| Level | Meaning |
| --- | --- |
| F0 converted | One reading, not corroborated. |
| F1 resolved | Every suspect in the node was answered against the page image and passed the gate. |
| F2 corroborated | An independent witness agrees, by `match_hash` and complex keys, or the text is born digital, as a Typst or LaTeX source is. |
| F3 proofread | A named person compared the node with the page image, and a valid receipt they signed lists the node at its current `node_hash`. |

Levels apply to leaves.
The pipeline records F0, F1 or F2 in a leaf's `fidelity`; F3 comes only from receipts.
A correction that changes a leaf drops it back: the gate's crop evidence makes it F1 at most, until it is corroborated or proofread again.

## 8.2 Levels

A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`.
A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does.
A document reports, in `document.json`, the lowest level among its leaves and the count at each level.

## 8.3 Receipts

A paper has thousands of nodes, and nobody signs thousands of times, so people sign receipts: one signature over a Merkle root of everything reviewed in a sitting.

| Property | Required | Holds |
| --- | --- | --- |
| `format` | Yes | `scienting-receipt/1`. |
| `id` | Yes | The record ID ([4.6](04-identity.md#46-record-ids)). |
| `doc` | Yes | The document key. |
| `rev` | Yes | The revision reviewed. |
| `level` | Yes | `F1`, `F2` or `F3`. |
| `scope` | Yes | `all`, `section:<id>`, `page:<n>`, `targeted:<name>` (such as `targeted:lode`), or `qa:<receipt id>` for a spot-check. |
| `nodes` | Yes | `[[id, node_hash], …]`, sorted by ID without repeats. |
| `records` | No | `[[kind, id, hash], …]` for other things signed in the sitting: `annotation`, `digitization`, `correction`, `layer` or `checklist`. |
| `root` | Yes | The Merkle root ([8.5](#85-merkle-roots)). |
| `evidence` | Yes | What the reviewer looked at: `{kind, sha256, node?, page_fp?, box?, note?}` with `kind` `crop`, `witness` or `source-file`, or `{kind: "person", by}`. A regenerated crop is different evidence. |
| `by` | Yes | The signer's roster key. |
| `at` | Yes | When it was signed. |
| `qa` | For `qa:` scopes | The spot-check ([8.8](#88-spot-checks)). |
| `export_of` | No | The receipt this one re-issues under a keyed scheme. |
| `keyed` | No | The scheme its values are keyed under, when it leaves the team. |
| `signature` | Yes | `{scheme, key, value}` ([8.4](#84-record-hashes-and-signatures)). |

A receipt lists nodes, records, or both.
A correction invalidates a receipt for the corrected node only, because the node's `node_hash` no longer matches; the rest still counts.

A receipt shared outside the team cannot carry plain node hashes, which would leak short texts, and its signature covers them.
So an export re-issues it: a new receipt with `export_of`, values under the export's key, signed by the original reviewer or a team key with the `export` purpose.

## 8.4 Record hashes and signatures

A signed record's record hash is SHA-256 of the canonical JSON of the record without `id` and `signature`.
Its `id` is the 13-character letter-first encoding of that hash.
Its signing payload is the canonical JSON of `{"format": <the record's format>, "hash": <the record hash>}`.

| Scheme | `value` |
| --- | --- |
| `ed25519` | The RFC 8032 signature of the payload, in standard Base64 with padding. |
| `ssh` | An SSH signature of the payload in the namespace `scienting`, armored. |
| `gpg` | A detached, armored OpenPGP signature of the payload. |
| `sigstore` | A Sigstore bundle for the payload, as JSON text. |

`signature.key` is the signer's roster key and equals `by`.
Self-hosted deployments sign with team keys or a private Sigstore deployment; the public Sigstore instance, whose log permanently records each signer's identity, is for public layers only.
`signatures.jsonl` verifies one receipt and the RFC 8032 test vector.

## 8.5 Merkle roots

A receipt's items are `["node", id, node_hash]` for each listed node, in order, followed by its `records`.
The root follows RFC 6962: a leaf hash is SHA-256 of the byte 0x00 and the item's canonical JSON; an interior hash is SHA-256 of 0x01 and the two child hashes; a list of *n* > 1 items splits after the largest power of two less than *n*.
RFC 6962 has since been replaced by RFC 9162, which defines the same tree.
A root over one item is that item's leaf hash; a receipt with no items is invalid.
The tree allows a proof that one node was covered without revealing the others; `merkle.jsonl` gives roots for one to seven items.

## 8.6 Receipt status

A receipt is:

1. **invalid** when its root, ID or signature does not verify, or its signer's key was not valid when it was signed (`registered_at` ≤ `at` < `revoked_at`);
2. otherwise **revoked** when a valid revocation names it, or names its signer with a period that contains its `at`;
3. otherwise **suspended** when a valid spot-check of it found an error;
4. otherwise **valid**.

Only valid receipts set levels.
A suspended receipt stays suspended; its reviewer reviews the sitting again and signs a new receipt, which needs its own spot-check.

## 8.7 Revocations and the roster

A revocation, `scienting-revocation/1`, names one `receipt`, or a `signer` and a `period` `{from, to}`, with a `reason`, and is signed like a receipt.
It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or an owner of the project.

The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
A package MAY carry the roster its receipts need as `roster.json`, so its receipts verify without the store.
The roster is meant to be the one a store verifies against; see the [rationale](../docs/rationale.md#known-limits).

## 8.8 Spot-checks

For each F3 receipt, a second reviewer checks a sample of its nodes and signs a `qa:` receipt.
A seed the reviewer controls could be ground, by varying the receipt until the sample falls on careful work, so the seed is a signature nobody can steer:

1. The seed is the checker's Ed25519 signature over the canonical JSON of `{"format": "scienting-spot-check/1", "receipt": <the original's record hash>}`, made with a key registered in the roster with the `spot-check` purpose before the original was signed.
   Ed25519 signatures are deterministic, so the checker cannot choose it, and the reviewer cannot predict it.
2. The sample size is the smaller of *n* and the larger of `minimum` and ⌈`rate` × *n*⌉, for the original's *n* nodes; the defaults are a rate of `0.05` and a minimum of 10.
3. Rank the original's node IDs by the hexadecimal SHA-256 of the seed's 64 bytes followed by the ID's UTF-8, then by ID, and take the first ones.

The `qa` object records `of`, `key`, `seed` (Base64), `rate`, `minimum`, the `sample` sorted by ID, and the `errors` found, each `{node, kind?, note?}`.
The `qa:` receipt's `nodes` are the sampled nodes at the hashes the original lists.
Anyone can recompute the sample; `spotcheck.jsonl` does.

## 8.9 Reviewer error bounds

One sample proves little; what someone relying on the work needs is a bound on each reviewer's error rate across all their samples.
With *k* errors in *n* sampled nodes, the bound is the one-sided Clopper–Pearson upper confidence bound at confidence *c*, by default 0.95: the rate *p* at which the probability of at most *k* errors in *n* is 1 − *c*, or 1 when *k* = *n*.
It is reported to six decimal places, rounded up.
With no errors the bound is about 3/*n*, so zero errors in 300 sampled nodes bounds a rate below 1%; `bounds.jsonl` gives cases.

A QA report, `scienting-qa-report/1`, states for each reviewer who signed a delivery the nodes sampled, the errors found, the confidence, the method `clopper-pearson` and the bound.

## 8.10 Targeted F3

The target set of a document is every node that an extract, a dataset value, a confirmed `claim-link`, a draft quotation or a pinned citation anchors, plus every display equation, plus every sentence with a digit in a `text` or `math` run.
Digits in `cite`, `xref` and `fnmark` runs do not count.
A project sets the digit rule to `all`, `anchored-sections` (only sentences in a section that holds an anchored node) or `none`.
Neighbors shown for context are not in the set, and nothing new comes to rest on a node below F3 without the node joining the set.