infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First implementation (#1)
check / check (push) Successful in 42s
Details
site / site (push) Successful in 3m4s
Details

Browse Source 
Reviewed-on: #1
This commit was merged in pull request #1.
This commit is contained in:
alexm committed 2026-10-15 05:47:05 -04:00
1 parent 34c0abb26a
commit 4fa326dc76
534 files changed
+28280 -37
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

spec/05-hashes.md
+196
View File
@@ -0,0 +1,196 @@
# 5. Content values
Each node carries five content values, each for one job.
No value covers the node's own ID, its geometry, its provenance or its fidelity.
So re-witnessing a page or proofreading a sentence changes none of them, while changing one character changes the `text_hash` and `node_hash` of that sentence.
In this chapter *H* is SHA-256 written as hexadecimal; under a keyed scheme, *H* is HMAC-SHA-256 under the key ([5.8](#58-keyed-hashes)).
## 5.1 The five values
| Value | Definition | Used for |
| --- | --- | --- |
| `text_hash` | *H*(normalize(plain projection)) | Quotations and verbatim checks, where every character counts. |
| `match_hash` | *H*(strict projection) | Certifying that two witnesses agree, and recognizing the same text elsewhere. |
| `align_key` | *H*(lossy projection) | Proposing alignments. It never certifies anything. |
| `own_hash` | *H*(canonical JSON of the node's own content, [5.2](#52-own-content)) | Anchors on containers that have content of their own. |
| `node_hash` | *H*(canonical JSON of `{"own": own_hash, "children": [each child's node_hash, in order]}`) | Pins on leaves; the root hash; knowing that something below has changed. |
Normalization is that of [1.4](01-conventions.md#14-text-normalization-for-hashing).
Because each `node_hash` covers its children's, the root's `node_hash` covers the whole tree (it is a hash over canonical JSON of `{own, children}`, not an RFC 6962 Merkle root, which [8.5](08-fidelity.md#85-merkle-roots) defines for receipts): a correction to one sentence changes that sentence's values and the `node_hash` of each ancestor, and nothing else.
`hashes.jsonl` gives every value for every node of a small document, with each node's own content.
## 5.2 Own content
A node's own content is an object with its `kind` and these members, and nothing else:
| Kinds | Members |
| --- | --- |
| Kinds that take a label | `label`: the label, or null. |
| `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content. |
| `equation`, `equation-line` | `notation`, `source`. |
| `code`, `algorithm` | `language` (or null), `text`. |
| `section` | `level`, and `heading`: the `text_hash` of its heading, or null. |
| `table` | `columns`, as stored. |
| `cell` | `column`, `rowspan`, `colspan` and `role`, with defaults 1, 1 and `data` when absent. |
| `theorem` | `variant`, or null. |
So paragraphs, rows, lists, items and the other containers without content have the same `own_hash` as every node of their kind, and are never judged by it.
Own content covers stored values exactly; it is not normalized.
A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table.
## 5.3 The strict projection
The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement.
It is computed over a node's runs, or its math, as follows; for other nodes it is the strict projections of the text leaves joined with U+0020, then step 3.
1. Apply NFC to every string, then project each run:
| Run | Strict projection |
| --- | --- |
| `text` with `sup` | `^{` + inner + `}`, or only primes when the content is only primes ([5.4](#54-simple-notation)) |
| `text` with `sub` | `_{` + inner + `}` |
| other `text`, `link` | The text, with Unicode script characters folded and dashes before digits folded ([5.4](#54-simple-notation)) |
| `math`, `equation` | The canonical linear form when the source is simple; U+FFFC when it is complex |
| `cite`, `xref`, `fnmark` | U+FFF9 + the printed form with whitespace removed + U+FFFB |
Concatenate the results.
2. Apply NFKC, which folds ligatures, the micro sign and Greek variant letters.
3. Collapse whitespace to one U+0020 and trim, as [1.4](01-conventions.md#14-text-normalization-for-hashing) does.
Folding scripts before NFKC matters: NFKC alone would turn a superscript ² into a plain 2 and erase the structure the projection exists to keep.
Marking citations matters for the same reason: "10" followed by citation 3 must not agree with the text "103".
Each complex run also yields a complex key, `notation:source`, with the source in NFC and every whitespace character removed.
A node's complex keys are listed in order.
Two readings agree only when their `match_hash` values are equal and their complex keys are equal, run by run; this is stage 1 of canonical math.
A later minor version may add stage 2, which replaces U+FFFC with canonical MathML; that would change `match_hash` and `align_key` values but no `text_hash`, `own_hash` or `node_hash`, so no anchor would go stale.
## 5.4 Simple notation
A character is simple, after folding, when it is:
1. a letter: General Category L in the Latin blocks U+0041–U+024F and U+1E00–U+1EFF, or in the Greek blocks U+0370–U+03FF, without the Coptic letters U+03E2–U+03EF, and U+1F00–U+1FFF; or U+00B5 MICRO SIGN;
2. a digit: General Category Nd; or
3. one of `+ − = × ⋅ / < > ≤ ≥ ± ( ) ′ . ,`.
Inside notation, characters fold: U+002D and U+2013 to U+2212 MINUS SIGN; U+00B7, U+2219 and U+22C5 to U+22C5 DOT OPERATOR; U+0027 and U+2032 to U+2032 PRIME.
**The canonical linear form.**
Notation is a sequence of atoms; an atom has a base (possibly empty), a count of primes, an optional subscript and an optional superscript, each itself a sequence of atoms.
An atom is written as its base, then its primes as U+2032, then `_{`, the subscript and `}`, then `^{`, the superscript and `}`.
A superscript that holds only bare primes becomes primes on its atom.
There are no spaces in the form, and a script with no base attaches to the character before it, so all of `$H_2O$`, `H$_2$O`, `$\mathrm{H}_{2}\mathrm{O}$`, `H<sub>2</sub>O` and `H₂O` project to `H_{2}O`.
**TeX sources** are read as tokens: control words (`\` and ASCII letters, with following whitespace skipped), control symbols (`\` and one other character), and single characters, with whitespace and `~` ignored.
| Token | Meaning |
| --- | --- |
| `{` … `}` | A group; its atoms are spliced in. A script on a group of more than one atom makes the source complex; a script after an empty group applies to the atom before it, or to an empty base, so `{}^{14}C` is `^{14}C`. |
| `_`, `^` | A subscript or superscript of the previous atom (or of an empty base), taking one argument: a group, a command or a character. A second script of the same kind is complex. |
| `'` | A prime on the previous atom. |
| `\mathrm`, `\text`, `\textrm`, `\mathup` | A wrapper: its argument's atoms are spliced in. |
| `\rm` | Ignored. |
| `\,` `\;` `\:` `\!`, a backslash before a space, `\quad` `\qquad` `\enspace` `\thinspace` `\medspace` `\thickspace` | Spacing, ignored. |
| `\alpha` … `\omega`, `\varepsilon`, `\vartheta`, `\varkappa`, `\varpi`, `\varrho`, `\varsigma`, `\varphi`, `\Gamma` … `\Omega` | The Greek letter TeX typesets; `\epsilon` is ϵ and `\phi` is ϕ. |
| `\times` `\cdot` `\pm` `\le` `\leq` `\ge` `\geq` `\lt` `\gt` `\prime` `\AA` | × ⋅ ± ≤ ≤ ≥ ≥ < > ′ Å |
| `\left(`, `\right)` | Parentheses; any other delimiter is complex. |
| A simple character | An atom. |
| Anything else | The source is complex. |
**Typst sources** are read as tokens: strings in double quotes (complex if they contain a backslash), identifiers (runs of letters, with `.` joining modifiers such as `plus.minus`), numbers (digits with an optional fraction), the two-character operators `<=` and `>=`, and single characters, with whitespace ignored.
| Token | Meaning |
| --- | --- |
| `_`, `^` | A script of the previous atom; its argument is a parenthesized group, whose parentheses are dropped, or one token. |
| `(` … `)` elsewhere | Literal parentheses around a group. |
| `'` | A prime. |
| A string | Its characters, with whitespace removed. |
| A one-letter identifier, a number | Its characters. |
| `upright(…)` | A wrapper. |
| `alpha` … `omega`, `Alpha` … `Omega`, and `epsilon.alt`, `theta.alt`, `phi.alt`, `pi.alt`, `rho.alt`, `sigma.alt`, `kappa.alt`, `beta.alt` | The Greek letter Typst renders; `epsilon` is ε and `phi` is φ. |
| `plus` `minus` `times` `dot` `dot.op` `dot.c` `plus.minus` `lt` `gt` `lt.eq` `gt.eq` `eq` `prime` `slash`, and `<=` `>=` | + − × ⋅ ⋅ ⋅ ± < > ≤ ≥ = ′ / ≤ ≥ |
| `thin` `med` `thick` `quad` `wide` `space` | Spacing, ignored. |
| `/`, `*`, `#`, `&`, brackets, braces, any other identifier, and the operators `->`, `=>`, `!=`, `<-`, `==`, `:=` | Complex. In Typst `/` is a fraction. |
NFKC later folds the variant letters (ϵ to ε, ϕ to φ, ϑ to θ), so TeX's and Typst's defaults agree in the projection.
**Styled scripts.**
The inner form of a `sup` or `sub` text run is its text with whitespace removed and characters folded, when every character is simple or a Unicode script character; otherwise it is the text with whitespace collapsed and trimmed.
**Unicode script characters** are those whose decomposition mapping is tagged `<super>` or `<sub>`, such as ², ⁻ and ₂.
In text, each maximal run of script characters of one kind folds to `^{` or `_{`, the folded bases, and `}`, when every base is simple; otherwise the run is left for NFKC to flatten.
**Dashes in text.**
Outside notation, U+002D and U+2013 immediately before a digit become U+2212, so "10-20" and "10–20" agree; "CT–CT" keeps its dash.
`notation.jsonl` covers every rule, and `projections.jsonl` with `projection-groups.jsonl` covers H₂O, cm⁻¹, 10⁻³ and K_r in every spelling, ligatures, the micro sign, citation marks and "103" against 10³.
## 5.5 The lossy projection
The lossy projection starts from the strict projection and:
1. removes each `_{` and `^{` and the `}` that closes it, leaving the script's content in place, while other braces stay;
2. removes each mark, from U+FFF9 to the next U+FFFB;
3. applies Unicode full case folding, then NFKC;
4. deletes every whitespace character, every character of General Category Pd, and U+00AD SOFT HYPHEN.
So "103" and 10³ share an `align_key` while their `match_hash` values differ, and "bond- length", "bond-length" and "bond length" share one too.
A node's prose length is the number of code points in its lossy projection other than U+FFFC.
## 5.6 Alignment
Two readings of a document are aligned as whole sentence sequences, never paragraph by paragraph, because backends disagree about paragraphs at page and column breaks.
Each reading is a list of sentences in document order, each with its strict projection, lossy projection, complex keys, container and context (the kind of its container: paragraph, heading, caption and so on).
1. **Anchors.** A sentence is a candidate when its lossy projection occurs exactly once in each reading and its prose length is at least 20.
Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections.
Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
3. **Compare** each pair.
They agree when their strict projections and complex keys are equal.
Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`.
The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`.
4. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
5. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted.
`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order.
## 5.7 Suspects
A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note.
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved.
## 5.8 Keyed hashes
A hash of a short text is not a secret: a withheld cell reading "310.0" is recovered by hashing guesses.
So every content value that leaves a team's store is keyed.
| Where the value goes | `hashing` | Key |
| --- | --- | --- |
| The team's own store and bundles | `{"scheme": "plain"}` | None. |
| An export to another machine of the team, whatever its text policy | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
| A public layer, or an export outside the team on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
The destination chooses the scheme, not the text policy.
A checks-only package ([12.5](12-package.md#125-text-policies)) sent between the team's own machines is keyed with the team key, as `hale2026harmonic-damb.checks-only` is; sent to anyone outside the team, it is keyed with the page key.
A keyed scheme replaces SHA-256 with HMAC-SHA-256 under its key in every content value, including the Merkle construction of `node_hash`, and in `meta_hash`.
**The team key** is HKDF-SHA-256 (RFC 5869) with the team secret, at least 32 random bytes, as input keying material, the salt `scienting-team-key/1`, the info `scienting-hash/1` and length 32.
Its `key_id` is the first 16 hexadecimal digits of SHA-256 of the bytes `scienting-key-id/1` followed by the key.
**The page key** is HKDF-SHA-256 with the concatenated 32-byte page fingerprints of the witness, in page order, as input keying material, the salt `scienting-page-key/1`, the UTF-8 of the context as info, and length 32.
The context names the export, such as `layer:smithlab/kp4m2x7q`, so two exports never share a key.
The export lists each page as its page hash, SHA-256 of `scienting-page-hash/1` followed by the fingerprint's 32 bytes, never the fingerprint itself.
A reader who holds the paper computes the fingerprints from their own copy, checks them against the page hashes and derives the key; someone who does not hold it sees only hashes of hashes.
In keyed anchors, quotations become MACs over the normalized quotation and context texts become MACs over their normalized text ([6.3](06-anchors.md#63-keyed-anchors)).
Open-access documents under a license that allows redistribution need none of this.
`keyed.jsonl` gives the derivations and a case a non-holder cannot verify.
## 5.9 The metadata hash
`meta_hash` is *H* of the canonical JSON of `{work, version, relations, meta, language, license}` from `document.json`, with `license` null when absent.
Fixing an affiliation changes `meta_hash` and not the root.