paperspec/spec/05-hashes.md

5. Content values

Each node carries five content values, each for one job. No value covers the node's own ID, its geometry, its provenance or its fidelity. So re-witnessing a page or proofreading a sentence changes none of them, while changing one character changes the text_hash and node_hash of that sentence.

In this chapter H is SHA-256 written as hexadecimal; under a keyed scheme, H is HMAC-SHA-256 under the key (5.8).

5.1 The five values

Value	Definition	Used for
text_hash	H(normalize(plain projection))	Quotations and verbatim checks, where every character counts.
match_hash	H(strict projection)	Certifying that two witnesses agree, and recognizing the same text elsewhere.
align_key	H(lossy projection)	Proposing alignments. It never certifies anything.
own_hash	H(canonical JSON of the node's own content, 5.2)	Anchors on containers that have content of their own.
node_hash	H(canonical JSON of {"own": own_hash, "children": [each child's node_hash, in order]})	Pins on leaves; the root hash; knowing that something below has changed.

Normalization is that of 1.4. Because each node_hash covers its children's, the root's node_hash covers the whole tree (it is a hash over canonical JSON of {own, children}, not an RFC 6962 Merkle root, which 8.5 defines for receipts): a correction to one sentence changes that sentence's values and the node_hash of each ancestor, and nothing else. hashes.jsonl gives every value for every node of a small document, with each node's own content.

5.2 Own content

A node's own content is an object with its kind and these members, and nothing else:

Kinds	Members
Kinds that take a label	label: the label, or null.
sentence, cell, reference, authors, keywords	runs: the runs as stored, except that each equation run is reduced to {"kind": "equation"}, since an ID is never content.
equation, equation-line	notation, source.
code, algorithm	language (or null), text.
section	level, and heading: the text_hash of its heading, or null.
table	columns, as stored.
cell	column, rowspan, colspan and role, with defaults 1, 1 and data when absent.
theorem	variant, or null.

So paragraphs, rows, lists, items and the other containers without content have the same own_hash as every node of their kind, and are never judged by it. Own content covers stored values exactly; it is not normalized. A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table.

5.3 The strict projection

The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement. It is computed over a node's runs, or its math, as follows; for other nodes it is the strict projections of the text leaves joined with U+0020, then step 3.

1. Apply NFC to every string, then project each run:

   Run	Strict projection
   text with sup	^{ + inner + }, or only primes when the content is only primes (5.4)
   text with sub	_{ + inner + }
   other text, link	The text, with Unicode script characters folded and dashes before digits folded (5.4)
   math, equation	The canonical linear form when the source is simple; U+FFFC when it is complex
   cite, xref, fnmark	U+FFF9 + the printed form with whitespace removed + U+FFFB

   Concatenate the results.

2. Apply NFKC, which folds ligatures, the micro sign and Greek variant letters.

3. Collapse whitespace to one U+0020 and trim, as 1.4 does.

Folding scripts before NFKC matters: NFKC alone would turn a superscript ² into a plain 2 and erase the structure the projection exists to keep. Marking citations matters for the same reason: "10" followed by citation 3 must not agree with the text "103".

Each complex run also yields a complex key, notation:source, with the source in NFC and every whitespace character removed. A node's complex keys are listed in order. Two readings agree only when their match_hash values are equal and their complex keys are equal, run by run; this is stage 1 of canonical math. A later minor version may add stage 2, which replaces U+FFFC with canonical MathML; that would change match_hash and align_key values but no text_hash, own_hash or node_hash, so no anchor would go stale.

5.4 Simple notation

A character is simple, after folding, when it is:

1. a letter: General Category L in the Latin blocks U+0041–U+024F and U+1E00–U+1EFF, or in the Greek blocks U+0370–U+03FF, without the Coptic letters U+03E2–U+03EF, and U+1F00–U+1FFF; or U+00B5 MICRO SIGN;
2. a digit: General Category Nd; or
3. one of + − = × ⋅ / < > ≤ ≥ ± ( ) ′ . ,.

Inside notation, characters fold: U+002D and U+2013 to U+2212 MINUS SIGN; U+00B7, U+2219 and U+22C5 to U+22C5 DOT OPERATOR; U+0027 and U+2032 to U+2032 PRIME.

The canonical linear form. Notation is a sequence of atoms; an atom has a base (possibly empty), a count of primes, an optional subscript and an optional superscript, each itself a sequence of atoms. An atom is written as its base, then its primes as U+2032, then _{, the subscript and }, then ^{, the superscript and }. A superscript that holds only bare primes becomes primes on its atom. There are no spaces in the form, and a script with no base attaches to the character before it, so all of $H_2O$, H$_2$O, $\mathrm{H}_{2}\mathrm{O}$, H<sub>2</sub>O and H₂O project to H_{2}O.

TeX sources are read as tokens: control words (\ and ASCII letters, with following whitespace skipped), control symbols (\ and one other character), and single characters, with whitespace and ~ ignored.

Token	Meaning
{ … }	A group; its atoms are spliced in. A script on a group of more than one atom makes the source complex; a script after an empty group applies to the atom before it, or to an empty base, so {}^{14}C is ^{14}C.
_, ^	A subscript or superscript of the previous atom (or of an empty base), taking one argument: a group, a command or a character. A second script of the same kind is complex.
'	A prime on the previous atom.
\mathrm, \text, \textrm, \mathup	A wrapper: its argument's atoms are spliced in.
\rm	Ignored.
\, \; \: \!, a backslash before a space, \quad \qquad \enspace \thinspace \medspace \thickspace	Spacing, ignored.
\alpha … \omega, \varepsilon, \vartheta, \varkappa, \varpi, \varrho, \varsigma, \varphi, \Gamma … \Omega	The Greek letter TeX typesets; \epsilon is ϵ and \phi is ϕ.
\times \cdot \pm \le \leq \ge \geq \lt \gt \prime \AA	× ⋅ ± ≤ ≤ ≥ ≥ < > ′ Å
\left(, \right)	Parentheses; any other delimiter is complex.
A simple character	An atom.
Anything else	The source is complex.

Typst sources are read as tokens: strings in double quotes (complex if they contain a backslash), identifiers (runs of letters, with . joining modifiers such as plus.minus), numbers (digits with an optional fraction), the two-character operators <= and >=, and single characters, with whitespace ignored.

Token	Meaning
_, ^	A script of the previous atom; its argument is a parenthesized group, whose parentheses are dropped, or one token.
( … ) elsewhere	Literal parentheses around a group.
'	A prime.
A string	Its characters, with whitespace removed.
A one-letter identifier, a number	Its characters.
upright(…)	A wrapper.
alpha … omega, Alpha … Omega, and epsilon.alt, theta.alt, phi.alt, pi.alt, rho.alt, sigma.alt, kappa.alt, beta.alt	The Greek letter Typst renders; epsilon is ε and phi is φ.
plus minus times dot dot.op dot.c plus.minus lt gt lt.eq gt.eq eq prime slash, and <= >=	+ − × ⋅ ⋅ ⋅ ± < > ≤ ≥ = ′ / ≤ ≥
thin med thick quad wide space	Spacing, ignored.
/, *, #, &, brackets, braces, any other identifier, and the operators ->, =>, !=, <-, ==, :=	Complex. In Typst / is a fraction.

NFKC later folds the variant letters (ϵ to ε, ϕ to φ, ϑ to θ), so TeX's and Typst's defaults agree in the projection.

Styled scripts. The inner form of a sup or sub text run is its text with whitespace removed and characters folded, when every character is simple or a Unicode script character; otherwise it is the text with whitespace collapsed and trimmed.

Unicode script characters are those whose decomposition mapping is tagged <super> or <sub>, such as ², ⁻ and ₂. In text, each maximal run of script characters of one kind folds to ^{ or _{, the folded bases, and }, when every base is simple; otherwise the run is left for NFKC to flatten.

Dashes in text. Outside notation, U+002D and U+2013 immediately before a digit become U+2212, so "10-20" and "10–20" agree; "CT–CT" keeps its dash.

notation.jsonl covers every rule, and projections.jsonl with projection-groups.jsonl covers H₂O, cm⁻¹, 10⁻³ and K_r in every spelling, ligatures, the micro sign, citation marks and "103" against 10³.

5.5 The lossy projection

The lossy projection starts from the strict projection and:

1. removes each _{ and ^{ and the } that closes it, leaving the script's content in place, while other braces stay;
2. removes each mark, from U+FFF9 to the next U+FFFB;
3. applies Unicode full case folding, then NFKC;
4. deletes every whitespace character, every character of General Category Pd, and U+00AD SOFT HYPHEN.

So "103" and 10³ share an align_key while their match_hash values differ, and "bond- length", "bond-length" and "bond length" share one too. A node's prose length is the number of code points in its lossy projection other than U+FFFC.

5.6 Alignment

Two readings of a document are aligned as whole sentence sequences, never paragraph by paragraph, because backends disagree about paragraphs at page and column breaks. Each reading is a list of sentences in document order, each with its strict projection, lossy projection, complex keys, container and context (the kind of its container: paragraph, heading, caption and so on).

1. Anchors. A sentence is a candidate when its lossy projection occurs exactly once in each reading and its prose length is at least 20. Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
2. Between anchors, and before the first and after the last, align by the longest common subsequence of lossy projections. Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
3. Compare each pair. They agree when their strict projections and complex keys are equal. Otherwise the pair is a suspect: of kind number when their numeric skeletons differ, else equation when their complex keys differ, else word. The numeric skeleton keeps only digits, U+2212, ., ,, +, _{, ^{ and }.
4. Structure. A pair whose contexts differ is also a structure suspect; two consecutive pairs that share a container in one reading but not in the other are a join suspect.
5. Unplaced sentences are dropped (in the first reading only) or added (in the second only).

Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted. alignment.jsonl aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order.

5.7 Suspects

A suspect, scienting-suspect/1, records a doubtful reading for review: its id, its kind (word, number, equation, structure, join, added, dropped, boundary for an uncertain sentence boundary, or reading for anything else), the node, the readings compared, an optional crop by SHA-256 and a note. Suspects may travel in a package's suspects.jsonl, sorted by ID, until they are resolved.

5.8 Keyed hashes

A hash of a short text is not a secret: a withheld cell reading "310.0" is recovered by hashing guesses. So every content value that leaves a team's store is keyed.

Where the value goes	hashing	Key
The team's own store and bundles	{"scheme": "plain"}	None.
An export to another machine of the team, whatever its text policy	{"scheme": "hmac-team", "key_id": …}	The team key.
A public layer, or an export outside the team on a work whose text is withheld	{"scheme": "hkdf-pages", "context": …, "pages": […]}	A key derived from the witness's page fingerprints.

The destination chooses the scheme, not the text policy. A checks-only package (12.5) sent between the team's own machines is keyed with the team key, as hale2026harmonic.checks-only is; sent to anyone outside the team, it is keyed with the page key.

A keyed scheme replaces SHA-256 with HMAC-SHA-256 under its key in every content value, including the Merkle construction of node_hash, and in meta_hash.

The team key is HKDF-SHA-256 (RFC 5869) with the team secret, at least 32 random bytes, as input keying material, the salt scienting-team-key/1, the info scienting-hash/1 and length 32. Its key_id is the first 16 hexadecimal digits of SHA-256 of the bytes scienting-key-id/1 followed by the key.

The page key is HKDF-SHA-256 with the concatenated 32-byte page fingerprints of the witness, in page order, as input keying material, the salt scienting-page-key/1, the UTF-8 of the context as info, and length 32. The context names the export, such as layer:smithlab/kp4m2x7q, so two exports never share a key. The export lists each page as its page hash, SHA-256 of scienting-page-hash/1 followed by the fingerprint's 32 bytes, never the fingerprint itself. A reader who holds the paper computes the fingerprints from their own copy, checks them against the page hashes and derives the key; someone who does not hold it sees only hashes of hashes.

In keyed anchors, quotations become MACs over the normalized quotation and context texts become MACs over their normalized text (6.3). Open-access documents under a license that allows redistribution need none of this. keyed.jsonl gives the derivations and a case a non-holder cannot verify.

5.9 The metadata hash

meta_hash is H of the canonical JSON of {work, version, relations, meta, language, license} from document.json, with license null when absent. Fixing an affiliation changes meta_hash and not the root.