197 lines
15 KiB
Markdown
197 lines
15 KiB
Markdown
# 5. Content values
|
||
|
||
Each node carries five content values, each for one job.
|
||
No value covers the node's own ID, its geometry, its provenance or its fidelity.
|
||
So re-witnessing a page or proofreading a sentence changes none of them, while changing one character changes the `text_hash` and `node_hash` of that sentence.
|
||
|
||
In this chapter *H* is SHA-256 written as hexadecimal; under a keyed scheme, *H* is HMAC-SHA-256 under the key ([5.8](#58-keyed-hashes)).
|
||
|
||
## 5.1 The five values
|
||
|
||
| Value | Definition | Used for |
|
||
| --- | --- | --- |
|
||
| `text_hash` | *H*(normalize(plain projection)) | Quotations and verbatim checks, where every character counts. |
|
||
| `match_hash` | *H*(strict projection) | Certifying that two witnesses agree, and recognizing the same text elsewhere. |
|
||
| `align_key` | *H*(lossy projection) | Proposing alignments. It never certifies anything. |
|
||
| `own_hash` | *H*(canonical JSON of the node's own content, [5.2](#52-own-content)) | Anchors on containers that have content of their own. |
|
||
| `node_hash` | *H*(canonical JSON of `{"own": own_hash, "children": [each child's node_hash, in order]}`) | Pins on leaves; the root hash; knowing that something below has changed. |
|
||
|
||
Normalization is that of [1.4](01-conventions.md#14-text-normalization-for-hashing).
|
||
Because each `node_hash` covers its children's, the root's `node_hash` covers the whole tree (it is a hash over canonical JSON of `{own, children}`, not an RFC 6962 Merkle root, which [8.5](08-fidelity.md#85-merkle-roots) defines for receipts): a correction to one sentence changes that sentence's values and the `node_hash` of each ancestor, and nothing else.
|
||
`hashes.jsonl` gives every value for every node of a small document, with each node's own content.
|
||
|
||
## 5.2 Own content
|
||
|
||
A node's own content is an object with its `kind` and these members, and nothing else:
|
||
|
||
| Kinds | Members |
|
||
| --- | --- |
|
||
| Kinds that take a label | `label`: the label, or null. |
|
||
| `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content. |
|
||
| `equation`, `equation-line` | `notation`, `source`. |
|
||
| `code`, `algorithm` | `language` (or null), `text`. |
|
||
| `section` | `level`, and `heading`: the `text_hash` of its heading, or null. |
|
||
| `table` | `columns`, as stored. |
|
||
| `cell` | `column`, `rowspan`, `colspan` and `role`, with defaults 1, 1 and `data` when absent. |
|
||
| `theorem` | `variant`, or null. |
|
||
|
||
So paragraphs, rows, lists, items and the other containers without content have the same `own_hash` as every node of their kind, and are never judged by it.
|
||
Own content covers stored values exactly; it is not normalized.
|
||
A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table.
|
||
|
||
## 5.3 The strict projection
|
||
|
||
The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement.
|
||
It is computed over a node's runs, or its math, as follows; for other nodes it is the strict projections of the text leaves joined with U+0020, then step 3.
|
||
|
||
1. Apply NFC to every string, then project each run:
|
||
|
||
| Run | Strict projection |
|
||
| --- | --- |
|
||
| `text` with `sup` | `^{` + inner + `}`, or only primes when the content is only primes ([5.4](#54-simple-notation)) |
|
||
| `text` with `sub` | `_{` + inner + `}` |
|
||
| other `text`, `link` | The text, with Unicode script characters folded and dashes before digits folded ([5.4](#54-simple-notation)) |
|
||
| `math`, `equation` | The canonical linear form when the source is simple; U+FFFC when it is complex |
|
||
| `cite`, `xref`, `fnmark` | U+FFF9 + the printed form with whitespace removed + U+FFFB |
|
||
|
||
Concatenate the results.
|
||
2. Apply NFKC, which folds ligatures, the micro sign and Greek variant letters.
|
||
3. Collapse whitespace to one U+0020 and trim, as [1.4](01-conventions.md#14-text-normalization-for-hashing) does.
|
||
|
||
Folding scripts before NFKC matters: NFKC alone would turn a superscript ² into a plain 2 and erase the structure the projection exists to keep.
|
||
Marking citations matters for the same reason: "10" followed by citation 3 must not agree with the text "103".
|
||
|
||
Each complex run also yields a complex key, `notation:source`, with the source in NFC and every whitespace character removed.
|
||
A node's complex keys are listed in order.
|
||
Two readings agree only when their `match_hash` values are equal and their complex keys are equal, run by run; this is stage 1 of canonical math.
|
||
A later minor version may add stage 2, which replaces U+FFFC with canonical MathML; that would change `match_hash` and `align_key` values but no `text_hash`, `own_hash` or `node_hash`, so no anchor would go stale.
|
||
|
||
## 5.4 Simple notation
|
||
|
||
A character is simple, after folding, when it is:
|
||
|
||
1. a letter: General Category L in the Latin blocks U+0041–U+024F and U+1E00–U+1EFF, or in the Greek blocks U+0370–U+03FF, without the Coptic letters U+03E2–U+03EF, and U+1F00–U+1FFF; or U+00B5 MICRO SIGN;
|
||
2. a digit: General Category Nd; or
|
||
3. one of `+ − = × ⋅ / < > ≤ ≥ ± ( ) ′ . ,`.
|
||
|
||
Inside notation, characters fold: U+002D and U+2013 to U+2212 MINUS SIGN; U+00B7, U+2219 and U+22C5 to U+22C5 DOT OPERATOR; U+0027 and U+2032 to U+2032 PRIME.
|
||
|
||
**The canonical linear form.**
|
||
Notation is a sequence of atoms; an atom has a base (possibly empty), a count of primes, an optional subscript and an optional superscript, each itself a sequence of atoms.
|
||
An atom is written as its base, then its primes as U+2032, then `_{`, the subscript and `}`, then `^{`, the superscript and `}`.
|
||
A superscript that holds only bare primes becomes primes on its atom.
|
||
There are no spaces in the form, and a script with no base attaches to the character before it, so all of `$H_2O$`, `H$_2$O`, `$\mathrm{H}_{2}\mathrm{O}$`, `H<sub>2</sub>O` and `H₂O` project to `H_{2}O`.
|
||
|
||
**TeX sources** are read as tokens: control words (`\` and ASCII letters, with following whitespace skipped), control symbols (`\` and one other character), and single characters, with whitespace and `~` ignored.
|
||
|
||
| Token | Meaning |
|
||
| --- | --- |
|
||
| `{` … `}` | A group; its atoms are spliced in. A script on a group of more than one atom makes the source complex; a script after an empty group applies to the atom before it, or to an empty base, so `{}^{14}C` is `^{14}C`. |
|
||
| `_`, `^` | A subscript or superscript of the previous atom (or of an empty base), taking one argument: a group, a command or a character. A second script of the same kind is complex. |
|
||
| `'` | A prime on the previous atom. |
|
||
| `\mathrm`, `\text`, `\textrm`, `\mathup` | A wrapper: its argument's atoms are spliced in. |
|
||
| `\rm` | Ignored. |
|
||
| `\,` `\;` `\:` `\!`, a backslash before a space, `\quad` `\qquad` `\enspace` `\thinspace` `\medspace` `\thickspace` | Spacing, ignored. |
|
||
| `\alpha` … `\omega`, `\varepsilon`, `\vartheta`, `\varkappa`, `\varpi`, `\varrho`, `\varsigma`, `\varphi`, `\Gamma` … `\Omega` | The Greek letter TeX typesets; `\epsilon` is ϵ and `\phi` is ϕ. |
|
||
| `\times` `\cdot` `\pm` `\le` `\leq` `\ge` `\geq` `\lt` `\gt` `\prime` `\AA` | × ⋅ ± ≤ ≤ ≥ ≥ < > ′ Å |
|
||
| `\left(`, `\right)` | Parentheses; any other delimiter is complex. |
|
||
| A simple character | An atom. |
|
||
| Anything else | The source is complex. |
|
||
|
||
**Typst sources** are read as tokens: strings in double quotes (complex if they contain a backslash), identifiers (runs of letters, with `.` joining modifiers such as `plus.minus`), numbers (digits with an optional fraction), the two-character operators `<=` and `>=`, and single characters, with whitespace ignored.
|
||
|
||
| Token | Meaning |
|
||
| --- | --- |
|
||
| `_`, `^` | A script of the previous atom; its argument is a parenthesized group, whose parentheses are dropped, or one token. |
|
||
| `(` … `)` elsewhere | Literal parentheses around a group. |
|
||
| `'` | A prime. |
|
||
| A string | Its characters, with whitespace removed. |
|
||
| A one-letter identifier, a number | Its characters. |
|
||
| `upright(…)` | A wrapper. |
|
||
| `alpha` … `omega`, `Alpha` … `Omega`, and `epsilon.alt`, `theta.alt`, `phi.alt`, `pi.alt`, `rho.alt`, `sigma.alt`, `kappa.alt`, `beta.alt` | The Greek letter Typst renders; `epsilon` is ε and `phi` is φ. |
|
||
| `plus` `minus` `times` `dot` `dot.op` `dot.c` `plus.minus` `lt` `gt` `lt.eq` `gt.eq` `eq` `prime` `slash`, and `<=` `>=` | + − × ⋅ ⋅ ⋅ ± < > ≤ ≥ = ′ / ≤ ≥ |
|
||
| `thin` `med` `thick` `quad` `wide` `space` | Spacing, ignored. |
|
||
| `/`, `*`, `#`, `&`, brackets, braces, any other identifier, and the operators `->`, `=>`, `!=`, `<-`, `==`, `:=` | Complex. In Typst `/` is a fraction. |
|
||
|
||
NFKC later folds the variant letters (ϵ to ε, ϕ to φ, ϑ to θ), so TeX's and Typst's defaults agree in the projection.
|
||
|
||
**Styled scripts.**
|
||
The inner form of a `sup` or `sub` text run is its text with whitespace removed and characters folded, when every character is simple or a Unicode script character; otherwise it is the text with whitespace collapsed and trimmed.
|
||
|
||
**Unicode script characters** are those whose decomposition mapping is tagged `<super>` or `<sub>`, such as ², ⁻ and ₂.
|
||
In text, each maximal run of script characters of one kind folds to `^{` or `_{`, the folded bases, and `}`, when every base is simple; otherwise the run is left for NFKC to flatten.
|
||
|
||
**Dashes in text.**
|
||
Outside notation, U+002D and U+2013 immediately before a digit become U+2212, so "10-20" and "10–20" agree; "CT–CT" keeps its dash.
|
||
|
||
`notation.jsonl` covers every rule, and `projections.jsonl` with `projection-groups.jsonl` covers H₂O, cm⁻¹, 10⁻³ and K_r in every spelling, ligatures, the micro sign, citation marks and "103" against 10³.
|
||
|
||
## 5.5 The lossy projection
|
||
|
||
The lossy projection starts from the strict projection and:
|
||
|
||
1. removes each `_{` and `^{` and the `}` that closes it, leaving the script's content in place, while other braces stay;
|
||
2. removes each mark, from U+FFF9 to the next U+FFFB;
|
||
3. applies Unicode full case folding, then NFKC;
|
||
4. deletes every whitespace character, every character of General Category Pd, and U+00AD SOFT HYPHEN.
|
||
|
||
So "103" and 10³ share an `align_key` while their `match_hash` values differ, and "bond- length", "bond-length" and "bond length" share one too.
|
||
A node's prose length is the number of code points in its lossy projection other than U+FFFC.
|
||
|
||
## 5.6 Alignment
|
||
|
||
Two readings of a document are aligned as whole sentence sequences, never paragraph by paragraph, because backends disagree about paragraphs at page and column breaks.
|
||
Each reading is a list of sentences in document order, each with its strict projection, lossy projection, complex keys, container and context (the kind of its container: paragraph, heading, caption and so on).
|
||
|
||
1. **Anchors.** A sentence is a candidate when its lossy projection occurs exactly once in each reading and its prose length is at least 20.
|
||
Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
|
||
2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections.
|
||
Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
|
||
3. **Compare** each pair.
|
||
They agree when their strict projections and complex keys are equal.
|
||
Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`.
|
||
The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`.
|
||
4. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
|
||
5. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
|
||
|
||
Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted.
|
||
`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order.
|
||
|
||
## 5.7 Suspects
|
||
|
||
A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note.
|
||
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved.
|
||
|
||
## 5.8 Keyed hashes
|
||
|
||
A hash of a short text is not a secret: a withheld cell reading "310.0" is recovered by hashing guesses.
|
||
So every content value that leaves a team's store is keyed.
|
||
|
||
| Where the value goes | `hashing` | Key |
|
||
| --- | --- | --- |
|
||
| The team's own store and bundles | `{"scheme": "plain"}` | None. |
|
||
| An export to another machine of the team, whatever its text policy | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
|
||
| A public layer, or an export outside the team on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
|
||
|
||
The destination chooses the scheme, not the text policy.
|
||
A checks-only package ([12.5](12-package.md#125-text-policies)) sent between the team's own machines is keyed with the team key, as `hale2026harmonic-damb.checks-only` is; sent to anyone outside the team, it is keyed with the page key.
|
||
|
||
A keyed scheme replaces SHA-256 with HMAC-SHA-256 under its key in every content value, including the Merkle construction of `node_hash`, and in `meta_hash`.
|
||
|
||
**The team key** is HKDF-SHA-256 (RFC 5869) with the team secret, at least 32 random bytes, as input keying material, the salt `scienting-team-key/1`, the info `scienting-hash/1` and length 32.
|
||
Its `key_id` is the first 16 hexadecimal digits of SHA-256 of the bytes `scienting-key-id/1` followed by the key.
|
||
|
||
**The page key** is HKDF-SHA-256 with the concatenated 32-byte page fingerprints of the witness, in page order, as input keying material, the salt `scienting-page-key/1`, the UTF-8 of the context as info, and length 32.
|
||
The context names the export, such as `layer:smithlab/kp4m2x7q`, so two exports never share a key.
|
||
The export lists each page as its page hash, SHA-256 of `scienting-page-hash/1` followed by the fingerprint's 32 bytes, never the fingerprint itself.
|
||
A reader who holds the paper computes the fingerprints from their own copy, checks them against the page hashes and derives the key; someone who does not hold it sees only hashes of hashes.
|
||
|
||
In keyed anchors, quotations become MACs over the normalized quotation and context texts become MACs over their normalized text ([6.3](06-anchors.md#63-keyed-anchors)).
|
||
Open-access documents under a license that allows redistribution need none of this.
|
||
`keyed.jsonl` gives the derivations and a case a non-holder cannot verify.
|
||
|
||
## 5.9 The metadata hash
|
||
|
||
`meta_hash` is *H* of the canonical JSON of `{work, version, relations, meta, language, license}` from `document.json`, with `license` null when absent.
|
||
Fixing an affiliation changes `meta_hash` and not the root.
|