151 lines
8.5 KiB
Markdown
151 lines
8.5 KiB
Markdown
# Rationale
|
||
|
||
Why the formats look the way they do.
|
||
Each choice below is one a reader is likely to question; the specification states the rule, and this page gives the reason.
|
||
|
||
## Identity
|
||
|
||
**IDs carry no meaning.**
|
||
An ID that encodes a page, an ordinal or a kind changes when a better converter orders blocks differently, or when a paragraph turns out to be a caption.
|
||
An opaque name minted from the text survives both, and a fresh conversion of the same input still gets the same IDs.
|
||
|
||
**IDs never move, and retired IDs leave a trail.**
|
||
Everything a team builds points at IDs: notes, claims, dataset values, citations in drafts.
|
||
So a merge records its successor, a deletion records that there is none, and legacy IDs live on as aliases.
|
||
|
||
**Revisions are counted, not timed.**
|
||
Anchors are made on different machines and packages are imported later, and clocks disagree; a correction index does not.
|
||
|
||
## Hashes and projections
|
||
|
||
**Five values, not one.**
|
||
Quotations need every character; certifying that two witnesses agree needs notation folded but structure kept; proposing that two sentences are the same needs something looser still; and containers need a hash of themselves apart from what they hold.
|
||
No single hash does all four jobs.
|
||
|
||
**Only the strict projection certifies.**
|
||
OCR drops superscripts, so a projection loose enough to match "103" with 10³ would certify a thousandfold error.
|
||
The lossy key may propose that match; only the strict hash may accept it.
|
||
|
||
**Citation marks stay marked in the strict projection.**
|
||
Otherwise "10" followed by citation 3 would agree with the text "103".
|
||
|
||
**Decimal points and commas count as simple notation.**
|
||
Otherwise `$-1.5$` in a LaTeX source would be complex math and could never agree with "−1.5" in the PDF.
|
||
|
||
**A hyphen or en dash before a digit is a minus sign in the strict projection.**
|
||
Text layers and OCR disagree about dashes in "10–20" and "−5"; folding them before digits makes witnesses agree without touching "CT–CT".
|
||
|
||
**The lossy key deletes whitespace and dashes.**
|
||
OCR merges and splits words and line-break hyphens; a key that only proposes alignments should forgive that.
|
||
|
||
**Offsets index the stored text, and structural hashes cover it exactly.**
|
||
When `node_hash` is unchanged, every stored offset is still exact, and a renderer can slice the stored text without normalizing it first.
|
||
Only the three text values normalize.
|
||
|
||
**Hashes that leave the team are keyed.**
|
||
A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages.
|
||
Inside a team, plain hashes are fine; anything exported is an HMAC under the team's key, or under a key derived from the paper's own pages, which only someone holding the paper can compute.
|
||
|
||
## Anchors
|
||
|
||
**Every anchor carries `own_hash`, and anchors on containers without content carry their leaves.**
|
||
Telling a relabel from a real change needs the anchored `own_hash`, and deciding whether a merged paragraph still holds "at least half" of what was anchored needs the original leaves.
|
||
|
||
**Long quotations become a hash and a length.**
|
||
Quotations are capped at 200 code points to keep anchors small and to limit how much text an anchor carries; a hashed window can still be found again in the node.
|
||
|
||
**Prefix and suffix pick between repeated quotes.**
|
||
"The value" may occur twice in a sentence; the context around it, as in W3C text-quote selectors, decides which one was meant.
|
||
|
||
**`changed-below` reports drift.**
|
||
A note on a section stays valid when a typo inside it is fixed, but not forever: once a large share of what is inside has changed, the note is flagged for review.
|
||
|
||
**A row's header path is not hashed.**
|
||
Rows have no content of their own, so inserting a dropped row-group header does not stale the anchors on every cell beneath it.
|
||
|
||
## Corrections
|
||
|
||
**Corrections are appended, never edited.**
|
||
A document's history is its first reading plus its corrections, so anyone can see what changed, when, why and on what evidence.
|
||
|
||
**There is a `set` operation.**
|
||
Readings drop spans, roles and header paths as often as they drop words, and those need a correction too.
|
||
|
||
**Applied corrections record what they touched.**
|
||
With `path`, `to_path` and `touched` in each record, drift is computed from the records alone, with no need to replay history.
|
||
|
||
**Digits need image evidence.**
|
||
A wrong digit is the most expensive error a scientific text can carry and the easiest for a model to introduce, so no digit changes without a crop of the page.
|
||
|
||
## Geometry
|
||
|
||
**Pages have fingerprints.**
|
||
Publishers stamp each download, so file hashes cannot identify a paper across teammates' copies; fingerprints with the stamps removed can.
|
||
|
||
**Fingerprint text deletes whitespace entirely.**
|
||
PDF text extractors disagree about where to insert spaces, and the fingerprint must come out the same in a converter and in a browser.
|
||
|
||
## Review
|
||
|
||
**People sign receipts, not nodes.**
|
||
Nobody signs three thousand times, so one signature covers a Merkle root of everything reviewed in a sitting.
|
||
|
||
**Spot-check samples come from a signature nobody can steer.**
|
||
A seed the reviewer could influence could be ground until the sample fell on careful work.
|
||
The checker's deterministic Ed25519 signature over the receipt is unpredictable to the reviewer and unchoosable by the checker, and anyone can recompute it.
|
||
|
||
**Error rates are reported as bounds.**
|
||
One clean sample of ten proves little; a Clopper–Pearson bound across all of a reviewer's samples says what can actually be claimed.
|
||
|
||
**Levels fall back.**
|
||
A level counts only at the current text, so a correction never leaves a stale "proofread" behind.
|
||
|
||
## Rights
|
||
|
||
**Rights are recorded per witness, not per paper.**
|
||
What a team may do depends on how it obtained the paper: a subscription, a mining license, an open license, its own work.
|
||
|
||
**AI uses have two axes.**
|
||
Reviewers ask not only whether a model may see the text, but where it runs and whether the text is kept there.
|
||
|
||
**Unknown means no.**
|
||
A stage whose rights are unknown does not run; a project owner settles the question with a signed policy, so the decision is the team's and is on the record.
|
||
A policy fills in unknowns and never overrides an explicit answer.
|
||
|
||
**Backend licenses are part of the gate.**
|
||
A converter's code and its model weights can be licensed differently, and the weights may exclude commercial or competing use.
|
||
|
||
## Packages and databases
|
||
|
||
**Canonical JSON, one record per line.**
|
||
A corrected sentence is a one-line diff, and every file has exactly one valid byte sequence, so a hash of the manifest pins the whole package.
|
||
|
||
**Checks-only packages drop everything that carries text.**
|
||
Corrections carry their find and replace strings, datasets carry values, figure text and reference records carry text; a checks-only export omits them all.
|
||
|
||
**SQL avoids reserved words in every engine.**
|
||
The same tables run on SQLite, Dolt (MySQL-compatible) and PostgreSQL, where `key`, `index`, `before` and `by` are reserved; hence `doc_key`, `idx`, `before_hash` and `signer`.
|
||
|
||
**Loading and unloading are exact inverses.**
|
||
A team can keep its documents in a database and still hand anyone the package, byte for byte.
|
||
|
||
## Questions the formats leave open
|
||
|
||
These are settled for now, but a future version may revisit them.
|
||
|
||
| Question | Current answer |
|
||
| --- | --- |
|
||
| Should node IDs carry a kind prefix, such as `s-k3m9q2a`, to help people debugging? | No: opaque IDs survive reclassification. |
|
||
| Should `text_hash` normalize to NFC, or only collapse whitespace as Lode's extracts do? | NFC and whitespace; existing Lode extracts keep Lode's rule. |
|
||
| How long may a quotation in an anchor be? | 200 code points. |
|
||
| How is complex math compared? | By its source, run by run; comparing canonical MathML is a possible later addition. |
|
||
| What extension and media type do zipped packages use? | `.sct.zip` and `application/vnd.scienting.doc+zip`. |
|
||
| Which drift threshold, spot-check rate and digit rule should teams use? | A quarter of a container's leaves; 5% of a receipt's nodes, at least ten; and `all`, `anchored-sections` or `none`, set per project. |
|
||
| How should rights combine when a document comes from witnesses held on different bases? | The most restrictive witness governs each use. |
|
||
|
||
## Limits worth knowing
|
||
|
||
- Notation drops spaces and text keeps them, so `$T = 300$` from a LaTeX source and "T = 300" from a PDF disagree, and become a `word` suspect for review.
|
||
- Page fingerprints are computed from extracted text lines; two programs that should agree on fingerprints must extract lines the same way.
|
||
- The roster is defined here and is meant to be the same one Lode verifies; the manifest is meant to have the shape of Lode's `lode-cache/1`.
|