infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: rewrite docs
check / check (default, macos-latest) (push) Canceled after 0s
Details
check / check (default, ubuntu-latest) (push) Canceled after 0s
Details
check / check (py310, macos-latest) (push) Canceled after 0s
Details
check / check (py310, ubuntu-latest) (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
alexm committed 2026-10-12 07:55:22 -04:00
1 parent 8b2df58266
commit 002e2c3552
140 files changed
+4555 -701
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/rationale.md
+208 -78
View File
@@ -1,150 +1,280 @@
# Rationale
Why the formats look the way they do.
Each choice below is one a reader is likely to question; the specification states the rule, and this page gives the reason.
Each choice below is one a reader is likely to question.
The specification states the rule, and this page gives the reason.
This page is for newcomers, implementers and experts.
Newcomers find each term defined where it first appears.
Implementers find the chapter that defines each rule.
Experts find the cost of each choice and, where a neighboring standard chose differently, how it differs (the full comparison is in [comparison.md](comparison.md)).
The formats are at version 0.1.0.
Of the 27 in `registry/formats.json`, 20 are `candidate` and 7 are `provisional`, and the provisional ones are named where they appear.
The repository is manuspec, at <https://git.scient.ing/infra/manuspec>.
The `scienting-` prefix in format names such as `scienting-anchor/1` is the formats' namespace, separate from the repository's name.
## Identity
**IDs carry no meaning.**
An ID that encodes a page, an ordinal or a kind changes when a better converter orders blocks differently, or when a paragraph turns out to be a caption.
An opaque name minted from the text survives both, and a fresh conversion of the same input still gets the same IDs.
### IDs carry no meaning
Every sentence, cell and other node has an ID, a short name that other records use to point at it.
An ID that encoded a page, an ordinal or a kind would change when a converter orders blocks differently, or when a paragraph turns out to be a caption.
An ID is minted once from the document key, the node's kind, the hash of its text and an occurrence count that separates identical texts, and is then frozen ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
A reclassification leaves it as it was, and a fresh conversion of the same input mints the same IDs.
A different converter mints different ones for some sentences: those the alignment pairs with the established reading keep their IDs, and every difference becomes a suspect for review ([spec/05 §5.6](../spec/05-hashes.md#56-alignment)).
A content address such as an [IPFS content identifier](https://docs.ipfs.tech/concepts/content-addressing/) or a [Software Heritage identifier](https://www.swhid.org/specification/v1.2/5.Core_identifiers/) is a function of content at all times.
That suits immutable data and archived source code.
Any correction would rename the sentence.
Here the ID is derived from content once and kept apart from the content hashes, so the pair of ID and hash says whether the text changed.
The cost is that two conversions that read a sentence differently by one character mint different IDs for it.
### IDs never move, and retired IDs leave a trail
**IDs never move, and retired IDs leave a trail.**
Everything a team builds points at IDs: notes, claims, dataset values, citations in drafts.
So a merge records its successor, a deletion records that there is none, and legacy IDs live on as aliases.
A merge records its successor, a deletion records that there is none, and legacy IDs live on as aliases, all in `ids.jsonl` ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
**Revisions are counted, not timed.**
Anchors are made on different machines and packages are imported later, and clocks disagree; a correction index does not.
### Revisions are counted, not timed
Anchors are made on different machines and packages are imported later, and clocks disagree.
A correction index does not disagree with itself.
A document's revision is the index of its last correction ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor), [spec/11](../spec/11-corrections.md)).
## Hashes and projections
**Five values, not one.**
Quotations need every character; certifying that two witnesses agree needs notation folded but structure kept; proposing that two sentences are the same needs something looser still; and containers need a hash of themselves apart from what they hold.
No single hash does all four jobs.
A hash is a fixed-length fingerprint of data: equal data gives equal hashes, and any change gives a different one.
Every hash here is SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
A projection is a rewritten form of a text, made only to be hashed.
**Only the strict projection certifies.**
OCR drops superscripts, so a projection loose enough to match "103" with 10³ would certify a thousandfold error.
The lossy key may propose that match; only the strict hash may accept it.
### Canonical JSON
**Citation marks stay marked in the strict projection.**
Otherwise "10" followed by citation 3 would agree with the text "103".
Canonical JSON is JSON (JavaScript Object Notation, a text format for structured data) written in exactly one way, so equal data always hashes the same.
The formats use the JSON Canonicalization Scheme, [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785) (an RFC, or Request for Comments, is a numbered Internet standards document), which requires numbers to be IEEE 754 doubles (IEEE is the Institute of Electrical and Electronics Engineers) (the standard binary floating-point numbers), and add one rule: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits ([spec/01 §1.1 and §1.2](../spec/01-conventions.md)).
The cost is that a reader must treat those strings as decimals itself.
**Decimal points and commas count as simple notation.**
Otherwise `$-1.5$` in a LaTeX source would be complex math and could never agree with "−1.5" in the PDF.
### Five values, not one
**A hyphen or en dash before a digit is a minus sign in the strict projection.**
Text layers and OCR disagree about dashes in "10–20" and "−5"; folding them before digits makes witnesses agree without touching "CT–CT".
A text node carries five values ([spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)), because no single hash does all of their jobs.
**The lossy key deletes whitespace and dashes.**
OCR merges and splits words and line-break hyphens; a key that only proposes alignments should forgive that.
- Quotations need every character, so `text_hash` covers the normalized text.
- Certifying that two witnesses agree needs notation folded but structure kept, which is `match_hash`.
A witness is one file the text was read from.
- Proposing that two sentences are the same needs something looser still, which is `align_key`.
- Containers need a hash of their own content, `own_hash`, apart from the `node_hash` that also covers what they hold.
**Offsets index the stored text, and structural hashes cover it exactly.**
When `node_hash` is unchanged, every stored offset is still exact, and a renderer can slice the stored text without normalizing it first.
Only the three text values normalize.
Table columns and figure-text items have IDs but are not nodes, so they carry no hashes ([spec/09 §9.1](../spec/09-figures-tables-equations.md)).
**Hashes that leave the team are keyed.**
A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages.
Inside a team, plain hashes are fine; anything exported is an HMAC under the team's key, or under a key derived from the paper's own pages, which only someone holding the paper can compute.
### Only the strict projection certifies
OCR (optical character recognition, which reads text from page images) drops superscripts.
A projection loose enough to match "103" with 10³ would certify an error, because the printed value is 1000 and the reading is 103.
The lossy key may propose that match.
Only the strict hash may accept it, and then only when the complex-math keys agree too ([spec/05 §5.3](../spec/05-hashes.md#53-the-strict-projection)).
### Citation marks stay marked in the strict projection
"10" followed by citation 3 would agree with the text "103" without the marks.
### Decimal points and commas count as simple notation
Simple math is rewritten to one linear form that can match plain text; anything else is complex and is compared by its source. `$-1.5$` in a LaTeX source would be complex math without the two characters, and could never agree with "−1.5" in the PDF ([spec/05 §5.4](../spec/05-hashes.md#54-simple-notation)).
### A hyphen or en dash before a digit is a minus sign in the strict projection
Text layers and OCR disagree about dashes in "10–20" and "−5".
Folding them before digits makes witnesses agree without touching a dash inside a name such as "CT–CT".
### The lossy key deletes whitespace and dashes
OCR merges and splits words and line-break hyphens, and a key that only proposes alignments should forgive that.
It also folds case ([spec/05 §5.5](../spec/05-hashes.md#55-the-lossy-projection)).
### Offsets index the stored text, and structural hashes cover it exactly
An offset counts Unicode code points, the numbered characters of Unicode, into the node's stored plain text ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)), and own content is hashed as stored.
Every stored offset is still exact when `node_hash` is unchanged, and a renderer can slice the stored text without normalizing it first.
Only the three text values, `text_hash`, `match_hash` and `align_key`, normalize.
### Hashes that leave the team are keyed
A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages [CHECK: the overlapping-anchors claim is the author's; spec/05 §5.8 gives only the guessing argument].
Inside a team, plain hashes are fine.
An export to another machine of the team uses an HMAC (hash-based message authentication code), a hash that needs a secret key, under the team key.
A public layer, or any export of a work whose text is withheld, uses a key derived from the paper's own page fingerprints, which only someone holding the paper can compute ([spec/05 §5.8](../spec/05-hashes.md#58-keyed-hashes)).
## Anchors
**Every anchor carries `own_hash`, and anchors on containers without content carry their leaves.**
Telling a relabel from a real change needs the anchored `own_hash`, and deciding whether a merged paragraph still holds "at least half" of what was anchored needs the original leaves.
An anchor is a stored pointer to a node or to a quoted span inside it, with enough context to find it again after the text changes ([spec/06](../spec/06-anchors.md)).
**Long quotations become a hash and a length.**
Quotations are capped at 200 code points to keep anchors small and to limit how much text an anchor carries; a hashed window can still be found again in the node.
The [World Wide Web Consortium (W3C) Web Annotation model](https://www.w3.org/TR/annotation-model/) also anchors with quotations: its `TextQuoteSelector` takes `exact`, `prefix` and `suffix`, and its `TextPositionSelector` is described there as very brittle.
An anchor here keeps the quote with up to 32 code points of prefix and suffix and adds hashes, so resolution has exact outcomes (`current`, `changed-below`, `moved`, `reattached`, `stale`).
Hypothesis, in [its description of its own anchoring](https://web.hypothes.is/blog/fuzzy-anchoring/), matches fuzzily.
**Prefix and suffix pick between repeated quotes.**
"The value" may occur twice in a sentence; the context around it, as in W3C text-quote selectors, decides which one was meant.
Annotation layers use a custom `AnchorSelector` followed by a standard `TextQuoteSelector` fallback for tools that do not know ours ([spec/18 §18.3](../spec/18-layers.md)).
The cost is a selector type no generic tool defines.
For annotation in general, the W3C model is the better choice.
**`changed-below` reports drift.**
A note on a section stays valid when a typo inside it is fixed, but not forever: once a large share of what is inside has changed, the note is flagged for review.
### Every anchor carries `own_hash`, and anchors on containers without content carry their leaves
**A row's header path is not hashed.**
Rows have no content of their own, so inserting a dropped row-group header does not stale the anchors on every cell beneath it.
Telling a relabel from a real change needs the anchored `own_hash`.
Deciding whether a merged paragraph still holds "at least half" of what was anchored needs the original leaves.
### Long quotations become a hash and a length
Quotations over 200 code points are replaced by a hash and a length, which keeps anchors small and limits how much text an anchor carries [CHECK: reason for the cap is the author's; spec/06 §6.2 gives the number but not the reason].
A hashed window can still be found again in the node.
### Prefix and suffix pick between repeated quotes
"The value" may occur twice in a sentence.
The context around it decides which one was meant.
### `changed-below` reports drift
A note on a section stays valid when a typo inside it is fixed.
By default the drift is `high` once the leaves touched by corrections since the anchor was made exceed a quarter of the leaves involved, and the note is then flagged for review ([spec/06 §6.6](../spec/06-anchors.md#66-drift)).
### A row's header path is not hashed
Rows have no content of their own, and a row's `header_path` is not part of it.
Inserting a dropped row-group header does not stale the anchors on every cell beneath it ([spec/09 §9.2](../spec/09-figures-tables-equations.md)).
## Corrections
**Corrections are appended, never edited.**
A document's history is its first reading plus its corrections, so anyone can see what changed, when, why and on what evidence.
### Corrections are appended, never edited
**There is a `set` operation.**
Readings drop spans, roles and header paths as often as they drop words, and those need a correction too.
A document's history is its first reading plus its corrections, so anyone can see what changed, when and why, and on what evidence ([spec/11](../spec/11-corrections.md)).
**Applied corrections record what they touched.**
With `path`, `to_path` and `touched` in each record, drift is computed from the records alone, with no need to replay history.
### Corrections include a `set` operation
**Digits need image evidence.**
A wrong digit is the most expensive error a scientific text can carry and the easiest for a model to introduce, so no digit changes without a crop of the page.
Readings drop spans, roles and header paths, not only words, and those need a correction too.
### Applied corrections record what they touched
Drift is computed from the records alone, with no need to replay history, because each record carries `path`, `to_path` and `touched` ([spec/11 §11.3](../spec/11-corrections.md#113-applied-fields)).
### Digits need image evidence
A wrong digit changes a result without looking wrong, so no digit changes without a crop of the page as evidence, and no leaf outside the correction's `touched` list may end up with different digits ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)).
## Geometry
**Pages have fingerprints.**
Publishers stamp each download, so file hashes cannot identify a paper across teammates' copies; fingerprints with the stamps removed can.
### Pages have fingerprints
**Fingerprint text deletes whitespace entirely.**
PDF text extractors disagree about where to insert spaces, and the fingerprint must come out the same in a converter and in a browser.
Publishers stamp each download, so file hashes cannot identify a paper across teammates' copies.
Fingerprints with the stamps removed can ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
### Fingerprint text deletes whitespace entirely
PDF text extractors disagree about where to insert spaces, and the fingerprint must come out the same in any two implementations.
## Review
**People sign receipts, not nodes.**
### People sign receipts
Nobody signs three thousand times, so one signature covers a Merkle root of everything reviewed in a sitting.
A Merkle tree hashes items in pairs, then those hashes in pairs, up to one root, and allows a proof that one node was covered without revealing the others.
The receipt tree follows [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), whose replacement [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162) defines the same tree.
The specification makes a receipt with no items invalid, where the RFC hashes the empty list ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
The tree is not the one behind `node_hash`, which hashes each node's `own_hash` with its children's `node_hash` values.
### Spot-check samples come from a signature nobody can steer
**Spot-check samples come from a signature nobody can steer.**
A seed the reviewer could influence could be ground until the sample fell on careful work.
The checker's deterministic Ed25519 signature over the receipt is unpredictable to the reviewer and unchoosable by the checker, and anyone can recompute it.
The checker's deterministic Ed25519 signature over the receipt is unpredictable to the reviewer and unchoosable by the checker, because the checker's key was registered for spot-checks before the original was signed.
Anyone can recompute the sample ([spec/08 §8.8](../spec/08-fidelity.md#88-spot-checks)).
**Error rates are reported as bounds.**
One clean sample of ten proves little; a Clopper–Pearson bound across all of a reviewer's samples says what can actually be claimed.
### Error rates are reported as bounds
**Levels fall back.**
A level counts only at the current text, so a correction never leaves a stale "proofread" behind.
One clean sample of ten proves little.
A one-sided Clopper–Pearson upper bound, by default at confidence 0.95, across all of a reviewer's samples says what can be claimed.
With no errors it is about 3 divided by the number sampled ([spec/08 §8.9](../spec/08-fidelity.md#89-reviewer-error-bounds)).
### Levels fall back
A node counts as proofread only through a valid receipt that lists it at its current `node_hash`, so a correction never leaves a stale "proofread" behind ([spec/08 §8.1](../spec/08-fidelity.md#81-levels-of-fidelity)).
## Rights
**Rights are recorded per witness, not per paper.**
### Rights are recorded per witness
What a team may do depends on how it obtained the paper: a subscription, a mining license, an open license, its own work.
**AI uses have two axes.**
Reviewers ask not only whether a model may see the text, but where it runs and whether the text is kept there.
### AI uses have two axes
**Unknown means no.**
A stage whose rights are unknown does not run; a project owner settles the question with a signed policy, so the decision is the team's and is on the record.
Reviewers ask whether a model may see the text, where it runs, and whether the text is kept there.
The record names a purpose, `internal` or `external` depending on whether results leave the organization, and a processor: `local`, `tenancy`, `outside-no-retention` or `outside` ([spec/14 §14.1](../spec/14-rights.md#141-the-rights-record)).
### Unknown means no
A stage whose rights are unknown does not run.
A project owner settles the question with a signed policy, so the decision is the team's and is on the record.
A policy fills in unknowns and never overrides an explicit answer.
**Backend licenses are part of the gate.**
A project setting can turn the refusal into a warning.
The refusal is the default, and the project format is `provisional` ([spec/20](../spec/20-projects.md)).
### Backend licenses are part of the gate
A converter's code and its model weights can be licensed differently, and the weights may exclude commercial or competing use.
A condition whose answer is unknown counts as not met ([spec/14 §14.4](../spec/14-rights.md#144-backends-and-profiles)).
## Packages and databases
**Canonical JSON, one record per line.**
A corrected sentence is a one-line diff, and every file has exactly one valid byte sequence, so a hash of the manifest pins the whole package.
### Canonical JSON, one record per line
**Checks-only packages drop everything that carries text.**
Corrections carry their find and replace strings, datasets carry values, figure text and reference records carry text; a checks-only export omits them all.
A corrected sentence is a one-line diff, and every file has exactly one valid byte sequence, so a hash of the manifest pins the whole package ([spec/01 §1.9](../spec/01-conventions.md#19-files), [spec/12 §12.3](../spec/12-package.md#123-the-manifest)).
**SQL avoids reserved words in every engine.**
The same tables run on SQLite, Dolt (MySQL-compatible) and PostgreSQL, where `key`, `index`, `before` and `by` are reserved; hence `doc_key`, `idx`, `before_hash` and `signer`.
A [Frictionless Data Package](https://datapackage.org/standard/data-package/) lists resources with an optional `hash` that defaults to MD5 (an older hash function).
The package manifest here lists every other file with SHA-256 and size.
The dataset format, which is `provisional`, is a Data Package with units, uncertainty, an origin and anchors to the printed values added ([spec/15](../spec/15-datasets.md)).
Two neighboring standards cover other parts of this ground, RO-Crate and the PROV Ontology.
[RO-Crate](https://www.researchobject.org/ro-crate/) describes research data with schema.org annotations in JSON-LD, and it is the better choice for general description and discovery.
No mapping between it and these packages is defined.
Provenance is a plain `provenance.json` of tools, backends, cached readings and stages.
It is not the [PROV Ontology](https://www.w3.org/TR/prov-o/), which other systems can query generically.
### Checks-only packages drop everything that carries text
Corrections carry their find and replace strings, datasets carry values, suspects carry the readings compared, and figure text and reference records carry text.
A checks-only export omits them all, withholds the content of every leaf, and keys its hashes ([spec/12 §12.5](../spec/12-package.md#125-text-policies)).
### SQL avoids reserved words in every engine
The same tables run on SQLite, Dolt (MySQL-compatible) and PostgreSQL, and no name is a word reserved in any of the three.
The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash` ([spec/13 §13.1](../spec/13-database.md#131-portability)).
### Loading and unloading are exact inverses
**Loading and unloading are exact inverses.**
A team can keep its documents in a database and still hand anyone the package, byte for byte.
The conformance suite checks this for every fixture ([spec/13 §13.3](../spec/13-database.md#133-load-and-unload)).
## Questions the formats leave open
These are settled for now, but a future version may revisit them.
These are settled for now, and a future version may revisit them.
| Question | Current answer |
| --- | --- |
| Should node IDs carry a kind prefix, such as `s-k3m9q2a`, to help people debugging? | No: opaque IDs survive reclassification. |
| Should `text_hash` normalize to NFC, or only collapse whitespace as Lode's extracts do? | NFC and whitespace; existing Lode extracts keep Lode's rule. |
| Should node IDs carry a kind prefix, such as `s-k3m9q2a`, to help people debugging? | No: a reclassified node keeps its ID, so a prefix would become wrong. |
| Should `text_hash` normalize Unicode, or only collapse whitespace? | Both: Unicode Normalization Form C, then whitespace collapsed to single spaces and trimmed ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)). |
| How long may a quotation in an anchor be? | 200 code points. |
| How is complex math compared? | By its source, run by run; comparing canonical MathML is a possible later addition. |
| What extension and media type do zipped packages use? | `.sct.zip` and `application/vnd.scienting.doc+zip`. |
| Which drift threshold, spot-check rate and digit rule should teams use? | A quarter of a container's leaves; 5% of a receipt's nodes, at least ten; and `all`, `anchored-sections` or `none`, set per project. |
| Which drift threshold, spot-check rate and digit rule should teams use? | By default, drift is high above a quarter of the leaves involved, and spot-checks sample 5% of a receipt's nodes, at least ten. The digit rule is `all`, `anchored-sections` or `none`; each is set per project. |
| How should rights combine when a document comes from witnesses held on different bases? | The most restrictive witness governs each use. |
## Limits worth knowing
## Known limits
- Notation drops spaces and text keeps them, so `$T = 300$` from a LaTeX source and "T = 300" from a PDF disagree, and become a `word` suspect for review.
- Page fingerprints are computed from extracted text lines; two programs that should agree on fingerprints must extract lines the same way.
- The roster is defined here and is meant to be the same one Lode verifies; the manifest is meant to have the shape of Lode's `lode-cache/1`.
- Page fingerprints are computed from extracted text lines.
Two implementations that should agree on fingerprints must extract lines the same way.
- The formats are built for a team-scale corpus of hundreds of documents.
They do not replace publisher interchange formats, and they do not fix what a converter reads wrongly.
They record it and make it checkable.