paperspec/docs/rationale.md

# Rationale

Each choice below is one a reader is likely to question.
The specification states the rule, and this page gives the reason.

This page is for newcomers, implementers and experts.
Newcomers find each term defined where it first appears.
Implementers find the chapter that defines each rule.
Experts find the cost of each choice and, where a neighboring standard chose differently, how it differs (the full comparison is in [comparison.md](comparison.md)).

The formats are at version 0.1.0.
Of the 29 in `registry/formats.json`, 19 are `candidate` and 13 are `provisional`, and the provisional ones are named where they appear.
The repository is paperspec, at <https://git.scient.ing/infra/paperspec>.
The `scienting-` prefix in format names such as `scienting-anchor/1` is the formats' namespace, separate from the repository's name.

## Identity

### IDs carry no meaning

Every sentence, cell and other node has an ID, a short name that other records use to point at it.
An ID that encoded a page, an ordinal or a kind would change when a converter orders blocks differently, or when a paragraph turns out to be a caption.

An ID is minted once from the document key, the node's kind, the hash of its text and an occurrence count that separates identical texts, and is then frozen ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
A reclassification leaves it as it was, and a fresh conversion of the same input mints the same IDs.
A different converter mints different ones for some sentences: those the alignment pairs with the established reading keep their IDs, and every difference becomes a suspect for review ([spec/05 §5.6](../spec/05-hashes.md#56-alignment)).

A content address such as an [IPFS content identifier](https://docs.ipfs.tech/concepts/content-addressing/) or a [Software Heritage identifier](https://www.swhid.org/specification/v1.2/5.Core_identifiers/) is a function of content at all times.
That suits immutable data and archived source code.
Any correction would rename the sentence.
Here the ID is derived from content once and kept apart from the content hashes, so the pair of ID and hash says whether the text changed.

The cost is that two conversions that read a sentence differently by one character mint different IDs for it.

### IDs never move, and retired IDs leave a trail

Everything a team builds points at IDs: notes, claims, dataset values, citations in drafts.
A merge records its successor, a deletion records that there is none, and legacy IDs live on as aliases, all in `ids.jsonl` ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).

### Work keys carry a hash

A citation key such as `vaswani2017attention` is what readers say, and it collides: a preprint and its published version, or two papers by one first author in one year with the same first title word, would share it.
Four characters of a hash of the work's identity, its DOI when it has one, settle the collision and let any tool holding the DOI recompute the key.
The key is minted once and never recomputed, so a corrected title cannot change a name someone has cited; when two works still share four characters, the later one gets six ([spec/02 §2.4](../spec/02-model.md#24-work-and-document-keys)).
The rule is the one the studio's reading-list tool already uses, so a paper keeps its key from the reading list to the store.

### Identifiers are tag URIs

Layers, annotations and documents name themselves with `tag:scient.ing,2026:` URIs (RFC 4151) rather than `urn:` names, because a URN namespace must be registered with IANA and this one is not.
A tag URI needs only a domain and a date, and these identifiers sit inside hashed annotations, so the cheap time to choose was before release.

### Revisions are counted, not timed

Anchors are made on different machines and packages are imported later, and clocks disagree.
A correction index does not disagree with itself.
A document's revision is the index of its last correction ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor), [spec/11](../spec/11-corrections.md)).

## Hashes and projections

A hash is a fixed-length fingerprint of data: equal data gives equal hashes, and any change gives a different one.
Every hash here is SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
A projection is a rewritten form of a text, made only to be hashed.

### Canonical JSON

Canonical JSON is JSON (JavaScript Object Notation, a text format for structured data) written in exactly one way, so equal data always hashes the same.
The formats use the JSON Canonicalization Scheme, [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785) (an RFC, or Request for Comments, is a numbered Internet standards document), which requires numbers to be IEEE 754 doubles (IEEE is the Institute of Electrical and Electronics Engineers) (the standard binary floating-point numbers), and add one rule: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits ([spec/01 §1.1 and §1.2](../spec/01-conventions.md)).
The cost is that a reader must treat those strings as decimals itself.

### Five values, not one

A text node carries five values ([spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)), because no single hash does all of their jobs.

- Quotations need every character, so `text_hash` covers the normalized text.
- Certifying that two witnesses agree needs notation folded but structure kept, which is `match_hash`.
  A witness is one file the text was read from.
- Proposing that two sentences are the same needs something looser still, which is `align_key`.
- Containers need a hash of their own content, `own_hash`, apart from the `node_hash` that also covers what they hold.

Table columns and figure-text items have IDs but are not nodes, so they carry no hashes ([spec/09 §9.1](../spec/09-figures-tables-equations.md)).

### Only the strict projection certifies

OCR (optical character recognition, which reads text from page images) drops superscripts.
A projection loose enough to match "103" with 10³ would certify an error, because the printed value is 1000 and the reading is 103.
The lossy key may propose that match.
Only the strict hash may accept it, and then only when the complex-math keys agree too ([spec/05 §5.3](../spec/05-hashes.md#53-the-strict-projection)).

### Citation marks stay marked in the strict projection

"10" followed by citation 3 would agree with the text "103" without the marks.

### Decimal points and commas count as simple notation

Simple math is rewritten to one linear form that can match plain text; anything else is complex and is compared by its source. `$-1.5$` in a LaTeX source would be complex math without the two characters, and could never agree with "−1.5" in the PDF ([spec/05 §5.4](../spec/05-hashes.md#54-simple-notation)).

### A hyphen or en dash before a digit is a minus sign in the strict projection

Text layers and OCR disagree about dashes in "10–20" and "−5".
Folding them before digits makes witnesses agree without touching a dash inside a name such as "CT–CT".

### The lossy key deletes whitespace and dashes

OCR merges and splits words and line-break hyphens, and a key that only proposes alignments should forgive that.
It also folds case ([spec/05 §5.5](../spec/05-hashes.md#55-the-lossy-projection)).

### Offsets index the stored text, and structural hashes cover it exactly

An offset counts Unicode code points, the numbered characters of Unicode, into the node's stored plain text ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)), and own content is hashed as stored.
Every stored offset is still exact when `node_hash` is unchanged, and a renderer can slice the stored text without normalizing it first.
Only the three text values, `text_hash`, `match_hash` and `align_key`, normalize.

### The character data is pinned to one Unicode version

Normalization and General Categories change between Unicode versions, and both enter every hash.
The chapters pin Unicode 16.0, the reference tools refuse to run on another version, and moving to a later one is a change to every text-hashing format ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).

### Hashes that leave the team are keyed

A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages.
Inside a team, plain hashes are fine.

An export to another machine of the team uses an HMAC (hash-based message authentication code), a hash that needs a secret key, under the team key.
A public layer, or any export of a work whose text is withheld, uses a key derived from the paper's own page fingerprints, which only someone holding the paper can compute ([spec/05 §5.8](../spec/05-hashes.md#58-keyed-hashes)).

## Anchors

An anchor is a stored pointer to a node or to a quoted span inside it, with enough context to find it again after the text changes ([spec/06](../spec/06-anchors.md)).

The [World Wide Web Consortium (W3C) Web Annotation model](https://www.w3.org/TR/annotation-model/) also anchors with quotations: its `TextQuoteSelector` takes `exact`, `prefix` and `suffix`, and its `TextPositionSelector` is described there as very brittle.
An anchor here keeps the quote with up to 32 code points of prefix and suffix and adds hashes, so resolution has exact outcomes (`current`, `changed-below`, `moved`, `reattached`, `stale`).
Hypothesis, in [its description of its own anchoring](https://web.hypothes.is/blog/fuzzy-anchoring/), matches fuzzily.

Annotation layers use a custom `AnchorSelector` followed by a standard `TextQuoteSelector` fallback for tools that do not know ours ([spec/18 §18.3](../spec/18-layers.md)).
The cost is a selector type no generic tool defines.
For annotation in general, the W3C model is the better choice.

### Every anchor carries `own_hash`, and anchors on containers without content carry their leaves

Telling a relabel from a real change needs the anchored `own_hash`.
Deciding whether a merged paragraph still holds "at least half" of what was anchored needs the original leaves.

### Long quotations become a hash and a length

Quotations over 200 code points are replaced by a hash and a length, which keeps anchors small and limits how much text an anchor carries.
A hashed window can still be found again in the node.

### Prefix and suffix pick between repeated quotes

"The value" may occur twice in a sentence.
The context around it decides which one was meant.

### `changed-below` reports drift

A note on a section stays valid when a typo inside it is fixed.
By default the drift is `high` once the leaves touched by corrections since the anchor was made exceed a quarter of the leaves involved, and the note is then flagged for review ([spec/06 §6.6](../spec/06-anchors.md#66-drift)).

### A row's header path is not hashed

Rows have no content of their own, and a row's `header_path` is not part of it.
Inserting a dropped row-group header does not stale the anchors on every cell beneath it ([spec/09 §9.2](../spec/09-figures-tables-equations.md)).

## Corrections

### Corrections are appended, never edited

A document's history is its first reading plus its corrections, so anyone can see what changed, when and why, and on what evidence ([spec/11](../spec/11-corrections.md)).

### Corrections include a `set` operation

Readings drop spans, roles and header paths, not only words, and those need a correction too.

### Applied corrections record what they touched

Drift is computed from the records alone, with no need to replay history, because each record carries `path`, `to_path` and `touched` ([spec/11 §11.3](../spec/11-corrections.md#113-applied-fields)).

### The history has a root

Corrections are append-only in principle, but nothing in a package proved that a later revision extends an earlier one rather than rewriting it.
A Merkle root over the corrections in `document.json`, with RFC 9162 consistency proofs between revisions, does ([spec/11 §11.5](../spec/11-corrections.md#115-the-history-root)).

### Digits need image evidence

A wrong digit changes a result without looking wrong, so no digit changes without a crop of the page as evidence, and no leaf outside the correction's `touched` list may end up with different digits ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)).

## Geometry

### Pages have fingerprints

Publishers stamp each download, so file hashes cannot identify a paper across teammates' copies.
Fingerprints with the stamps removed can ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).

### Fingerprint text deletes whitespace entirely

PDF text extractors disagree about where to insert spaces, and the fingerprint must come out the same in any two implementations.

## Review

### People sign receipts

Nobody signs three thousand times, so one signature covers a Merkle root of everything reviewed in a sitting.
A Merkle tree hashes items in pairs, then those hashes in pairs, up to one root, and allows a proof that one node was covered without revealing the others.

The receipt tree follows [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), whose replacement [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162) defines the same tree.
The specification makes a receipt with no items invalid, where the RFC hashes the empty list ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
The tree is not the one behind `node_hash`, which hashes each node's `own_hash` with its children's `node_hash` values.

### Spot-check seeds are VRF proofs

A seed the reviewer could influence could be ground until the sample fell on careful work, and a seed the checker could vary could be ground by the checker.
An earlier draft used the checker's Ed25519 signature and called it unchoosable because signing is deterministic; that holds only for an honest checker, since a verifier accepts any valid signature and a dishonest one can sign with any nonce.
A verifiable random function (RFC 9381) has exactly one valid proof for a key and an input, which is the property the rule needs, and its keys are ordinary Ed25519 keys, so the roster needs nothing new.
Anyone can verify the proof and recompute the sample ([spec/08 §8.8](../spec/08-fidelity.md#88-spot-checks)).

### A receipt can be proven without being shown

The receipt tree follows RFC 9162 so that an inclusion proof, a dozen hashes, shows a buyer that one cell was proofread without handing over the receipt or the other nodes ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
A transparency log of receipts, revocations and rosters ([spec/23](../spec/23-log.md)) uses the same tree and the checkpoint format transparency logs share, so that a receipt's time and the roster that governed it stop being self-asserted.

### Error rates are reported as bounds

One clean sample of ten proves little.
A one-sided Clopper–Pearson upper bound, by default at confidence 0.95, across all of a reviewer's samples says what can be claimed.
With no errors it is about 3 divided by the number sampled ([spec/08 §8.9](../spec/08-fidelity.md#89-reviewer-error-bounds)).

### Levels fall back

A node counts as proofread only through a valid receipt that lists it at its current `node_hash`, so a correction never leaves a stale "proofread" behind ([spec/08 §8.1](../spec/08-fidelity.md#81-levels-of-fidelity)).

## Rights

### Rights are recorded per witness

What a team may do depends on how it obtained the paper: a subscription, a mining license, an open license, its own work.

### AI uses have two axes

Reviewers ask whether a model may see the text, where it runs, and whether the text is kept there.
The record names a purpose, `internal` or `external` depending on whether results leave the organization, and a processor: `local`, `tenancy`, `outside-no-retention` or `outside` ([spec/14 §14.1](../spec/14-rights.md#141-the-rights-record)).

### Unknown means no

A stage whose rights are unknown does not run.
A project owner settles the question with a signed policy, so the decision is the team's and is on the record.
A policy fills in unknowns and never overrides an explicit answer.

A project setting can turn the refusal into a warning.
The refusal is the default, and the project format is `provisional` ([spec/20](../spec/20-projects.md)).

### Backend licenses are part of the gate

A converter's code and its model weights can be licensed differently, and the weights may exclude commercial or competing use.
A condition whose answer is unknown counts as not met ([spec/14 §14.4](../spec/14-rights.md#144-backends-and-profiles)).

## Packages and databases

### Canonical JSON, one record per line

A corrected sentence is a one-line diff, and every file has exactly one valid byte sequence, so a hash of the manifest pins the whole package ([spec/01 §1.9](../spec/01-conventions.md#19-files), [spec/12 §12.3](../spec/12-package.md#123-the-manifest)).

A [Frictionless Data Package](https://datapackage.org/standard/data-package/) lists resources with an optional `hash` that defaults to MD5 (an older hash function).
The package manifest here lists every other file with SHA-256 and size.
The dataset format, which is `provisional`, is a Data Package with units, uncertainty, an origin and anchors to the printed values added ([spec/15](../spec/15-datasets.md)).

Two neighboring standards cover other parts of this ground, RO-Crate and the PROV Ontology.

[RO-Crate](https://www.researchobject.org/ro-crate/) describes research data with schema.org annotations in JSON-LD, and it is the better choice for general description and discovery.
No mapping between it and these packages is defined.

Provenance is a plain `provenance.json` of tools, backends, cached readings and stages.
It is not the [PROV Ontology](https://www.w3.org/TR/prov-o/), which other systems can query generically.

### Checks-only packages drop everything that carries text

Corrections carry their find and replace strings, datasets carry values, suspects carry the readings compared, and figure text and reference records carry text.
A checks-only export omits them all, withholds the content of every leaf, and keys its hashes ([spec/12 §12.5](../spec/12-package.md#125-text-policies)).

### SQL avoids reserved words in every engine

The same tables run on SQLite, Dolt (MySQL-compatible) and PostgreSQL, and no name is a word reserved in any of the three.
The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash` ([spec/13 §13.1](../spec/13-database.md#131-portability)).

### Loading and unloading are exact inverses

A team can keep its documents in a database and still hand anyone the package, byte for byte.
The conformance suite checks this for every fixture ([spec/13 §13.3](../spec/13-database.md#133-load-and-unload)).

## Questions the formats leave open

These are settled for now, and a future version may revisit them.

| Question | Current answer |
| --- | --- |
| Should node IDs carry a kind prefix, such as `s-k3m9q2a`, to help people debugging? | No: a reclassified node keeps its ID, so a prefix would become wrong. |
| Should `text_hash` normalize Unicode, or only collapse whitespace? | Both: Unicode Normalization Form C, then whitespace collapsed to single spaces and trimmed ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)). |
| How long may a quotation in an anchor be? | 200 code points. |
| How is complex math compared? | By its source, run by run; comparing canonical MathML is a possible later addition. |
| What extension and media type do zipped packages use? | `.sct.zip` and `application/vnd.scienting.doc+zip`. |
| Which drift threshold, spot-check rate and digit rule should teams use? | By default, drift is high above a quarter of the leaves involved, and spot-checks sample 5% of a receipt's nodes, at least ten. The digit rule is `all`, `anchored-sections` or `none`; each is set per project. |
| How should rights combine when a document comes from witnesses held on different bases? | The most restrictive witness governs each use. |

## Looking ahead

Choices made so that the formats still serve in the 2030s, and the ones deliberately left open.

- Signature schemes `cose` and `ml-dsa-65` are reserved: COSE so that the IETF's SCITT transparency services can verify receipts, ML-DSA (FIPS 204) for the time Ed25519 is no longer considered safe.
  A consumer treats an unimplemented scheme as invalid, so adopting either is a minor version ([spec/08 §8.4](../spec/08-fidelity.md#84-record-hashes-and-signatures)).
- Every hash is SHA-256, and the hashing scheme names only how a value is keyed, not the function.
  A hash function change would be a new major version of every format; the formats accept that rather than carrying agility they cannot test.
- Roster keys are named by short keys, not by decentralized identifiers; a `did:key` form could be added as a key property without a new version.
- A correction's author may be a model and its run; a fuller provenance for model output (the model's identity, weights and prompt by hash) is left to the project that first needs it.
- A transparency log is specified but no deployment exists yet; its tile layout is left to the first one ([spec/23](../spec/23-log.md)).

## Known limits

- Notation drops spaces and text keeps them, so `$T = 300$` from a LaTeX source and "T = 300" from a PDF disagree, and become a `word` suspect for review.
- Page fingerprints are computed from extracted text lines.
  Two implementations that should agree on fingerprints must extract lines the same way.
- The formats are built for a team-scale corpus of hundreds of documents.
  They do not replace publisher interchange formats, and they do not fix what a converter reads wrongly.
  They record it and make it checkable.