paperspec/docs/rationale.md

Rationale

Each choice below is one a reader is likely to question. The specification states the rule, and this page gives the reason.

This page is for newcomers, implementers and experts. Newcomers find each term defined where it first appears. Implementers find the chapter that defines each rule. Experts find the cost of each choice and, where a neighboring standard chose differently, how it differs (the full comparison is in comparison.md).

The formats are at version 0.1.0. Of the 29 in registry/formats.json, 18 are candidate and 11 are provisional, and the provisional ones are named where they appear. The repository is paperspec, at https://git.scient.ing/infra/paperspec. The scienting- prefix in format names such as scienting-anchor/1 is the formats' namespace, separate from the repository's name.

Identity

IDs carry no meaning

Every sentence, cell and other node has an ID, a short name that other records use to point at it. An ID that encoded a page, an ordinal or a kind would change when a converter orders blocks differently, or when a paragraph turns out to be a caption.

An ID is minted once from the document key, the node's kind, the hash of its text and an occurrence count that separates identical texts, and is then frozen (spec/04 §4.2). A reclassification leaves it as it was, and a fresh conversion of the same input mints the same IDs. A different converter mints different ones for some sentences: those the alignment pairs with the established reading keep their IDs, and every difference becomes a suspect for review (spec/05 §5.6).

A content address such as an IPFS content identifier or a Software Heritage identifier is a function of content at all times. That suits immutable data and archived source code. Any correction would rename the sentence. Here the ID is derived from content once and kept apart from the content hashes, so the pair of ID and hash says whether the text changed.

The cost is that two conversions that read a sentence differently by one character mint different IDs for it.

IDs never move, and retired IDs leave a trail

Everything a team builds points at IDs: notes, claims, dataset values, citations in drafts. A merge records its successor, a deletion records that there is none, and legacy IDs live on as aliases, all in ids.jsonl (spec/04 §4.3).

Revisions are counted, not timed

Anchors are made on different machines and packages are imported later, and clocks disagree. A correction index does not disagree with itself. A document's revision is the index of its last correction (spec/06 §6.2, spec/11).

Hashes and projections

A hash is a fixed-length fingerprint of data: equal data gives equal hashes, and any change gives a different one. Every hash here is SHA-256 (spec/01 §1.3). A projection is a rewritten form of a text, made only to be hashed.

Canonical JSON

Canonical JSON is JSON (JavaScript Object Notation, a text format for structured data) written in exactly one way, so equal data always hashes the same. The formats use the JSON Canonicalization Scheme, RFC 8785 (an RFC, or Request for Comments, is a numbered Internet standards document), which requires numbers to be IEEE 754 doubles (IEEE is the Institute of Electrical and Electronics Engineers) (the standard binary floating-point numbers), and add one rule: a decimal that carries meaning is a string, so 310.0 and 1.5260 keep their printed digits (spec/01 §1.1 and §1.2). The cost is that a reader must treat those strings as decimals itself.

Five values, not one

A text node carries five values (spec/05 §5.1), because no single hash does all of their jobs.

• Quotations need every character, so text_hash covers the normalized text.
• Certifying that two witnesses agree needs notation folded but structure kept, which is match_hash. A witness is one file the text was read from.
• Proposing that two sentences are the same needs something looser still, which is align_key.
• Containers need a hash of their own content, own_hash, apart from the node_hash that also covers what they hold.

Table columns and figure-text items have IDs but are not nodes, so they carry no hashes (spec/09 §9.1).

Only the strict projection certifies

OCR (optical character recognition, which reads text from page images) drops superscripts. A projection loose enough to match "103" with 10³ would certify an error, because the printed value is 1000 and the reading is 103. The lossy key may propose that match. Only the strict hash may accept it, and then only when the complex-math keys agree too (spec/05 §5.3).

Citation marks stay marked in the strict projection

"10" followed by citation 3 would agree with the text "103" without the marks.

Decimal points and commas count as simple notation

Simple math is rewritten to one linear form that can match plain text; anything else is complex and is compared by its source. $-1.5$ in a LaTeX source would be complex math without the two characters, and could never agree with "−1.5" in the PDF (spec/05 §5.4).

A hyphen or en dash before a digit is a minus sign in the strict projection

Text layers and OCR disagree about dashes in "10–20" and "−5". Folding them before digits makes witnesses agree without touching a dash inside a name such as "CT–CT".

The lossy key deletes whitespace and dashes

OCR merges and splits words and line-break hyphens, and a key that only proposes alignments should forgive that. It also folds case (spec/05 §5.5).

Offsets index the stored text, and structural hashes cover it exactly

An offset counts Unicode code points, the numbered characters of Unicode, into the node's stored plain text (spec/01 §1.5), and own content is hashed as stored. Every stored offset is still exact when node_hash is unchanged, and a renderer can slice the stored text without normalizing it first. Only the three text values, text_hash, match_hash and align_key, normalize.

Hashes that leave the team are keyed

A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages. Inside a team, plain hashes are fine.

An export to another machine of the team uses an HMAC (hash-based message authentication code), a hash that needs a secret key, under the team key. A public layer, or any export of a work whose text is withheld, uses a key derived from the paper's own page fingerprints, which only someone holding the paper can compute (spec/05 §5.8).

Anchors

An anchor is a stored pointer to a node or to a quoted span inside it, with enough context to find it again after the text changes (spec/06).

The World Wide Web Consortium (W3C) Web Annotation model also anchors with quotations: its TextQuoteSelector takes exact, prefix and suffix, and its TextPositionSelector is described there as very brittle. An anchor here keeps the quote with up to 32 code points of prefix and suffix and adds hashes, so resolution has exact outcomes (current, changed-below, moved, reattached, stale). Hypothesis, in its description of its own anchoring, matches fuzzily.

Annotation layers use a custom AnchorSelector followed by a standard TextQuoteSelector fallback for tools that do not know ours (spec/18 §18.3). The cost is a selector type no generic tool defines. For annotation in general, the W3C model is the better choice.

Every anchor carries own_hash, and anchors on containers without content carry their leaves

Telling a relabel from a real change needs the anchored own_hash. Deciding whether a merged paragraph still holds "at least half" of what was anchored needs the original leaves.

Long quotations become a hash and a length

Quotations over 200 code points are replaced by a hash and a length, which keeps anchors small and limits how much text an anchor carries. A hashed window can still be found again in the node.

Prefix and suffix pick between repeated quotes

"The value" may occur twice in a sentence. The context around it decides which one was meant.

changed-below reports drift

A note on a section stays valid when a typo inside it is fixed. By default the drift is high once the leaves touched by corrections since the anchor was made exceed a quarter of the leaves involved, and the note is then flagged for review (spec/06 §6.6).

A row's header path is not hashed

Rows have no content of their own, and a row's header_path is not part of it. Inserting a dropped row-group header does not stale the anchors on every cell beneath it (spec/09 §9.2).

Corrections

Corrections are appended, never edited

A document's history is its first reading plus its corrections, so anyone can see what changed, when and why, and on what evidence (spec/11).

Corrections include a set operation

Readings drop spans, roles and header paths, not only words, and those need a correction too.

Applied corrections record what they touched

Drift is computed from the records alone, with no need to replay history, because each record carries path, to_path and touched (spec/11 §11.3).

Digits need image evidence

A wrong digit changes a result without looking wrong, so no digit changes without a crop of the page as evidence, and no leaf outside the correction's touched list may end up with different digits (spec/11 §11.4).

Geometry

Pages have fingerprints

Publishers stamp each download, so file hashes cannot identify a paper across teammates' copies. Fingerprints with the stamps removed can (spec/07 §7.1).

Fingerprint text deletes whitespace entirely

PDF text extractors disagree about where to insert spaces, and the fingerprint must come out the same in any two implementations.

Review

People sign receipts

Nobody signs three thousand times, so one signature covers a Merkle root of everything reviewed in a sitting. A Merkle tree hashes items in pairs, then those hashes in pairs, up to one root, and allows a proof that one node was covered without revealing the others.

The receipt tree follows RFC 6962, whose replacement RFC 9162 defines the same tree. The specification makes a receipt with no items invalid, where the RFC hashes the empty list (spec/08 §8.5). The tree is not the one behind node_hash, which hashes each node's own_hash with its children's node_hash values.

Spot-check samples come from a signature nobody can steer

A seed the reviewer could influence could be ground until the sample fell on careful work. The checker's deterministic Ed25519 signature over the receipt is unpredictable to the reviewer and unchoosable by the checker, because the checker's key was registered for spot-checks before the original was signed. Anyone can recompute the sample (spec/08 §8.8).

Error rates are reported as bounds

One clean sample of ten proves little. A one-sided Clopper–Pearson upper bound, by default at confidence 0.95, across all of a reviewer's samples says what can be claimed. With no errors it is about 3 divided by the number sampled (spec/08 §8.9).

Levels fall back

A node counts as proofread only through a valid receipt that lists it at its current node_hash, so a correction never leaves a stale "proofread" behind (spec/08 §8.1).

Rights

Rights are recorded per witness

What a team may do depends on how it obtained the paper: a subscription, a mining license, an open license, its own work.

AI uses have two axes

Reviewers ask whether a model may see the text, where it runs, and whether the text is kept there. The record names a purpose, internal or external depending on whether results leave the organization, and a processor: local, tenancy, outside-no-retention or outside (spec/14 §14.1).

Unknown means no

A stage whose rights are unknown does not run. A project owner settles the question with a signed policy, so the decision is the team's and is on the record. A policy fills in unknowns and never overrides an explicit answer.

A project setting can turn the refusal into a warning. The refusal is the default, and the project format is provisional (spec/20).

Backend licenses are part of the gate

A converter's code and its model weights can be licensed differently, and the weights may exclude commercial or competing use. A condition whose answer is unknown counts as not met (spec/14 §14.4).

Packages and databases

Canonical JSON, one record per line

A corrected sentence is a one-line diff, and every file has exactly one valid byte sequence, so a hash of the manifest pins the whole package (spec/01 §1.9, spec/12 §12.3).

A Frictionless Data Package lists resources with an optional hash that defaults to MD5 (an older hash function). The package manifest here lists every other file with SHA-256 and size. The dataset format, which is provisional, is a Data Package with units, uncertainty, an origin and anchors to the printed values added (spec/15).

Two neighboring standards cover other parts of this ground, RO-Crate and the PROV Ontology.

RO-Crate describes research data with schema.org annotations in JSON-LD, and it is the better choice for general description and discovery. No mapping between it and these packages is defined.

Provenance is a plain provenance.json of tools, backends, cached readings and stages. It is not the PROV Ontology, which other systems can query generically.

Checks-only packages drop everything that carries text

Corrections carry their find and replace strings, datasets carry values, suspects carry the readings compared, and figure text and reference records carry text. A checks-only export omits them all, withholds the content of every leaf, and keys its hashes (spec/12 §12.5).

SQL avoids reserved words in every engine

The same tables run on SQLite, Dolt (MySQL-compatible) and PostgreSQL, and no name is a word reserved in any of the three. The formats' key, index, at, by, before and after become doc_key, idx, made_at, signer, before_hash and after_hash (spec/13 §13.1).

Loading and unloading are exact inverses

A team can keep its documents in a database and still hand anyone the package, byte for byte. The conformance suite checks this for every fixture (spec/13 §13.3).

Questions the formats leave open

These are settled for now, and a future version may revisit them.

Question	Current answer
Should node IDs carry a kind prefix, such as s-k3m9q2a, to help people debugging?	No: a reclassified node keeps its ID, so a prefix would become wrong.
Should text_hash normalize Unicode, or only collapse whitespace?	Both: Unicode Normalization Form C, then whitespace collapsed to single spaces and trimmed (spec/01 §1.4).
How long may a quotation in an anchor be?	200 code points.
How is complex math compared?	By its source, run by run; comparing canonical MathML is a possible later addition.
What extension and media type do zipped packages use?	.sct.zip and application/vnd.scienting.doc+zip.
Which drift threshold, spot-check rate and digit rule should teams use?	By default, drift is high above a quarter of the leaves involved, and spot-checks sample 5% of a receipt's nodes, at least ten. The digit rule is all, anchored-sections or none; each is set per project.
How should rights combine when a document comes from witnesses held on different bases?	The most restrictive witness governs each use.

Known limits

• Notation drops spaces and text keeps them, so $T = 300$ from a LaTeX source and "T = 300" from a PDF disagree, and become a word suspect for review.
• Page fingerprints are computed from extracted text lines. Two implementations that should agree on fingerprints must extract lines the same way.
• The formats are built for a team-scale corpus of hundreds of documents. They do not replace publisher interchange formats, and they do not fix what a converter reads wrongly. They record it and make it checkable.