infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8c8b51cc3f8fc39a0870d02fa7913e22710b324a
paperspec/docs/rationale.md
T
alexmandClaude 8c8b51cc3f feat: validator, bundle builder and release checks for 0.1.0 
- Add tools/validate.py: validates packages, and with --correction runs
  proposed corrections through the gate against a package without
  writing anything (spec/11 §11.4). The gate now refuses a proposal that
  fails the correction schema, rule correction.schema.
- tools/load.py validates every package before loading, and with
  --bundle --built-from builds a scienting-bundle/1 with the FTS5 search
  index and bundle_meta (spec/13 §13.5). Search rows are defined per
  unit (sentence, cell, equation outside a sentence, figure-text) with
  the parent kind as context.
- Roster keys carry a role, and a key with the role owner may revoke any
  receipt (spec/08 §8.7, spec/20); roster schema, store table and the
  fixture roster updated. The checker signs a trial revocation to prove
  the rule both ways.
- A qa: receipt raises the nodes it sampled except those in its errors
  (spec/08 §8.2); the example paper's fidelity counts become 27 F3,
  13 F2, 4 F1 and 1 F0.
- The destination chooses the keyed scheme, not the text policy:
  hmac-team between the team's machines, hkdf-pages outside the team
  (spec/05 §5.8, spec/12 §12.5).
- A TextQuoteSelector must repeat the anchor's plain quote and is absent
  for hashed or keyed anchors (spec/18 §18.3).
- Add fixtures/records/suspect.json, a scienting-suspect/1 record.
- Oracle: hashes are kept incrementally and checked after every
  correction against a full recomputation; load and write refuse unsafe
  manifest paths; Ed25519, HKDF and schema validation come from the
  cryptography and jsonschema packages; canonical JSON uses the built-in
  encoder with an exact RFC 8785 fallback.
- The tools need Python 3.14 with cryptography and jsonschema; the py310
  CI environment is dropped and the workflow runs the default only.
- Resolve every [CHECK] note in the docs, READMEs and context; add
  CONTRIBUTING.md; every tool answers --help and refuses unknown options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-14 18:58:00 -04:00

281 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Rationale
Each choice below is one a reader is likely to question.
The specification states the rule, and this page gives the reason.
This page is for newcomers, implementers and experts.
Newcomers find each term defined where it first appears.
Implementers find the chapter that defines each rule.
Experts find the cost of each choice and, where a neighboring standard chose differently, how it differs (the full comparison is in [comparison.md](comparison.md)).
The formats are at version 0.1.0.
Of the 27 in `registry/formats.json`, 20 are `candidate` and 7 are `provisional`, and the provisional ones are named where they appear.
The repository is manuspec, at <https://git.scient.ing/infra/manuspec>.
The `scienting-` prefix in format names such as `scienting-anchor/1` is the formats' namespace, separate from the repository's name.
## Identity
### IDs carry no meaning
Every sentence, cell and other node has an ID, a short name that other records use to point at it.
An ID that encoded a page, an ordinal or a kind would change when a converter orders blocks differently, or when a paragraph turns out to be a caption.
An ID is minted once from the document key, the node's kind, the hash of its text and an occurrence count that separates identical texts, and is then frozen ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
A reclassification leaves it as it was, and a fresh conversion of the same input mints the same IDs.
A different converter mints different ones for some sentences: those the alignment pairs with the established reading keep their IDs, and every difference becomes a suspect for review ([spec/05 §5.6](../spec/05-hashes.md#56-alignment)).
A content address such as an [IPFS content identifier](https://docs.ipfs.tech/concepts/content-addressing/) or a [Software Heritage identifier](https://www.swhid.org/specification/v1.2/5.Core_identifiers/) is a function of content at all times.
That suits immutable data and archived source code.
Any correction would rename the sentence.
Here the ID is derived from content once and kept apart from the content hashes, so the pair of ID and hash says whether the text changed.
The cost is that two conversions that read a sentence differently by one character mint different IDs for it.
### IDs never move, and retired IDs leave a trail
Everything a team builds points at IDs: notes, claims, dataset values, citations in drafts.
A merge records its successor, a deletion records that there is none, and legacy IDs live on as aliases, all in `ids.jsonl` ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
### Revisions are counted, not timed
Anchors are made on different machines and packages are imported later, and clocks disagree.
A correction index does not disagree with itself.
A document's revision is the index of its last correction ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor), [spec/11](../spec/11-corrections.md)).
## Hashes and projections
A hash is a fixed-length fingerprint of data: equal data gives equal hashes, and any change gives a different one.
Every hash here is SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
A projection is a rewritten form of a text, made only to be hashed.
### Canonical JSON
Canonical JSON is JSON (JavaScript Object Notation, a text format for structured data) written in exactly one way, so equal data always hashes the same.
The formats use the JSON Canonicalization Scheme, [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785) (an RFC, or Request for Comments, is a numbered Internet standards document), which requires numbers to be IEEE 754 doubles (IEEE is the Institute of Electrical and Electronics Engineers) (the standard binary floating-point numbers), and add one rule: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits ([spec/01 §1.1 and §1.2](../spec/01-conventions.md)).
The cost is that a reader must treat those strings as decimals itself.
### Five values, not one
A text node carries five values ([spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)), because no single hash does all of their jobs.
- Quotations need every character, so `text_hash` covers the normalized text.
- Certifying that two witnesses agree needs notation folded but structure kept, which is `match_hash`.
A witness is one file the text was read from.
- Proposing that two sentences are the same needs something looser still, which is `align_key`.
- Containers need a hash of their own content, `own_hash`, apart from the `node_hash` that also covers what they hold.
Table columns and figure-text items have IDs but are not nodes, so they carry no hashes ([spec/09 §9.1](../spec/09-figures-tables-equations.md)).
### Only the strict projection certifies
OCR (optical character recognition, which reads text from page images) drops superscripts.
A projection loose enough to match "103" with 10³ would certify an error, because the printed value is 1000 and the reading is 103.
The lossy key may propose that match.
Only the strict hash may accept it, and then only when the complex-math keys agree too ([spec/05 §5.3](../spec/05-hashes.md#53-the-strict-projection)).
### Citation marks stay marked in the strict projection
"10" followed by citation 3 would agree with the text "103" without the marks.
### Decimal points and commas count as simple notation
Simple math is rewritten to one linear form that can match plain text; anything else is complex and is compared by its source. `$-1.5$` in a LaTeX source would be complex math without the two characters, and could never agree with "−1.5" in the PDF ([spec/05 §5.4](../spec/05-hashes.md#54-simple-notation)).
### A hyphen or en dash before a digit is a minus sign in the strict projection
Text layers and OCR disagree about dashes in "10–20" and "−5".
Folding them before digits makes witnesses agree without touching a dash inside a name such as "CT–CT".
### The lossy key deletes whitespace and dashes
OCR merges and splits words and line-break hyphens, and a key that only proposes alignments should forgive that.
It also folds case ([spec/05 §5.5](../spec/05-hashes.md#55-the-lossy-projection)).
### Offsets index the stored text, and structural hashes cover it exactly
An offset counts Unicode code points, the numbered characters of Unicode, into the node's stored plain text ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)), and own content is hashed as stored.
Every stored offset is still exact when `node_hash` is unchanged, and a renderer can slice the stored text without normalizing it first.
Only the three text values, `text_hash`, `match_hash` and `align_key`, normalize.
### Hashes that leave the team are keyed
A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages.
Inside a team, plain hashes are fine.
An export to another machine of the team uses an HMAC (hash-based message authentication code), a hash that needs a secret key, under the team key.
A public layer, or any export of a work whose text is withheld, uses a key derived from the paper's own page fingerprints, which only someone holding the paper can compute ([spec/05 §5.8](../spec/05-hashes.md#58-keyed-hashes)).
## Anchors
An anchor is a stored pointer to a node or to a quoted span inside it, with enough context to find it again after the text changes ([spec/06](../spec/06-anchors.md)).
The [World Wide Web Consortium (W3C) Web Annotation model](https://www.w3.org/TR/annotation-model/) also anchors with quotations: its `TextQuoteSelector` takes `exact`, `prefix` and `suffix`, and its `TextPositionSelector` is described there as very brittle.
An anchor here keeps the quote with up to 32 code points of prefix and suffix and adds hashes, so resolution has exact outcomes (`current`, `changed-below`, `moved`, `reattached`, `stale`).
Hypothesis, in [its description of its own anchoring](https://web.hypothes.is/blog/fuzzy-anchoring/), matches fuzzily.
Annotation layers use a custom `AnchorSelector` followed by a standard `TextQuoteSelector` fallback for tools that do not know ours ([spec/18 §18.3](../spec/18-layers.md)).
The cost is a selector type no generic tool defines.
For annotation in general, the W3C model is the better choice.
### Every anchor carries `own_hash`, and anchors on containers without content carry their leaves
Telling a relabel from a real change needs the anchored `own_hash`.
Deciding whether a merged paragraph still holds "at least half" of what was anchored needs the original leaves.
### Long quotations become a hash and a length
Quotations over 200 code points are replaced by a hash and a length, which keeps anchors small and limits how much text an anchor carries.
A hashed window can still be found again in the node.
### Prefix and suffix pick between repeated quotes
"The value" may occur twice in a sentence.
The context around it decides which one was meant.
### `changed-below` reports drift
A note on a section stays valid when a typo inside it is fixed.
By default the drift is `high` once the leaves touched by corrections since the anchor was made exceed a quarter of the leaves involved, and the note is then flagged for review ([spec/06 §6.6](../spec/06-anchors.md#66-drift)).
### A row's header path is not hashed
Rows have no content of their own, and a row's `header_path` is not part of it.
Inserting a dropped row-group header does not stale the anchors on every cell beneath it ([spec/09 §9.2](../spec/09-figures-tables-equations.md)).
## Corrections
### Corrections are appended, never edited
A document's history is its first reading plus its corrections, so anyone can see what changed, when and why, and on what evidence ([spec/11](../spec/11-corrections.md)).
### Corrections include a `set` operation
Readings drop spans, roles and header paths, not only words, and those need a correction too.
### Applied corrections record what they touched
Drift is computed from the records alone, with no need to replay history, because each record carries `path`, `to_path` and `touched` ([spec/11 §11.3](../spec/11-corrections.md#113-applied-fields)).
### Digits need image evidence
A wrong digit changes a result without looking wrong, so no digit changes without a crop of the page as evidence, and no leaf outside the correction's `touched` list may end up with different digits ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)).
## Geometry
### Pages have fingerprints
Publishers stamp each download, so file hashes cannot identify a paper across teammates' copies.
Fingerprints with the stamps removed can ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
### Fingerprint text deletes whitespace entirely
PDF text extractors disagree about where to insert spaces, and the fingerprint must come out the same in any two implementations.
## Review
### People sign receipts
Nobody signs three thousand times, so one signature covers a Merkle root of everything reviewed in a sitting.
A Merkle tree hashes items in pairs, then those hashes in pairs, up to one root, and allows a proof that one node was covered without revealing the others.
The receipt tree follows [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), whose replacement [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162) defines the same tree.
The specification makes a receipt with no items invalid, where the RFC hashes the empty list ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
The tree is not the one behind `node_hash`, which hashes each node's `own_hash` with its children's `node_hash` values.
### Spot-check samples come from a signature nobody can steer
A seed the reviewer could influence could be ground until the sample fell on careful work.
The checker's deterministic Ed25519 signature over the receipt is unpredictable to the reviewer and unchoosable by the checker, because the checker's key was registered for spot-checks before the original was signed.
Anyone can recompute the sample ([spec/08 §8.8](../spec/08-fidelity.md#88-spot-checks)).
### Error rates are reported as bounds
One clean sample of ten proves little.
A one-sided Clopper–Pearson upper bound, by default at confidence 0.95, across all of a reviewer's samples says what can be claimed.
With no errors it is about 3 divided by the number sampled ([spec/08 §8.9](../spec/08-fidelity.md#89-reviewer-error-bounds)).
### Levels fall back
A node counts as proofread only through a valid receipt that lists it at its current `node_hash`, so a correction never leaves a stale "proofread" behind ([spec/08 §8.1](../spec/08-fidelity.md#81-levels-of-fidelity)).
## Rights
### Rights are recorded per witness
What a team may do depends on how it obtained the paper: a subscription, a mining license, an open license, its own work.
### AI uses have two axes
Reviewers ask whether a model may see the text, where it runs, and whether the text is kept there.
The record names a purpose, `internal` or `external` depending on whether results leave the organization, and a processor: `local`, `tenancy`, `outside-no-retention` or `outside` ([spec/14 §14.1](../spec/14-rights.md#141-the-rights-record)).
### Unknown means no
A stage whose rights are unknown does not run.
A project owner settles the question with a signed policy, so the decision is the team's and is on the record.
A policy fills in unknowns and never overrides an explicit answer.
A project setting can turn the refusal into a warning.
The refusal is the default, and the project format is `provisional` ([spec/20](../spec/20-projects.md)).
### Backend licenses are part of the gate
A converter's code and its model weights can be licensed differently, and the weights may exclude commercial or competing use.
A condition whose answer is unknown counts as not met ([spec/14 §14.4](../spec/14-rights.md#144-backends-and-profiles)).
## Packages and databases
### Canonical JSON, one record per line
A corrected sentence is a one-line diff, and every file has exactly one valid byte sequence, so a hash of the manifest pins the whole package ([spec/01 §1.9](../spec/01-conventions.md#19-files), [spec/12 §12.3](../spec/12-package.md#123-the-manifest)).
A [Frictionless Data Package](https://datapackage.org/standard/data-package/) lists resources with an optional `hash` that defaults to MD5 (an older hash function).
The package manifest here lists every other file with SHA-256 and size.
The dataset format, which is `provisional`, is a Data Package with units, uncertainty, an origin and anchors to the printed values added ([spec/15](../spec/15-datasets.md)).
Two neighboring standards cover other parts of this ground, RO-Crate and the PROV Ontology.
[RO-Crate](https://www.researchobject.org/ro-crate/) describes research data with schema.org annotations in JSON-LD, and it is the better choice for general description and discovery.
No mapping between it and these packages is defined.
Provenance is a plain `provenance.json` of tools, backends, cached readings and stages.
It is not the [PROV Ontology](https://www.w3.org/TR/prov-o/), which other systems can query generically.
### Checks-only packages drop everything that carries text
Corrections carry their find and replace strings, datasets carry values, suspects carry the readings compared, and figure text and reference records carry text.
A checks-only export omits them all, withholds the content of every leaf, and keys its hashes ([spec/12 §12.5](../spec/12-package.md#125-text-policies)).
### SQL avoids reserved words in every engine
The same tables run on SQLite, Dolt (MySQL-compatible) and PostgreSQL, and no name is a word reserved in any of the three.
The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash` ([spec/13 §13.1](../spec/13-database.md#131-portability)).
### Loading and unloading are exact inverses
A team can keep its documents in a database and still hand anyone the package, byte for byte.
The conformance suite checks this for every fixture ([spec/13 §13.3](../spec/13-database.md#133-load-and-unload)).
## Questions the formats leave open
These are settled for now, and a future version may revisit them.
| Question | Current answer |
| --- | --- |
| Should node IDs carry a kind prefix, such as `s-k3m9q2a`, to help people debugging? | No: a reclassified node keeps its ID, so a prefix would become wrong. |
| Should `text_hash` normalize Unicode, or only collapse whitespace? | Both: Unicode Normalization Form C, then whitespace collapsed to single spaces and trimmed ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)). |
| How long may a quotation in an anchor be? | 200 code points. |
| How is complex math compared? | By its source, run by run; comparing canonical MathML is a possible later addition. |
| What extension and media type do zipped packages use? | `.sct.zip` and `application/vnd.scienting.doc+zip`. |
| Which drift threshold, spot-check rate and digit rule should teams use? | By default, drift is high above a quarter of the leaves involved, and spot-checks sample 5% of a receipt's nodes, at least ten. The digit rule is `all`, `anchored-sections` or `none`; each is set per project. |
| How should rights combine when a document comes from witnesses held on different bases? | The most restrictive witness governs each use. |
## Known limits
- Notation drops spaces and text keeps them, so `$T = 300$` from a LaTeX source and "T = 300" from a PDF disagree, and become a `word` suspect for review.
- Page fingerprints are computed from extracted text lines.
Two implementations that should agree on fingerprints must extract lines the same way.
- The formats are built for a team-scale corpus of hundreds of documents.
They do not replace publisher interchange formats, and they do not fix what a converter reads wrongly.
They record it and make it checkable.
Reference in new issue View Git Blame Copy Permalink