paperspec/docs/concepts.md

Concepts

Each section introduces one idea, shows it in the example paper hale2026harmonic-damb (a synthetic chemistry paper in fixtures/packages/), and links to the chapter that defines it exactly.

This page is written for three readers. A newcomer, such as a student who knows what a paper and a citation are and is new to hashes, JSON and annotation standards, gets each term defined where it first appears, with the reason before the rule. An implementer, who will write a reader, writer, converter or checker, gets the chapter that defines each behavior, the test vector file that pins it, and the traps. An expert, such as a researcher, standards author or reviewer who knows the neighboring standards, gets the limits and the status of each format, and a short note wherever a close relative exists elsewhere.

The longer comparisons are on the comparison page. For a gentler start, read the overview first; for the exact rules, use the specification.

Two facts apply to everything below. The specification is at version 0.1.0. It defines 32 formats, 19 of them candidate and 13 provisional, as registry/formats.json lists. The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint formats; the sections below say so where they describe them.

Format identifiers such as scienting-anchor/1 and scienting-correction/1 begin with scienting-, which is the formats' namespace. The repository's name, paperspec, is a separate matter, and the formats are named after what they hold. The repository is at https://git.scient.ing/infra/paperspec.

Hashes and canonical JSON

Two tools appear in almost every section.

A hash is a short fingerprint computed from data. This specification uses SHA-256, which turns any sequence of bytes into 64 hexadecimal digits. The same bytes always give the same digits, and changing one character gives unrelated digits. Comparing two fingerprints tells you whether two texts are identical without comparing the texts. The fingerprint cannot be turned back into the text. The SHA-256 of the cell text 310.0 in the example begins b478fd…. Hashes are written as 64 lowercase hexadecimal digits; six digits are used only for display and for pins in addresses (chapter 1.3).

JSON is a plain-text format for structured data: objects with named members, lists, strings and numbers. The same JSON data can be written in many ways (members in any order, spaces anywhere), and a hash of each way would differ. Canonical JSON fixes one way, so that every writer produces the same bytes and a hash of the data means the same thing everywhere.

The specification uses the JSON Canonicalization Scheme of RFC 8785: no whitespace, members sorted by name, and numbers written as ECMAScript writes them. Strings are not Unicode-normalized, and a document containing a lone surrogate, NaN or an infinity has no canonical form and is invalid. The specification adds one rule on top: a decimal that carries meaning is written as a string, so that 310.0 and 1.5260 keep their printed digits. Every hash of a JSON value here is a hash of its canonical JSON encoded as UTF-8.

RFC 8785 is a building block and says nothing about what to put in the JSON. This specification adopts it for the canonical form and adds the decimal and invalid-value rules above (chapter 1.1, 1.2, 1.9).

Pinned by vectors/canonical-json.jsonl. A trap for implementers: the vectors parse every number as an IEEE 754 double, and sort member names as UTF-16 code units, not as bytes.

Works, versions, witnesses and documents

Work, version, witness and document are easy to confuse, so the formats keep them apart.

Object	In the example
A work is the paper across all its versions.	Hale and Osei's paper on harmonic bond parameters, doi:10.5555/example.hale.2026.
A version is one expression of it: a preprint, the version of record, an erratum, the Supporting Information.	The version of record.
A witness is one file of a version, identified by its SHA-256.	The publisher's PDF, witness qptsanv.
A document is the text established for one version, from one or more witnesses.	hale2026harmonic-damb.

A document's key is its work's key, with a suffix for anything other than the version of record: hale2026harmonic-damb.si would be its Supporting Information. A witness can be a PDF, a JATS file, a LaTeX archive, an HTML page, an EPUB, a Typst project, other XML, an image, or another file; these are the values of its kind. See chapter 2.

Keys

A work key is a work's permanent name, built so that a reader can say it and so that it never collides. It is a Scholar-style citation key (first author, year, first title word), a hyphen, and four characters of a hash of the work's identity, which is its DOI when it has one:

hale2026harmonic-damb

Two works that a citation key alone would name alike, a preprint and its published version, say, get different hashes. A key is minted once, when the work enters a store, and never recomputed, so a corrected title changes nothing anyone has cited; if two works share four characters, the later one gets six. A document key adds a suffix for a version other than the version of record: hale2026harmonic-damb.si would be the paper's Supporting Information. See spec/02 §2.4; keys.jsonl pins the cases.

A document is a tree of nodes

Every part of the text is a node: the document, its sections, headings, paragraphs and sentences, figures and their panels and captions, tables with their rows and cells, equations, footnotes and references. The example has 75 nodes, 45 of them leaves (nodes that hold text themselves). This outline shows its two sections and the equation, figure and table in them:

p36k8t6 1 Introduction
tkxsmts    equation (1)
d400rz3 2 Methods
v7cx6qs    figure Figure 1
hj4s7pb    table Table 1

nodes.jsonl holds one node per line, in document order: the root first, then each node before its children. Each line has the node's kind, parent, position among its siblings, content and hashes. A sentence's content is a list of runs: plain text, styled text, inline math, citations, cross-references, footnote marks, links, and display equations the sentence runs through. This is one table cell, shortened:

{"id":"c20cna6","kind":"cell","parent":"g8d2pjb","ord":1,
 "attrs":{"column":"…","role":"data","rowspan":1,"colspan":1},
 "content":[{"kind":"text","text":"310.0"}],
 "fidelity":"F1","text_hash":"…","match_hash":"…","align_key":"…","own_hash":"…","node_hash":"21f13b…"}

Page breaks, running heads and columns are not nodes; they are geometry (chapter 7). See chapter 3.

Close relatives: JATS 1.4 is an XML format in which publishers and archives exchange journal content, and TEI is a set of guidelines for encoding scholarly text. Each is the better choice for its own job, and the paperspec formats do not replace them; a JATS file can be one witness of a document. The element list in the JATS 1.4 archiving tag library shows sections, paragraphs, table wrappers and cells and has no sentence element, while TEI has an <s> element. Here the sentence is the unit that gets an ID and hashes. See the comparison page.

IDs are names, not positions

c20cna6 says nothing about where the cell is. That is deliberate. An ID that encoded a page or an ordinal would change when a better converter ordered blocks differently, and one that encoded a kind would change when a paragraph turned out to be a caption. So an ID encodes no page, ordinal or kind. It is seven characters, the first a letter, so it is valid as an HTML id, a CSS selector and a URL fragment.

An ID is minted once, from a hash of the document key, the node's kind, the hash of its text and an occurrence count (the number of earlier nodes with the same kind and text, so that repeated sentences get different IDs). Then it is frozen: it is never recomputed, reassigned or reused. A fresh conversion of the same input mints the same IDs. A different converter is not guaranteed to, because two readings that differ by one character would mint different IDs for that sentence.

A second reading is aligned with the document sentence by sentence. Sentences the alignment pairs keep the document's IDs, and every difference becomes a suspect, a recorded doubtful reading that a person resolves (chapter 5.6 and 5.7).

One ID is retired and points at its successor when two nodes merge; an artifact's ID is retired with no successor when the artifact is deleted. Legacy IDs, such as a block ID from an earlier numbering scheme, live on as aliases: in the example, b-p01-text-003 still leads to sentence hca6dc5. Both the retirements and the aliases are recorded in ids.jsonl.

Not every addressable thing is a node. The columns of a table and the text items found inside a figure panel get IDs by the same rule and carry no node hashes, so they do not have the five hashes described below. In the example, Table 1's second column has the ID wdtrcqt, and the axis label "r (Å)" in Figure 1's first panel has the ID rtxzjj9.

Pinned by vectors/ids.jsonl. See chapter 4.

Close relative: in TEI, xml:id is assigned by the author, and the Guidelines recommend it "For durability in the presence of editing"; JATS has an id attribute. TEI's recommendation gives no rule for minting an ID from content or for retiring one with a successor. For a text a person edits by hand, an author-assigned xml:id is the simpler tool. See the comparison page.

Addresses

An address names a node, and optionally pins it to a state and narrows it to a span:

Address	Names
hale2026harmonic-damb#hca6dc5	The sentence about water, whatever it says now.
hale2026harmonic-damb#hca6dc5@cd6cdb	That sentence as it read when its node_hash began cd6cdb.
hale2026harmonic-damb#hca6dc5@cd6cdb[29:44]	Code points 29 to 44 of it in that state: "first test case".
smithlab/hale2026harmonic-damb#hca6dc5	The same sentence in the smithlab store.

The offsets count Unicode code points in the node's plain projection, which is its text as stored with math written between dollar signs. The dollar signs count, which is why the span starts at 29: the sentence reads Water, H$_2$O, served as the first test case, ….

On the web the ID is the fragment, so a link to …/smithlab/hale2026harmonic-damb#hca6dc5 lands on the sentence. Pinned by vectors/addresses.jsonl. See chapter 6.

Five hashes, five jobs

Every node carries five content values. Table columns and figure-text items, from the previous section, do not.

Value	Covers	Answers
text_hash	The text exactly, normalized.	Is this the same quotation, character for character?
match_hash	The text with notation folded to one form.	Do two witnesses really agree?
align_key	The text folded loosely.	Is this probably the same sentence?
own_hash	The node's own content, without its children.	Has this section itself changed, as opposed to something inside it?
node_hash	The node and everything below it.	Has anything here changed at all?

"Normalized" means Unicode normalization form C, runs of whitespace collapsed to one space, and leading and trailing space removed (chapter 1.4). own_hash is a hash of the canonical JSON of the node's own content, such as its runs or its label. node_hash is a hash of the canonical JSON of {"own": own_hash, "children": [each child's node_hash, in order]}.

Each node_hash covers its children's, so a correction to one sentence changes that sentence's values and the node_hash of each ancestor, and nothing else. This design is the idea of a Merkle tree, a tree in which each hash covers the hashes below it. It is not the RFC 6962 construction that receipts use (see Receipts and spot-checks).

The middle two exist because one hash cannot do both of their jobs. Chemistry writes the same thing many ways, and every one of these sentences has the same match_hash:

As stored	Strict projection
The solvent is $H_2O$.	The solvent is H_{2}O.
The solvent is H$_2$O.	The solvent is H_{2}O.
The solvent is H and a subscript 2, then O.	The solvent is H_{2}O.
The solvent is H₂O.	The solvent is H_{2}O.
The solvent is $H_2 O$. in Typst	The solvent is H_{2}O.

(The vectors also include a sixth spelling, $\mathrm{H}_{2}\mathrm{O}$, with the same match_hash.)

OCR often drops a superscript, and "103" must never be certified as agreeing with 10³. So "A rate of 103 per second." and "A rate of 10³ per second." share an align_key, which proposes that they are the same sentence, while their match_hash values differ, which raises a suspect of kind number for a person to resolve. The same care keeps "The energy" followed by citation 3 from matching the text "The energy3".

The cost is that notation too complex to fold, such as a fraction, is compared by its source text alone, so two spellings of one complex formula do not match (chapter 5.3).

Pinned by vectors/projections.jsonl, vectors/projection-groups.jsonl and vectors/hashes.jsonl. See chapter 5.

Corrections

A document is never edited in place. Every fix is a correction: an operation with its evidence, its author and a reason, appended to corrections.jsonl.

{"format":"scienting-correction/1","index":1,"op":"replace","target":"c20cna6",
 "find":"3l0.0","replace":"310.0",
 "evidence":[{"kind":"crop","sha256":"9b6fe6…"}],"author":{"key":"ana"},
 "reason":"OCR read the digit 1 as the letter l.","at":"2026-10-02T10:00:00Z",
 "before":"2fd1e9…","after":"21f13b…","touched":["c20cna6"]}

A document's history is its first reading plus its corrections, and its revision is the index of the last one. The operations are replace, insert, delete, split, merge, move, reclassify, relabel and set. Evidence is a crop (an image of the page region), a witness, a source file by SHA-256, or a named person.

A gate refuses any correction that would change a digit without a crop of the page as evidence, or that would let a digit change anywhere it did not declare. The same gate also refuses an insert without a crop and a find that does not occur exactly once. Digits are the characters of Unicode category Nd.

Pinned by vectors/corrections.jsonl, which applies each operation and each refusal to a small document. See chapter 11.

Anchors

Corrections change nodes and their hashes, so a note that points at a node has to survive them. An anchor is how anything points into a document: a quotation that must match the paper word for word, an annotation, a dataset value, a draft's citation. It is a pinned address with enough context to find its text again after the document changes:

{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic-damb","node":"hca6dc5","kind":"sentence",
 "own_hash":"d5b2eb…","node_hash":"cd6cdb…","span":[29,44],
 "quote":{"exact":"first test case","prefix":"Water, H$_2$O, served as the ","suffix":", with a bending band near 1595 "},
 "rev":12}

rev is the document's revision when the anchor was made. It counts corrections, not clock time, because anchors are made on different machines and clocks disagree. The prefix and suffix are up to 32 code points of context on each side of the quote.

An anchor resolves one of five ways when the document changes. Resolution first follows the ID table (aliases and successors), then looks for the quote, using the prefix and suffix to choose among repeats, then judges by the kind of node:

Result	Means	In the example
current	Nothing it points at has changed.	A note on the water sentence.
changed-below	A container is unchanged in itself while something inside it changed; a drift figure says how much.	A note on Table 1 after rows were replaced and added.
moved	Its node was merged into another that still holds the quote.	A note on "least squares" in half of a sentence that was split at a line break, then rejoined.
reattached	Its node changed and the quote is still there exactly once, so it is re-pinned.	A note on "treated as zero" in a sentence whose superscript was fixed.
stale	It can no longer be trusted; a person must re-anchor it.	A note on the cell that read "3l0.0" before it was corrected to "310.0".

Drift is the share of the leaves below a container that corrections have touched since the anchor's revision; the default threshold for "high" is one quarter (chapter 6.6). Pinned by vectors/resolve.jsonl, which resolves 17 anchors against the example. See chapter 6; the walkthrough shows all five.

Close relative: the W3C Web Annotation Data Model defines selectors, including a TextQuoteSelector (exact text with optional prefix and suffix) and a TextPositionSelector that the same document calls very brittle when the resource changes. Its remedy for change is a state that pins a representation of the whole resource. An anchor adds a node ID, two hash pins and the five defined outcomes. The Hypothesis anchoring approach also stores a quote with 32 characters of context, and reattaches by fuzzy matching where this specification defines exact outcomes. For annotation in general, the W3C model and its existing clients are the better choice; the Layers section below uses it directly. See the comparison page.

Fidelity

Every leaf has a level that says how far its text was checked.

Level	Means
F0 converted	One reading, not corroborated.
F1 resolved	Every doubt about it was answered against the page image.
F2 corroborated	An independent witness agrees exactly, or the text was born digital.
F3 proofread	A named person compared it with the page and signed for its current text.

A level counts only at the node's current hash, so a correction drops a proofread sentence back until someone signs for it again. The pipeline records F0, F1 or F2; F3 comes only from a valid receipt. The example reports 27 leaves at F3, 13 at F2, 4 at F1 and 1 at F0, which is its 45 leaves. See chapter 8.

Receipts and spot-checks

Nobody signs three thousand nodes one at a time (a 30-page paper has about that many). A reviewer signs a receipt instead: one signature over a Merkle root of everything reviewed in a sitting, with the evidence looked at. A Merkle root is the single hash at the top of a tree of hashes, so one signature commits to every item below it. In the example, the reviewer ana signs one receipt for 23 nodes: the cells of Table 1's three data rows and its four column headers, the equation and the sentences around it.

Receipts use an RFC 6962 style tree over the reviewed nodes (chapter 8.5). RFC 6962 has since been replaced by RFC 9162, which gives the same tree definition. One difference: this specification makes a receipt with no items invalid, while the RFC defines a hash for an empty list.

Signing once per sitting invites rubber-stamping, so a second reviewer spot-checks a sample. The sample is chosen by the checker's Ed25519 signature over the receipt, which neither reviewer can steer and anyone can recompute. By default the sample is 5% of the nodes with a minimum of 10, so for the 23-node receipt it is 10 nodes.

The record states a bound on each reviewer's error rate across samples: zero errors in 10 sampled nodes bounds the rate below 26%, and zero errors in 300 bounds it below 1%. These are one-sided upper confidence bounds at 95% confidence (Clopper-Pearson); with no errors the bound is about 3 divided by the sample size.

A spot-check that finds an error suspends the receipt; a revocation withdraws it. Either way the nodes fall back to the level they had without it. Pinned by vectors/merkle.jsonl, vectors/spotcheck.jsonl and vectors/bounds.jsonl. See chapter 8.

Page fingerprints

Several publishers stamp each download with the institution and time, so two teammates' copies of one paper rarely share a SHA-256. Each page also has a fingerprint: a hash of its size, its text with the stamp lines removed, and its images.

Two downloads of the same page stamped "Downloaded via UNIVERSITY OF EXAMPLE on October 11, 2026" and "Downloaded via EXAMPLE PHARMA INC on March 2, 2027" share the fingerprint 6272e6f3…; change one digit of the text and it becomes b545d285…. Geometry and digitizations point at fingerprints, so a re-download keeps every box and every value read off a plot.

The limit is in the stamp rules. A missed stamp splits one edition into two, and an over-broad rule hides a real difference. Pinned by vectors/fingerprints.jsonl. See chapter 7.

Keeping text private

A team may annotate a paper it may not share. A checks-only package keeps every ID, label, page and box, and withholds the text. It also keys every hash, because a plain hash of a short text such as "310.0" is recovered by hashing guesses. A keyed hash (an HMAC) needs a secret key to compute, so guessing does not work without the key.

Values are plain SHA-256 inside the team. Exported to another of the team's machines, they are HMACs under the team's key. In a public layer, or in an export outside the team, the key is derived from the paper's page fingerprints (see the previous section), so only someone holding the paper can check them. Where the values go chooses the key, not whether the text is withheld. fixtures/packages/hale2026harmonic-damb.checks-only is the example paper exported to another of the team's machines, so it uses the team key.

A checks-only package also has no corrections, suspects or datasets. Pinned by vectors/keyed.jsonl. See chapter 5.8.

Rights

What a team may do with a paper depends on how it holds the paper. Every witness carries a rights record: the license as stated, the basis (an open license, a subscription, a collective license, the team's own work), and which uses that allows, including AI uses by purpose and by where the processing runs. Every processing stage declares what it needs and runs only when the rights allow it. A use that is not recorded counts as unknown, and unknown counts as no.

The example PDF is held on a subscription that allows text mining and internal AI use on the team's own machines. Conversion by a local model is allowed. Conversion by an outside service is refused for lacking ai/internal/outside. Model review in the team's cloud is refused for lacking ai/internal/tenancy? (the question mark means unknown) until a project owner signs a policy that settles it. A policy can settle an unknown use and cannot override an explicit yes or no. The refusal for ai/internal/outside, which the record sets to false, stands.

The record also has an end date, after which every use is false. Nothing here is legal advice. Pinned by vectors/rights.jsonl. See chapter 14.

Layers and annotations

Everything people and models add is an annotation, grouped in layers. The example layer is "Smith lab reading of Hale and Osei", for a student audience, visible to the team.

Annotations are W3C Web Annotations whose selector is an anchor. A custom AnchorSelector carries the anchor, followed by a standard TextQuoteSelector as a fallback for readers that do not know the first; there is no TextPositionSelector or FragmentSelector. This gloss, shortened, explains a phrase for a first-year student:

{"type":"Annotation","motivation":"describing","status":"confirmed",
 "body":{"type":"TextualBody","kind":"gloss","audience":"novice",
         "value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},
 "target":{"type":"SpecificResource","source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb",
           "selector":[{"type":"AnchorSelector","anchor":{"node":"rr20tev","span":[111,130],"…":"…"}},
                       {"type":"TextQuoteSelector","exact":"bond force constant"}]}}

One rule decides what is a claim and what is a layer. A statement true independently of the paper is a claim, which is checked and leveled. A statement about reading the paper belongs in a layer, where it is attributed and signed and not checked. Claims are not defined in this repository.

Model-written text is always a generated annotation, never source, and a model's annotation stays proposed until a person confirms it. Whether a generic Web Annotation client displays an AnchorSelector has not been tested; the fallback exists so that it can still use the quote. Layers are candidate. Pinned by the layer records in fixtures/records/layer/. See chapter 18.

Exercises

Every sentence and cell has an address, so the answer to "find the evidence" is an address, and an address can be graded. An exercise asks a student to find the cell giving the Set A bond constant for CT–CT. A student who points at c20cna6 scores 1 of 1 with "That is the cell."; one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."

Students submit under pseudonyms, and a key whose anchors have gone stale is never used to grade. Pinned by vectors/grading.jsonl. The exercise, submission and grade formats are provisional: no course has used them yet, and the first that does will shape them. See chapter 19.

Datasets, digitizations and views

Numbers live in prose, in tables and inside figures; all three should end up as data a team can plot and cite. The dataset, digitization and view formats are all provisional: complete, and expected to change as implementations gain experience.

A dataset is a Frictionless Data Package with units, uncertainty, and an origin for each resource or row. The example's Table 1 becomes a CSV whose every value is anchored to the cell that prints it, digits exactly as printed:

key,bond,set,k_r,r_eq
ct-ct-a,CT–CT,A,310.0,1.526
ct-ct-b,CT–CT,B,305.2,1.5260

A digitization records how values were read off a plot (axes, calibration, series, uncertainty) and counts only once a person confirms it against an overlay. A view is a Vega-Lite specification that names its data by dataset ID and hash. See chapters 15, 16 and 17.

Close relative: a Frictionless Data Package is "a simple container format for describing a coherent collection of data in a single package". The dataset format adopts it and adds units, uncertainty, origin and per-value anchors. The spec does not state which Data Package version it uses. See the comparison page.

Packages, databases and bundles

A document travels as a package: a folder of canonical JSON files, or the same folder zipped.

hale2026harmonic-damb/
  MANIFEST.json          every other file, with its SHA-256; its own hash pins the package
  document.json          the work, version, metadata, policy, root hash and fidelity
  nodes.jsonl            one node per line, in reading order
  witnesses.json         each witness and its rights
  pages.jsonl            each page and its fingerprint
  locations.jsonl        where each node is on each page
  references.jsonl       parsed references
  corrections.jsonl      every correction, in order
  receipts.jsonl         receipts and revocations
  ids.jsonl              retired IDs and legacy aliases
  suspects.jsonl         suspects not yet resolved
  roster.json            the keys that signed
  provenance.json        tools, versions and where each stage ran
  assets/                images, by SHA-256
  datasets/              datasets reported from the document
  derived/               MathML and other renderings, rebuildable

Only MANIFEST.json, document.json, nodes.jsonl, witnesses.json and provenance.json are required, and an optional file with no records is left out; the example package omits suspects.jsonl. Every line is canonical JSON, so a corrected sentence is a one-line diff, and every file has exactly one valid byte sequence. This specification defines provenance.json, which records the tools and versions used and, for each stage, the processor where it ran.

The same data loads into tables that run on SQLite, Dolt and PostgreSQL, and unloads back to the same bytes. A bundle is one SQLite file with every table and a full-text index, which a reader or a server can query offline. The bundle and project formats are provisional; the package format is candidate. See chapters 12 and 13.

Close relatives: BagIt (RFC 8493) also packages files with a manifest of checksums, and is the better choice for transferring arbitrary payloads to archives; MANIFEST.json also lists files with checksums, in a different layout. RO-Crate describes a folder's contents with schema.org terms in JSON-LD and suits general research-object description; neither it nor BagIt defines addressing of sentences or cells. For provenance, W3C PROV-O is a general W3C vocabulary (entities, activities, agents) that other systems can query generically; this specification does not use it, and no mapping between it and provenance.json is defined. See the comparison page.