paperspec/docs/concepts.md

# Concepts

Each section introduces one idea, shows it in the example paper `hale2026harmonic-damb` (a synthetic chemistry paper in [fixtures/packages/](../fixtures/packages/)), and links to the chapter that defines it exactly.

This page is written for three readers.
A newcomer, such as a student who knows what a paper and a citation are and is new to hashes, JSON and annotation standards, gets each term defined where it first appears, with the reason before the rule.
An implementer, who will write a reader, writer, converter or checker, gets the chapter that defines each behavior, the test vector file that pins it, and the traps.
An expert, such as a researcher, standards author or reviewer who knows the neighboring standards, gets the limits and the status of each format, and a short note wherever a close relative exists elsewhere.

The longer comparisons are on [the comparison page](comparison.md).
For a gentler start, read the [overview](overview.md) first; for the exact rules, use the [specification](../spec/README.md).

Two facts apply to everything below.
The specification is at version 0.1.0.
It defines 32 formats, 19 of them `candidate` and 13 `provisional`, as [registry/formats.json](../registry/formats.json) lists.
The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint formats; the sections below say so where they describe them.

Format identifiers such as `scienting-anchor/1` and `scienting-correction/1` begin with `scienting-`, which is the formats' namespace.
The repository's name, paperspec, is a separate matter, and the formats are named after what they hold.
The repository is at <https://git.scient.ing/infra/paperspec>.

## Hashes and canonical JSON

Two tools appear in almost every section.

A *hash* is a short fingerprint computed from data.
This specification uses SHA-256, which turns any sequence of bytes into 64 hexadecimal digits.
The same bytes always give the same digits, and changing one character gives unrelated digits.
Comparing two fingerprints tells you whether two texts are identical without comparing the texts.
The fingerprint cannot be turned back into the text.
The SHA-256 of the cell text `310.0` in the example begins `b478fd…`.
Hashes are written as 64 lowercase hexadecimal digits; six digits are used only for display and for pins in addresses ([chapter 1.3](../spec/01-conventions.md#13-hashes)).

*JSON* is a plain-text format for structured data: objects with named members, lists, strings and numbers.
The same JSON data can be written in many ways (members in any order, spaces anywhere), and a hash of each way would differ. **Canonical JSON** fixes one way, so that every writer produces the same bytes and a hash of the data means the same thing everywhere.

The specification uses the JSON Canonicalization Scheme of [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785): no whitespace, members sorted by name, and numbers written as ECMAScript writes them.
Strings are not Unicode-normalized, and a document containing a lone surrogate, `NaN` or an infinity has no canonical form and is invalid.
The specification adds one rule on top: a decimal that carries meaning is written as a string, so that `310.0` and `1.5260` keep their printed digits.
Every hash of a JSON value here is a hash of its canonical JSON encoded as UTF-8.

RFC 8785 is a building block and says nothing about what to put in the JSON. This specification adopts it for the canonical form and adds the decimal and invalid-value rules above ([chapter 1.1](../spec/01-conventions.md#11-canonical-json), [1.2](../spec/01-conventions.md#12-numbers), [1.9](../spec/01-conventions.md#19-files)).

Pinned by [vectors/canonical-json.jsonl](../vectors/canonical-json.jsonl).
A trap for implementers: the vectors parse every number as an IEEE 754 double, and sort member names as UTF-16 code units, not as bytes.

## Works, versions, witnesses and documents

Work, version, witness and document are easy to confuse, so the formats keep them apart.

| Object | In the example |
| --- | --- |
| A work is the paper across all its versions. | Hale and Osei's paper on harmonic bond parameters, `doi:10.5555/example.hale.2026`. |
| A version is one expression of it: a preprint, the version of record, an erratum, the Supporting Information. | The version of record. |
| A witness is one file of a version, identified by its SHA-256. | The publisher's PDF, witness `qptsanv`. |
| A document is the text established for one version, from one or more witnesses. | `hale2026harmonic-damb`. |

A document's key is its work's key, with a suffix for anything other than the version of record: `hale2026harmonic-damb.si` would be its Supporting Information.
A witness can be a PDF, a JATS file, a LaTeX archive, an HTML page, an EPUB, a Typst project, other XML, an image, or another file; these are the values of its `kind`.
See [chapter 2](../spec/02-model.md).

## Keys

A work key is a work's permanent name, built so that a reader can say it and so that it never collides.
It is a Scholar-style citation key (first author, year, first title word), a hyphen, and four characters of a hash of the work's identity, which is its DOI when it has one:

```text
hale2026harmonic-damb
```

Two works that a citation key alone would name alike, a preprint and its published version, say, get different hashes.
A key is minted once, when the work enters a store, and never recomputed, so a corrected title changes nothing anyone has cited; if two works share four characters, the later one gets six.
A document key adds a suffix for a version other than the version of record: `hale2026harmonic-damb.si` would be the paper's Supporting Information.
See [spec/02 §2.4](../spec/02-model.md#24-work-and-document-keys); `keys.jsonl` pins the cases.

## A document is a tree of nodes

Every part of the text is a node: the document, its sections, headings, paragraphs and sentences, figures and their panels and captions, tables with their rows and cells, equations, footnotes and references.
The example has 75 nodes, 45 of them leaves (nodes that hold text themselves).
This outline shows its two sections and the equation, figure and table in them:

```text
p36k8t6 1 Introduction
tkxsmts    equation (1)
d400rz3 2 Methods
v7cx6qs    figure Figure 1
hj4s7pb    table Table 1
```

`nodes.jsonl` holds one node per line, in document order: the root first, then each node before its children.
Each line has the node's kind, parent, position among its siblings, content and hashes.
A sentence's content is a list of *runs*: plain text, styled text, inline math, citations, cross-references, footnote marks, links, and display equations the sentence runs through.
This is one table cell, shortened:

```json
{"id":"c20cna6","kind":"cell","parent":"g8d2pjb","ord":1,
 "attrs":{"column":"…","role":"data","rowspan":1,"colspan":1},
 "content":[{"kind":"text","text":"310.0"}],
 "fidelity":"F1","text_hash":"…","match_hash":"…","align_key":"…","own_hash":"…","node_hash":"21f13b…"}
```

Page breaks, running heads and columns are not nodes; they are geometry ([chapter 7](../spec/07-geometry.md)).
See [chapter 3](../spec/03-nodes.md).

Close relatives: [JATS 1.4](https://www.niso.org/publications/z3996-2024-jats) is an XML format in which publishers and archives exchange journal content, and TEI is a set of guidelines for encoding scholarly text.
Each is the better choice for its own job, and the paperspec formats do not replace them; a JATS file can be one witness of a document.
The element list in the [JATS 1.4 archiving tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) shows sections, paragraphs, table wrappers and cells and has no sentence element, while [TEI has an `<s>` element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html).
Here the sentence is the unit that gets an ID and hashes.
See [the comparison page](comparison.md).

## IDs are names, not positions

`c20cna6` says nothing about where the cell is.
That is deliberate.
An ID that encoded a page or an ordinal would change when a better converter ordered blocks differently, and one that encoded a kind would change when a paragraph turned out to be a caption.
So an ID encodes no page, ordinal or kind.
It is seven characters, the first a letter, so it is valid as an HTML `id`, a CSS selector and a URL fragment.

An ID is minted once, from a hash of the document key, the node's kind, the hash of its text and an occurrence count (the number of earlier nodes with the same kind and text, so that repeated sentences get different IDs).
Then it is frozen: it is never recomputed, reassigned or reused.
A fresh conversion of the same input mints the same IDs.
A different converter is not guaranteed to, because two readings that differ by one character would mint different IDs for that sentence.

A second reading is aligned with the document sentence by sentence.
Sentences the alignment pairs keep the document's IDs, and every difference becomes a suspect, a recorded doubtful reading that a person resolves ([chapter 5.6](../spec/05-hashes.md#56-alignment) and [5.7](../spec/05-hashes.md#57-suspects)).

One ID is retired and points at its successor when two nodes merge; an artifact's ID is retired with no successor when the artifact is deleted.
Legacy IDs, such as a block ID from an earlier numbering scheme, live on as aliases: in the example, `b-p01-text-003` still leads to sentence `hca6dc5`.
Both the retirements and the aliases are recorded in `ids.jsonl`.

Not every addressable thing is a node.
The columns of a table and the text items found inside a figure panel get IDs by the same rule and carry no node hashes, so they do not have the five hashes described below.
In the example, Table 1's second column has the ID `wdtrcqt`, and the axis label "r (Å)" in Figure 1's first panel has the ID `rtxzjj9`.

Pinned by [vectors/ids.jsonl](../vectors/ids.jsonl).
See [chapter 4](../spec/04-identity.md).

Close relative: in TEI, `xml:id` is assigned by the author, and the Guidelines recommend it ["For durability in the presence of editing"](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html); JATS has an [`id` attribute](https://jats.nlm.nih.gov/archiving/tag-library/1.4/attribute/id.html).
TEI's recommendation gives no rule for minting an ID from content or for retiring one with a successor.
For a text a person edits by hand, an author-assigned `xml:id` is the simpler tool.
See [the comparison page](comparison.md).

## Addresses

An address names a node, and optionally pins it to a state and narrows it to a span:

| Address | Names |
| --- | --- |
| `hale2026harmonic-damb#hca6dc5` | The sentence about water, whatever it says now. |
| `hale2026harmonic-damb#hca6dc5@cd6cdb` | That sentence as it read when its `node_hash` began `cd6cdb`. |
| `hale2026harmonic-damb#hca6dc5@cd6cdb[29:44]` | Code points 29 to 44 of it in that state: "first test case". |
| `smithlab/hale2026harmonic-damb#hca6dc5` | The same sentence in the `smithlab` store. |

The offsets count Unicode code points in the node's plain projection, which is its text as stored with math written between dollar signs.
The dollar signs count, which is why the span starts at 29: the sentence reads `Water, H$_2$O, served as the first test case, …`.

On the web the ID is the fragment, so a link to `…/smithlab/hale2026harmonic-damb#hca6dc5` lands on the sentence.
Pinned by [vectors/addresses.jsonl](../vectors/addresses.jsonl).
See [chapter 6](../spec/06-anchors.md#61-addresses).

## Five hashes, five jobs

Every node carries five content values.
Table columns and figure-text items, from the previous section, do not.

| Value | Covers | Answers |
| --- | --- | --- |
| `text_hash` | The text exactly, normalized. | Is this the same quotation, character for character? |
| `match_hash` | The text with notation folded to one form. | Do two witnesses really agree? |
| `align_key` | The text folded loosely. | Is this probably the same sentence? |
| `own_hash` | The node's own content, without its children. | Has this section itself changed, as opposed to something inside it? |
| `node_hash` | The node and everything below it. | Has anything here changed at all? |

"Normalized" means Unicode normalization form C, runs of whitespace collapsed to one space, and leading and trailing space removed ([chapter 1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)). `own_hash` is a hash of the canonical JSON of the node's own content, such as its runs or its label. `node_hash` is a hash of the canonical JSON of `{"own": own_hash, "children": [each child's node_hash, in order]}`.

Each `node_hash` covers its children's, so a correction to one sentence changes that sentence's values and the `node_hash` of each ancestor, and nothing else.
This design is the idea of a Merkle tree, a tree in which each hash covers the hashes below it.
It is not the RFC 6962 construction that receipts use (see Receipts and spot-checks).

The middle two exist because one hash cannot do both of their jobs.
Chemistry writes the same thing many ways, and every one of these sentences has the same `match_hash`:

| As stored | Strict projection |
| --- | --- |
| `The solvent is $H_2O$.` | `The solvent is H_{2}O.` |
| `The solvent is H$_2$O.` | `The solvent is H_{2}O.` |
| `The solvent is H` and a subscript `2`, then `O.` | `The solvent is H_{2}O.` |
| `The solvent is H₂O.` | `The solvent is H_{2}O.` |
| `The solvent is $H_2 O$.` in Typst | `The solvent is H_{2}O.` |

(The vectors also include a sixth spelling, `$\mathrm{H}_{2}\mathrm{O}$`, with the same `match_hash`.)

OCR often drops a superscript, and "103" must never be certified as agreeing with 10³. So "A rate of 103 per second." and "A rate of 10³ per second." share an `align_key`, which proposes that they are the same sentence, while their `match_hash` values differ, which raises a suspect of kind `number` for a person to resolve.
The same care keeps "The energy" followed by citation 3 from matching the text "The energy3".

The cost is that notation too complex to fold, such as a fraction, is compared by its source text alone, so two spellings of one complex formula do not match ([chapter 5.3](../spec/05-hashes.md#53-the-strict-projection)).

Pinned by [vectors/projections.jsonl](../vectors/projections.jsonl), [vectors/projection-groups.jsonl](../vectors/projection-groups.jsonl) and [vectors/hashes.jsonl](../vectors/hashes.jsonl).
See [chapter 5](../spec/05-hashes.md).

## Corrections

A document is never edited in place.
Every fix is a correction: an operation with its evidence, its author and a reason, appended to `corrections.jsonl`.

```json
{"format":"scienting-correction/1","index":1,"op":"replace","target":"c20cna6",
 "find":"3l0.0","replace":"310.0",
 "evidence":[{"kind":"crop","sha256":"9b6fe6…"}],"author":{"key":"ana"},
 "reason":"OCR read the digit 1 as the letter l.","at":"2026-10-02T10:00:00Z",
 "before":"2fd1e9…","after":"21f13b…","touched":["c20cna6"]}
```

A document's history is its first reading plus its corrections, and its revision is the index of the last one.
The operations are `replace`, `insert`, `delete`, `split`, `merge`, `move`, `reclassify`, `relabel` and `set`.
Evidence is a crop (an image of the page region), a witness, a source file by SHA-256, or a named person.

A gate refuses any correction that would change a digit without a crop of the page as evidence, or that would let a digit change anywhere it did not declare.
The same gate also refuses an `insert` without a crop and a `find` that does not occur exactly once.
Digits are the characters of Unicode category Nd.

Pinned by [vectors/corrections.jsonl](../vectors/corrections.jsonl), which applies each operation and each refusal to a small document.
See [chapter 11](../spec/11-corrections.md).

## Anchors

Corrections change nodes and their hashes, so a note that points at a node has to survive them.
An anchor is how anything points into a document: a quotation that must match the paper word for word, an annotation, a dataset value, a draft's citation.
It is a pinned address with enough context to find its text again after the document changes:

```json
{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic-damb","node":"hca6dc5","kind":"sentence",
 "own_hash":"d5b2eb…","node_hash":"cd6cdb…","span":[29,44],
 "quote":{"exact":"first test case","prefix":"Water, H$_2$O, served as the ","suffix":", with a bending band near 1595 "},
 "rev":12}
```

`rev` is the document's revision when the anchor was made.
It counts corrections, not clock time, because anchors are made on different machines and clocks disagree.
The `prefix` and `suffix` are up to 32 code points of context on each side of the quote.

An anchor resolves one of five ways when the document changes.
Resolution first follows the ID table (aliases and successors), then looks for the quote, using the prefix and suffix to choose among repeats, then judges by the kind of node:

| Result | Means | In the example |
| --- | --- | --- |
| `current` | Nothing it points at has changed. | A note on the water sentence. |
| `changed-below` | A container is unchanged in itself while something inside it changed; a drift figure says how much. | A note on Table 1 after rows were replaced and added. |
| `moved` | Its node was merged into another that still holds the quote. | A note on "least squares" in half of a sentence that was split at a line break, then rejoined. |
| `reattached` | Its node changed and the quote is still there exactly once, so it is re-pinned. | A note on "treated as zero" in a sentence whose superscript was fixed. |
| `stale` | It can no longer be trusted; a person must re-anchor it. | A note on the cell that read "3l0.0" before it was corrected to "310.0". |

Drift is the share of the leaves below a container that corrections have touched since the anchor's revision; the default threshold for "high" is one quarter ([chapter 6.6](../spec/06-anchors.md#66-drift)).
Pinned by [vectors/resolve.jsonl](../vectors/resolve.jsonl), which resolves 17 anchors against the example.
See [chapter 6](../spec/06-anchors.md); the [walkthrough](walkthrough.md#5-what-happened-to-the-notes) shows all five.

Close relative: the W3C Web Annotation Data Model defines selectors, including a [`TextQuoteSelector`](https://www.w3.org/TR/annotation-model/) (exact text with optional prefix and suffix) and a `TextPositionSelector` that the same document calls very brittle when the resource changes.
Its remedy for change is a state that pins a representation of the whole resource.
An anchor adds a node ID, two hash pins and the five defined outcomes.
The [Hypothesis anchoring approach](https://web.hypothes.is/blog/fuzzy-anchoring/) also stores a quote with 32 characters of context, and reattaches by fuzzy matching where this specification defines exact outcomes.
For annotation in general, the W3C model and its existing clients are the better choice; the Layers section below uses it directly.
See [the comparison page](comparison.md).

## Fidelity

Every leaf has a level that says how far its text was checked.

| Level | Means |
| --- | --- |
| F0 converted | One reading, not corroborated. |
| F1 resolved | Every doubt about it was answered against the page image. |
| F2 corroborated | An independent witness agrees exactly, or the text was born digital. |
| F3 proofread | A named person compared it with the page and signed for its current text. |

A level counts only at the node's current hash, so a correction drops a proofread sentence back until someone signs for it again.
The pipeline records F0, F1 or F2; F3 comes only from a valid receipt.
The example reports 27 leaves at F3, 13 at F2, 4 at F1 and 1 at F0, which is its 45 leaves.
See [chapter 8](../spec/08-fidelity.md).

## Receipts and spot-checks

Nobody signs three thousand nodes one at a time (a 30-page paper has about that many).
A reviewer signs a **receipt** instead: one signature over a Merkle root of everything reviewed in a sitting, with the evidence looked at.
A Merkle root is the single hash at the top of a tree of hashes, so one signature commits to every item below it.
In the example, the reviewer `ana` signs one receipt for 23 nodes: the cells of Table 1's three data rows and its four column headers, the equation and the sentences around it.

Receipts use an RFC 6962 style tree over the reviewed nodes ([chapter 8.5](../spec/08-fidelity.md#85-merkle-roots)).
RFC 6962 has since been replaced by [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), which gives the same tree definition.
One difference: this specification makes a receipt with no items invalid, while the RFC defines a hash for an empty list.

Signing once per sitting invites rubber-stamping, so a second reviewer spot-checks a sample.
The sample is chosen by the checker's Ed25519 signature over the receipt, which neither reviewer can steer and anyone can recompute.
By default the sample is 5% of the nodes with a minimum of 10, so for the 23-node receipt it is 10 nodes.

The record states a bound on each reviewer's error rate across samples: zero errors in 10 sampled nodes bounds the rate below 26%, and zero errors in 300 bounds it below 1%. These are one-sided upper confidence bounds at 95% confidence (Clopper-Pearson); with no errors the bound is about 3 divided by the sample size.

A spot-check that finds an error suspends the receipt; a revocation withdraws it.
Either way the nodes fall back to the level they had without it.
Pinned by [vectors/merkle.jsonl](../vectors/merkle.jsonl), [vectors/spotcheck.jsonl](../vectors/spotcheck.jsonl) and [vectors/bounds.jsonl](../vectors/bounds.jsonl).
See [chapter 8](../spec/08-fidelity.md#83-receipts).

## Page fingerprints

Several publishers stamp each download with the institution and time, so two teammates' copies of one paper rarely share a SHA-256.
Each page also has a fingerprint: a hash of its size, its text with the stamp lines removed, and its images.

Two downloads of the same page stamped "Downloaded via UNIVERSITY OF EXAMPLE on October 11, 2026" and "Downloaded via EXAMPLE PHARMA INC on March 2, 2027" share the fingerprint `6272e6f3…`; change one digit of the text and it becomes `b545d285…`.
Geometry and digitizations point at fingerprints, so a re-download keeps every box and every value read off a plot.

The limit is in the stamp rules.
A missed stamp splits one edition into two, and an over-broad rule hides a real difference.
Pinned by [vectors/fingerprints.jsonl](../vectors/fingerprints.jsonl).
See [chapter 7](../spec/07-geometry.md).

## Keeping text private

A team may annotate a paper it may not share.
A **checks-only** package keeps every ID, label, page and box, and withholds the text.
It also keys every hash, because a plain hash of a short text such as "310.0" is recovered by hashing guesses.
A keyed hash (an HMAC) needs a secret key to compute, so guessing does not work without the key.

Values are plain SHA-256 inside the team.
Exported to another of the team's machines, they are HMACs under the team's key.
In a public layer, or in an export outside the team, the key is derived from the paper's page fingerprints (see the previous section), so only someone holding the paper can check them.
Where the values go chooses the key, not whether the text is withheld.
[fixtures/packages/hale2026harmonic-damb.checks-only](../fixtures/packages/hale2026harmonic-damb.checks-only) is the example paper exported to another of the team's machines, so it uses the team key.

A checks-only package also has no corrections, suspects or datasets.
Pinned by [vectors/keyed.jsonl](../vectors/keyed.jsonl).
See [chapter 5.8](../spec/05-hashes.md#58-keyed-hashes).

## Rights

What a team may do with a paper depends on how it holds the paper.
Every witness carries a rights record: the license as stated, the basis (an open license, a subscription, a collective license, the team's own work), and which uses that allows, including AI uses by purpose and by where the processing runs.
Every processing stage declares what it needs and runs only when the rights allow it.
A use that is not recorded counts as unknown, and unknown counts as no.

The example PDF is held on a subscription that allows text mining and internal AI use on the team's own machines.
Conversion by a local model is allowed.
Conversion by an outside service is refused for lacking `ai/internal/outside`.
Model review in the team's cloud is refused for lacking `ai/internal/tenancy?` (the question mark means unknown) until a project owner signs a policy that settles it.
A policy can settle an unknown use and cannot override an explicit yes or no.
The refusal for `ai/internal/outside`, which the record sets to false, stands.

The record also has an end date, after which every use is false.
Nothing here is legal advice.
Pinned by [vectors/rights.jsonl](../vectors/rights.jsonl).
See [chapter 14](../spec/14-rights.md).

## Layers and annotations

Everything people and models add is an annotation, grouped in layers.
The example layer is "Smith lab reading of Hale and Osei", for a student audience, visible to the team.

Annotations are W3C Web Annotations whose selector is an anchor.
A custom `AnchorSelector` carries the anchor, followed by a standard `TextQuoteSelector` as a fallback for readers that do not know the first; there is no `TextPositionSelector` or `FragmentSelector`.
This gloss, shortened, explains a phrase for a first-year student:

```json
{"type":"Annotation","motivation":"describing","status":"confirmed",
 "body":{"type":"TextualBody","kind":"gloss","audience":"novice",
         "value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},
 "target":{"type":"SpecificResource","source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb",
           "selector":[{"type":"AnchorSelector","anchor":{"node":"rr20tev","span":[111,130],"…":"…"}},
                       {"type":"TextQuoteSelector","exact":"bond force constant"}]}}
```

One rule decides what is a claim and what is a layer.
A statement true independently of the paper is a claim, which is checked and leveled.
A statement about reading the paper belongs in a layer, where it is attributed and signed and not checked.
Claims are not defined in this repository.

Model-written text is always a `generated` annotation, never source, and a model's annotation stays `proposed` until a person confirms it.
Whether a generic Web Annotation client displays an `AnchorSelector` has not been tested; the fallback exists so that it can still use the quote.
Layers are `candidate`.
Pinned by the layer records in `fixtures/records/layer/`.
See [chapter 18](../spec/18-layers.md).

## Exercises

Every sentence and cell has an address, so the answer to "find the evidence" is an address, and an address can be graded.
An exercise asks a student to find the cell giving the Set A bond constant for CT–CT. A student who points at `c20cna6` scores 1 of 1 with "That is the cell."; one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."

Students submit under pseudonyms, and a key whose anchors have gone stale is never used to grade.
Pinned by [vectors/grading.jsonl](../vectors/grading.jsonl).
The exercise, submission and grade formats are `provisional`: no course has used them yet, and the first that does will shape them.
See [chapter 19](../spec/19-exercises.md).

## Datasets, digitizations and views

Numbers live in prose, in tables and inside figures; all three should end up as data a team can plot and cite.
The dataset, digitization and view formats are all `provisional`: complete, and expected to change as implementations gain experience.

A dataset is a Frictionless Data Package with units, uncertainty, and an origin for each resource or row.
The example's Table 1 becomes a CSV whose every value is anchored to the cell that prints it, digits exactly as printed:

```text
key,bond,set,k_r,r_eq
ct-ct-a,CT–CT,A,310.0,1.526
ct-ct-b,CT–CT,B,305.2,1.5260
```

A digitization records how values were read off a plot (axes, calibration, series, uncertainty) and counts only once a person confirms it against an overlay.
A view is a Vega-Lite specification that names its data by dataset ID and hash.
See chapters [15](../spec/15-datasets.md), [16](../spec/16-digitization.md) and [17](../spec/17-views.md).

Close relative: a [Frictionless Data Package](https://datapackage.org/standard/data-package/) is "a simple container format for describing a coherent collection of data in a single package".
The dataset format adopts it and adds units, uncertainty, origin and per-value anchors.
The spec does not state which Data Package version it uses.
See [the comparison page](comparison.md).

## Packages, databases and bundles

A document travels as a package: a folder of canonical JSON files, or the same folder zipped.

```text
hale2026harmonic-damb/
  MANIFEST.json          every other file, with its SHA-256; its own hash pins the package
  document.json          the work, version, metadata, policy, root hash and fidelity
  nodes.jsonl            one node per line, in reading order
  witnesses.json         each witness and its rights
  pages.jsonl            each page and its fingerprint
  locations.jsonl        where each node is on each page
  references.jsonl       parsed references
  corrections.jsonl      every correction, in order
  receipts.jsonl         receipts and revocations
  ids.jsonl              retired IDs and legacy aliases
  suspects.jsonl         suspects not yet resolved
  roster.json            the keys that signed
  provenance.json        tools, versions and where each stage ran
  assets/                images, by SHA-256
  datasets/              datasets reported from the document
  derived/               MathML and other renderings, rebuildable
```

Only `MANIFEST.json`, `document.json`, `nodes.jsonl`, `witnesses.json` and `provenance.json` are required, and an optional file with no records is left out; the example package omits `suspects.jsonl`.
Every line is canonical JSON, so a corrected sentence is a one-line diff, and every file has exactly one valid byte sequence.
This specification defines `provenance.json`, which records the tools and versions used and, for each stage, the processor where it ran.

The same data loads into tables that run on SQLite, Dolt and PostgreSQL, and unloads back to the same bytes.
A bundle is one SQLite file with every table and a full-text index, which a reader or a server can query offline.
The bundle and project formats are `provisional`; the package format is `candidate`.
See chapters [12](../spec/12-package.md) and [13](../spec/13-database.md).

Close relatives: [BagIt](https://www.rfc-editor.org/rfc/rfc8493) (RFC 8493) also packages files with a manifest of checksums, and is the better choice for transferring arbitrary payloads to archives; `MANIFEST.json` also lists files with checksums, in a different layout. [RO-Crate](https://www.researchobject.org/ro-crate/) describes a folder's contents with schema.org terms in JSON-LD and suits general research-object description; neither it nor BagIt defines addressing of sentences or cells.
For provenance, [W3C PROV-O](https://www.w3.org/TR/prov-o/) is a general W3C vocabulary (entities, activities, agents) that other systems can query generically; this specification does not use it, and no mapping between it and `provenance.json` is defined.
See [the comparison page](comparison.md).