infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: minted keys, VRF seeds, RFC 9162 proofs, history roots, tag URIs and a transparency log

Browse Source 
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
  Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
  where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
  openalex in same_as, else the folded title, year and family name. A
  collision lengthens the later key by two characters; a key is minted
  once and never recomputed. A document key adds .si, .v<seq> or .<type>
  for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
  okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
  every node and record ID was re-minted and the docs rewritten to match.
  The validator checks the document key against the work key and version
  (doc.key.work) and the hash part against the identifier (doc.key.suffix).
  vectors/keys.jsonl pins the cases paperstack's tests pin.

Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
  implemented in tools/oracle/vrf.py in plain Python and reproduced
  against the RFC's Example 16; the sample ranks by the VRF output. The
  earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
  verify_inclusion, consistency_proof, verify_consistency) with
  scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
  the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
  checkpoints with key hashes as transparency logs share them, the rule
  that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).

Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
  namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
  the checker refuses another version.

Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
  checkpoint records; a check that every ID and hash the prose quotes
  exists in the fixtures or vectors; three more broken packages
  (history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
  keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
  section, media-type registration and OCI distribution notes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 23:21:57 -04:00
1 parent 76d76353d9
commit a09d158e16
307 files changed
+2698 -1497
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/comparison.md
+9 -9
View File
@@ -19,11 +19,11 @@ The identifiers appear inside hashed data and test vectors, so they are not rena
Suppose a research team keeps fifty papers it needs to understand completely.
A page number names a region of a PDF and cannot name a statement, and it changes between a preprint and the published version.
The team wants to point at one sentence or one table cell, such as `hale2026harmonic#sxgcc9h@76e06b[111:130]` in the fixtures, and have that pointer keep working after someone fixes a misread digit elsewhere in the paper.
The team wants to point at one sentence or one table cell, such as `hale2026harmonic-damb#rr20tev@76e06b[111:130]` in the fixtures, and have that pointer keep working after someone fixes a misread digit elsewhere in the paper.
The pointer has four parts:
- The document key, `hale2026harmonic`.
- After `#`, the node ID, `sxgcc9h`.
- The document key, `hale2026harmonic-damb`.
- After `#`, the node ID, `rr20tev`.
- After `@`, the first six digits of the node's hash, `76e06b`, so the pointer names the sentence as it then read.
- In brackets, the code-point range of the words inside the sentence, `[111:130]`.
It also wants to know who compared a passage with the page image, and how far.
@@ -67,7 +67,7 @@ The design choices that matter for comparison:
Resolving it gives one of five results: `current`, `changed-below`, `moved`, `reattached` or `stale`, with a drift measure for `changed-below` ([spec/06](../spec/06-anchors.md)).
- **Corrections:** Corrections are appended, never edited.
Each carries evidence, an author and a reason, and a gate refuses a digit change without image evidence ([spec/11](../spec/11-corrections.md)).
In `fixtures/packages/hale2026harmonic`, correction 1 replaces `3l0.0` with `310.0` because "OCR (optical character recognition) read the digit 1 as the letter l", with a crop of the page as evidence.
In `fixtures/packages/hale2026harmonic-damb`, correction 1 replaces `3l0.0` with `310.0` because "OCR (optical character recognition) read the digit 1 as the letter l", with a crop of the page as evidence.
- **Fidelity and receipts:** Levels F0 (converted) to F3 (proofread).
F3 comes only from a signed receipt that lists the node at its current `node_hash`; a correction drops the node back ([spec/08](../spec/08-fidelity.md)).
- **Scale:** The formats are built for a team of roughly five to 20 people with a few dozen to a few hundred papers.
@@ -135,7 +135,7 @@ paperspec is not a publisher interchange format and does not replace JATS ([spec
*Interoperation.*
JATS is an input.
`jats` is a witness kind, and the fixture `okafor2025open` was established from a JATS file and a PDF that agree ([fixtures/README.md](../fixtures/README.md)).
`jats` is a witness kind, and the fixture `okafor2025solvent-acth` was established from a JATS file and a PDF that agree ([fixtures/README.md](../fixtures/README.md)).
The document's `meta` is one CSL-JSON item, not JATS front matter ([spec/02 §2.2](../spec/02-model.md#22-documentjson)).
The specification defines no JATS exporter.
@@ -427,8 +427,8 @@ Keep using it as a reading and distribution format.
*Status.*
The specification is version 0.1.0.
`registry/formats.json` lists 29 formats: 18 are `candidate` (complete, with vectors and fixtures, ready to implement) and 11 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade and checklist ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
The repository holds 23 chapters, 39 JSON Schemas, 21 vector files, five valid fixture packages, one of them also zipped, and 28 invalid ones, each naming the rule it breaks, and a reference checker, `tools/check.py`.
`registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `tools/check.py`.
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; tools Apache-2.0 ([README.md](../README.md#licenses)).
*No adoption claim.*
@@ -466,12 +466,12 @@ These are the places where the guides and the specification differ, and this pag
4. **Five hashes everywhere:** `docs/overview.md` lists the things that get "a permanent ID and five content hashes" without naming them.
Table columns and figure-text items have IDs, are not nodes, and carry no five hashes ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids), [spec/09 §9.1](../spec/09-figures-tables-equations.md#91-figures-and-panels)).
5. **Dataset digits:** spec/15 says values keep their printed digits, and in JSON they are strings.
The fixture descriptor `fixtures/packages/hale2026harmonic/datasets/ds-hale-bond-params/datapackage.json` types the field `k_r` as `number`.
The fixture descriptor `fixtures/packages/hale2026harmonic-damb/datasets/ds-hale-bond-params/datapackage.json` types the field `k_r` as `number`.
How Frictionless tools treat that was not verified, so take care when comparing with Frictionless Table Schema.
6. **Package tree:** `docs/concepts.md` omits `suspects.jsonl` from its package tree, and [spec/12 §12.1](../spec/12-package.md#121-files) lists it.
7. **RFC 6962 is obsolete:** spec/08 §8.5 cites it; RFC 9162 obsoletes it (see above).
8. **Status:** The guides describe the formats as one set.
Of the 29 formats, 11 are `provisional`, including `scienting-dataset/1`, which the datasets paragraphs in `docs/concepts.md` present without that qualifier.
Of the 32 formats, 13 are `provisional`, including `scienting-dataset/1`, which the datasets paragraphs in `docs/concepts.md` present without that qualifier.
*The distinguishing combination, and what it costs.*
paperspec puts in one package permanent sentence- and cell-level IDs, five hashes per node, anchors with explicit resolution outcomes and drift, append-only evidence-gated corrections, signed fidelity receipts with spot-check bounds, per-witness rights with AI-use axes, and value-to-cell anchors.
docs/concepts.md
+41 -27
View File
@@ -1,6 +1,6 @@
# Concepts
Each section introduces one idea, shows it in the example paper `hale2026harmonic` (a synthetic chemistry paper in [fixtures/packages/](../fixtures/packages/)), and links to the chapter that defines it exactly.
Each section introduces one idea, shows it in the example paper `hale2026harmonic-damb` (a synthetic chemistry paper in [fixtures/packages/](../fixtures/packages/)), and links to the chapter that defines it exactly.
This page is written for three readers.
A newcomer, such as a student who knows what a paper and a citation are and is new to hashes, JSON and annotation standards, gets each term defined where it first appears, with the reason before the rule.
@@ -12,8 +12,8 @@ For a gentler start, read the [overview](overview.md) first; for the exact rules
Two facts apply to everything below.
The specification is at version 0.1.0.
It defines 29 formats, 18 of them `candidate` and 11 `provisional`, as [registry/formats.json](../registry/formats.json) lists.
The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade and checklist formats; the sections below say so where they describe them.
It defines 32 formats, 19 of them `candidate` and 13 `provisional`, as [registry/formats.json](../registry/formats.json) lists.
The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint formats; the sections below say so where they describe them.
Format identifiers such as `scienting-anchor/1` and `scienting-correction/1` begin with `scienting-`, which is the formats' namespace.
The repository's name, paperspec, is a separate matter, and the formats are named after what they hold.
@@ -53,12 +53,26 @@ Work, version, witness and document are easy to confuse, so the formats keep the
| A work is the paper across all its versions. | Hale and Osei's paper on harmonic bond parameters, `doi:10.5555/example.hale.2026`. |
| A version is one expression of it: a preprint, the version of record, an erratum, the Supporting Information. | The version of record. |
| A witness is one file of a version, identified by its SHA-256. | The publisher's PDF, witness `qptsanv`. |
| A document is the text established for one version, from one or more witnesses. | `hale2026harmonic`. |
| A document is the text established for one version, from one or more witnesses. | `hale2026harmonic-damb`. |
A document's key is its work's key, with a suffix for anything other than the version of record: `hale2026harmonic.si` would be its Supporting Information.
A document's key is its work's key, with a suffix for anything other than the version of record: `hale2026harmonic-damb.si` would be its Supporting Information.
A witness can be a PDF, a JATS file, a LaTeX archive, an HTML page, an EPUB, a Typst project, other XML, an image, or another file; these are the values of its `kind`.
See [chapter 2](../spec/02-model.md).
## Keys
A work key is a work's permanent name, built so that a reader can say it and so that it never collides.
It is a Scholar-style citation key (first author, year, first title word), a hyphen, and four characters of a hash of the work's identity, which is its DOI when it has one:
```text
hale2026harmonic-damb
```
Two works that a citation key alone would name alike, a preprint and its published version, say, get different hashes.
A key is minted once, when the work enters a store, and never recomputed, so a corrected title changes nothing anyone has cited; if two works share four characters, the later one gets six.
A document key adds a suffix for a version other than the version of record: `hale2026harmonic-damb.si` would be the paper's Supporting Information.
See [spec/02 §2.4](../spec/02-model.md#24-work-and-document-keys); `keys.jsonl` pins the cases.
## A document is a tree of nodes
Every part of the text is a node: the document, its sections, headings, paragraphs and sentences, figures and their panels and captions, tables with their rows and cells, equations, footnotes and references.
@@ -66,11 +80,11 @@ The example has 75 nodes, 45 of them leaves (nodes that hold text themselves).
This outline shows its two sections and the equation, figure and table in them:
```text
bzqvbhd 1 Introduction
qn08tj9 equation (1)
f0tj6ca 2 Methods
ajfvq4s figure Figure 1
gm5z4rx table Table 1
p36k8t6 1 Introduction
tkxsmts equation (1)
d400rz3 2 Methods
v7cx6qs figure Figure 1
hj4s7pb table Table 1
```
`nodes.jsonl` holds one node per line, in document order: the root first, then each node before its children.
@@ -79,7 +93,7 @@ A sentence's content is a list of *runs*: plain text, styled text, inline math,
This is one table cell, shortened:
```json
{"id":"nkm3xn5","kind":"cell","parent":"zhe4g0k","ord":1,
{"id":"c20cna6","kind":"cell","parent":"g8d2pjb","ord":1,
"attrs":{"column":"…","role":"data","rowspan":1,"colspan":1},
"content":[{"kind":"text","text":"310.0"}],
"fidelity":"F1","text_hash":"…","match_hash":"…","align_key":"…","own_hash":"…","node_hash":"21f13b…"}
@@ -96,7 +110,7 @@ See [the comparison page](comparison.md).
## IDs are names, not positions
`nkm3xn5` says nothing about where the cell is.
`c20cna6` says nothing about where the cell is.
That is deliberate.
An ID that encoded a page or an ordinal would change when a better converter ordered blocks differently, and one that encoded a kind would change when a paragraph turned out to be a caption.
So an ID encodes no page, ordinal or kind.
@@ -111,12 +125,12 @@ A second reading is aligned with the document sentence by sentence.
Sentences the alignment pairs keep the document's IDs, and every difference becomes a suspect, a recorded doubtful reading that a person resolves ([chapter 5.6](../spec/05-hashes.md#56-alignment) and [5.7](../spec/05-hashes.md#57-suspects)).
One ID is retired and points at its successor when two nodes merge; an artifact's ID is retired with no successor when the artifact is deleted.
Legacy IDs, such as a block ID from an earlier numbering scheme, live on as aliases: in the example, `b-p01-text-003` still leads to sentence `f929eq9`.
Legacy IDs, such as a block ID from an earlier numbering scheme, live on as aliases: in the example, `b-p01-text-003` still leads to sentence `hca6dc5`.
Both the retirements and the aliases are recorded in `ids.jsonl`.
Not every addressable thing is a node.
The columns of a table and the text items found inside a figure panel get IDs by the same rule and carry no node hashes, so they do not have the five hashes described below.
In the example, Table 1's second column has the ID `kp6grmd`, and the axis label "r (Å)" in Figure 1's first panel has the ID `nhk15ag`.
In the example, Table 1's second column has the ID `wdtrcqt`, and the axis label "r (Å)" in Figure 1's first panel has the ID `rtxzjj9`.
Pinned by [vectors/ids.jsonl](../vectors/ids.jsonl).
See [chapter 4](../spec/04-identity.md).
@@ -132,15 +146,15 @@ An address names a node, and optionally pins it to a state and narrows it to a s
| Address | Names |
| --- | --- |
| `hale2026harmonic#f929eq9` | The sentence about water, whatever it says now. |
| `hale2026harmonic#f929eq9@cd6cdb` | That sentence as it read when its `node_hash` began `cd6cdb`. |
| `hale2026harmonic#f929eq9@cd6cdb[29:44]` | Code points 29 to 44 of it in that state: "first test case". |
| `smithlab/hale2026harmonic#f929eq9` | The same sentence in the `smithlab` store. |
| `hale2026harmonic-damb#hca6dc5` | The sentence about water, whatever it says now. |
| `hale2026harmonic-damb#hca6dc5@cd6cdb` | That sentence as it read when its `node_hash` began `cd6cdb`. |
| `hale2026harmonic-damb#hca6dc5@cd6cdb[29:44]` | Code points 29 to 44 of it in that state: "first test case". |
| `smithlab/hale2026harmonic-damb#hca6dc5` | The same sentence in the `smithlab` store. |
The offsets count Unicode code points in the node's plain projection, which is its text as stored with math written between dollar signs.
The dollar signs count, which is why the span starts at 29: the sentence reads `Water, H$_2$O, served as the first test case, …`.
On the web the ID is the fragment, so a link to `…/smithlab/hale2026harmonic#f929eq9` lands on the sentence.
On the web the ID is the fragment, so a link to `…/smithlab/hale2026harmonic-damb#hca6dc5` lands on the sentence.
Pinned by [vectors/addresses.jsonl](../vectors/addresses.jsonl).
See [chapter 6](../spec/06-anchors.md#61-addresses).
@@ -190,11 +204,11 @@ A document is never edited in place.
Every fix is a correction: an operation with its evidence, its author and a reason, appended to `corrections.jsonl`.
```json
{"format":"scienting-correction/1","index":1,"op":"replace","target":"nkm3xn5",
{"format":"scienting-correction/1","index":1,"op":"replace","target":"c20cna6",
"find":"3l0.0","replace":"310.0",
"evidence":[{"kind":"crop","sha256":"9b6fe6…"}],"author":{"key":"ana"},
"reason":"OCR read the digit 1 as the letter l.","at":"2026-10-02T10:00:00Z",
"before":"2fd1e9…","after":"21f13b…","touched":["nkm3xn5"]}
"before":"2fd1e9…","after":"21f13b…","touched":["c20cna6"]}
```
A document's history is its first reading plus its corrections, and its revision is the index of the last one.
@@ -215,7 +229,7 @@ An anchor is how anything points into a document: a quotation that must match th
It is a pinned address with enough context to find its text again after the document changes:
```json
{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic","node":"f929eq9","kind":"sentence",
{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic-damb","node":"hca6dc5","kind":"sentence",
"own_hash":"d5b2eb…","node_hash":"cd6cdb…","span":[29,44],
"quote":{"exact":"first test case","prefix":"Water, H$_2$O, served as the ","suffix":", with a bending band near 1595 "},
"rev":12}
@@ -309,7 +323,7 @@ Values are plain SHA-256 inside the team.
Exported to another of the team's machines, they are HMACs under the team's key.
In a public layer, or in an export outside the team, the key is derived from the paper's page fingerprints (see the previous section), so only someone holding the paper can check them.
Where the values go chooses the key, not whether the text is withheld.
[fixtures/packages/hale2026harmonic.checks-only](../fixtures/packages/hale2026harmonic.checks-only) is the example paper exported to another of the team's machines, so it uses the team key.
[fixtures/packages/hale2026harmonic-damb.checks-only](../fixtures/packages/hale2026harmonic-damb.checks-only) is the example paper exported to another of the team's machines, so it uses the team key.
A checks-only package also has no corrections, suspects or datasets.
Pinned by [vectors/keyed.jsonl](../vectors/keyed.jsonl).
@@ -347,8 +361,8 @@ This gloss, shortened, explains a phrase for a first-year student:
{"type":"Annotation","motivation":"describing","status":"confirmed",
"body":{"type":"TextualBody","kind":"gloss","audience":"novice",
"value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},
"target":{"type":"SpecificResource","source":"urn:scienting:doc:smithlab/hale2026harmonic",
"selector":[{"type":"AnchorSelector","anchor":{"node":"sxgcc9h","span":[111,130],"…":"…"}},
"target":{"type":"SpecificResource","source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb",
"selector":[{"type":"AnchorSelector","anchor":{"node":"rr20tev","span":[111,130],"…":"…"}},
{"type":"TextQuoteSelector","exact":"bond force constant"}]}}
```
@@ -366,7 +380,7 @@ See [chapter 18](../spec/18-layers.md).
## Exercises
Every sentence and cell has an address, so the answer to "find the evidence" is an address, and an address can be graded.
An exercise asks a student to find the cell giving the Set A bond constant for CT–CT. A student who points at `nkm3xn5` scores 1 of 1 with "That is the cell."; one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."
An exercise asks a student to find the cell giving the Set A bond constant for CT–CT. A student who points at `c20cna6` scores 1 of 1 with "That is the cell."; one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."
Students submit under pseudonyms, and a key whose anchors have gone stale is never used to grade.
Pinned by [vectors/grading.jsonl](../vectors/grading.jsonl).
@@ -401,7 +415,7 @@ See [the comparison page](comparison.md).
A document travels as a package: a folder of canonical JSON files, or the same folder zipped.
```text
hale2026harmonic/
hale2026harmonic-damb/
MANIFEST.json every other file, with its SHA-256; its own hash pins the package
document.json the work, version, metadata, policy, root hash and fidelity
nodes.jsonl one node per line, in reading order
docs/examples.md
+31 -31
View File
@@ -41,10 +41,10 @@ printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
```
The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic`:
The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic-damb`:
```sh
grep -o '"id":"j06tfhn".*' fixtures/packages/hale2026harmonic/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
grep -o '"id":"v3063qv".*' fixtures/packages/hale2026harmonic-damb/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
```
```text
@@ -58,14 +58,14 @@ A package is a folder of JSON (JavaScript Object Notation, a plain-text format f
`MANIFEST.json` lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package ([chapter 12](../spec/12-package.md#123-the-manifest)).
```sh
cd fixtures/packages/okafor2025open
cd fixtures/packages/okafor2025solvent-acth
sha256sum nodes.jsonl
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
```
```text
0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94 nodes.jsonl
"path":"nodes.jsonl","sha256":"0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94"
09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3 nodes.jsonl
"path":"nodes.jsonl","sha256":"09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3"
```
## Read a package with nothing but a JSON parser
@@ -108,26 +108,26 @@ for n in nodes:
```
```sh
python3 outline.py fixtures/packages/hale2026harmonic
python3 outline.py fixtures/packages/hale2026harmonic-damb
```
```text
bzqvbhd 1 Introduction
qn08tj9 equation (1)
f0tj6ca 2 Methods
ajfvq4s figure Figure 1
gm5z4rx table Table 1
p36k8t6 1 Introduction
tkxsmts equation (1)
d400rz3 2 Methods
v7cx6qs figure Figure 1
hj4s7pb table Table 1
```
## Follow an address
This shows how a citation points inside a sentence.
An address names a node, and optionally a span of its text ([chapter 6](../spec/06-anchors.md#61-addresses)).
`hale2026harmonic#f929eq9[29:44]` names code points 29 to 44 of sentence `f929eq9`.
`hale2026harmonic-damb#hca6dc5[29:44]` names code points 29 to 44 of sentence `hca6dc5`.
With `text()` from the script above:
```python
print(text(by_id["f929eq9"])[29:44])
print(text(by_id["hca6dc5"])[29:44])
```
```text
@@ -143,15 +143,15 @@ For text with characters outside the Basic Multilingual Plane the two counts dif
This shows that the same packages can live in a database and be queried with SQL.
```sh
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic fixtures/packages/okafor2025open fixtures/packages/smithlab2027charges
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
```
Without pixi, run `python3 tools/load.py` with the same arguments.
```text
hale2026harmonic: 75 nodes
okafor2025open: 15 nodes
smithlab2027charges: 15 nodes
hale2026harmonic-damb: 75 nodes
okafor2025solvent-acth: 15 nodes
writer2027fixed-ynr7.v3: 15 nodes
```
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
@@ -163,14 +163,14 @@ import json, sqlite3
db = sqlite3.connect("lab.db")
# Sentences the converter left below F2. Receipts can raise them; see below.
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic' "
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic-damb' "
"AND kind = 'sentence' AND fidelity IN ('F0', 'F1') ORDER BY seq"):
print(row)
# Every sentence that cites reference 2: the citation graph is sentence-level.
for row in db.execute("SELECT c.sentence FROM citations c JOIN refs r "
"ON r.doc_key = c.doc_key AND r.node = c.ref_node "
"WHERE c.doc_key = 'hale2026harmonic' AND r.label = '2'"):
"WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2'"):
print(row)
# Who signed what.
@@ -180,26 +180,26 @@ for row in db.execute("SELECT r.signer, r.scope, COUNT(*) FROM receipts r "
# Table 1's body, row by row.
rows = db.execute("SELECT r.id FROM nodes r JOIN nodes b ON b.doc_key = r.doc_key AND b.id = r.parent "
"WHERE r.doc_key = 'hale2026harmonic' AND r.kind = 'row' AND b.kind = 'table-body' "
"WHERE r.doc_key = 'hale2026harmonic-damb' AND r.kind = 'row' AND b.kind = 'table-body' "
"ORDER BY r.seq").fetchall()
for (row,) in rows:
cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic' AND parent = ? ORDER BY ord",
cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic-damb' AND parent = ? ORDER BY ord",
(row,)).fetchall()
print(" | ".join("".join(r.get("text", "") for r in json.loads(c)) for (c,) in cells))
```
```text
('e4fkseg', 'F1')
('r1gacps', 'F0')
('eajq93t', 'F0')
('f929eq9',)
('ec5jc93', 'F1')
('a4jrz2t', 'F0')
('xjns57z', 'F0')
('hca6dc5',)
('ana', 'targeted:lode', 3)
('ana', 'targeted:lode', 23)
('ana', 'section:bzqvbhd', 2)
('ana', 'section:p36k8t6', 2)
('cara', 'page:2', 3)
('ana', 'targeted:captions', 3)
('ben', 'qa:eqgbbpwb6725v', 10)
('ben', 'qa:gena76nkb6cnc', 3)
('ben', 'qa:y2zw8ctg5s8dp', 10)
('ben', 'qa:gx2s84z9jf9xv', 3)
Bond parameters
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
CT–HC | 340.0 | 1.090 | 338.7 | 1.0900
@@ -208,7 +208,7 @@ OW–HW | 553.0 | 0.9572 | 548.1 | 0.9570
The first query reads the level the converter recorded, not the final one.
A receipt is a signed record that a reviewer checked certain nodes against the page.
Receipts by `ana` raise `e4fkseg` and `r1gacps` to F3; `eajq93t` stays at F0 because the only receipt that covered it, by `cara`, was revoked.
Receipts by `ana` raise `ec5jc93` and `a4jrz2t` to F3; `xjns57z` stays at F0 because the only receipt that covered it, by `cara`, was revoked.
To compute the final level, check each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
The `scope` column of the third query shows the value `targeted:lode`, which is data in the fixture, copied here exactly as the query prints it.
@@ -239,11 +239,11 @@ This shows what the checks catch.
Change one digit in a fixture and run the checks:
```sh
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025open/nodes.jsonl
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025solvent-acth/nodes.jsonl
pixi run quick
```
The checker reports three failures: the SHA-256 of `nodes.jsonl` no longer matches the manifest, unloading the package from the database no longer gives the same bytes, and the package hash differs after unloading.
Restore the file with `git checkout fixtures/packages/okafor2025open/nodes.jsonl`.
Restore the file with `git checkout fixtures/packages/okafor2025solvent-acth/nodes.jsonl`.
The broken packages in [fixtures/invalid/](../fixtures/invalid/) are this kind of change, each with the rule it breaks.
`git checkout` restores a tracked file, so run this only in a clone you can reset.
docs/faq.md
+10 -4
View File
@@ -13,12 +13,18 @@ Every file is JSON or JSON Lines (one JSON value per line), and [spec/](../spec/
## Why not keep using PDFs and page numbers?
A page number names a region, not a statement, and it changes between a preprint and the version of record.
An address names a sentence or a cell by its ID, such as `hale2026harmonic#j06tfhn`.
An address names a sentence or a cell by its ID, such as `hale2026harmonic-damb#v3063qv`.
It can add a pin (the first six hex digits of the node's hash) and character offsets, which lets a reader check the address against the exact text ([spec/06 §6.1](../spec/06-anchors.md#61-addresses)).
An ID encodes no page, position or kind.
The system mints it once from the document key, kind, text hash and occurrence, then freezes it ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
## Why does a key end in four odd characters?
`hale2026harmonic-damb` is the paper's key: the citation key a reader would use, then four characters of a hash of its DOI.
Citation keys alone collide, between a preprint and its published version or two papers by one author in one year; the hash settles that, and any tool that knows the DOI can recompute it.
A key is minted once and never recomputed, so a corrected title does not change a key someone has cited ([spec/02 §2.4](../spec/02-model.md#24-work-and-document-keys)).
## Why are numbers stored as strings?
Databases and JSON libraries rewrite numbers: `310.0` becomes `310` and `1.5260` becomes `1.526`, and the printed digits are part of what a paper says.
@@ -96,12 +102,12 @@ The full comparison is [docs/comparison.md](comparison.md)
## What is its status and maturity?
Version 0.1.0 has 29 formats: 18 are `candidate` and 11 are `provisional`, as [registry/formats.json](../registry/formats.json) lists.
Version 0.1.0 has 32 formats: 19 are `candidate` and 13 are `provisional`, as [registry/formats.json](../registry/formats.json) lists.
A `candidate` format is complete, with vectors and fixtures, and ready to implement.
A `provisional` one is complete and expected to change as implementations gain experience.
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade and checklist.
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint.
The repository holds a reference checker (`tools/check.py`), 21 vector files, five valid packages, one of them also as a zip, and 28 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
The repository holds a reference checker (`tools/check.py`), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
## What does it not do?
docs/glossary.md
+21 -1
View File
@@ -36,6 +36,10 @@ Checklist
: A record of checking an export, such as accessible HTML, against a named standard, criterion by criterion; it is covered by the receipt of the person who checked.
Format `scienting-checklist/1`, `provisional`. ([8.11](../spec/08-fidelity.md#811-checklists))
Checkpoint
: A signed note that states a transparency log's size and root hash, which a verifier checks against the last checkpoint it saw.
Format `scienting-checkpoint/1`, `provisional`. ([23.2](../spec/23-log.md#232-checkpoints))
Correction
: An operation on an established document, with its evidence and author; never edited, only appended. ([11](../spec/11-corrections.md))
@@ -45,6 +49,9 @@ Description
Document
: The text established for one version of one work, from one or more witnesses, as a tree of nodes. ([2.1](../spec/02-model.md#21-four-objects), [3](../spec/03-nodes.md))
Document key
: The work key, with a dotted suffix for any version other than the version of record: `.si` for Supporting Information, `.v2` for a second preprint. ([2.4](../spec/02-model.md#24-work-and-document-keys))
Drift
: The amount that has changed below a container since an anchor on it was made, counted from the corrections: the share of the container's leaves touched by later corrections.
It is reported as `none`, `low` or `high`, with a default threshold of a quarter. ([6.6](../spec/06-anchors.md#66-drift))
@@ -56,10 +63,13 @@ The key is anchors, a verdict, a chain, a value or planted errors, by exercise k
Fidelity level
: How far a leaf's text was checked against its source, from F0 converted to F3 proofread. ([8.1](../spec/08-fidelity.md#81-levels-of-fidelity))
History root
: The root of the Merkle tree over a document's corrections, kept in `document.json`, so a consistency proof can show that a later revision extends an earlier one. ([11.5](../spec/11-corrections.md#115-the-history-root))
JATS
: Outside term.
Journal Article Tag Suite, ANSI/NISO Z39.96-2024, an XML format in which publishers and archives exchange journal content ([the NISO page](https://www.niso.org/publications/z3996-2024-jats)).
In paperspec, `jats` is a witness kind: the fixture `okafor2025open` was established from a JATS file and a PDF that agree.
In paperspec, `jats` is a witness kind: the fixture `okafor2025solvent-acth` was established from a JATS file and a PDF that agree.
The formats ingest JATS and do not replace it. ([2.3](../spec/02-model.md#23-witnessesjson))
Keyed hash
@@ -101,6 +111,10 @@ Policy
: A project owner's signed decision that settles rights a record leaves unknown.
It never overrides a use the record sets to true or false. ([14.2](../spec/14-rights.md#142-effective-uses))
Proof
: An RFC 9162 inclusion proof that one item was covered by a receipt, or a consistency proof that one history extends another, each a short list of hashes.
Format `scienting-proof/1`. ([8.5](../spec/08-fidelity.md#85-merkle-roots))
Project
: A team's curated set of documents, with its members, layers, data, reading paths and drafts.
Format `scienting-project/1`, `provisional`. ([20](../spec/20-projects.md))
@@ -156,6 +170,9 @@ Verdict
Version
: One expression of a work: a preprint, the version of record, an erratum, the Supporting Information. ([2.1](../spec/02-model.md#21-four-objects))
VRF
: A verifiable random function (RFC 9381): a keyed function with exactly one valid proof for each key and input, so that a spot-check seed can be checked but not chosen. ([8.8](../spec/08-fidelity.md#88-spot-checks))
Web Annotation
: Outside term.
The W3C Recommendation of 23 February 2017 for annotations with a body, a target and a motivation ([the W3C model](https://www.w3.org/TR/annotation-model/)).
@@ -167,3 +184,6 @@ Witness
Work
: A piece of scholarship across all its versions. ([2.1](../spec/02-model.md#21-four-objects))
Work key
: A work's permanent name: a Scholar-style citation key, a hyphen, and the first characters of a hash of the work's identity, such as `vaswani2017attention-a241`; minted once and never recomputed. ([2.4](../spec/02-model.md#24-work-and-document-keys))
docs/overview.md
+6 -6
View File
@@ -26,7 +26,7 @@ The formats supply those three, and they are worth most at the scale of one team
paperspec is a set of file formats for scientific papers that have been converted, split into addressable pieces and checked.
The repository is at <https://git.scient.ing/infra/paperspec>.
It is version 0.1.0 and defines 29 formats; in [registry/formats.json](../registry/formats.json), 18 are `candidate` and 11 are `provisional`.
It is version 0.1.0 and defines 32 formats; in [registry/formats.json](../registry/formats.json), 19 are `candidate` and 13 are `provisional`.
## Who it is for
@@ -51,12 +51,12 @@ It appears inside hashed data and test vectors, and it is separate from the repo
| Folder | Holds |
| --- | --- |
| [spec/](../spec/README.md) | The specification, in 23 chapters (00 to 22). Normative. |
| [schemas/](../schemas/README.md) | 39 JSON (plain-text data format) Schemas for the files and records. |
| [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. |
| [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. |
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
| [tools/](../tools/README.md) | Reference programs: `check.py` checks everything, `build.py` rebuilds the fixtures and vectors, `load.py` loads packages into SQLite, and `oracle/` is a compact implementation of every rule. |
| [vectors/](../vectors/README.md) | 21 files of test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 28 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
| [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. |
| [docs/](README.md) | Guides, including this page. |
@@ -107,7 +107,7 @@ The [specification's reading paths](../spec/README.md) say which chapters each r
A store loads the package into a database.
A reader renders it with layers of annotations.
Anchors (stored references to a sentence, with its hash and a quotation) let a claim, a dataset value or a draft's citation point at a passage.
Datasets, drafts, views, projects, exercises and checklists are `provisional` formats.
Datasets, drafts, views, projects, exercises, checklists and the transparency log are `provisional` formats.
7.
Check and sign.
People proofread what matters and sign receipts; second reviewers spot-check them.
docs/rationale.md
+45 -5
View File
@@ -9,7 +9,7 @@ Implementers find the chapter that defines each rule.
Experts find the cost of each choice and, where a neighboring standard chose differently, how it differs (the full comparison is in [comparison.md](comparison.md)).
The formats are at version 0.1.0.
Of the 29 in `registry/formats.json`, 18 are `candidate` and 11 are `provisional`, and the provisional ones are named where they appear.
Of the 29 in `registry/formats.json`, 19 are `candidate` and 13 are `provisional`, and the provisional ones are named where they appear.
The repository is paperspec, at <https://git.scient.ing/infra/paperspec>.
The `scienting-` prefix in format names such as `scienting-anchor/1` is the formats' namespace, separate from the repository's name.
@@ -36,6 +36,18 @@ The cost is that two conversions that read a sentence differently by one charact
Everything a team builds points at IDs: notes, claims, dataset values, citations in drafts.
A merge records its successor, a deletion records that there is none, and legacy IDs live on as aliases, all in `ids.jsonl` ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
### Work keys carry a hash
A citation key such as `vaswani2017attention` is what readers say, and it collides: a preprint and its published version, or two papers by one first author in one year with the same first title word, would share it.
Four characters of a hash of the work's identity, its DOI when it has one, settle the collision and let any tool holding the DOI recompute the key.
The key is minted once and never recomputed, so a corrected title cannot change a name someone has cited; when two works still share four characters, the later one gets six ([spec/02 §2.4](../spec/02-model.md#24-work-and-document-keys)).
The rule is the one the studio's reading-list tool already uses, so a paper keeps its key from the reading list to the store.
### Identifiers are tag URIs
Layers, annotations and documents name themselves with `tag:scient.ing,2026:` URIs (RFC 4151) rather than `urn:` names, because a URN namespace must be registered with IANA and this one is not.
A tag URI needs only a domain and a date, and these identifiers sit inside hashed annotations, so the cheap time to choose was before release.
### Revisions are counted, not timed
Anchors are made on different machines and packages are imported later, and clocks disagree.
@@ -97,6 +109,11 @@ An offset counts Unicode code points, the numbered characters of Unicode, into t
Every stored offset is still exact when `node_hash` is unchanged, and a renderer can slice the stored text without normalizing it first.
Only the three text values, `text_hash`, `match_hash` and `align_key`, normalize.
### The character data is pinned to one Unicode version
Normalization and General Categories change between Unicode versions, and both enter every hash.
The chapters pin Unicode 16.0, the reference tools refuse to run on another version, and moving to a later one is a change to every text-hashing format ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
### Hashes that leave the team are keyed
A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages.
@@ -156,6 +173,11 @@ Readings drop spans, roles and header paths, not only words, and those need a co
Drift is computed from the records alone, with no need to replay history, because each record carries `path`, `to_path` and `touched` ([spec/11 §11.3](../spec/11-corrections.md#113-applied-fields)).
### The history has a root
Corrections are append-only in principle, but nothing in a package proved that a later revision extends an earlier one rather than rewriting it.
A Merkle root over the corrections in `document.json`, with RFC 9162 consistency proofs between revisions, does ([spec/11 §11.5](../spec/11-corrections.md#115-the-history-root)).
### Digits need image evidence
A wrong digit changes a result without looking wrong, so no digit changes without a crop of the page as evidence, and no leaf outside the correction's `touched` list may end up with different digits ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)).
@@ -182,11 +204,17 @@ The receipt tree follows [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), who
The specification makes a receipt with no items invalid, where the RFC hashes the empty list ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
The tree is not the one behind `node_hash`, which hashes each node's `own_hash` with its children's `node_hash` values.
### Spot-check samples come from a signature nobody can steer
### Spot-check seeds are VRF proofs
A seed the reviewer could influence could be ground until the sample fell on careful work.
The checker's deterministic Ed25519 signature over the receipt is unpredictable to the reviewer and unchoosable by the checker, because the checker's key was registered for spot-checks before the original was signed.
Anyone can recompute the sample ([spec/08 §8.8](../spec/08-fidelity.md#88-spot-checks)).
A seed the reviewer could influence could be ground until the sample fell on careful work, and a seed the checker could vary could be ground by the checker.
An earlier draft used the checker's Ed25519 signature and called it unchoosable because signing is deterministic; that holds only for an honest checker, since a verifier accepts any valid signature and a dishonest one can sign with any nonce.
A verifiable random function (RFC 9381) has exactly one valid proof for a key and an input, which is the property the rule needs, and its keys are ordinary Ed25519 keys, so the roster needs nothing new.
Anyone can verify the proof and recompute the sample ([spec/08 §8.8](../spec/08-fidelity.md#88-spot-checks)).
### A receipt can be proven without being shown
The receipt tree follows RFC 9162 so that an inclusion proof, a dozen hashes, shows a buyer that one cell was proofread without handing over the receipt or the other nodes ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
A transparency log of receipts, revocations and rosters ([spec/23](../spec/23-log.md)) uses the same tree and the checkpoint format transparency logs share, so that a receipt's time and the roster that governed it stop being self-asserted.
### Error rates are reported as bounds
@@ -270,6 +298,18 @@ These are settled for now, and a future version may revisit them.
| Which drift threshold, spot-check rate and digit rule should teams use? | By default, drift is high above a quarter of the leaves involved, and spot-checks sample 5% of a receipt's nodes, at least ten. The digit rule is `all`, `anchored-sections` or `none`; each is set per project. |
| How should rights combine when a document comes from witnesses held on different bases? | The most restrictive witness governs each use. |
## Looking ahead
Choices made so that the formats still serve in the 2030s, and the ones deliberately left open.
- Signature schemes `cose` and `ml-dsa-65` are reserved: COSE so that the IETF's SCITT transparency services can verify receipts, ML-DSA (FIPS 204) for the time Ed25519 is no longer considered safe.
A consumer treats an unimplemented scheme as invalid, so adopting either is a minor version ([spec/08 §8.4](../spec/08-fidelity.md#84-record-hashes-and-signatures)).
- Every hash is SHA-256, and the hashing scheme names only how a value is keyed, not the function.
A hash function change would be a new major version of every format; the formats accept that rather than carrying agility they cannot test.
- Roster keys are named by short keys, not by decentralized identifiers; a `did:key` form could be added as a key property without a new version.
- A correction's author may be a model and its run; a fuller provenance for model output (the model's identity, weights and prompt by hash) is left to the project that first needs it.
- A transparency log is specified but no deployment exists yet; its tile layout is left to the first one ([spec/23](../spec/23-log.md)).
## Known limits
- Notation drops spaces and text keeps them, so `$T = 300$` from a LaTeX source and "T = 300" from a PDF disagree, and become a `word` suspect for review.
docs/walkthrough.md
+37 -37
View File
@@ -1,17 +1,17 @@
# Walkthrough: one paper, start to finish
This guide follows the example paper `hale2026harmonic` from a flawed first reading to a corrected, reviewed and signed document.
This guide follows the example paper `hale2026harmonic-damb` from a flawed first reading to a corrected, reviewed and signed document.
The paper is synthetic, written for the fixtures.
Its problems are modeled on the failures that the specification's rules address.
Three readers can use the guide, and each gets something different.
A newcomer knows what a paper and a citation are, not hashes or JSON, so every term is defined where it first appears.
An implementer who will write a reader, writer, converter or checker gets the chapter and test vector for each step.
An expert gets the status: [paperspec](https://git.scient.ing/infra/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (18 candidate, 11 provisional).
An expert gets the status: [paperspec](https://git.scient.ing/infra/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (19 candidate, 13 provisional).
The `scienting-` prefix on format names, as in `scienting-doc/1`, is the formats' namespace.
It has nothing to do with the repository's name.
Open [fixtures/packages/hale2026harmonic/](../fixtures/packages/hale2026harmonic/) beside this page, because every ID below is in it.
Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it.
A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line).
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`tools/build.py`](../tools/build.py) expands it into the package.
Each step says what a converter, a person or a checker does.
@@ -26,7 +26,7 @@ Whether it allows processing in the team's cloud is unknown.
The publisher stamped the download in the margin with the institution and the time, so two downloads rarely share a SHA-256.
A page fingerprint avoids the problem of downloads that differ in their stamps.
It is a hash of a page's text, size and images after the stamp lines are removed ([spec/07](../spec/07-geometry.md)), and the rules `downloaded-via` and `sharing-guidelines` fired on both pages (see [`pages.jsonl`](../fixtures/packages/hale2026harmonic/pages.jsonl)).
It is a hash of a page's text, size and images after the stamp lines are removed ([spec/07](../spec/07-geometry.md)), and the rules `downloaded-via` and `sharing-guidelines` fired on both pages (see [`pages.jsonl`](../fixtures/packages/hale2026harmonic-damb/pages.jsonl)).
Page 101's fingerprint is `6272e6f3…` in this copy, stamped October 11, 2026, and in a second download from another institution, stamped March 2, 2027 ([vectors/fingerprints.jsonl](../vectors/fingerprints.jsonl)).
## 2. The first reading, with its mistakes
@@ -57,34 +57,34 @@ Nothing changes until a correction passes the gate ([spec/11](../spec/11-correct
## 3. Twelve corrections
Each fix is appended to [`corrections.jsonl`](../fixtures/packages/hale2026harmonic/corrections.jsonl), numbered, with its evidence (a crop, or a person who read the page), its author (`ana` throughout) and its reason.
Each fix is appended to [`corrections.jsonl`](../fixtures/packages/hale2026harmonic-damb/corrections.jsonl), numbered, with its evidence (a crop, or a person who read the page), its author (`ana` throughout) and its reason.
| # | Operation | On | What it did |
| --- | --- | --- | --- |
| 1 | `replace` | cell `nkm3xn5` | `3l0.0` to `310.0`. A digit changed, so the gate required a crop, and it has one. |
| 2 | `replace` | sentence `e4fkseg` | `10-3` to the runs `10` and superscript `−3`. |
| 3 | `split` | sentence `e4fkseg` | Split before "Table 1"; the first part keeps `e4fkseg`, the second is new: `r1gacps`. |
| 4 | `merge` | sentence `eajq93t` | Joined with `e9rh611`, which is retired with `eajq93t` as its successor. |
| 5 | `merge` | paragraph `zyqr0yc` | Joined with `g44k023`, retired the same way. |
| 6 | `delete` | paragraph `nm7vyzk` | The running head is gone; its IDs are retired with no successor. |
| 7 | `relabel` | equation `qn08tj9` | `(l)` to `(1)`. |
| 8 | `delete` | row `tfjge3c` | The garbled CT–HC row and its four cells. |
| 9 | `insert` | row `fbg2mwq` | The CT–HC row, read again: five new cells. |
| 10 | `insert` | row `j0x5twd` | The missing OW–HW row. |
| 11 | `set` | cell `vtszvw9` | Set B's `colspan` from 1 to 2. |
| 12 | `relabel` | figure `ajfvq4s` | `Figure l` to `Figure 1`. |
| 1 | `replace` | cell `c20cna6` | `3l0.0` to `310.0`. A digit changed, so the gate required a crop, and it has one. |
| 2 | `replace` | sentence `ec5jc93` | `10-3` to the runs `10` and superscript `−3`. |
| 3 | `split` | sentence `ec5jc93` | Split before "Table 1"; the first part keeps `ec5jc93`, the second is new: `a4jrz2t`. |
| 4 | `merge` | sentence `xjns57z` | Joined with `ayea71r`, which is retired with `xjns57z` as its successor. |
| 5 | `merge` | paragraph `gwzbj3s` | Joined with `ncx2v4f`, retired the same way. |
| 6 | `delete` | paragraph `v8e4z6z` | The running head is gone; its IDs are retired with no successor. |
| 7 | `relabel` | equation `tkxsmts` | `(l)` to `(1)`. |
| 8 | `delete` | row `rncchma` | The garbled CT–HC row and its four cells. |
| 9 | `insert` | row `kx2kz2y` | The CT–HC row, read again: five new cells. |
| 10 | `insert` | row `ke0jxpn` | The missing OW–HW row. |
| 11 | `set` | cell `xyy7x6c` | Set B's `colspan` from 1 to 2. |
| 12 | `relabel` | figure `v7cx6qs` | `Figure l` to `Figure 1`. |
The document is now at revision 12.
No uncorrected cell, sentence or row changed its ID or its hashes.
Only corrected nodes and their ancestors got new `node_hash` values; the sentence `sxgcc9h` is one of the ancestors, because it holds the relabeled equation.
Only corrected nodes and their ancestors got new `node_hash` values; the sentence `rr20tev` is one of the ancestors, because it holds the relabeled equation.
Inserting a row adds a row node.
The table's own hash (`own_hash`) covers only its label and column definitions, so every other cell, and every value already taken from them, is untouched.
## 4. IDs that never move
The sentence about water is still `f929eq9` after twelve corrections, the corrected cell is still `nkm3xn5`, and the sentence that lost its superscript is still `e4fkseg`.
The retired IDs are recorded in [`ids.jsonl`](../fixtures/packages/hale2026harmonic/ids.jsonl), so anything that pointed at them can follow the trail.
Legacy aliases are recorded there too: `b-p01-text-003`, the ID an older tool gave the water sentence, still leads to `f929eq9`.
The sentence about water is still `hca6dc5` after twelve corrections, the corrected cell is still `c20cna6`, and the sentence that lost its superscript is still `ec5jc93`.
The retired IDs are recorded in [`ids.jsonl`](../fixtures/packages/hale2026harmonic-damb/ids.jsonl), so anything that pointed at them can follow the trail.
Legacy aliases are recorded there too: `b-p01-text-003`, the ID an older tool gave the water sentence, still leads to `hca6dc5`.
A fresh conversion of the same input mints the same IDs.
A reading by a different converter is aligned with this document sentence by sentence ([spec/05 §5.6](../spec/05-hashes.md), pinned by [vectors/alignment.jsonl](../vectors/alignment.jsonl)).
@@ -116,13 +116,13 @@ The table shows how each anchor resolves at revision 12 ([vectors/resolve.jsonl]
| The water sentence, through the alias `b-p01-text-003` | `current` | The alias leads to the same node. |
| The cell that read `3l0.0` | `stale` | Its text changed and "3l0.0" is no longer in it; a person must look again. |
| "treated as zero" | `reattached` | The sentence changed twice, and the quote occurs in it exactly once, so the anchor is re-pinned. |
| "fitted constants" | `stale` | After the split, those words are in the new sentence `r1gacps`, not in `e4fkseg`. |
| "least squares", in the half-sentence `e9rh611` | `moved` | That ID was merged into `eajq93t`, which still holds the quote. |
| The paragraph `g44k023` | `moved` | Merged into `zyqr0yc`, which holds all of its sentences. |
| "fitted constants" | `stale` | After the split, those words are in the new sentence `a4jrz2t`, not in `ec5jc93`. |
| "least squares", in the half-sentence `ayea71r` | `moved` | That ID was merged into `xjns57z`, which still holds the quote. |
| The paragraph `ncx2v4f` | `moved` | Merged into `gwzbj3s`, which holds all of its sentences. |
| The running head | `stale` | Deleted, with no successor. |
| The equation | `reattached` | Only its number changed. |
| The figure | `reattached` | Only its label changed, by a `relabel`. |
| The CT–HC cell `1.090` in the garbled row | `reattached` | Its row was deleted, and exactly one cell in Table 1 has the same row and column headers and the same text: the new `e0n07n0`. |
| The CT–HC cell `1.090` in the garbled row | `reattached` | Its row was deleted, and exactly one cell in Table 1 has the same row and column headers and the same text: the new `b5gd0h1`. |
| Section 2 | `changed-below`, drift high | The section itself is unchanged, and 53% of what it holds or held was touched. |
| Table 1 | `changed-below`, drift high | Rows were replaced and added. |
| The table body | `changed-below`, drift high | 75% of the cells it holds or held were touched. |
@@ -141,23 +141,23 @@ See [comparison.md](comparison.md).
A receipt is a signed record that a person compared a set of nodes with the page.
It lifts a node to F3, the top fidelity level (F0 is one reading; F1, every doubtful place answered; F2, an independent witness agrees) ([spec/08](../spec/08-fidelity.md)).
One signature covers a Merkle tree, a hash over everything reviewed in a sitting, built as in RFC 6962 (a Request for Comments standard; replaced by [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), which defines the same tree).
The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic/roster.json).
The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic-damb/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic-damb/roster.json).
- **Before the corrections**, `ana` had signed receipt `c5cx7n4tga6y9` for three cells at revision 0.
- **Before the corrections**, `ana` had signed receipt `npc15pn6675z5` for three cells at revision 0.
Two of them never changed, so the receipt still counts for them.
For the corrected cell it no longer counts, because the cell's hash moved on.
- After the corrections, `ana` signs `eqgbbpwb6725v` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`e4fkseg`, `f929eq9` and `sxgcc9h`, the one holding the equation).
`ben` spot-checks it with receipt `h6yr2ddx6d3ef`.
- After the corrections, `ana` signs `y2zw8ctg5s8dp` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`ec5jc93`, `hca6dc5` and `rr20tev`, the one holding the equation).
`ben` spot-checks it with receipt `mg18668nzxvap`.
His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes nobody could have chosen, and he finds no errors.
- `ana` signs `zaxky8c1y12tc` for two sentences of section 1, including the new `r1gacps`.
- `cara` signs `adtrpq41np7sd` for three sentences on page 2, then revokes it herself (`q0yg0w6ygwwxh`) because she had compared them with the crops of another paper.
Those sentences fall back to their earlier levels, which is why `eajq93t` is still F0.
- `ana` signs `gena76nkb6cnc` for the figure's three caption sentences. `ben`'s spot-check `tejzs01eqn2ym` finds a missing period in `nmm4md0`, which suspends the receipt until she reviews the sitting again and signs a new one.
- `ana` signs `mvjmb94nz7rdh` for two sentences of section 1, including the new `a4jrz2t`.
- `cara` signs `q465jpwz8afhe` for three sentences on page 2, then revokes it herself (`pcbhtpwvysgq6`) because she had compared them with the crops of another paper.
Those sentences fall back to their earlier levels, which is why `xjns57z` is still F0.
- `ana` signs `gx2s84z9jf9xv` for the figure's three caption sentences. `ben`'s spot-check `yt70fcc3qm0xb` finds a missing period in `caehhp6`, which suspends the receipt until she reviews the sitting again and signs a new one.
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
The 27 are the 23 nodes of `eqgbbpwb6725v`, the 2 of `zaxky8c1y12tc`, and 2 of the 3 caption sentences.
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic-damb/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
The 27 are the 23 nodes of `y2zw8ctg5s8dp`, the 2 of `mvjmb94nz7rdh`, and 2 of the 3 caption sentences.
Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right.
The third, `nmm4md0`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
The third, `caehhp6`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
## 7. Data out of the table
@@ -171,7 +171,7 @@ The reference checker, [`tools/check.py`](../tools/check.py), confirms each CSV
## 8. Sharing without the text
The team wants to move its notes to another of its own machines without copying the paper there.
[`hale2026harmonic.checks-only`](../fixtures/packages/hale2026harmonic.checks-only/) is the same document with every leaf's text withheld and every hash replaced by an HMAC under the team's key ([spec/12 §12.5](../spec/12-package.md), [spec/05 §5.8](../spec/05-hashes.md)).
[`hale2026harmonic-damb.checks-only`](../fixtures/packages/hale2026harmonic-damb.checks-only/) is the same document with every leaf's text withheld and every hash replaced by an HMAC under the team's key ([spec/12 §12.5](../spec/12-package.md), [spec/05 §5.8](../spec/05-hashes.md)).
An HMAC is a hash mixed with a secret key.
Anyone can recover a plain hash of a short text such as "310.0" by hashing guesses; a keyed hash resists that.
Corrections, receipts, the roster and the dataset are left out too.