infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: minted keys, VRF seeds, RFC 9162 proofs, history roots, tag URIs and a transparency log

Browse Source 
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
  Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
  where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
  openalex in same_as, else the folded title, year and family name. A
  collision lengthens the later key by two characters; a key is minted
  once and never recomputed. A document key adds .si, .v<seq> or .<type>
  for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
  okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
  every node and record ID was re-minted and the docs rewritten to match.
  The validator checks the document key against the work key and version
  (doc.key.work) and the hash part against the identifier (doc.key.suffix).
  vectors/keys.jsonl pins the cases paperstack's tests pin.

Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
  implemented in tools/oracle/vrf.py in plain Python and reproduced
  against the RFC's Example 16; the sample ranks by the VRF output. The
  earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
  verify_inclusion, consistency_proof, verify_consistency) with
  scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
  the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
  checkpoints with key hashes as transparency logs share them, the rule
  that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).

Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
  namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
  the checker refuses another version.

Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
  checkpoint records; a check that every ID and hash the prose quotes
  exists in the fixtures or vectors; three more broken packages
  (history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
  keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
  section, media-type registration and OCI distribution notes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 23:21:57 -04:00
1 parent 76d76353d9
commit a09d158e16
307 files changed
+2698 -1497
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/concepts.md
+41 -27
View File
@@ -1,6 +1,6 @@
# Concepts
Each section introduces one idea, shows it in the example paper `hale2026harmonic` (a synthetic chemistry paper in [fixtures/packages/](../fixtures/packages/)), and links to the chapter that defines it exactly.
Each section introduces one idea, shows it in the example paper `hale2026harmonic-damb` (a synthetic chemistry paper in [fixtures/packages/](../fixtures/packages/)), and links to the chapter that defines it exactly.
This page is written for three readers.
A newcomer, such as a student who knows what a paper and a citation are and is new to hashes, JSON and annotation standards, gets each term defined where it first appears, with the reason before the rule.
@@ -12,8 +12,8 @@ For a gentler start, read the [overview](overview.md) first; for the exact rules
Two facts apply to everything below.
The specification is at version 0.1.0.
It defines 29 formats, 18 of them `candidate` and 11 `provisional`, as [registry/formats.json](../registry/formats.json) lists.
The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade and checklist formats; the sections below say so where they describe them.
It defines 32 formats, 19 of them `candidate` and 13 `provisional`, as [registry/formats.json](../registry/formats.json) lists.
The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint formats; the sections below say so where they describe them.
Format identifiers such as `scienting-anchor/1` and `scienting-correction/1` begin with `scienting-`, which is the formats' namespace.
The repository's name, paperspec, is a separate matter, and the formats are named after what they hold.
@@ -53,12 +53,26 @@ Work, version, witness and document are easy to confuse, so the formats keep the
| A work is the paper across all its versions. | Hale and Osei's paper on harmonic bond parameters, `doi:10.5555/example.hale.2026`. |
| A version is one expression of it: a preprint, the version of record, an erratum, the Supporting Information. | The version of record. |
| A witness is one file of a version, identified by its SHA-256. | The publisher's PDF, witness `qptsanv`. |
| A document is the text established for one version, from one or more witnesses. | `hale2026harmonic`. |
| A document is the text established for one version, from one or more witnesses. | `hale2026harmonic-damb`. |
A document's key is its work's key, with a suffix for anything other than the version of record: `hale2026harmonic.si` would be its Supporting Information.
A document's key is its work's key, with a suffix for anything other than the version of record: `hale2026harmonic-damb.si` would be its Supporting Information.
A witness can be a PDF, a JATS file, a LaTeX archive, an HTML page, an EPUB, a Typst project, other XML, an image, or another file; these are the values of its `kind`.
See [chapter 2](../spec/02-model.md).
## Keys
A work key is a work's permanent name, built so that a reader can say it and so that it never collides.
It is a Scholar-style citation key (first author, year, first title word), a hyphen, and four characters of a hash of the work's identity, which is its DOI when it has one:
```text
hale2026harmonic-damb
```
Two works that a citation key alone would name alike, a preprint and its published version, say, get different hashes.
A key is minted once, when the work enters a store, and never recomputed, so a corrected title changes nothing anyone has cited; if two works share four characters, the later one gets six.
A document key adds a suffix for a version other than the version of record: `hale2026harmonic-damb.si` would be the paper's Supporting Information.
See [spec/02 §2.4](../spec/02-model.md#24-work-and-document-keys); `keys.jsonl` pins the cases.
## A document is a tree of nodes
Every part of the text is a node: the document, its sections, headings, paragraphs and sentences, figures and their panels and captions, tables with their rows and cells, equations, footnotes and references.
@@ -66,11 +80,11 @@ The example has 75 nodes, 45 of them leaves (nodes that hold text themselves).
This outline shows its two sections and the equation, figure and table in them:
```text
bzqvbhd 1 Introduction
qn08tj9 equation (1)
f0tj6ca 2 Methods
ajfvq4s figure Figure 1
gm5z4rx table Table 1
p36k8t6 1 Introduction
tkxsmts equation (1)
d400rz3 2 Methods
v7cx6qs figure Figure 1
hj4s7pb table Table 1
```
`nodes.jsonl` holds one node per line, in document order: the root first, then each node before its children.
@@ -79,7 +93,7 @@ A sentence's content is a list of *runs*: plain text, styled text, inline math,
This is one table cell, shortened:
```json
{"id":"nkm3xn5","kind":"cell","parent":"zhe4g0k","ord":1,
{"id":"c20cna6","kind":"cell","parent":"g8d2pjb","ord":1,
"attrs":{"column":"…","role":"data","rowspan":1,"colspan":1},
"content":[{"kind":"text","text":"310.0"}],
"fidelity":"F1","text_hash":"…","match_hash":"…","align_key":"…","own_hash":"…","node_hash":"21f13b…"}
@@ -96,7 +110,7 @@ See [the comparison page](comparison.md).
## IDs are names, not positions
`nkm3xn5` says nothing about where the cell is.
`c20cna6` says nothing about where the cell is.
That is deliberate.
An ID that encoded a page or an ordinal would change when a better converter ordered blocks differently, and one that encoded a kind would change when a paragraph turned out to be a caption.
So an ID encodes no page, ordinal or kind.
@@ -111,12 +125,12 @@ A second reading is aligned with the document sentence by sentence.
Sentences the alignment pairs keep the document's IDs, and every difference becomes a suspect, a recorded doubtful reading that a person resolves ([chapter 5.6](../spec/05-hashes.md#56-alignment) and [5.7](../spec/05-hashes.md#57-suspects)).
One ID is retired and points at its successor when two nodes merge; an artifact's ID is retired with no successor when the artifact is deleted.
Legacy IDs, such as a block ID from an earlier numbering scheme, live on as aliases: in the example, `b-p01-text-003` still leads to sentence `f929eq9`.
Legacy IDs, such as a block ID from an earlier numbering scheme, live on as aliases: in the example, `b-p01-text-003` still leads to sentence `hca6dc5`.
Both the retirements and the aliases are recorded in `ids.jsonl`.
Not every addressable thing is a node.
The columns of a table and the text items found inside a figure panel get IDs by the same rule and carry no node hashes, so they do not have the five hashes described below.
In the example, Table 1's second column has the ID `kp6grmd`, and the axis label "r (Å)" in Figure 1's first panel has the ID `nhk15ag`.
In the example, Table 1's second column has the ID `wdtrcqt`, and the axis label "r (Å)" in Figure 1's first panel has the ID `rtxzjj9`.
Pinned by [vectors/ids.jsonl](../vectors/ids.jsonl).
See [chapter 4](../spec/04-identity.md).
@@ -132,15 +146,15 @@ An address names a node, and optionally pins it to a state and narrows it to a s
| Address | Names |
| --- | --- |
| `hale2026harmonic#f929eq9` | The sentence about water, whatever it says now. |
| `hale2026harmonic#f929eq9@cd6cdb` | That sentence as it read when its `node_hash` began `cd6cdb`. |
| `hale2026harmonic#f929eq9@cd6cdb[29:44]` | Code points 29 to 44 of it in that state: "first test case". |
| `smithlab/hale2026harmonic#f929eq9` | The same sentence in the `smithlab` store. |
| `hale2026harmonic-damb#hca6dc5` | The sentence about water, whatever it says now. |
| `hale2026harmonic-damb#hca6dc5@cd6cdb` | That sentence as it read when its `node_hash` began `cd6cdb`. |
| `hale2026harmonic-damb#hca6dc5@cd6cdb[29:44]` | Code points 29 to 44 of it in that state: "first test case". |
| `smithlab/hale2026harmonic-damb#hca6dc5` | The same sentence in the `smithlab` store. |
The offsets count Unicode code points in the node's plain projection, which is its text as stored with math written between dollar signs.
The dollar signs count, which is why the span starts at 29: the sentence reads `Water, H$_2$O, served as the first test case, …`.
On the web the ID is the fragment, so a link to `…/smithlab/hale2026harmonic#f929eq9` lands on the sentence.
On the web the ID is the fragment, so a link to `…/smithlab/hale2026harmonic-damb#hca6dc5` lands on the sentence.
Pinned by [vectors/addresses.jsonl](../vectors/addresses.jsonl).
See [chapter 6](../spec/06-anchors.md#61-addresses).
@@ -190,11 +204,11 @@ A document is never edited in place.
Every fix is a correction: an operation with its evidence, its author and a reason, appended to `corrections.jsonl`.
```json
{"format":"scienting-correction/1","index":1,"op":"replace","target":"nkm3xn5",
{"format":"scienting-correction/1","index":1,"op":"replace","target":"c20cna6",
"find":"3l0.0","replace":"310.0",
"evidence":[{"kind":"crop","sha256":"9b6fe6…"}],"author":{"key":"ana"},
"reason":"OCR read the digit 1 as the letter l.","at":"2026-10-02T10:00:00Z",
"before":"2fd1e9…","after":"21f13b…","touched":["nkm3xn5"]}
"before":"2fd1e9…","after":"21f13b…","touched":["c20cna6"]}
```
A document's history is its first reading plus its corrections, and its revision is the index of the last one.
@@ -215,7 +229,7 @@ An anchor is how anything points into a document: a quotation that must match th
It is a pinned address with enough context to find its text again after the document changes:
```json
{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic","node":"f929eq9","kind":"sentence",
{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic-damb","node":"hca6dc5","kind":"sentence",
"own_hash":"d5b2eb…","node_hash":"cd6cdb…","span":[29,44],
"quote":{"exact":"first test case","prefix":"Water, H$_2$O, served as the ","suffix":", with a bending band near 1595 "},
"rev":12}
@@ -309,7 +323,7 @@ Values are plain SHA-256 inside the team.
Exported to another of the team's machines, they are HMACs under the team's key.
In a public layer, or in an export outside the team, the key is derived from the paper's page fingerprints (see the previous section), so only someone holding the paper can check them.
Where the values go chooses the key, not whether the text is withheld.
[fixtures/packages/hale2026harmonic.checks-only](../fixtures/packages/hale2026harmonic.checks-only) is the example paper exported to another of the team's machines, so it uses the team key.
[fixtures/packages/hale2026harmonic-damb.checks-only](../fixtures/packages/hale2026harmonic-damb.checks-only) is the example paper exported to another of the team's machines, so it uses the team key.
A checks-only package also has no corrections, suspects or datasets.
Pinned by [vectors/keyed.jsonl](../vectors/keyed.jsonl).
@@ -347,8 +361,8 @@ This gloss, shortened, explains a phrase for a first-year student:
{"type":"Annotation","motivation":"describing","status":"confirmed",
"body":{"type":"TextualBody","kind":"gloss","audience":"novice",
"value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},
"target":{"type":"SpecificResource","source":"urn:scienting:doc:smithlab/hale2026harmonic",
"selector":[{"type":"AnchorSelector","anchor":{"node":"sxgcc9h","span":[111,130],"…":"…"}},
"target":{"type":"SpecificResource","source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb",
"selector":[{"type":"AnchorSelector","anchor":{"node":"rr20tev","span":[111,130],"…":"…"}},
{"type":"TextQuoteSelector","exact":"bond force constant"}]}}
```
@@ -366,7 +380,7 @@ See [chapter 18](../spec/18-layers.md).
## Exercises
Every sentence and cell has an address, so the answer to "find the evidence" is an address, and an address can be graded.
An exercise asks a student to find the cell giving the Set A bond constant for CT–CT. A student who points at `nkm3xn5` scores 1 of 1 with "That is the cell."; one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."
An exercise asks a student to find the cell giving the Set A bond constant for CT–CT. A student who points at `c20cna6` scores 1 of 1 with "That is the cell."; one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."
Students submit under pseudonyms, and a key whose anchors have gone stale is never used to grade.
Pinned by [vectors/grading.jsonl](../vectors/grading.jsonl).
@@ -401,7 +415,7 @@ See [the comparison page](comparison.md).
A document travels as a package: a folder of canonical JSON files, or the same folder zipped.
```text
hale2026harmonic/
hale2026harmonic-damb/
MANIFEST.json every other file, with its SHA-256; its own hash pins the package
document.json the work, version, metadata, policy, root hash and fidelity
nodes.jsonl one node per line, in reading order