infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: minted keys, VRF seeds, RFC 9162 proofs, history roots, tag URIs and a transparency log

Browse Source 
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
  Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
  where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
  openalex in same_as, else the folded title, year and family name. A
  collision lengthens the later key by two characters; a key is minted
  once and never recomputed. A document key adds .si, .v<seq> or .<type>
  for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
  okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
  every node and record ID was re-minted and the docs rewritten to match.
  The validator checks the document key against the work key and version
  (doc.key.work) and the hash part against the identifier (doc.key.suffix).
  vectors/keys.jsonl pins the cases paperstack's tests pin.

Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
  implemented in tools/oracle/vrf.py in plain Python and reproduced
  against the RFC's Example 16; the sample ranks by the VRF output. The
  earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
  verify_inclusion, consistency_proof, verify_consistency) with
  scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
  the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
  checkpoints with key hashes as transparency logs share them, the rule
  that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).

Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
  namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
  the checker refuses another version.

Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
  checkpoint records; a check that every ID and hash the prose quotes
  exists in the fixtures or vectors; three more broken packages
  (history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
  keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
  section, media-type registration and OCI distribution notes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 23:21:57 -04:00
1 parent 76d76353d9
commit a09d158e16
307 files changed
+2698 -1497
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/walkthrough.md
+37 -37
View File
@@ -1,17 +1,17 @@
# Walkthrough: one paper, start to finish
This guide follows the example paper `hale2026harmonic` from a flawed first reading to a corrected, reviewed and signed document.
This guide follows the example paper `hale2026harmonic-damb` from a flawed first reading to a corrected, reviewed and signed document.
The paper is synthetic, written for the fixtures.
Its problems are modeled on the failures that the specification's rules address.
Three readers can use the guide, and each gets something different.
A newcomer knows what a paper and a citation are, not hashes or JSON, so every term is defined where it first appears.
An implementer who will write a reader, writer, converter or checker gets the chapter and test vector for each step.
An expert gets the status: [paperspec](https://git.scient.ing/infra/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (18 candidate, 11 provisional).
An expert gets the status: [paperspec](https://git.scient.ing/infra/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (19 candidate, 13 provisional).
The `scienting-` prefix on format names, as in `scienting-doc/1`, is the formats' namespace.
It has nothing to do with the repository's name.
Open [fixtures/packages/hale2026harmonic/](../fixtures/packages/hale2026harmonic/) beside this page, because every ID below is in it.
Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it.
A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line).
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`tools/build.py`](../tools/build.py) expands it into the package.
Each step says what a converter, a person or a checker does.
@@ -26,7 +26,7 @@ Whether it allows processing in the team's cloud is unknown.
The publisher stamped the download in the margin with the institution and the time, so two downloads rarely share a SHA-256.
A page fingerprint avoids the problem of downloads that differ in their stamps.
It is a hash of a page's text, size and images after the stamp lines are removed ([spec/07](../spec/07-geometry.md)), and the rules `downloaded-via` and `sharing-guidelines` fired on both pages (see [`pages.jsonl`](../fixtures/packages/hale2026harmonic/pages.jsonl)).
It is a hash of a page's text, size and images after the stamp lines are removed ([spec/07](../spec/07-geometry.md)), and the rules `downloaded-via` and `sharing-guidelines` fired on both pages (see [`pages.jsonl`](../fixtures/packages/hale2026harmonic-damb/pages.jsonl)).
Page 101's fingerprint is `6272e6f3…` in this copy, stamped October 11, 2026, and in a second download from another institution, stamped March 2, 2027 ([vectors/fingerprints.jsonl](../vectors/fingerprints.jsonl)).
## 2. The first reading, with its mistakes
@@ -57,34 +57,34 @@ Nothing changes until a correction passes the gate ([spec/11](../spec/11-correct
## 3. Twelve corrections
Each fix is appended to [`corrections.jsonl`](../fixtures/packages/hale2026harmonic/corrections.jsonl), numbered, with its evidence (a crop, or a person who read the page), its author (`ana` throughout) and its reason.
Each fix is appended to [`corrections.jsonl`](../fixtures/packages/hale2026harmonic-damb/corrections.jsonl), numbered, with its evidence (a crop, or a person who read the page), its author (`ana` throughout) and its reason.
| # | Operation | On | What it did |
| --- | --- | --- | --- |
| 1 | `replace` | cell `nkm3xn5` | `3l0.0` to `310.0`. A digit changed, so the gate required a crop, and it has one. |
| 2 | `replace` | sentence `e4fkseg` | `10-3` to the runs `10` and superscript `−3`. |
| 3 | `split` | sentence `e4fkseg` | Split before "Table 1"; the first part keeps `e4fkseg`, the second is new: `r1gacps`. |
| 4 | `merge` | sentence `eajq93t` | Joined with `e9rh611`, which is retired with `eajq93t` as its successor. |
| 5 | `merge` | paragraph `zyqr0yc` | Joined with `g44k023`, retired the same way. |
| 6 | `delete` | paragraph `nm7vyzk` | The running head is gone; its IDs are retired with no successor. |
| 7 | `relabel` | equation `qn08tj9` | `(l)` to `(1)`. |
| 8 | `delete` | row `tfjge3c` | The garbled CT–HC row and its four cells. |
| 9 | `insert` | row `fbg2mwq` | The CT–HC row, read again: five new cells. |
| 10 | `insert` | row `j0x5twd` | The missing OW–HW row. |
| 11 | `set` | cell `vtszvw9` | Set B's `colspan` from 1 to 2. |
| 12 | `relabel` | figure `ajfvq4s` | `Figure l` to `Figure 1`. |
| 1 | `replace` | cell `c20cna6` | `3l0.0` to `310.0`. A digit changed, so the gate required a crop, and it has one. |
| 2 | `replace` | sentence `ec5jc93` | `10-3` to the runs `10` and superscript `−3`. |
| 3 | `split` | sentence `ec5jc93` | Split before "Table 1"; the first part keeps `ec5jc93`, the second is new: `a4jrz2t`. |
| 4 | `merge` | sentence `xjns57z` | Joined with `ayea71r`, which is retired with `xjns57z` as its successor. |
| 5 | `merge` | paragraph `gwzbj3s` | Joined with `ncx2v4f`, retired the same way. |
| 6 | `delete` | paragraph `v8e4z6z` | The running head is gone; its IDs are retired with no successor. |
| 7 | `relabel` | equation `tkxsmts` | `(l)` to `(1)`. |
| 8 | `delete` | row `rncchma` | The garbled CT–HC row and its four cells. |
| 9 | `insert` | row `kx2kz2y` | The CT–HC row, read again: five new cells. |
| 10 | `insert` | row `ke0jxpn` | The missing OW–HW row. |
| 11 | `set` | cell `xyy7x6c` | Set B's `colspan` from 1 to 2. |
| 12 | `relabel` | figure `v7cx6qs` | `Figure l` to `Figure 1`. |
The document is now at revision 12.
No uncorrected cell, sentence or row changed its ID or its hashes.
Only corrected nodes and their ancestors got new `node_hash` values; the sentence `sxgcc9h` is one of the ancestors, because it holds the relabeled equation.
Only corrected nodes and their ancestors got new `node_hash` values; the sentence `rr20tev` is one of the ancestors, because it holds the relabeled equation.
Inserting a row adds a row node.
The table's own hash (`own_hash`) covers only its label and column definitions, so every other cell, and every value already taken from them, is untouched.
## 4. IDs that never move
The sentence about water is still `f929eq9` after twelve corrections, the corrected cell is still `nkm3xn5`, and the sentence that lost its superscript is still `e4fkseg`.
The retired IDs are recorded in [`ids.jsonl`](../fixtures/packages/hale2026harmonic/ids.jsonl), so anything that pointed at them can follow the trail.
Legacy aliases are recorded there too: `b-p01-text-003`, the ID an older tool gave the water sentence, still leads to `f929eq9`.
The sentence about water is still `hca6dc5` after twelve corrections, the corrected cell is still `c20cna6`, and the sentence that lost its superscript is still `ec5jc93`.
The retired IDs are recorded in [`ids.jsonl`](../fixtures/packages/hale2026harmonic-damb/ids.jsonl), so anything that pointed at them can follow the trail.
Legacy aliases are recorded there too: `b-p01-text-003`, the ID an older tool gave the water sentence, still leads to `hca6dc5`.
A fresh conversion of the same input mints the same IDs.
A reading by a different converter is aligned with this document sentence by sentence ([spec/05 §5.6](../spec/05-hashes.md), pinned by [vectors/alignment.jsonl](../vectors/alignment.jsonl)).
@@ -116,13 +116,13 @@ The table shows how each anchor resolves at revision 12 ([vectors/resolve.jsonl]
| The water sentence, through the alias `b-p01-text-003` | `current` | The alias leads to the same node. |
| The cell that read `3l0.0` | `stale` | Its text changed and "3l0.0" is no longer in it; a person must look again. |
| "treated as zero" | `reattached` | The sentence changed twice, and the quote occurs in it exactly once, so the anchor is re-pinned. |
| "fitted constants" | `stale` | After the split, those words are in the new sentence `r1gacps`, not in `e4fkseg`. |
| "least squares", in the half-sentence `e9rh611` | `moved` | That ID was merged into `eajq93t`, which still holds the quote. |
| The paragraph `g44k023` | `moved` | Merged into `zyqr0yc`, which holds all of its sentences. |
| "fitted constants" | `stale` | After the split, those words are in the new sentence `a4jrz2t`, not in `ec5jc93`. |
| "least squares", in the half-sentence `ayea71r` | `moved` | That ID was merged into `xjns57z`, which still holds the quote. |
| The paragraph `ncx2v4f` | `moved` | Merged into `gwzbj3s`, which holds all of its sentences. |
| The running head | `stale` | Deleted, with no successor. |
| The equation | `reattached` | Only its number changed. |
| The figure | `reattached` | Only its label changed, by a `relabel`. |
| The CT–HC cell `1.090` in the garbled row | `reattached` | Its row was deleted, and exactly one cell in Table 1 has the same row and column headers and the same text: the new `e0n07n0`. |
| The CT–HC cell `1.090` in the garbled row | `reattached` | Its row was deleted, and exactly one cell in Table 1 has the same row and column headers and the same text: the new `b5gd0h1`. |
| Section 2 | `changed-below`, drift high | The section itself is unchanged, and 53% of what it holds or held was touched. |
| Table 1 | `changed-below`, drift high | Rows were replaced and added. |
| The table body | `changed-below`, drift high | 75% of the cells it holds or held were touched. |
@@ -141,23 +141,23 @@ See [comparison.md](comparison.md).
A receipt is a signed record that a person compared a set of nodes with the page.
It lifts a node to F3, the top fidelity level (F0 is one reading; F1, every doubtful place answered; F2, an independent witness agrees) ([spec/08](../spec/08-fidelity.md)).
One signature covers a Merkle tree, a hash over everything reviewed in a sitting, built as in RFC 6962 (a Request for Comments standard; replaced by [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), which defines the same tree).
The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic/roster.json).
The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic-damb/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic-damb/roster.json).
- **Before the corrections**, `ana` had signed receipt `c5cx7n4tga6y9` for three cells at revision 0.
- **Before the corrections**, `ana` had signed receipt `npc15pn6675z5` for three cells at revision 0.
Two of them never changed, so the receipt still counts for them.
For the corrected cell it no longer counts, because the cell's hash moved on.
- After the corrections, `ana` signs `eqgbbpwb6725v` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`e4fkseg`, `f929eq9` and `sxgcc9h`, the one holding the equation).
`ben` spot-checks it with receipt `h6yr2ddx6d3ef`.
- After the corrections, `ana` signs `y2zw8ctg5s8dp` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`ec5jc93`, `hca6dc5` and `rr20tev`, the one holding the equation).
`ben` spot-checks it with receipt `mg18668nzxvap`.
His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes nobody could have chosen, and he finds no errors.
- `ana` signs `zaxky8c1y12tc` for two sentences of section 1, including the new `r1gacps`.
- `cara` signs `adtrpq41np7sd` for three sentences on page 2, then revokes it herself (`q0yg0w6ygwwxh`) because she had compared them with the crops of another paper.
Those sentences fall back to their earlier levels, which is why `eajq93t` is still F0.
- `ana` signs `gena76nkb6cnc` for the figure's three caption sentences. `ben`'s spot-check `tejzs01eqn2ym` finds a missing period in `nmm4md0`, which suspends the receipt until she reviews the sitting again and signs a new one.
- `ana` signs `mvjmb94nz7rdh` for two sentences of section 1, including the new `a4jrz2t`.
- `cara` signs `q465jpwz8afhe` for three sentences on page 2, then revokes it herself (`pcbhtpwvysgq6`) because she had compared them with the crops of another paper.
Those sentences fall back to their earlier levels, which is why `xjns57z` is still F0.
- `ana` signs `gx2s84z9jf9xv` for the figure's three caption sentences. `ben`'s spot-check `yt70fcc3qm0xb` finds a missing period in `caehhp6`, which suspends the receipt until she reviews the sitting again and signs a new one.
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
The 27 are the 23 nodes of `eqgbbpwb6725v`, the 2 of `zaxky8c1y12tc`, and 2 of the 3 caption sentences.
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic-damb/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
The 27 are the 23 nodes of `y2zw8ctg5s8dp`, the 2 of `mvjmb94nz7rdh`, and 2 of the 3 caption sentences.
Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right.
The third, `nmm4md0`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
The third, `caehhp6`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
## 7. Data out of the table
@@ -171,7 +171,7 @@ The reference checker, [`tools/check.py`](../tools/check.py), confirms each CSV
## 8. Sharing without the text
The team wants to move its notes to another of its own machines without copying the paper there.
[`hale2026harmonic.checks-only`](../fixtures/packages/hale2026harmonic.checks-only/) is the same document with every leaf's text withheld and every hash replaced by an HMAC under the team's key ([spec/12 §12.5](../spec/12-package.md), [spec/05 §5.8](../spec/05-hashes.md)).
[`hale2026harmonic-damb.checks-only`](../fixtures/packages/hale2026harmonic-damb.checks-only/) is the same document with every leaf's text withheld and every hash replaced by an HMAC under the team's key ([spec/12 §12.5](../spec/12-package.md), [spec/05 §5.8](../spec/05-hashes.md)).
An HMAC is a hash mixed with a secret key.
Anyone can recover a plain hash of a short text such as "310.0" by hashing guesses; a keyed hash resists that.
Corrections, receipts, the roster and the dataset are left out too.