feat: minted keys, VRF seeds, RFC 9162 proofs, history roots, tag URIs and a transparency log

Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
  Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
  where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
  openalex in same_as, else the folded title, year and family name. A
  collision lengthens the later key by two characters; a key is minted
  once and never recomputed. A document key adds .si, .v<seq> or .<type>
  for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
  okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
  every node and record ID was re-minted and the docs rewritten to match.
  The validator checks the document key against the work key and version
  (doc.key.work) and the hash part against the identifier (doc.key.suffix).
  vectors/keys.jsonl pins the cases paperstack's tests pin.

Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
  implemented in tools/oracle/vrf.py in plain Python and reproduced
  against the RFC's Example 16; the sample ranks by the VRF output. The
  earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
  verify_inclusion, consistency_proof, verify_consistency) with
  scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
  the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
  checkpoints with key hashes as transparency logs share them, the rule
  that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).

Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
  namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
  the checker refuses another version.

Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
  checkpoint records; a check that every ID and hash the prose quotes
  exists in the fixtures or vectors; three more broken packages
  (history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
  keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
  section, media-type registration and OCI distribution notes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 23:21:57 -04:00
1 parent 76d76353d9
commit a09d158e16
307 files changed
+2698 -1497

No files matched your search

+7 -6
View File
@@ -21,14 +21,15 @@ Every file is JSON (a plain-text format for structured data) or JSON Lines (one
A package is the unit these formats store a paper in: a folder or a zip file of such files holding the text as nodes (sections, sentences, cells, equations), page geometry, references, corrections, review receipts and the list of suspects ([spec/12](spec/12-package.md)).
```text
cornell1995second#k3m9q2a@4f1a0c[12:57]
└──── document ──┘└ node ┘└ pin ┘└span┘
cornell1995second-saqk#k3m9q2a@4f1a0c[12:57]
└────── document ────┘└ node ┘└ pin ┘└span┘
```
A hash is a short fingerprint computed from text: the same text always gives the same fingerprint, and changing one character changes the hash.
The string above names code points 12 to 57 of one sentence, as it read when its hash began `4f1a0c`.
The node ID `k3m9q2a` encodes no page, position or kind.
It is minted once from the document key, the node's kind, its text hash and its occurrence, and then frozen.
The document key's last four characters, `saqk`, are the start of a hash of the paper's DOI, so two papers that a citation key alone would name alike never share a key ([spec/02 §2.4](spec/02-model.md#24-work-and-document-keys)).
Fixing a typo elsewhere in the paper leaves the ID alone, and converting the same input again mints the same IDs.
A teammate's download of the same PDF, stamped differently by the publisher, has the same page fingerprints.
@@ -53,7 +54,7 @@ The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and
| Folder | Holds |
| --- | --- |
| [spec/](spec/README.md) | The specification, in 23 chapters. Normative. |
| [spec/](spec/README.md) | The specification, in 24 chapters. Normative. |
| [schemas/](schemas/README.md) | JSON Schemas, which state what a valid file or record looks like, for every file and record. |
| [sql/](sql/README.md) | Tables for SQLite (a database kept in one file), Dolt and PostgreSQL, and the SQLite bundle with full-text search. |
| [context/](context/README.md) | The JSON-LD context (a file that gives JSON terms shared meanings) that makes annotation layers W3C Web Annotations. |
@@ -87,7 +88,7 @@ Without pixi, install those two with pip and run `python3 tools/check.py`.
To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from.
```sh
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
```
## Format names
@@ -101,8 +102,8 @@ Schemas use the identifier base `https://paperspec.scient.ing/schemas/`.
## Status and limits
This is version 0.1.0.
Of the 29 formats, 18 are `candidate` and 11 are `provisional`.
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade and checklist; where those are described, they are marked as such.
Of the 32 formats, 19 are `candidate` and 13 are `provisional`.
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint; where those are described, they are marked as such.
The repository includes a reference checker, test vectors and fixtures.
The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.