alexmandClaude
a09d158e16
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
openalex in same_as, else the folded title, year and family name. A
collision lengthens the later key by two characters; a key is minted
once and never recomputed. A document key adds .si, .v<seq> or .<type>
for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
every node and record ID was re-minted and the docs rewritten to match.
The validator checks the document key against the work key and version
(doc.key.work) and the hash part against the identifier (doc.key.suffix).
vectors/keys.jsonl pins the cases paperstack's tests pin.
Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
implemented in tools/oracle/vrf.py in plain Python and reproduced
against the RFC's Example 16; the sample ranks by the VRF output. The
earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
verify_inclusion, consistency_proof, verify_consistency) with
scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
checkpoints with key hashes as transparency logs share them, the rule
that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).
Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
the checker refuses another version.
Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
checkpoint records; a check that every ID and hash the prose quotes
exists in the fixtures or vectors; three more broken packages
(history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
section, media-type registration and OCI distribution notes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Tools
This page is for anyone who wants to run the repository's checks, rebuild its examples or load a document into a database. Newcomers get each term defined where it first appears. Implementers get the commands and the traps.
These programs are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
They need Python 3.14 and two packages, jsonschema and cryptography, and are licensed Apache-2.0.
A package is the folder of files that holds one document, such as fixtures/packages/hale2026harmonic.
A vector is a test case: an input and the exact output every implementation must compute (see the vectors).
A fixture is an example file or package under fixtures/.
| Program | Does |
|---|---|
check.py |
Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. --quick skips the rebuild. |
build.py |
Builds fixtures/ and vectors/ from the hand-written sources in fixtures/src/ and the cases it defines. --out DIR builds into another folder. |
validate.py |
Validates packages, as folders or as zipped packages (.sct.zip, unpacked under the rules of spec/12 §12.2): python3 tools/validate.py PACKAGE [PACKAGE ...]. With --correction PROPOSALS it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything (spec/11 §11.4). |
load.py |
Validates packages, folders or zipped, then loads them into a SQLite database with the tables of sql/: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]. With --bundle --built-from REV it builds a new bundle instead, with the full-text search index (spec/13 §13.5). |
site.py |
Builds the website into _site/: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. pixi run check builds it into a temporary folder and checks every link (deploy/). |
stress.py |
Stresses the oracle with random inputs from a seeded generator: python3 tools/stress.py [--seed N] [--rounds N]. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
oracle/ |
A compact implementation of every rule. build.py uses it to compute expected values, and check.py uses it to recompute them. |
When to run what
Each program is also a pixi task.
Pixi sets up a pinned Python and the other packages the repository uses.
The commands pixi run check, pixi run quick, pixi run build and pixi run load then work from anywhere in the repository.
pixi run check lints the Markdown first.
Run the programs directly from the repository root if you do not use pixi:
python3 tools/check.py # everything
python3 tools/check.py --quick # everything except the rebuild
python3 tools/build.py --out /tmp/rebuilt
python3 tools/validate.py fixtures/packages/hale2026harmonic
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
A passing check ends with All checks pass.
The load command prints hale2026harmonic: 75 nodes, one line per package, and writes lab.db with a folder lab.db.assets/ beside it for files that have no table of their own.
It validates every package first and loads nothing if one fails, printing each failure as package: where: rule: message.
The full and checks-only forms of one document share its key, so they go into separate databases.
Each program prints its options with --help and refuses options it does not know.
build.py rebuilds fixtures/ and vectors/ in place unless you pass --out.
Run pixi run check before every commit.
Run pixi run build after you change a fixture source, a rule in oracle/ or a case in build.py, and commit the regenerated files with the change.
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
About the oracle
An oracle is a reference computation: a slow, plain program whose answers count as the expected ones. It is not a production reader or writer. The oracle exists so that no expected value in this repository is typed by hand.
It favors clarity, and leaves the cryptography and schema validation to established packages.
Ed25519 (a digital signature scheme) and HKDF come from cryptography, and JSON Schema (a format that describes the shape of JSON data) validation from jsonschema.
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
Hashes are kept incrementally.
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
check.py compares the result after every correction with a slow computation that projects each node from its whole subtree.
Do not build on it as a library. Implementations follow the specification and test themselves against the vectors.