paperspec/tools/README.md

Tools

This page is for anyone who wants to run the repository's checks, rebuild its examples or load a document into a database. Newcomers get each term defined where it first appears. Implementers get the commands and the traps.

These programs are the repository's reference implementation. They apply every rule of the specification, so the rest of the repository can be checked against them. Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file. They need Python 3.14 and two packages, jsonschema and cryptography, and are licensed Apache-2.0.

A package is the folder of files that holds one document, such as fixtures/packages/hale2026harmonic. A vector is a test case: an input and the exact output every implementation must compute (see the vectors). A fixture is an example file or package under fixtures/.

Program	Does
check.py	Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. --quick skips the rebuild.
build.py	Builds fixtures/ and vectors/ from the hand-written sources in fixtures/src/ and the cases it defines. --out DIR builds into another folder.
validate.py	Validates packages, as folders or as zipped packages (.sct.zip, unpacked under the rules of spec/12 §12.2): python3 tools/validate.py PACKAGE [PACKAGE ...]. With --correction PROPOSALS it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything (spec/11 §11.4).
load.py	Validates packages, folders or zipped, then loads them into a SQLite database with the tables of sql/: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]. With --bundle --built-from REV it builds a new bundle instead, with the full-text search index (spec/13 §13.5).
site.py	Builds the website into _site/: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. pixi run check builds it into a temporary folder and checks every link (deploy/).
stress.py	Stresses the oracle with random inputs from a seeded generator: python3 tools/stress.py [--seed N] [--rounds N]. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused.
oracle/	A compact implementation of every rule. build.py uses it to compute expected values, and check.py uses it to recompute them.

When to run what

Each program is also a pixi task. Pixi sets up a pinned Python and the other packages the repository uses. The commands pixi run check, pixi run quick, pixi run build and pixi run load then work from anywhere in the repository. pixi run check lints the Markdown first.

Run the programs directly from the repository root if you do not use pixi:

python3 tools/check.py            # everything
python3 tools/check.py --quick    # everything except the rebuild
python3 tools/build.py --out /tmp/rebuilt
python3 tools/validate.py fixtures/packages/hale2026harmonic
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic

A passing check ends with All checks pass. The load command prints hale2026harmonic: 75 nodes, one line per package, and writes lab.db with a folder lab.db.assets/ beside it for files that have no table of their own. It validates every package first and loads nothing if one fails, printing each failure as package: where: rule: message. The full and checks-only forms of one document share its key, so they go into separate databases. Each program prints its options with --help and refuses options it does not know. build.py rebuilds fixtures/ and vectors/ in place unless you pass --out.

Run pixi run check before every commit. Run pixi run build after you change a fixture source, a rule in oracle/ or a case in build.py, and commit the regenerated files with the change. The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.

About the oracle

An oracle is a reference computation: a slow, plain program whose answers count as the expected ones. It is not a production reader or writer. The oracle exists so that no expected value in this repository is typed by hand.

It favors clarity, and leaves the cryptography and schema validation to established packages. Ed25519 (a digital signature scheme) and HKDF come from cryptography, and JSON Schema (a format that describes the shape of JSON data) validation from jsonschema. The VRF of RFC 9381 (oracle/vrf.py) and ML-DSA-65 of FIPS 204 (oracle/mldsa.py) are written out in plain Python, slow and clear, and checked against the standards' own test vectors. The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to. Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.

Hashes are kept incrementally. A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined. check.py compares the result after every correction with a slow computation that projects each node from its whole subtree.

Other repositories of the project do not reimplement the specification: they install this repository as the Python package paperspec (pip install ., or a git pin) and import the oracle, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code. The library surface they use is keys, text, jcs, package, db, schema and rights, with the schemas, SQL and vectors as package data (importlib.resources.files("paperspec") / "sql" / "doc.sql"). Those modules run on Python 3.12 with the standard library alone, except that schema needs paperspec[schema] (jsonschema and referencing) and the HKDF and Ed25519 functions of crypto need paperspec[crypto]; paperspec[tools] installs both. When scienting-core exists it takes this role behind the same vectors, and the import changes without any computed value changing. Independent implementations follow the specification and test themselves against the vectors.