paperspec/tools/README.md

Tools

This page is for anyone who wants to run the repository's checks, rebuild its examples or load a document into a database. Newcomers get each term defined where it first appears. Implementers get the commands and the traps.

These commands are the repository's reference implementation. They apply every rule of the specification, so the rest of the repository can be checked against them. Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website. They are subcommands of one command, paperspec, which the Python package paperspec installs; its code is in src/paperspec/, with the commands in src/paperspec/cli/. paperspec --help lists them and says where each works, and paperspec COMMAND --help gives a command's options, advice and examples. They need Python 3.14 and the package's tools extra (jsonschema, referencing and cryptography), except paperspec site, which needs only the standard library, and they are licensed Apache-2.0.

A package is the folder of files that holds one document, such as fixtures/packages/hale2026harmonic-damb. A vector is a test case: an input and the exact output every implementation must compute (see the vectors). A fixture is an example file or package under fixtures/.

Command	Does
paperspec check	Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. --quick skips the rebuild.
paperspec build	Builds fixtures/ and vectors/ from the hand-written sources in fixtures/src/ and the cases it defines. --out DIR builds into another folder.
paperspec validate	Validates packages, as folders or as zipped packages (.sct.zip, unpacked under the rules of spec/12 §12.2): paperspec validate PACKAGE [PACKAGE ...]. With --correction PROPOSALS it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything (spec/11 §11.4).
paperspec load	Validates packages, folders or zipped, then loads them into a SQLite database with the tables of sql/: paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]. With --bundle --built-from REV it builds a new bundle instead, with the full-text search index (spec/13 §13.5).
paperspec site	Builds the website into _site/: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. pixi run check builds it into a temporary folder and checks every link (deploy/).
paperspec stress	Stresses the oracle with random inputs from a seeded generator: paperspec stress [--seed N] [--rounds N]. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused.

paperspec validate and paperspec load work anywhere the package is installed with its tools extra, using the schemas and SQL it carries. The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.

When to run what

Each command is also a pixi task. Pixi sets up a pinned Python and the other packages the repository uses, and installs this checkout as the package, editable, so a change to the code takes effect at once. The commands pixi run check, pixi run quick, pixi run build and pixi run load then work from anywhere in the repository. pixi run check lints the Markdown first.

If you do not use pixi, install the checkout editable with its extras on Python 3.14, then run the commands from the repository root:

pip install -e ".[tools]"
paperspec check            # everything
paperspec check --quick    # everything except the rebuild
paperspec build --out /tmp/rebuilt
paperspec validate fixtures/packages/hale2026harmonic-damb
paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb

python -m paperspec check runs the same program as paperspec check, which helps where the installed command is not on the path.

A passing check ends with All checks pass. The load command prints hale2026harmonic-damb: 75 nodes, one line per package, and writes lab.db with a folder lab.db.assets/ beside it for files that have no table of their own. It validates every package first and loads nothing if one fails, printing each failure as package: where: rule: message, or package: rule: message for a zipped package that cannot be unpacked. The full and checks-only forms of one document share its key, so they go into separate databases. Each command prints its options with --help and refuses options it does not know. paperspec build rebuilds fixtures/ and vectors/ in place unless you pass --out.

Run pixi run check before every commit. Run pixi run build after you change a fixture source, a rule in src/paperspec/ or a case in src/paperspec/cli/build.py, and commit the regenerated files with the change. The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.

About the oracle

An oracle is a reference computation: a slow, plain program whose answers count as the expected ones. It is not a production reader or writer. The oracle exists so that no expected value in this repository is typed by hand.

It favors clarity, and leaves the cryptography and schema validation to established packages. Ed25519 (a digital signature scheme) and HKDF come from cryptography, and JSON Schema (a format that describes the shape of JSON data) validation from jsonschema. The VRF of RFC 9381 (src/paperspec/vrf.py) and ML-DSA-65 of FIPS 204 (src/paperspec/mldsa.py) are written out in plain Python, slow and clear, and checked against the standards' own test vectors. The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to. Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.

Hashes are kept incrementally. A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined. paperspec check compares the result after every correction with a slow computation that projects each node from its whole subtree.

Other repositories of the project do not reimplement the specification: they install this repository as the Python package paperspec, from the Gitea package registry or as a git pin (deploy/), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code. The library surface they use is keys, text, jcs, package, db, schema and rights, with the schemas, SQL and vectors as package data: paperspec.resources.path("sql") finds a folder in an installed wheel and in an editable install alike. The vectors travel with the package, but resolve.jsonl, grading.jsonl and targets.jsonl name a fixture package, and the fixtures stay in the repository. The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other. Importing any of its modules needs only the standard library. Schema validation (schema, and with it package.validate and corrections.gate) needs paperspec[schema] (jsonschema and referencing), and HKDF and Ed25519 in crypto, which the signature and receipt checks use, need paperspec[crypto]. paperspec[tools] installs both, and is what a program that validates packages needs. The package version comes from git tags through setuptools-scm, and pixi run -e dev dist builds the sdist and the wheel into dist/. When scienting-core exists it takes this role behind the same vectors, and the import changes without any computed value changing. Independent implementations follow the specification and test themselves against the vectors.