paperspec/tools/README.md

Tools

This page is for anyone who wants to run the repository's checks, rebuild its examples or load a document into a database. Newcomers get each term defined where it first appears. Implementers get the commands and the traps.

These commands are the repository's reference implementation. They apply every rule of the specification, so the rest of the repository can be checked against them. Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file. They are console scripts of the Python package paperspec, whose code is in src/paperspec/, with the commands in src/paperspec/cli/. They need Python 3.14 and two packages, jsonschema and cryptography, and are licensed Apache-2.0.

A package is the folder of files that holds one document, such as fixtures/packages/hale2026harmonic-damb. A vector is a test case: an input and the exact output every implementation must compute (see the vectors). A fixture is an example file or package under fixtures/.

Command	Does
paperspec-check	Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. --quick skips the rebuild.
paperspec-build	Builds fixtures/ and vectors/ from the hand-written sources in fixtures/src/ and the cases it defines. --out DIR builds into another folder.
paperspec-validate	Validates packages, as folders or as zipped packages (.sct.zip, unpacked under the rules of spec/12 §12.2): paperspec-validate PACKAGE [PACKAGE ...]. With --correction PROPOSALS it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything (spec/11 §11.4).
paperspec-load	Validates packages, folders or zipped, then loads them into a SQLite database with the tables of sql/: paperspec-load --sqlite lab.db PACKAGE [PACKAGE ...]. With --bundle --built-from REV it builds a new bundle instead, with the full-text search index (spec/13 §13.5).
paperspec-site	Builds the website into _site/: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. pixi run check builds it into a temporary folder and checks every link (deploy/).
paperspec-stress	Stresses the oracle with random inputs from a seeded generator: paperspec-stress [--seed N] [--rounds N]. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused.

paperspec-validate and paperspec-load work anywhere the package is installed, with the schemas and SQL it carries. The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.

When to run what

Each command is also a pixi task. Pixi sets up a pinned Python and the other packages the repository uses, and installs this checkout as the package, editable, so a change to the code takes effect at once. The commands pixi run check, pixi run quick, pixi run build and pixi run load then work from anywhere in the repository. pixi run check lints the Markdown first.

If you do not use pixi, install the checkout editable with its extras on Python 3.14, then run the commands from the repository root:

pip install -e ".[tools]"
paperspec-check            # everything
paperspec-check --quick    # everything except the rebuild
paperspec-build --out /tmp/rebuilt
paperspec-validate fixtures/packages/hale2026harmonic-damb
paperspec-load --sqlite lab.db fixtures/packages/hale2026harmonic-damb

Each command is also a module, so python -m paperspec.cli.check runs the same program.

A passing check ends with All checks pass. The load command prints hale2026harmonic-damb: 75 nodes, one line per package, and writes lab.db with a folder lab.db.assets/ beside it for files that have no table of their own. It validates every package first and loads nothing if one fails, printing each failure as package: where: rule: message. The full and checks-only forms of one document share its key, so they go into separate databases. Each command prints its options with --help and refuses options it does not know. paperspec-build rebuilds fixtures/ and vectors/ in place unless you pass --out.

Run pixi run check before every commit. Run pixi run build after you change a fixture source, a rule in src/paperspec/ or a case in src/paperspec/cli/build.py, and commit the regenerated files with the change. The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.

About the oracle

An oracle is a reference computation: a slow, plain program whose answers count as the expected ones. It is not a production reader or writer. The oracle exists so that no expected value in this repository is typed by hand.

It favors clarity, and leaves the cryptography and schema validation to established packages. Ed25519 (a digital signature scheme) and HKDF come from cryptography, and JSON Schema (a format that describes the shape of JSON data) validation from jsonschema. The VRF of RFC 9381 (src/paperspec/vrf.py) and ML-DSA-65 of FIPS 204 (src/paperspec/mldsa.py) are written out in plain Python, slow and clear, and checked against the standards' own test vectors. The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to. Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.

Hashes are kept incrementally. A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined. paperspec-check compares the result after every correction with a slow computation that projects each node from its whole subtree.

Other repositories of the project do not reimplement the specification: they install this repository as the Python package paperspec (pip install ., or a git pin) and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code. The library surface they use is keys, text, jcs, package, db, schema and rights, with the schemas, SQL and vectors as package data: paperspec.resources.path("sql") finds a folder in an installed wheel and in an editable install alike. The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other. Otherwise it needs only the standard library, except that schema needs paperspec[schema] (jsonschema and referencing) and the HKDF and Ed25519 functions of crypto need paperspec[crypto]; paperspec[tools] installs both. The package version comes from git tags through setuptools-scm, and pixi run -e dev wheel builds the wheel into dist/. When scienting-core exists it takes this role behind the same vectors, and the import changes without any computed value changing. Independent implementations follow the specification and test themselves against the vectors.