156 lines
15 KiB
Markdown
156 lines
15 KiB
Markdown
# Tools
|
|
|
|
This page is for anyone who wants to run the repository's checks, rebuild its examples or load a document into a database.
|
|
Newcomers get each term defined where it first appears.
|
|
Implementers get the commands and the traps.
|
|
|
|
<<<<<<< HEAD
|
|
These commands are the repository's reference implementation.
|
|
They apply every rule of the specification, so the rest of the repository can be checked against them.
|
|
Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
|
|
They are subcommands of one command, `paperspec`, which the Python package `paperspec` installs; its code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
|
|
`paperspec --help` lists them and says where each works, and `paperspec COMMAND --help` gives a command's options, advice and examples.
|
|
They need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), except `paperspec site`, which needs only the standard library, and they are licensed Apache-2.0.
|
|
|
|
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic-damb`.
|
|
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
|
|
A fixture is an example file or package under `fixtures/`.
|
|
|
|
| Command | Does |
|
|
| --- | --- |
|
|
| `paperspec check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
|
|
| `paperspec build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
|
| `paperspec validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
|
| `paperspec load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
|
| `paperspec site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
|
|
| `paperspec stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
|
|
|
|
`paperspec validate` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries.
|
|
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
|
|
|
|
## When to run what
|
|
|
|
Each command is also a pixi task.
|
|
Pixi sets up a pinned Python and the other packages the repository uses, and installs this checkout as the package, editable, so a change to the code takes effect at once.
|
|
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
|
|
`pixi run check` lints the Markdown first.
|
|
|
|
If you do not use pixi, install the checkout editable with its extras on Python 3.14, then run the commands from the repository root:
|
|
|
|
```sh
|
|
pip install -e ".[tools]"
|
|
paperspec check # everything
|
|
paperspec check --quick # everything except the rebuild
|
|
paperspec build --out /tmp/rebuilt
|
|
paperspec validate fixtures/packages/hale2026harmonic-damb
|
|
paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
|
```
|
|
|
|
`python -m paperspec check` runs the same program as `paperspec check`, which helps where the installed command is not on the path.
|
|
|
|
A passing check ends with `All checks pass.`
|
|
The load command prints `hale2026harmonic-damb: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
|
|
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`, or `package: rule: message` for a zipped package that cannot be unpacked.
|
|
The full and checks-only forms of one document share its key, so they go into separate databases.
|
|
Each command prints its options with `--help` and refuses options it does not know.
|
|
`paperspec build` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
|
|
|
Run `pixi run check` before every commit.
|
|
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `src/paperspec/cli/build.py`, and commit the regenerated files with the change.
|
|
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
|
|
|
|
## About the oracle
|
|
|
|
An oracle is a reference computation: a slow, plain program whose answers count as the expected ones.
|
|
It is not a production reader or writer.
|
|
The oracle exists so that no expected value in this repository is typed by hand.
|
|
|
|
It favors clarity, and leaves the cryptography and schema validation to established packages.
|
|
Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
|
|
The VRF of RFC 9381 (`src/paperspec/vrf.py`) and ML-DSA-65 of FIPS 204 (`src/paperspec/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors.
|
|
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
|
|
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
|
|
|
|
Hashes are kept incrementally.
|
|
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
|
|
`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
|
|
|
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
|
|
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
|
|
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl` and `targets.jsonl` name a fixture package, and the fixtures stay in the repository.
|
|
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
|
|
Importing any of its modules needs only the standard library.
|
|
Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`.
|
|
`paperspec[tools]` installs both, and is what a program that validates packages needs.
|
|
The package version comes from git tags through setuptools-scm, and `pixi run -e dev dist` builds the sdist and the wheel into `dist/`.
|
|
When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
|
|
Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
|
|
||||||| parent of 4fa326d (First implementation (#1))
|
|
=======
|
|
These programs are the repository's reference implementation.
|
|
They apply every rule of the specification, so the rest of the repository can be checked against them.
|
|
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
|
|
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
|
|
|
|
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
|
|
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
|
|
A fixture is an example file or package under `fixtures/`.
|
|
|
|
| Program | Does |
|
|
| --- | --- |
|
|
| `check.py` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
|
|
| `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
|
| `validate.py` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
|
| `load.py` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
|
| `site.py` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
|
|
| `stress.py` | Stresses the oracle with random inputs from a seeded generator: `python3 tools/stress.py [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
|
|
| `oracle/` | A compact implementation of every rule. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. |
|
|
|
|
## When to run what
|
|
|
|
Each program is also a pixi task.
|
|
Pixi sets up a pinned Python and the other packages the repository uses.
|
|
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
|
|
`pixi run check` lints the Markdown first.
|
|
|
|
Run the programs directly from the repository root if you do not use pixi:
|
|
|
|
```sh
|
|
python3 tools/check.py # everything
|
|
python3 tools/check.py --quick # everything except the rebuild
|
|
python3 tools/build.py --out /tmp/rebuilt
|
|
python3 tools/validate.py fixtures/packages/hale2026harmonic
|
|
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
|
|
```
|
|
|
|
A passing check ends with `All checks pass.`
|
|
The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
|
|
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
|
|
The full and checks-only forms of one document share its key, so they go into separate databases.
|
|
Each program prints its options with `--help` and refuses options it does not know.
|
|
`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
|
|
|
Run `pixi run check` before every commit.
|
|
Run `pixi run build` after you change a fixture source, a rule in `oracle/` or a case in `build.py`, and commit the regenerated files with the change.
|
|
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
|
|
|
|
## About the oracle
|
|
|
|
An oracle is a reference computation: a slow, plain program whose answers count as the expected ones.
|
|
It is not a production reader or writer.
|
|
The oracle exists so that no expected value in this repository is typed by hand.
|
|
|
|
It favors clarity, and leaves the cryptography and schema validation to established packages.
|
|
Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
|
|
The VRF of RFC 9381 (`oracle/vrf.py`) and ML-DSA-65 of FIPS 204 (`oracle/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors.
|
|
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
|
|
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
|
|
|
|
Hashes are kept incrementally.
|
|
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
|
|
`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
|
|
|
Do not build on it as a library.
|
|
Implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
|
|
>>>>>>> 4fa326d (First implementation (#1))
|