Reorganize paperspec commands and pin dependency versions (#2)

Reviewed-on: #2

tools/README.md

@@ -4,51 +4,58 @@ This page is for anyone who wants to run the repository's checks, rebuild its ex
 Newcomers get each term defined where it first appears.
 Implementers get the commands and the traps.
 
-These programs are the repository's reference implementation.
+These commands are the repository's reference implementation.
 They apply every rule of the specification, so the rest of the repository can be checked against them.
-Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
-They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
+Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
+They are subcommands of one command, `paperspec`, which the Python package `paperspec` installs; its code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
+`paperspec --help` lists them and says where each works, and `paperspec COMMAND --help` gives a command's options, advice and examples.
+They need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), except `paperspec site`, which needs only the standard library, and they are licensed Apache-2.0.
 
-A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
+A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic-damb`.
 A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
 A fixture is an example file or package under `fixtures/`.
 
-| Program | Does |
+| Command | Does |
 | --- | --- |
-| `check.py` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
-| `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
-| `validate.py` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
-| `load.py` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
-| `site.py` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
-| `stress.py` | Stresses the oracle with random inputs from a seeded generator: `python3 tools/stress.py [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
-| `oracle/` | A compact implementation of every rule. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. |
+| `paperspec check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
+| `paperspec build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
+| `paperspec validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
+| `paperspec load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
+| `paperspec site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
+| `paperspec stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
+
+`paperspec validate` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries.
+The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
 
 ## When to run what
 
-Each program is also a pixi task.
-Pixi sets up a pinned Python and the other packages the repository uses.
+Each command is also a pixi task.
+Pixi sets up a pinned Python and the other packages the repository uses, and installs this checkout as the package, editable, so a change to the code takes effect at once.
 The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
 `pixi run check` lints the Markdown first.
 
-Run the programs directly from the repository root if you do not use pixi:
+If you do not use pixi, install the checkout editable with its extras on Python 3.14, then run the commands from the repository root:
 
 ```sh
-python3 tools/check.py            # everything
-python3 tools/check.py --quick    # everything except the rebuild
-python3 tools/build.py --out /tmp/rebuilt
-python3 tools/validate.py fixtures/packages/hale2026harmonic
-python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
+pip install -e ".[tools]"
+paperspec check            # everything
+paperspec check --quick    # everything except the rebuild
+paperspec build --out /tmp/rebuilt
+paperspec validate fixtures/packages/hale2026harmonic-damb
+paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
 ```
 
+`python -m paperspec check` runs the same program as `paperspec check`, which helps where the installed command is not on the path.
+
 A passing check ends with `All checks pass.`
-The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
-It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
+The load command prints `hale2026harmonic-damb: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
+It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`, or `package: rule: message` for a zipped package that cannot be unpacked.
 The full and checks-only forms of one document share its key, so they go into separate databases.
-Each program prints its options with `--help` and refuses options it does not know.
-`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
+Each command prints its options with `--help` and refuses options it does not know.
+`paperspec build` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
 
 Run `pixi run check` before every commit.
-Run `pixi run build` after you change a fixture source, a rule in `oracle/` or a case in `build.py`, and commit the regenerated files with the change.
+Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `src/paperspec/cli/build.py`, and commit the regenerated files with the change.
 The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
 
 ## About the oracle
@@ -59,13 +66,21 @@ The oracle exists so that no expected value in this repository is typed by hand.
 
 It favors clarity, and leaves the cryptography and schema validation to established packages.
 Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
-The VRF of RFC 9381 (`oracle/vrf.py`) and ML-DSA-65 of FIPS 204 (`oracle/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors.
+The VRF of RFC 9381 (`src/paperspec/vrf.py`) and ML-DSA-65 of FIPS 204 (`src/paperspec/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors.
 The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
 Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
 
 Hashes are kept incrementally.
 A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
-`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
+`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
 
-Do not build on it as a library.
-Implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
+Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
+The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
+The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl` and `targets.jsonl` name a fixture package, and the fixtures stay in the repository.
+The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
+Importing any of its modules needs only the standard library.
+Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`.
+`paperspec[tools]` installs both, and is what a program that validates packages needs.
+The package version comes from git tags through setuptools-scm, and `pixi run -e dev dist` builds the sdist and the wheel into `dist/`.
+When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
+Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).