infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat!: pin Python 3.14 and Unicode 16.0, and make the tools commands

Browse Source 
Python and Unicode
- requires-python is >=3.14,<3.15: 3.14 is the only Python with the
  Unicode 16.0 data the specification pins (3.13 has 15.1, 3.15 has 17.0).
- `import paperspec` raises ImportError unless unicodedata reports
  paperspec.UNICODE_VERSION (16.0.0), so a source checkout or an install
  that bypasses requires-python cannot compute values on other data.

Commands
- tools/{check,build,validate,load,stress,site}.py move with history to
  src/paperspec/cli/ and install as paperspec-check, -build, -validate,
  -load, -stress and -site; each also runs as `python -m paperspec.cli.X`.
- paperspec.cli.require() exits with "install paperspec[tools]" when
  cryptography or jsonschema is missing; paperspec.cli.checkout() finds
  the repository the check, build, stress and site commands work on (the
  editable install's own checkout, else the one around the current
  folder) and exits with a message outside one. tools/__init__.py is gone.
- validate and load read the schemas and SQL through the new
  paperspec.resources.path(), which finds them in a wheel and in an
  editable install alike, so they work anywhere the package is installed.
- check runs the rebuild as `python -m paperspec.cli.build`.
- build still writes the provenance tool label "paperspec tools/build.py":
  it is part of every hashed and signed fixture.
- pixi tasks and the check workflow run the commands; tools/README.md
  documents them, and the other docs name the commands instead of paths.

License
- LICENSE.md gives `src/` Apache-2.0 alongside `tools/`. Since the
  previous commit moved the code to src/, no row had covered it.

pixi run check passes (run as `python -m paperspec.cli.check` with src/ on
the path, since this environment cannot install the entry points); the
byte-for-byte rebuild matches, so no fixture or vector changed.
pixi.lock still needs `pixi lock`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-15 09:17:27 -04:00
1 parent cfe1f39099
commit 607062056f
33 files changed
+227 -149
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tools/README.md
+32 -25
View File
@@ -4,51 +4,57 @@ This page is for anyone who wants to run the repository's checks, rebuild its ex
Newcomers get each term defined where it first appears.
Implementers get the commands and the traps.
These programs are the repository's reference implementation.
These commands are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
They are console scripts of the Python package `paperspec`, whose code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic-damb`.
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
A fixture is an example file or package under `fixtures/`.
| Program | Does |
| Command | Does |
| --- | --- |
| `check.py` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
| `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
| `validate.py` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
| `load.py` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
| `site.py` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
| `stress.py` | Stresses the oracle with random inputs from a seeded generator: `python3 tools/stress.py [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
| [src/paperspec/](../src/paperspec/) | The oracle: a compact implementation of every rule, and the Python package `paperspec`. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. The programs put `src/` first on the path, so they run this checkout's code with or without pixi. |
| `paperspec-check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
| `paperspec-build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
| `paperspec-validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec-validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
| `paperspec-load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec-load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
| `paperspec-site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
| `paperspec-stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec-stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
`paperspec-validate` and `paperspec-load` work anywhere the package is installed, with the schemas and SQL it carries.
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
## When to run what
Each program is also a pixi task.
Pixi sets up a pinned Python and the other packages the repository uses.
Each command is also a pixi task.
Pixi sets up a pinned Python and the other packages the repository uses, and installs this checkout as the package, editable, so a change to the code takes effect at once.
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
`pixi run check` lints the Markdown first.
Run the programs directly from the repository root if you do not use pixi:
If you do not use pixi, install the checkout editable with its extras on Python 3.14, then run the commands from the repository root:
```sh
python3 tools/check.py # everything
python3 tools/check.py --quick # everything except the rebuild
python3 tools/build.py --out /tmp/rebuilt
python3 tools/validate.py fixtures/packages/hale2026harmonic
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
pip install -e ".[tools]"
paperspec-check # everything
paperspec-check --quick # everything except the rebuild
paperspec-build --out /tmp/rebuilt
paperspec-validate fixtures/packages/hale2026harmonic-damb
paperspec-load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
```
Each command is also a module, so `python -m paperspec.cli.check` runs the same program.
A passing check ends with `All checks pass.`
The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
The load command prints `hale2026harmonic-damb: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
The full and checks-only forms of one document share its key, so they go into separate databases.
Each program prints its options with `--help` and refuses options it does not know.
`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
Each command prints its options with `--help` and refuses options it does not know.
`paperspec-build` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
Run `pixi run check` before every commit.
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `build.py`, and commit the regenerated files with the change.
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `src/paperspec/cli/build.py`, and commit the regenerated files with the change.
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
## About the oracle
@@ -65,11 +71,12 @@ Canonical JSON goes through Python's built-in JSON encoder, and falls back to an
Hashes are kept incrementally.
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
`paperspec-check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec` (`pip install .`, or a git pin) and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data (`importlib.resources.files("paperspec") / "sql" / "doc.sql"`).
Those modules need Python 3.14, the first with the Unicode 16.0 character data the specification pins, and otherwise only the standard library, except that `schema` needs `paperspec[schema]` (`jsonschema` and `referencing`) and the HKDF and Ed25519 functions of `crypto` need `paperspec[crypto]`; `paperspec[tools]` installs both.
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
Otherwise it needs only the standard library, except that `schema` needs `paperspec[schema]` (`jsonschema` and `referencing`) and the HKDF and Ed25519 functions of `crypto` need `paperspec[crypto]`; `paperspec[tools]` installs both.
The package version comes from git tags through setuptools-scm, and `pixi run -e dev wheel` builds the wheel into `dist/`.
When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).