infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ad65fcc014384be922ddb69418f7926141fb790f
paperspec/tools/README.md
T
alexmandClaude 516148146f feat: publish the package from CI, and clean up its references 
Publishing
- .gitea/workflows/publish.yml publishes paperspec to the instance's PyPI
  registry (/api/packages/infra/pypi) when a v* tag is pushed: it checks
  the repository, builds the sdist and wheel, refuses to upload unless the
  built version equals the tag, runs twine check --strict, and uploads
  with the secrets PACKAGE_USER and PACKAGE_TOKEN (--skip-existing, so a
  rerun on the same tag is safe). Run by hand, it refuses anything but a
  tag.
- pixi dev environment: twine >= 6.1 (the first that reads Metadata 2.4),
  and `dist` (sdist and wheel) replaces the `wheel` task.
- deploy/README.md: how to set up the secrets, release a version, and
  install from the registry with pixi, uv or pip, tying paperspec to the
  registry where the installer allows it.
- CONTRIBUTING.md "Releasing": a tag versions the package; a format
  change takes the new format version as the top CHANGELOG heading, and a
  code-only release takes the next patch as a subsection, as 0.1.1 does.
- pyproject: fallback_version 0+unknown for source trees without git; the
  license expression is Apache-2.0 AND CC0-1.0, for the code and the data
  the wheel carries.

Cleanup (from an audit of every reference to the package)
- spec/00 lists src/ (the package) and tools/ (how to run it), and no
  longer says the reference implementation is not a library.
- Stale explanations fixed in pixi.toml, .gitignore, the Caddyfile, the
  vectors, context, sql and tools READMEs, docs/examples.md, docs/faq.md,
  spec/01, docs/rationale.md and docs/comparison.md: the commands, what
  they need (the tools extra), the fixture folder hale2026harmonic-damb,
  and three lines that held two sentences.
- tools/README.md counts all six commands, says what needs which extra
  (package.validate and corrections.gate need [schema]), and that the
  vectors ship but the fixtures they name do not.
- The CLI's install and clone hints point at the registry instructions and
  an https clone URL; schema.py raises an ImportError naming
  paperspec[schema], as crypto.py does for [crypto].
- check's Unicode check now guards the package's pinned constant.
- site skips build/, dist/ and *.egg-info.
- CHANGELOG 0.1.1 describes the final state.

pixi run check passes (as `python -m paperspec check`); the rebuild is
byte-identical. pixi.lock needs `pixi lock` for twine before pushing: the
publish workflow installs with --locked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 09:41:26 -04:00

87 lines
8.9 KiB
Markdown
Raw Blame History

# Tools
This page is for anyone who wants to run the repository's checks, rebuild its examples or load a document into a database.
Newcomers get each term defined where it first appears.
Implementers get the commands and the traps.
These commands are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
They are subcommands of one command, `paperspec`, which the Python package `paperspec` installs; its code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
`paperspec --help` lists them and says where each works, and `paperspec COMMAND --help` gives a command's options, advice and examples.
They need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), except `paperspec site`, which needs only the standard library, and they are licensed Apache-2.0.
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic-damb`.
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
A fixture is an example file or package under `fixtures/`.
| Command | Does |
| --- | --- |
| `paperspec check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
| `paperspec build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
| `paperspec validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
| `paperspec load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
| `paperspec site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
| `paperspec stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
`paperspec validate` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries.
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
## When to run what
Each command is also a pixi task.
Pixi sets up a pinned Python and the other packages the repository uses, and installs this checkout as the package, editable, so a change to the code takes effect at once.
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
`pixi run check` lints the Markdown first.
If you do not use pixi, install the checkout editable with its extras on Python 3.14, then run the commands from the repository root:
```sh
pip install -e ".[tools]"
paperspec check # everything
paperspec check --quick # everything except the rebuild
paperspec build --out /tmp/rebuilt
paperspec validate fixtures/packages/hale2026harmonic-damb
paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
```
`python -m paperspec check` runs the same program as `paperspec check`, which helps where the installed command is not on the path.
A passing check ends with `All checks pass.`
The load command prints `hale2026harmonic-damb: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`, or `package: rule: message` for a zipped package that cannot be unpacked.
The full and checks-only forms of one document share its key, so they go into separate databases.
Each command prints its options with `--help` and refuses options it does not know.
`paperspec build` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
Run `pixi run check` before every commit.
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `src/paperspec/cli/build.py`, and commit the regenerated files with the change.
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
## About the oracle
An oracle is a reference computation: a slow, plain program whose answers count as the expected ones.
It is not a production reader or writer.
The oracle exists so that no expected value in this repository is typed by hand.
It favors clarity, and leaves the cryptography and schema validation to established packages.
Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
The VRF of RFC 9381 (`src/paperspec/vrf.py`) and ML-DSA-65 of FIPS 204 (`src/paperspec/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors.
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
Hashes are kept incrementally.
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl` and `targets.jsonl` name a fixture package, and the fixtures stay in the repository.
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
Importing any of its modules needs only the standard library.
Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`.
`paperspec[tools]` installs both, and is what a program that validates packages needs.
The package version comes from git tags through setuptools-scm, and `pixi run -e dev dist` builds the sdist and the wheel into `dist/`.
When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
Reference in new issue View Git Blame Copy Permalink