Add paperspec correct, check math forms and datasets, and give the tools public names

The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
This commit is contained in:
alexmandClaude committed 2026-10-20 22:10:17 -04:00
1 parent 8c07dd7c15
commit 4fb8b4f5a7
70 files changed
+4470 -270

No files matched your search

+6 -5
View File
@@ -6,7 +6,7 @@ Implementers get the commands and the traps.
These commands are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
Two build and check the repository, one validates packages and proposed corrections, one applies corrections to a package, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
They are subcommands of one command, `paperspec`, which the Python package `paperspec` installs; its code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
`paperspec --help` lists them and says where each works, and `paperspec COMMAND --help` gives a command's options, advice and examples.
They need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), except `paperspec site`, which needs only the standard library, and they are licensed Apache-2.0.
@@ -20,11 +20,12 @@ A fixture is an example file or package under `fixtures/`.
| `paperspec check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
| `paperspec build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
| `paperspec validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
| `paperspec correct` | Applies proposed corrections to a package, folder or zipped, and writes the package they make: `paperspec correct --correction PROPOSALS [--out PATH] PACKAGE`. Every proposal runs through the gate first and nothing is written unless all pass; the writer recomputes the nodes' values and `document.json`, records the corrections and retired IDs, lowers the changed leaves' levels, and leaves out the derived lines and datasets that read corrected text ([spec/11](../spec/11-corrections.md), [spec/12 §12.7](../spec/12-package.md#127-derived-files)). |
| `paperspec load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
| `paperspec site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
| `paperspec stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
`paperspec validate` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries.
`paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries.
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
## When to run what
@@ -75,12 +76,12 @@ A node's five values depend only on its own subtree, so after a correction the o
`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema`, `rights`, `traces`, `structures`, `sequences` and `data`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository.
paperstack, paperform, paperdesk and paperclaim import it, and paperchem, paperbio and paperdata test their annotation bodies against it.
The library surface they use is `keys`, `text`, `jcs`, `hashing`, `tree`, `anchors`, `package`, `revision`, `corrections`, `db`, `schema`, `fingerprint`, `rights`, `layers`, `claims`, `traces`, `structures`, `sequences` and `data`, with the schemas, SQL, vectors, registries and JSON-LD context as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository, where `paperspec.resources.path("fixtures")` finds them in a checkout or an editable install.
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
Importing any of its modules needs only the standard library.
Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`.
`paperspec[tools]` installs both, and is what a program that validates packages needs.
The package version comes from git tags through setuptools-scm, and `pixi run -e release dist` builds the sdist and the wheel into `dist/`.
When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).