Add paperspec correct, check math forms and datasets, and give the tools public names
The work the other paper tools planned for paperspec (paperdesk PLAN M5.4, paperform PLAN 4.2): - paperspec correct and paperspec.revision apply proposed corrections to a package, folder or zip, and write it only once it validates. - doc.math.form is checked: label and style commands always, the form's own spelling when papermath is installed or forms= is passed. - paperspec.datasets and seven doc.dataset.* rules check a package's datasets. - fixtures/invalid gains suspect-witness and math-form-label (33 in all). - Canonical JSON reads numbers as doubles, so 1e20 round-trips. - The registries and the JSON-LD context ship in the wheel. - Public names for what paperdesk and paperform imported as private ones, with the old names kept as aliases. - The text and lossy-projection functions are about three times faster through the gate, tested equal to the old code on every code point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
This commit is contained in:
70 files changed
+4470
-270
No files matched your search
+6
-5
@@ -6,7 +6,7 @@ Implementers get the commands and the traps.
|
||||
|
||||
These commands are the repository's reference implementation.
|
||||
They apply every rule of the specification, so the rest of the repository can be checked against them.
|
||||
Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
|
||||
Two build and check the repository, one validates packages and proposed corrections, one applies corrections to a package, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
|
||||
They are subcommands of one command, `paperspec`, which the Python package `paperspec` installs; its code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
|
||||
`paperspec --help` lists them and says where each works, and `paperspec COMMAND --help` gives a command's options, advice and examples.
|
||||
They need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), except `paperspec site`, which needs only the standard library, and they are licensed Apache-2.0.
|
||||
@@ -20,11 +20,12 @@ A fixture is an example file or package under `fixtures/`.
|
||||
| `paperspec check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
|
||||
| `paperspec build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
||||
| `paperspec validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
||||
| `paperspec correct` | Applies proposed corrections to a package, folder or zipped, and writes the package they make: `paperspec correct --correction PROPOSALS [--out PATH] PACKAGE`. Every proposal runs through the gate first and nothing is written unless all pass; the writer recomputes the nodes' values and `document.json`, records the corrections and retired IDs, lowers the changed leaves' levels, and leaves out the derived lines and datasets that read corrected text ([spec/11](../spec/11-corrections.md), [spec/12 §12.7](../spec/12-package.md#127-derived-files)). |
|
||||
| `paperspec load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
||||
| `paperspec site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
|
||||
| `paperspec stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
|
||||
|
||||
`paperspec validate` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries.
|
||||
`paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries.
|
||||
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
|
||||
|
||||
## When to run what
|
||||
@@ -75,12 +76,12 @@ A node's five values depend only on its own subtree, so after a correction the o
|
||||
`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
||||
|
||||
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
|
||||
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema`, `rights`, `traces`, `structures`, `sequences` and `data`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
|
||||
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository.
|
||||
paperstack, paperform, paperdesk and paperclaim import it, and paperchem, paperbio and paperdata test their annotation bodies against it.
|
||||
The library surface they use is `keys`, `text`, `jcs`, `hashing`, `tree`, `anchors`, `package`, `revision`, `corrections`, `db`, `schema`, `fingerprint`, `rights`, `layers`, `claims`, `traces`, `structures`, `sequences` and `data`, with the schemas, SQL, vectors, registries and JSON-LD context as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
|
||||
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository, where `paperspec.resources.path("fixtures")` finds them in a checkout or an editable install.
|
||||
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
|
||||
Importing any of its modules needs only the standard library.
|
||||
Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`.
|
||||
`paperspec[tools]` installs both, and is what a program that validates packages needs.
|
||||
The package version comes from git tags through setuptools-scm, and `pixi run -e release dist` builds the sdist and the wheel into `dist/`.
|
||||
When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
|
||||
Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
|
||||
Reference in new issue
Block a user