Add paperspec correct, check math forms and datasets, and give the tools public names

The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
This commit is contained in:
alexmandClaude committed 2026-10-20 22:10:17 -04:00
1 parent 8c07dd7c15
commit 4fb8b4f5a7
70 files changed
+4470 -270

No files matched your search

+13
View File
@@ -41,3 +41,16 @@ paperspec validate --correction fix.json fixtures/packages/tiny2026tiny-j4qm
The command prints the record the correction would append to `corrections.jsonl`, with its computed `index`, `id`, `path`, `before` and `after`, and writes nothing.
A refused proposal prints `file:N node: rule: message` on standard error and stops: with this proposal, changing `second` to `2nd` is refused with `correction.digits`, because an emendation changes no digit.
`--correction` takes exactly one package, and it refuses a package that withholds its text.
## Apply the correction
When the proposal passes, `paperspec correct` applies it and writes the corrected package:
```sh
paperspec correct --correction fix.json --out fixed/tiny2026tiny-j4qm fixtures/packages/tiny2026tiny-j4qm
```
It runs the same gate, then rewrites every file the correction touches: the nodes and their values, `corrections.jsonl`, `ids.jsonl`, `references.jsonl`, `locations.jsonl`, `suspects.jsonl`, `document.json` and the manifest.
The corrected package is validated before it is written; without `--out` it replaces the package given.
The printed record is the one `validate --correction` printed.
A leaf the correction changed drops to F1 when the evidence includes a crop of the page, and to F0 otherwise; an emendation like this one keeps the leaf's level.
+41 -10
View File
@@ -9,15 +9,16 @@ Two aliases are shared by the type annotations: `jsontypes.Record`, one parsed J
A function that takes a path returns text unless its docstring says otherwise.
A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the rule as the specification's conformance chapter lists it.
`paperspec.resources.path(name)` returns the folder `schemas`, `sql` or `vectors`, in an installed wheel and in a checkout alike.
`paperspec.resources.path(name)` returns the folder `schemas`, `sql`, `vectors`, `registry` or `context`, in an installed wheel and in a checkout alike, and `fixtures` in a checkout or an editable install of one.
`paperspec.resources.registry(name)` reads a registry file by its path under `registry/`, such as `rules.json` or `stamps/scienting-stamps-1.json`.
## Text, hashes and canonical JSON
| Module | For | Main functions |
| --- | --- | --- |
| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` |
| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads_line` |
| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `lossy`, `own_content`, `meta_hash`, `Hasher`, `Index` |
| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `digits`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` |
| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads`, `loads_line`, `parse_int` |
| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `strict_runs`, `lossy`, `lossy_from_strict`, `own_content`, `meta_hash`, `Hasher`, `Index` |
| `notation` | Simple math notation and its canonical linear form. | `fold_math`, `complex_key`, `fold_plain_text` |
| `keys` | Work and document keys. | `work_key`, `unique_work_key`, `document_key`, `split_key` |
@@ -26,20 +27,22 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru
| Module | For | Main functions |
| --- | --- | --- |
| `model` | The node tree of `scienting-doc/1`. | `Doc`, `anchor_class`, the tables of allowed children and attributes |
| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` |
| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve` |
| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `establish_with_side`, `flatten`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` |
| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve`, `resolve_address`, `follow` |
| `align` | Alignment of two readings of one paper. | `align`, `classify`, `similarity` |
| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp` |
| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp`, `published_rules` |
| `targets` | The target set of targeted F3. | `target_set` |
## Packages, corrections and the database
| Module | For | Main functions |
| --- | --- | --- |
| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `read_state`, `write`, `write_zip`, `read_zip`, `read_json`, `read_jsonl` |
| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State` |
| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `check_forms`, `read_state`, `write`, `write_zip`, `read_zip`, `read_zip_files`, `check_zip_entries`, `read_json`, `read_jsonl`, `parse_json`, `parse_jsonl` |
| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State`, `OPS`, `KEEPS_LEVEL` |
| `revision` | Applying corrections to a package and writing the package they make. | `revise`, `revise_zip`, `revise_files`, `summarize`, `level_after`, `Refused`, `Invalid` |
| `datasets` | A package's datasets, checked against its text. | `problems`, `content_hash`, `folders` |
| `schema` | JSON Schema validation with the keyword subset the schemas are restricted to. | `Store`, `validate`, `lint` |
| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `index` |
| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `read_state`, `delete`, `load_layer`, `upgrade`, `index` |
## Signatures, receipts and the log
@@ -56,6 +59,8 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru
| Module | For | Main functions |
| --- | --- | --- |
| `rights` | Rights records, policies and stage checks. | `effective`, `combine`, `check`, `backend_allowed` |
| `layers` | What every annotation shares: its record ID, its anchors and the lists its fields take. | `annotation_id`, `anchors_of`, `BODY_KINDS`, `CATEGORIES`, `VERDICTS`, `STATUSES` |
| `claims` | References to paperclaim claims and their pins. | `parse_ref`, `parse_pin`, `format_ref`, `format_pin` |
| `traces` | Traced citations. | `resolve`, `problems`, `disputed` |
| `structures` | Chemical structures tied to words. | `scan`, `problems` |
| `sequences` | Biological sequences tied to words. | `residues`, `problems` |
@@ -64,6 +69,32 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru
The `problems` functions of the four annotation modules return the rules an annotation breaks, as a list of `{"rule", "message"}` records in the order checked.
## Correcting a package
`revision.revise(root, proposals, store)` validates the package folder at `root`, runs each proposed correction through the gate in order, and writes the corrected package in place, or to `out` when it is given; `revision.revise_zip` does the same for a zipped package.
Nothing is written unless every proposal passes and the package written validates.
The writer recomputes every node's values and `document.json`, and records the corrections, the retired IDs and the resolved references.
A leaf a correction changed is F1 when the correction's evidence includes a crop of the page and F0 otherwise, and is no longer corroborated; an `emend`, `relink` or `resolve` keeps its level ([spec/08 §8.1](https://paperspec.scient.ing/spec/08-fidelity.html#81-levels-of-fidelity)).
A merged node's boxes go to the node it merged into, suspects at removed nodes and those named in `answered` leave `suspects.jsonl`, and the lines of `derived/` files and the datasets that read corrected text are left out and reported, since only the tools that made them can rebuild them ([spec/12 §12.7](https://paperspec.scient.ing/spec/12-package.html#127-derived-files)).
With no proposals, the package written is the package read, byte for byte.
```python
from paperspec import resources, revision
from paperspec.schema import Store
result = revision.revise("paper/", [proposal], Store(resources.path("schemas")))
result["rev"], result["package"], result["dropped"]
```
A proposal the gate refuses raises `revision.Refused`, with the proposal's `index` and the `rule` it breaks, and a package that does not validate raises `revision.Invalid` with its errors.
`revision.revise_files` works on a package held in memory, as `{path: bytes}`, and `revision.summarize` recomputes what `document.json` derives from the nodes, for a producer that writes packages itself.
## Math forms
`package.validate` reports `doc.math.form` for a source with `form` that is not the form's own spelling ([spec/03 §3.7](https://paperspec.scient.ing/spec/03-nodes.html#37-math-forms)).
Without help it knows only what `scienting-math/1` never holds, a `\tag`, `\label`, `\nonumber`, `\notag`, style command, `\dfrac` or `\tfrac`; with papermath installed it spells every source again and requires it back unchanged.
Pass `forms={}` to check only the first, or a mapping from a form to a function that spells a source in it.
## The command
`paperspec.cli` holds the subcommands of the `paperspec` command, one module each, with a `run(args)` function that returns the exit status; see [Command line](cli.md).
+22 -1
View File
@@ -1,12 +1,13 @@
# Command line
One command, `paperspec`, with six subcommands.
One command, `paperspec`, with seven subcommands.
`python -m paperspec` runs the same program.
`paperspec --version` prints the package version and the Unicode version it pins, and `paperspec COMMAND --help` prints a command's options, advice and examples.
| Command | Works | Does |
| --- | --- | --- |
| [`validate`](#validate) | anywhere the package is installed with its `tools` extra | Validate packages, or run proposed corrections through the gate. |
| [`correct`](#correct) | anywhere the package is installed with its `tools` extra | Apply proposed corrections to a package and write the corrected package. |
| [`load`](#load) | anywhere the package is installed with its `tools` extra | Validate packages and load them into a SQLite database. |
| [`check`](#check) | in a checkout | Check the whole repository. |
| [`build`](#build) | in a checkout | Rebuild `fixtures/` and `vectors/` from their sources. |
@@ -31,6 +32,26 @@ paperspec validate [-h] [--correction PROPOSALS] [--max-size BYTES] PACKAGE [PAC
Each failure is printed to standard error as `package: where: rule: message`.
Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.
## correct
```text
paperspec correct [-h] --correction PROPOSALS [--out PATH] [--answered SUSPECT] [--asset FILE] [--max-size BYTES] PACKAGE
```
| Option | Meaning |
| --- | --- |
| `PACKAGE` | A package folder or a zipped package. It is validated first. |
| `--correction PROPOSALS` | A JSON file holding one proposed correction, or a JSON Lines file of several, applied in order. Nothing is written unless every one passes the gate. |
| `--out PATH` | Write the corrected package here, a folder, or a zip for a zipped package, which must not exist. By default `PACKAGE` is replaced. |
| `--answered SUSPECT` | The ID of a suspect the corrections answer, which leaves `suspects.jsonl`. Repeat for several. |
| `--asset FILE` | A file to add under `assets/` as `<sha256>.<extension>`, such as a crop a correction's evidence names. Repeat for several. |
| `--max-size BYTES` | As for `validate`. |
The corrected package is validated before it replaces anything.
Each record appended to `corrections.jsonl` is printed, one per line; standard error names the derived lines and datasets left out because they read corrected text, and ends with the document's new revision and package hash.
A refusal is printed as `validate` prints it, and nothing is written.
Exit status: 0 when the corrections were applied, 1 when the package or a proposal is refused, 2 for a usage error.
## load
```text