Add paperspec correct, check math forms and datasets, and give the tools public names

The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
This commit is contained in:
alexmandClaude committed 2026-10-20 22:10:17 -04:00
1 parent 8c07dd7c15
commit 4fb8b4f5a7
70 files changed
+4470 -270

No files matched your search

+1 -1
View File
@@ -428,7 +428,7 @@ Keep using it as a reading and distribution format.
*Status.*
The specification is version 0.1.0.
`registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
The repository holds 24 chapters, 43 JSON Schemas, 29 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`.
The repository holds 24 chapters, 43 JSON Schemas, 29 vector files, five valid fixture packages, one of them also zipped, and 33 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`.
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; the code in `src/` Apache-2.0 ([README.md](../README.md#licenses)).
*No adoption claim.*
+1 -1
View File
@@ -107,7 +107,7 @@ A `candidate` format is complete, with vectors and fixtures, and ready to implem
A `provisional` one is complete and expected to change as implementations gain experience.
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint.
The repository holds a reference checker (`paperspec check`), 29 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
The repository holds a reference checker (`paperspec check`), 29 vector files, five valid packages, one of them also as a zip, and 33 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
## What does it not do?
+1 -1
View File
@@ -57,7 +57,7 @@ It appears inside hashed data and test vectors, and it is separate from the repo
| [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. |
| [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. |
| [vectors/](../vectors/README.md) | 29 files of test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 33 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
| [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. |
| [docs/](README.md) | Guides, including this page. |
+13
View File
@@ -41,3 +41,16 @@ paperspec validate --correction fix.json fixtures/packages/tiny2026tiny-j4qm
The command prints the record the correction would append to `corrections.jsonl`, with its computed `index`, `id`, `path`, `before` and `after`, and writes nothing.
A refused proposal prints `file:N node: rule: message` on standard error and stops: with this proposal, changing `second` to `2nd` is refused with `correction.digits`, because an emendation changes no digit.
`--correction` takes exactly one package, and it refuses a package that withholds its text.
## Apply the correction
When the proposal passes, `paperspec correct` applies it and writes the corrected package:
```sh
paperspec correct --correction fix.json --out fixed/tiny2026tiny-j4qm fixtures/packages/tiny2026tiny-j4qm
```
It runs the same gate, then rewrites every file the correction touches: the nodes and their values, `corrections.jsonl`, `ids.jsonl`, `references.jsonl`, `locations.jsonl`, `suspects.jsonl`, `document.json` and the manifest.
The corrected package is validated before it is written; without `--out` it replaces the package given.
The printed record is the one `validate --correction` printed.
A leaf the correction changed drops to F1 when the evidence includes a crop of the page, and to F0 otherwise; an emendation like this one keeps the leaf's level.
+41 -10
View File
@@ -9,15 +9,16 @@ Two aliases are shared by the type annotations: `jsontypes.Record`, one parsed J
A function that takes a path returns text unless its docstring says otherwise.
A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the rule as the specification's conformance chapter lists it.
`paperspec.resources.path(name)` returns the folder `schemas`, `sql` or `vectors`, in an installed wheel and in a checkout alike.
`paperspec.resources.path(name)` returns the folder `schemas`, `sql`, `vectors`, `registry` or `context`, in an installed wheel and in a checkout alike, and `fixtures` in a checkout or an editable install of one.
`paperspec.resources.registry(name)` reads a registry file by its path under `registry/`, such as `rules.json` or `stamps/scienting-stamps-1.json`.
## Text, hashes and canonical JSON
| Module | For | Main functions |
| --- | --- | --- |
| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` |
| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads_line` |
| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `lossy`, `own_content`, `meta_hash`, `Hasher`, `Index` |
| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `digits`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` |
| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads`, `loads_line`, `parse_int` |
| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `strict_runs`, `lossy`, `lossy_from_strict`, `own_content`, `meta_hash`, `Hasher`, `Index` |
| `notation` | Simple math notation and its canonical linear form. | `fold_math`, `complex_key`, `fold_plain_text` |
| `keys` | Work and document keys. | `work_key`, `unique_work_key`, `document_key`, `split_key` |
@@ -26,20 +27,22 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru
| Module | For | Main functions |
| --- | --- | --- |
| `model` | The node tree of `scienting-doc/1`. | `Doc`, `anchor_class`, the tables of allowed children and attributes |
| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` |
| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve` |
| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `establish_with_side`, `flatten`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` |
| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve`, `resolve_address`, `follow` |
| `align` | Alignment of two readings of one paper. | `align`, `classify`, `similarity` |
| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp` |
| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp`, `published_rules` |
| `targets` | The target set of targeted F3. | `target_set` |
## Packages, corrections and the database
| Module | For | Main functions |
| --- | --- | --- |
| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `read_state`, `write`, `write_zip`, `read_zip`, `read_json`, `read_jsonl` |
| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State` |
| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `check_forms`, `read_state`, `write`, `write_zip`, `read_zip`, `read_zip_files`, `check_zip_entries`, `read_json`, `read_jsonl`, `parse_json`, `parse_jsonl` |
| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State`, `OPS`, `KEEPS_LEVEL` |
| `revision` | Applying corrections to a package and writing the package they make. | `revise`, `revise_zip`, `revise_files`, `summarize`, `level_after`, `Refused`, `Invalid` |
| `datasets` | A package's datasets, checked against its text. | `problems`, `content_hash`, `folders` |
| `schema` | JSON Schema validation with the keyword subset the schemas are restricted to. | `Store`, `validate`, `lint` |
| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `index` |
| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `read_state`, `delete`, `load_layer`, `upgrade`, `index` |
## Signatures, receipts and the log
@@ -56,6 +59,8 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru
| Module | For | Main functions |
| --- | --- | --- |
| `rights` | Rights records, policies and stage checks. | `effective`, `combine`, `check`, `backend_allowed` |
| `layers` | What every annotation shares: its record ID, its anchors and the lists its fields take. | `annotation_id`, `anchors_of`, `BODY_KINDS`, `CATEGORIES`, `VERDICTS`, `STATUSES` |
| `claims` | References to paperclaim claims and their pins. | `parse_ref`, `parse_pin`, `format_ref`, `format_pin` |
| `traces` | Traced citations. | `resolve`, `problems`, `disputed` |
| `structures` | Chemical structures tied to words. | `scan`, `problems` |
| `sequences` | Biological sequences tied to words. | `residues`, `problems` |
@@ -64,6 +69,32 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru
The `problems` functions of the four annotation modules return the rules an annotation breaks, as a list of `{"rule", "message"}` records in the order checked.
## Correcting a package
`revision.revise(root, proposals, store)` validates the package folder at `root`, runs each proposed correction through the gate in order, and writes the corrected package in place, or to `out` when it is given; `revision.revise_zip` does the same for a zipped package.
Nothing is written unless every proposal passes and the package written validates.
The writer recomputes every node's values and `document.json`, and records the corrections, the retired IDs and the resolved references.
A leaf a correction changed is F1 when the correction's evidence includes a crop of the page and F0 otherwise, and is no longer corroborated; an `emend`, `relink` or `resolve` keeps its level ([spec/08 §8.1](https://paperspec.scient.ing/spec/08-fidelity.html#81-levels-of-fidelity)).
A merged node's boxes go to the node it merged into, suspects at removed nodes and those named in `answered` leave `suspects.jsonl`, and the lines of `derived/` files and the datasets that read corrected text are left out and reported, since only the tools that made them can rebuild them ([spec/12 §12.7](https://paperspec.scient.ing/spec/12-package.html#127-derived-files)).
With no proposals, the package written is the package read, byte for byte.
```python
from paperspec import resources, revision
from paperspec.schema import Store
result = revision.revise("paper/", [proposal], Store(resources.path("schemas")))
result["rev"], result["package"], result["dropped"]
```
A proposal the gate refuses raises `revision.Refused`, with the proposal's `index` and the `rule` it breaks, and a package that does not validate raises `revision.Invalid` with its errors.
`revision.revise_files` works on a package held in memory, as `{path: bytes}`, and `revision.summarize` recomputes what `document.json` derives from the nodes, for a producer that writes packages itself.
## Math forms
`package.validate` reports `doc.math.form` for a source with `form` that is not the form's own spelling ([spec/03 §3.7](https://paperspec.scient.ing/spec/03-nodes.html#37-math-forms)).
Without help it knows only what `scienting-math/1` never holds, a `\tag`, `\label`, `\nonumber`, `\notag`, style command, `\dfrac` or `\tfrac`; with papermath installed it spells every source again and requires it back unchanged.
Pass `forms={}` to check only the first, or a mapping from a form to a function that spells a source in it.
## The command
`paperspec.cli` holds the subcommands of the `paperspec` command, one module each, with a `run(args)` function that returns the exit status; see [Command line](cli.md).
+22 -1
View File
@@ -1,12 +1,13 @@
# Command line
One command, `paperspec`, with six subcommands.
One command, `paperspec`, with seven subcommands.
`python -m paperspec` runs the same program.
`paperspec --version` prints the package version and the Unicode version it pins, and `paperspec COMMAND --help` prints a command's options, advice and examples.
| Command | Works | Does |
| --- | --- | --- |
| [`validate`](#validate) | anywhere the package is installed with its `tools` extra | Validate packages, or run proposed corrections through the gate. |
| [`correct`](#correct) | anywhere the package is installed with its `tools` extra | Apply proposed corrections to a package and write the corrected package. |
| [`load`](#load) | anywhere the package is installed with its `tools` extra | Validate packages and load them into a SQLite database. |
| [`check`](#check) | in a checkout | Check the whole repository. |
| [`build`](#build) | in a checkout | Rebuild `fixtures/` and `vectors/` from their sources. |
@@ -31,6 +32,26 @@ paperspec validate [-h] [--correction PROPOSALS] [--max-size BYTES] PACKAGE [PAC
Each failure is printed to standard error as `package: where: rule: message`.
Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.
## correct
```text
paperspec correct [-h] --correction PROPOSALS [--out PATH] [--answered SUSPECT] [--asset FILE] [--max-size BYTES] PACKAGE
```
| Option | Meaning |
| --- | --- |
| `PACKAGE` | A package folder or a zipped package. It is validated first. |
| `--correction PROPOSALS` | A JSON file holding one proposed correction, or a JSON Lines file of several, applied in order. Nothing is written unless every one passes the gate. |
| `--out PATH` | Write the corrected package here, a folder, or a zip for a zipped package, which must not exist. By default `PACKAGE` is replaced. |
| `--answered SUSPECT` | The ID of a suspect the corrections answer, which leaves `suspects.jsonl`. Repeat for several. |
| `--asset FILE` | A file to add under `assets/` as `<sha256>.<extension>`, such as a crop a correction's evidence names. Repeat for several. |
| `--max-size BYTES` | As for `validate`. |
The corrected package is validated before it replaces anything.
Each record appended to `corrections.jsonl` is printed, one per line; standard error names the derived lines and datasets left out because they read corrected text, and ends with the document's new revision and package hash.
A refusal is printed as `validate` prints it, and nothing is written.
Exit status: 0 when the corrections were applied, 1 when the package or a proposal is refused, 2 for a usage error.
## load
```text