Add paperspec correct, check math forms and datasets, and give the tools public names

The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
This commit is contained in:
alexmandClaude committed 2026-10-20 22:10:17 -04:00
1 parent 8c07dd7c15
commit 4fb8b4f5a7
70 files changed
+4470 -270

No files matched your search

+23 -4
View File
@@ -11,7 +11,7 @@ The first release.
- Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search.
- The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements.
- Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3.
- Example packages (a converted chemistry paper with corrections and receipts, its checks-only export, an open-access paper, a Typst draft and a tiny document, also as a zip), a record of every other format, and 31 broken packages, each breaking one named rule.
- Example packages (a converted chemistry paper with corrections and receipts, its checks-only export, an open-access paper, a Typst draft and a tiny document, also as a zip), a record of every other format, and 33 broken packages, each breaking one named rule.
- Work keys minted from a Scholar-style citation key and a hash of the work's identity, with document keys for versions; `tag:` URIs for layers, annotations and documents.
- Spot-check seeds as VRF proofs (RFC 9381), RFC 9162 inclusion and consistency proofs, a history root over every document's corrections, and a provisional transparency log with signed checkpoints.
- Signed QA reports, a conformance checklist covered by a receipt, and a `description` annotation for the figure descriptions an accessible export needs.
@@ -45,7 +45,7 @@ Both are minor changes in the sense of [spec/00](spec/00-introduction.md): no va
A sentence one reading misread is now one `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one, so a producer re-aligning the same two readings can get different suspects than before; the pairs and suspects of readings with no such sentence are unchanged, as the first alignment vector is.
`paperspec.align` gains `lcs_length` (bit-parallel, Hyyrö 2004) and `similarity`, and `alignment.jsonl` a second case, `substitutions`: a misread number, a misread word, a sentence too short to be paired this way and two sentences too unlike to be one.
- A suspect may name the witness its second reading came from (`witness`, a witness ID; [spec/05 §5.7](spec/05-hashes.md#57-suspects), `suspect.schema.json`), and a package whose suspect names a witness it does not have breaks `doc.suspect.witness`.
No broken package exercises the rule yet: adding one changes the count of broken packages that the 0.1.0 entry above states, so it waits for the release.
`fixtures/invalid/suspect-witness/` exercises the rule (see the last section).
Four more, asked for by editors who correct and discuss papers rather than readings.
They are minor changes too: new operations, enumeration values and optional properties, with no valid file made invalid, no value of an existing file computed differently, and no existing vector case or package changed.
@@ -70,7 +70,7 @@ The last two make some files that were valid invalid, which [spec/00](spec/00-in
A consumer may rely on the source being canonical and idempotent, carrying no label, and on everything else being derived from it.
`form` enters no content value ([spec/05 §5.2](spec/05-hashes.md#52-own-content)): own content leaves it out of math runs (`paperspec.hashing`), an equation's own content never held more than its notation and source, and no projection reads it, so adding `form` to a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does.
Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first.
`hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator does not.
`hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator now does in part (see the last section).
- `derived/math.jsonl` ([spec/12 §12.7](spec/12-package.md#127-derived-files), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `math.schema.json`): one line per formula, `{node, run, tool, form, tree, symbols, complete, issues, label?}`, papermath's reading without the canonical TeX, which is the source; `run` is null for an equation, as in `derived/mathml.jsonl`, and `label` is omitted when the source has no `\tag`.
A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 49 kinds and six codes of version 1.
The validator now checks derived files: each line against its schema, that it names a formula of the package (`doc.derived.node`), that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`), and that a checks-only package carries none (`doc.policy.withheld`), since a tree or MathML reads the withheld text.
@@ -103,7 +103,7 @@ It is a minor change in the sense of [spec/00](spec/00-introduction.md): a new b
`paperspec check` runs both layers through the record checks and the store.
The example project holds the new layer, which changes `project.json`; no other fixture changes.
One more, asked for by paperchem and a sequence tool to come, which tie molecules and sequences to the words of a paper.
One more, asked for by paperchem and paperbio, which tie molecules and sequences to the words of a paper.
It is a minor change in the sense of [spec/00](spec/00-introduction.md): two new body kinds, optional properties and seven rules, with no valid file made invalid and no computed value changed; the example layers, the traces and every receipt are as they were.
- Chemical structures ([spec/18 §18.6](spec/18-layers.md#186-chemical-structures), `annotation.schema.json`): a `structure` body ties a SMILES or SMARTS to a span, with a `role` (`names`, `depicts`, `parametrizes` or `uses`), an `identity` that finds the same structure however it was written (the InChIKey of a SMILES, else `smiles:` or `smarts:` and twelve hex digits of the SHA-256 of the value), the paper's own `written` notation, a `label`, outside `accessions` and `terms`: the bonds, angles, torsions and impropers a paper gives constants for.
@@ -153,3 +153,22 @@ Tooling and code quality only; no computed value, vector, fixture, schema or SQL
- The publish workflow also creates the Gitea release, with the files attached and the notes taken from this changelog; CI runs Scienting's checks (`.gitea/workflows/ci.yml`) on `main`, `dev` and pull requests.
- Python files open with an `SPDX-License-Identifier: Apache-2.0` header, which ruff and a test check.
- The checker and the site builder skip `work/`, caches and build output.
One more, the work the other paper tools planned for paperspec (paperdesk's plan, item M5.4, and paperform's, §4.2): a command that applies corrections, two checks the validator had left to producers, and public names for what the tools had been importing as private ones.
It is a minor change in the sense of [spec/00](spec/00-introduction.md): new rules, functions and a command, with no fixture package made invalid and no computed value changed.
- `paperspec correct` and `paperspec.revision` apply proposed corrections to a package, a folder or a zip, through the gate ([spec/11](spec/11-corrections.md)), and write the package they make, in place or to `--out`, only once it validates.
A changed leaf becomes F1 when a crop of it is the evidence and F0 otherwise ([spec/08 §8.1](spec/08-fidelity.md#81-levels-of-fidelity)), and an `emend`, `relink` or `resolve` keeps its level.
A retired node's locations move to its successor, suspects at removed nodes and those named by `--answered` are dropped, derived lines and datasets that read the corrected text are left out and reported, and provenance gains a `review` stage.
With no corrections it writes every example package and every valid package of paperform's bench back byte for byte (`test_revision.py`, `test_cli_correct.py`).
- `doc.math.form` is checked ([spec/03 §3.7](spec/03-nodes.md#37-math-forms)).
The validator refuses a source in the form that holds a label or style command the form never holds, such as `\tag`, `\label`, `\nonumber`, `\displaystyle` or `\dfrac`.
When papermath is installed, or a caller passes `forms=`, it also refuses a source that is not the form's own spelling or that the form cannot read (`test_math_form.py`).
- The validator checks a package's datasets ([spec/15](spec/15-datasets.md), `paperspec.datasets`): the descriptor against its schema, the folder named by the ID, a unique primary key, each value anchor resolving as current to the text the CSV holds, and every numeric value of a reported resource anchored.
These are the rules `doc.dataset.schema`, `.id`, `.file`, `.key`, `.anchor`, `.value` and `.unanchored` (`test_datasets.py`).
- `fixtures/invalid/` gains `suspect-witness` and `math-form-label`, so there are 33 broken packages, which the 0.1.0 entry above states too (`paperspec check`).
- Canonical JSON reads numbers as doubles, so `1e20` is read back as it was written (`test_jcs.py`).
- The registries and the JSON-LD context ship in the wheel as `paperspec.registry` and `paperspec.context`, with `paperspec.resources.registry` and `paperspec.fingerprint.published_rules` for the download-stamp rules (`test_resources.py`, `test_fingerprint.py`).
- Public names, with the old private names kept as aliases: `package.parse_canonical`, `parse_json`, `parse_jsonl`, `check_zip_entries` and `read_zip_files`; `tree.flatten` and `establish_with_side`, which keeps a producer's private keys; `hashing.strict_runs`; and `corrections.OPS` and `EMEND_KINDS` (`test_public_api.py`).
Also `anchors.follow` and `resolve_address`, `claims.parse_ref`, `parse_pin`, `format_ref` and `format_pin`, `layers.annotation_id`, `anchors_of` and the lists of body kinds, categories, verdicts and statuses, and `db.load_layer`, `read_state`, `delete` and `upgrade` (`test_anchors.py`, `test_claims.py`, `test_layers.py`, `test_db.py`).
- `text.collapse_ws`, `trim`, `remove_ws` and `digits` and `hashing.lossy_from_strict` are rewritten for speed, so a correction goes through the gate about three times faster, and are tested equal to the old code on every code point (`test_text.py`, `test_hashing.py`).