Files
paperspec/docs/package/reference/api.md
T
alexmandClaude 4fb8b4f5a7 Add paperspec correct, check math forms and datasets, and give the tools public names
The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
2026-10-20 22:10:17 -04:00

102 lines
8.2 KiB
Markdown

# Python API
The package `paperspec` is a set of small modules, one for each part of the specification, and each opens with a docstring that says what it is for.
This page is written by hand from those docstrings and lists the modules and the functions most programs call; `help(paperspec.MODULE)` has the rest.
The base package needs only the standard library.
`schema` needs `paperspec[schema]`, and the Ed25519 and HKDF functions of `crypto` need `paperspec[crypto]`; both import their packages on first use.
Two aliases are shared by the type annotations: `jsontypes.Record`, one parsed JSON object, and `jsontypes.StrPath`, a path given as text or as a path object.
A function that takes a path returns text unless its docstring says otherwise.
A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the rule as the specification's conformance chapter lists it.
`paperspec.resources.path(name)` returns the folder `schemas`, `sql`, `vectors`, `registry` or `context`, in an installed wheel and in a checkout alike, and `fixtures` in a checkout or an editable install of one.
`paperspec.resources.registry(name)` reads a registry file by its path under `registry/`, such as `rules.json` or `stamps/scienting-stamps-1.json`.
## Text, hashes and canonical JSON
| Module | For | Main functions |
| --- | --- | --- |
| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `digits`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` |
| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads`, `loads_line`, `parse_int` |
| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `strict_runs`, `lossy`, `lossy_from_strict`, `own_content`, `meta_hash`, `Hasher`, `Index` |
| `notation` | Simple math notation and its canonical linear form. | `fold_math`, `complex_key`, `fold_plain_text` |
| `keys` | Work and document keys. | `work_key`, `unique_work_key`, `document_key`, `split_key` |
## The document
| Module | For | Main functions |
| --- | --- | --- |
| `model` | The node tree of `scienting-doc/1`. | `Doc`, `anchor_class`, the tables of allowed children and attributes |
| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `establish_with_side`, `flatten`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` |
| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve`, `resolve_address`, `follow` |
| `align` | Alignment of two readings of one paper. | `align`, `classify`, `similarity` |
| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp`, `published_rules` |
| `targets` | The target set of targeted F3. | `target_set` |
## Packages, corrections and the database
| Module | For | Main functions |
| --- | --- | --- |
| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `check_forms`, `read_state`, `write`, `write_zip`, `read_zip`, `read_zip_files`, `check_zip_entries`, `read_json`, `read_jsonl`, `parse_json`, `parse_jsonl` |
| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State`, `OPS`, `KEEPS_LEVEL` |
| `revision` | Applying corrections to a package and writing the package they make. | `revise`, `revise_zip`, `revise_files`, `summarize`, `level_after`, `Refused`, `Invalid` |
| `datasets` | A package's datasets, checked against its text. | `problems`, `content_hash`, `folders` |
| `schema` | JSON Schema validation with the keyword subset the schemas are restricted to. | `Store`, `validate`, `lint` |
| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `read_state`, `delete`, `load_layer`, `upgrade`, `index` |
## Signatures, receipts and the log
| Module | For | Main functions |
| --- | --- | --- |
| `crypto` | Keyed hashing, Merkle roots and proofs, Ed25519, HKDF and `did:key`. | `merkle_root`, `inclusion_path`, `verify_inclusion`, `consistency_proof`, `verify_consistency`, `ed25519_sign`, `ed25519_verify`, `did_key` |
| `receipts` | Receipts, revocations, levels and spot-check sampling. | `seal`, `verify_signature`, `statuses`, `effective_levels`, `sample`, `upper_bound` |
| `log` | The team's transparency log and its signed checkpoints. | `sign_checkpoint`, `verify_checkpoint`, `write_log`, `read_log` |
| `vrf` | ECVRF-EDWARDS25519-SHA512-TAI, RFC 9381. | `prove`, `verify`, `proof_to_hash` |
| `mldsa` | ML-DSA-65, FIPS 204, in plain Python. | `keygen`, `sign`, `verify` |
## Layers, rights and grading
| Module | For | Main functions |
| --- | --- | --- |
| `rights` | Rights records, policies and stage checks. | `effective`, `combine`, `check`, `backend_allowed` |
| `layers` | What every annotation shares: its record ID, its anchors and the lists its fields take. | `annotation_id`, `anchors_of`, `BODY_KINDS`, `CATEGORIES`, `VERDICTS`, `STATUSES` |
| `claims` | References to paperclaim claims and their pins. | `parse_ref`, `parse_pin`, `format_ref`, `format_pin` |
| `traces` | Traced citations. | `resolve`, `problems`, `disputed` |
| `structures` | Chemical structures tied to words. | `scan`, `problems` |
| `sequences` | Biological sequences tied to words. | `residues`, `problems` |
| `data` | Data tied to words. | `problems` |
| `grade` | Grading a submission against an exercise key. | `grade`, `key_anchors` |
The `problems` functions of the four annotation modules return the rules an annotation breaks, as a list of `{"rule", "message"}` records in the order checked.
## Correcting a package
`revision.revise(root, proposals, store)` validates the package folder at `root`, runs each proposed correction through the gate in order, and writes the corrected package in place, or to `out` when it is given; `revision.revise_zip` does the same for a zipped package.
Nothing is written unless every proposal passes and the package written validates.
The writer recomputes every node's values and `document.json`, and records the corrections, the retired IDs and the resolved references.
A leaf a correction changed is F1 when the correction's evidence includes a crop of the page and F0 otherwise, and is no longer corroborated; an `emend`, `relink` or `resolve` keeps its level ([spec/08 §8.1](https://paperspec.scient.ing/spec/08-fidelity.html#81-levels-of-fidelity)).
A merged node's boxes go to the node it merged into, suspects at removed nodes and those named in `answered` leave `suspects.jsonl`, and the lines of `derived/` files and the datasets that read corrected text are left out and reported, since only the tools that made them can rebuild them ([spec/12 §12.7](https://paperspec.scient.ing/spec/12-package.html#127-derived-files)).
With no proposals, the package written is the package read, byte for byte.
```python
from paperspec import resources, revision
from paperspec.schema import Store
result = revision.revise("paper/", [proposal], Store(resources.path("schemas")))
result["rev"], result["package"], result["dropped"]
```
A proposal the gate refuses raises `revision.Refused`, with the proposal's `index` and the `rule` it breaks, and a package that does not validate raises `revision.Invalid` with its errors.
`revision.revise_files` works on a package held in memory, as `{path: bytes}`, and `revision.summarize` recomputes what `document.json` derives from the nodes, for a producer that writes packages itself.
## Math forms
`package.validate` reports `doc.math.form` for a source with `form` that is not the form's own spelling ([spec/03 §3.7](https://paperspec.scient.ing/spec/03-nodes.html#37-math-forms)).
Without help it knows only what `scienting-math/1` never holds, a `\tag`, `\label`, `\nonumber`, `\notag`, style command, `\dfrac` or `\tfrac`; with papermath installed it spells every source again and requires it back unchanged.
Pass `forms={}` to check only the first, or a mapping from a form to a function that spells a source in it.
## The command
`paperspec.cli` holds the subcommands of the `paperspec` command, one module each, with a `run(args)` function that returns the exit status; see [Command line](cli.md).
`cli.checkout()` finds the repository root that `check`, `build`, `stress` and `site` work on.