From 4fb8b4f5a7735d86161af2b516b22f47d588a19a Mon Sep 17 00:00:00 2001 From: Alex Maldonado Date: Tue, 20 Oct 2026 22:10:17 -0400 Subject: [PATCH] Add paperspec correct, check math forms and datasets, and give the tools public names The work the other paper tools planned for paperspec (paperdesk PLAN M5.4, paperform PLAN 4.2): - paperspec correct and paperspec.revision apply proposed corrections to a package, folder or zip, and write it only once it validates. - doc.math.form is checked: label and style commands always, the form's own spelling when papermath is installed or forms= is passed. - paperspec.datasets and seven doc.dataset.* rules check a package's datasets. - fixtures/invalid gains suspect-witness and math-form-label (33 in all). - Canonical JSON reads numbers as doubles, so 1e20 round-trips. - The registries and the JSON-LD context ship in the wheel. - Public names for what paperdesk and paperform imported as private ones, with the old names kept as aliases. - The text and lossy-projection functions are about three times faster through the gate, tested equal to the old code on every code point. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E --- CHANGELOG.md | 27 +- README.md | 6 +- docs/comparison.md | 2 +- docs/faq.md | 2 +- docs/overview.md | 2 +- docs/package/guides/validate-a-package.md | 13 + docs/package/reference/api.md | 51 +- docs/package/reference/cli.md | 23 +- fixtures/README.md | 16 +- fixtures/invalid/math-form-label/expect.json | 1 + .../math-form-label/package/MANIFEST.json | 1 + .../math-form-label/package/document.json | 1 + .../math-form-label/package/nodes.jsonl | 22 + .../math-form-label/package/provenance.json | 1 + .../math-form-label/package/receipts.jsonl | 1 + .../math-form-label/package/references.jsonl | 1 + .../math-form-label/package/roster.json | 1 + .../math-form-label/package/witnesses.json | 1 + fixtures/invalid/suspect-witness/expect.json | 1 + .../suspect-witness/package/MANIFEST.json | 1 + .../suspect-witness/package/document.json | 1 + .../suspect-witness/package/nodes.jsonl | 22 + .../suspect-witness/package/provenance.json | 1 + .../suspect-witness/package/receipts.jsonl | 1 + .../suspect-witness/package/references.jsonl | 1 + .../suspect-witness/package/roster.json | 1 + .../suspect-witness/package/suspects.jsonl | 1 + .../suspect-witness/package/witnesses.json | 1 + pyproject.toml | 26 +- registry/rules.json | 37 +- spec/03-nodes.md | 1 + spec/08-fidelity.md | 3 +- spec/15-datasets.md | 6 + spec/22-conformance.md | 1 + src/paperspec/anchors.py | 94 ++- src/paperspec/claims.py | 102 ++++ src/paperspec/cli/__init__.py | 9 +- src/paperspec/cli/build.py | 66 +++ src/paperspec/cli/check.py | 40 +- src/paperspec/cli/correct.py | 108 ++++ src/paperspec/cli/main.py | 83 ++- src/paperspec/corrections.py | 17 + src/paperspec/datasets.py | 180 ++++++ src/paperspec/db.py | 424 ++++++++++++-- src/paperspec/fingerprint.py | 15 +- src/paperspec/hashing.py | 85 ++- src/paperspec/jcs.py | 38 +- src/paperspec/layers.py | 88 +++ src/paperspec/package.py | 296 ++++++++-- src/paperspec/resources.py | 37 +- src/paperspec/revision.py | 554 ++++++++++++++++++ src/paperspec/text.py | 31 +- src/paperspec/traces.py | 18 +- src/paperspec/tree.py | 37 +- tests/test_anchors.py | 160 +++++ tests/test_claims.py | 203 +++++++ tests/test_cli_correct.py | 194 ++++++ tests/test_cli_main.py | 2 +- tests/test_datasets.py | 190 ++++++ tests/test_db.py | 375 ++++++++++++ tests/test_fingerprint.py | 5 + tests/test_hashing.py | 63 ++ tests/test_jcs.py | 31 + tests/test_layers.py | 143 +++++ tests/test_math_form.py | 172 ++++++ tests/test_public_api.py | 114 ++++ tests/test_resources.py | 52 +- tests/test_revision.py | 378 ++++++++++++ tests/test_text.py | 48 ++ tools/README.md | 11 +- 70 files changed, 4470 insertions(+), 270 deletions(-) create mode 100644 fixtures/invalid/math-form-label/expect.json create mode 100644 fixtures/invalid/math-form-label/package/MANIFEST.json create mode 100644 fixtures/invalid/math-form-label/package/document.json create mode 100644 fixtures/invalid/math-form-label/package/nodes.jsonl create mode 100644 fixtures/invalid/math-form-label/package/provenance.json create mode 100644 fixtures/invalid/math-form-label/package/receipts.jsonl create mode 100644 fixtures/invalid/math-form-label/package/references.jsonl create mode 100644 fixtures/invalid/math-form-label/package/roster.json create mode 100644 fixtures/invalid/math-form-label/package/witnesses.json create mode 100644 fixtures/invalid/suspect-witness/expect.json create mode 100644 fixtures/invalid/suspect-witness/package/MANIFEST.json create mode 100644 fixtures/invalid/suspect-witness/package/document.json create mode 100644 fixtures/invalid/suspect-witness/package/nodes.jsonl create mode 100644 fixtures/invalid/suspect-witness/package/provenance.json create mode 100644 fixtures/invalid/suspect-witness/package/receipts.jsonl create mode 100644 fixtures/invalid/suspect-witness/package/references.jsonl create mode 100644 fixtures/invalid/suspect-witness/package/roster.json create mode 100644 fixtures/invalid/suspect-witness/package/suspects.jsonl create mode 100644 fixtures/invalid/suspect-witness/package/witnesses.json create mode 100644 src/paperspec/claims.py create mode 100644 src/paperspec/cli/correct.py create mode 100644 src/paperspec/datasets.py create mode 100644 src/paperspec/layers.py create mode 100644 src/paperspec/revision.py create mode 100644 tests/test_claims.py create mode 100644 tests/test_cli_correct.py create mode 100644 tests/test_datasets.py create mode 100644 tests/test_layers.py create mode 100644 tests/test_math_form.py create mode 100644 tests/test_public_api.py create mode 100644 tests/test_revision.py diff --git a/CHANGELOG.md b/CHANGELOG.md index ae5dc14..f97450d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ The first release. - Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search. - The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements. - Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3. -- Example packages (a converted chemistry paper with corrections and receipts, its checks-only export, an open-access paper, a Typst draft and a tiny document, also as a zip), a record of every other format, and 31 broken packages, each breaking one named rule. +- Example packages (a converted chemistry paper with corrections and receipts, its checks-only export, an open-access paper, a Typst draft and a tiny document, also as a zip), a record of every other format, and 33 broken packages, each breaking one named rule. - Work keys minted from a Scholar-style citation key and a hash of the work's identity, with document keys for versions; `tag:` URIs for layers, annotations and documents. - Spot-check seeds as VRF proofs (RFC 9381), RFC 9162 inclusion and consistency proofs, a history root over every document's corrections, and a provisional transparency log with signed checkpoints. - Signed QA reports, a conformance checklist covered by a receipt, and a `description` annotation for the figure descriptions an accessible export needs. @@ -45,7 +45,7 @@ Both are minor changes in the sense of [spec/00](spec/00-introduction.md): no va A sentence one reading misread is now one `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one, so a producer re-aligning the same two readings can get different suspects than before; the pairs and suspects of readings with no such sentence are unchanged, as the first alignment vector is. `paperspec.align` gains `lcs_length` (bit-parallel, Hyyrö 2004) and `similarity`, and `alignment.jsonl` a second case, `substitutions`: a misread number, a misread word, a sentence too short to be paired this way and two sentences too unlike to be one. - A suspect may name the witness its second reading came from (`witness`, a witness ID; [spec/05 §5.7](spec/05-hashes.md#57-suspects), `suspect.schema.json`), and a package whose suspect names a witness it does not have breaks `doc.suspect.witness`. - No broken package exercises the rule yet: adding one changes the count of broken packages that the 0.1.0 entry above states, so it waits for the release. + `fixtures/invalid/suspect-witness/` exercises the rule (see the last section). Four more, asked for by editors who correct and discuss papers rather than readings. They are minor changes too: new operations, enumeration values and optional properties, with no valid file made invalid, no value of an existing file computed differently, and no existing vector case or package changed. @@ -70,7 +70,7 @@ The last two make some files that were valid invalid, which [spec/00](spec/00-in A consumer may rely on the source being canonical and idempotent, carrying no label, and on everything else being derived from it. `form` enters no content value ([spec/05 §5.2](spec/05-hashes.md#52-own-content)): own content leaves it out of math runs (`paperspec.hashing`), an equation's own content never held more than its notation and source, and no projection reads it, so adding `form` to a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does. Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first. - `hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator does not. + `hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator now does in part (see the last section). - `derived/math.jsonl` ([spec/12 §12.7](spec/12-package.md#127-derived-files), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `math.schema.json`): one line per formula, `{node, run, tool, form, tree, symbols, complete, issues, label?}`, papermath's reading without the canonical TeX, which is the source; `run` is null for an equation, as in `derived/mathml.jsonl`, and `label` is omitted when the source has no `\tag`. A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 49 kinds and six codes of version 1. The validator now checks derived files: each line against its schema, that it names a formula of the package (`doc.derived.node`), that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`), and that a checks-only package carries none (`doc.policy.withheld`), since a tree or MathML reads the withheld text. @@ -103,7 +103,7 @@ It is a minor change in the sense of [spec/00](spec/00-introduction.md): a new b `paperspec check` runs both layers through the record checks and the store. The example project holds the new layer, which changes `project.json`; no other fixture changes. -One more, asked for by paperchem and a sequence tool to come, which tie molecules and sequences to the words of a paper. +One more, asked for by paperchem and paperbio, which tie molecules and sequences to the words of a paper. It is a minor change in the sense of [spec/00](spec/00-introduction.md): two new body kinds, optional properties and seven rules, with no valid file made invalid and no computed value changed; the example layers, the traces and every receipt are as they were. - Chemical structures ([spec/18 §18.6](spec/18-layers.md#186-chemical-structures), `annotation.schema.json`): a `structure` body ties a SMILES or SMARTS to a span, with a `role` (`names`, `depicts`, `parametrizes` or `uses`), an `identity` that finds the same structure however it was written (the InChIKey of a SMILES, else `smiles:` or `smarts:` and twelve hex digits of the SHA-256 of the value), the paper's own `written` notation, a `label`, outside `accessions` and `terms`: the bonds, angles, torsions and impropers a paper gives constants for. @@ -153,3 +153,22 @@ Tooling and code quality only; no computed value, vector, fixture, schema or SQL - The publish workflow also creates the Gitea release, with the files attached and the notes taken from this changelog; CI runs Scienting's checks (`.gitea/workflows/ci.yml`) on `main`, `dev` and pull requests. - Python files open with an `SPDX-License-Identifier: Apache-2.0` header, which ruff and a test check. - The checker and the site builder skip `work/`, caches and build output. + +One more, the work the other paper tools planned for paperspec (paperdesk's plan, item M5.4, and paperform's, §4.2): a command that applies corrections, two checks the validator had left to producers, and public names for what the tools had been importing as private ones. +It is a minor change in the sense of [spec/00](spec/00-introduction.md): new rules, functions and a command, with no fixture package made invalid and no computed value changed. + +- `paperspec correct` and `paperspec.revision` apply proposed corrections to a package, a folder or a zip, through the gate ([spec/11](spec/11-corrections.md)), and write the package they make, in place or to `--out`, only once it validates. + A changed leaf becomes F1 when a crop of it is the evidence and F0 otherwise ([spec/08 §8.1](spec/08-fidelity.md#81-levels-of-fidelity)), and an `emend`, `relink` or `resolve` keeps its level. + A retired node's locations move to its successor, suspects at removed nodes and those named by `--answered` are dropped, derived lines and datasets that read the corrected text are left out and reported, and provenance gains a `review` stage. + With no corrections it writes every example package and every valid package of paperform's bench back byte for byte (`test_revision.py`, `test_cli_correct.py`). +- `doc.math.form` is checked ([spec/03 §3.7](spec/03-nodes.md#37-math-forms)). + The validator refuses a source in the form that holds a label or style command the form never holds, such as `\tag`, `\label`, `\nonumber`, `\displaystyle` or `\dfrac`. + When papermath is installed, or a caller passes `forms=`, it also refuses a source that is not the form's own spelling or that the form cannot read (`test_math_form.py`). +- The validator checks a package's datasets ([spec/15](spec/15-datasets.md), `paperspec.datasets`): the descriptor against its schema, the folder named by the ID, a unique primary key, each value anchor resolving as current to the text the CSV holds, and every numeric value of a reported resource anchored. + These are the rules `doc.dataset.schema`, `.id`, `.file`, `.key`, `.anchor`, `.value` and `.unanchored` (`test_datasets.py`). +- `fixtures/invalid/` gains `suspect-witness` and `math-form-label`, so there are 33 broken packages, which the 0.1.0 entry above states too (`paperspec check`). +- Canonical JSON reads numbers as doubles, so `1e20` is read back as it was written (`test_jcs.py`). +- The registries and the JSON-LD context ship in the wheel as `paperspec.registry` and `paperspec.context`, with `paperspec.resources.registry` and `paperspec.fingerprint.published_rules` for the download-stamp rules (`test_resources.py`, `test_fingerprint.py`). +- Public names, with the old private names kept as aliases: `package.parse_canonical`, `parse_json`, `parse_jsonl`, `check_zip_entries` and `read_zip_files`; `tree.flatten` and `establish_with_side`, which keeps a producer's private keys; `hashing.strict_runs`; and `corrections.OPS` and `EMEND_KINDS` (`test_public_api.py`). + Also `anchors.follow` and `resolve_address`, `claims.parse_ref`, `parse_pin`, `format_ref` and `format_pin`, `layers.annotation_id`, `anchors_of` and the lists of body kinds, categories, verdicts and statuses, and `db.load_layer`, `read_state`, `delete` and `upgrade` (`test_anchors.py`, `test_claims.py`, `test_layers.py`, `test_db.py`). +- `text.collapse_ws`, `trim`, `remove_ws` and `digits` and `hashing.lossy_from_strict` are rewritten for speed, so a correction goes through the gate about three times faster, and are tested equal to the old code on every code point (`test_text.py`, `test_hashing.py`). diff --git a/README.md b/README.md index 3fd2009..7a1f0cc 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,7 @@ pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb ``` [docs/examples.md](docs/examples.md) has queries to start from. +To correct a package, try the proposal with `paperspec validate --correction fix.json PACKAGE`, then apply it with `paperspec correct --correction fix.json PACKAGE`, which writes the corrected package. `paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)). `pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)). @@ -209,7 +210,10 @@ Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2]( The Python package is an oracle, not a production reader or writer: it favors clarity over speed, and its VRF and ML-DSA-65 are written out in plain Python ([tools/](tools/README.md#about-the-oracle)). It needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other. -Only `paperspec validate` and `paperspec load` work anywhere the package is installed; `check`, `build`, `stress` and `site` work on a checkout of this repository. +Only `paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is installed; `check`, `build`, `stress` and `site` work on a checkout of this repository. +`paperspec correct` cannot rebuild what pinned tools derived from the text: it leaves out the lines of `derived/` files and the datasets that read corrected text, and says which, for the tool that made them to rebuild. +The validator checks the math form `scienting-math/1` in full only when papermath is installed; without it, it refuses only what the form never holds, such as a `\tag`. +It accepts more than one location record for a node on one page, which [spec/07 §7.4](spec/07-geometry.md#74-locationsjsonl) rules out. ## Licenses diff --git a/docs/comparison.md b/docs/comparison.md index 67a6cee..4f51af2 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -428,7 +428,7 @@ Keep using it as a reading and distribution format. *Status.* The specification is version 0.1.0. `registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)). -The repository holds 24 chapters, 43 JSON Schemas, 29 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`. +The repository holds 24 chapters, 43 JSON Schemas, 29 vector files, five valid fixture packages, one of them also zipped, and 33 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`. Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; the code in `src/` Apache-2.0 ([README.md](../README.md#licenses)). *No adoption claim.* diff --git a/docs/faq.md b/docs/faq.md index f14409e..bbfa68d 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -107,7 +107,7 @@ A `candidate` format is complete, with vectors and fixtures, and ready to implem A `provisional` one is complete and expected to change as implementations gain experience. The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint. -The repository holds a reference checker (`paperspec check`), 29 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)). +The repository holds a reference checker (`paperspec check`), 29 vector files, five valid packages, one of them also as a zip, and 33 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)). ## What does it not do? diff --git a/docs/overview.md b/docs/overview.md index f355997..bc2863e 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -57,7 +57,7 @@ It appears inside hashed data and test vectors, and it is separate from the repo | [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. | | [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. | | [vectors/](../vectors/README.md) | 29 files of test vectors: inputs and the exact outputs every implementation must compute. | -| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. | +| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 33 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. | | [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. | | [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. | | [docs/](README.md) | Guides, including this page. | diff --git a/docs/package/guides/validate-a-package.md b/docs/package/guides/validate-a-package.md index 9fd925c..2b7275b 100644 --- a/docs/package/guides/validate-a-package.md +++ b/docs/package/guides/validate-a-package.md @@ -41,3 +41,16 @@ paperspec validate --correction fix.json fixtures/packages/tiny2026tiny-j4qm The command prints the record the correction would append to `corrections.jsonl`, with its computed `index`, `id`, `path`, `before` and `after`, and writes nothing. A refused proposal prints `file:N node: rule: message` on standard error and stops: with this proposal, changing `second` to `2nd` is refused with `correction.digits`, because an emendation changes no digit. `--correction` takes exactly one package, and it refuses a package that withholds its text. + +## Apply the correction + +When the proposal passes, `paperspec correct` applies it and writes the corrected package: + +```sh +paperspec correct --correction fix.json --out fixed/tiny2026tiny-j4qm fixtures/packages/tiny2026tiny-j4qm +``` + +It runs the same gate, then rewrites every file the correction touches: the nodes and their values, `corrections.jsonl`, `ids.jsonl`, `references.jsonl`, `locations.jsonl`, `suspects.jsonl`, `document.json` and the manifest. +The corrected package is validated before it is written; without `--out` it replaces the package given. +The printed record is the one `validate --correction` printed. +A leaf the correction changed drops to F1 when the evidence includes a crop of the page, and to F0 otherwise; an emendation like this one keeps the leaf's level. diff --git a/docs/package/reference/api.md b/docs/package/reference/api.md index 0077851..41c0eb2 100644 --- a/docs/package/reference/api.md +++ b/docs/package/reference/api.md @@ -9,15 +9,16 @@ Two aliases are shared by the type annotations: `jsontypes.Record`, one parsed J A function that takes a path returns text unless its docstring says otherwise. A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the rule as the specification's conformance chapter lists it. -`paperspec.resources.path(name)` returns the folder `schemas`, `sql` or `vectors`, in an installed wheel and in a checkout alike. +`paperspec.resources.path(name)` returns the folder `schemas`, `sql`, `vectors`, `registry` or `context`, in an installed wheel and in a checkout alike, and `fixtures` in a checkout or an editable install of one. +`paperspec.resources.registry(name)` reads a registry file by its path under `registry/`, such as `rules.json` or `stamps/scienting-stamps-1.json`. ## Text, hashes and canonical JSON | Module | For | Main functions | | --- | --- | --- | -| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` | -| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads_line` | -| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `lossy`, `own_content`, `meta_hash`, `Hasher`, `Index` | +| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `digits`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` | +| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads`, `loads_line`, `parse_int` | +| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `strict_runs`, `lossy`, `lossy_from_strict`, `own_content`, `meta_hash`, `Hasher`, `Index` | | `notation` | Simple math notation and its canonical linear form. | `fold_math`, `complex_key`, `fold_plain_text` | | `keys` | Work and document keys. | `work_key`, `unique_work_key`, `document_key`, `split_key` | @@ -26,20 +27,22 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru | Module | For | Main functions | | --- | --- | --- | | `model` | The node tree of `scienting-doc/1`. | `Doc`, `anchor_class`, the tables of allowed children and attributes | -| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` | -| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve` | +| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `establish_with_side`, `flatten`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` | +| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve`, `resolve_address`, `follow` | | `align` | Alignment of two readings of one paper. | `align`, `classify`, `similarity` | -| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp` | +| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp`, `published_rules` | | `targets` | The target set of targeted F3. | `target_set` | ## Packages, corrections and the database | Module | For | Main functions | | --- | --- | --- | -| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `read_state`, `write`, `write_zip`, `read_zip`, `read_json`, `read_jsonl` | -| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State` | +| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `check_forms`, `read_state`, `write`, `write_zip`, `read_zip`, `read_zip_files`, `check_zip_entries`, `read_json`, `read_jsonl`, `parse_json`, `parse_jsonl` | +| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State`, `OPS`, `KEEPS_LEVEL` | +| `revision` | Applying corrections to a package and writing the package they make. | `revise`, `revise_zip`, `revise_files`, `summarize`, `level_after`, `Refused`, `Invalid` | +| `datasets` | A package's datasets, checked against its text. | `problems`, `content_hash`, `folders` | | `schema` | JSON Schema validation with the keyword subset the schemas are restricted to. | `Store`, `validate`, `lint` | -| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `index` | +| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `read_state`, `delete`, `load_layer`, `upgrade`, `index` | ## Signatures, receipts and the log @@ -56,6 +59,8 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru | Module | For | Main functions | | --- | --- | --- | | `rights` | Rights records, policies and stage checks. | `effective`, `combine`, `check`, `backend_allowed` | +| `layers` | What every annotation shares: its record ID, its anchors and the lists its fields take. | `annotation_id`, `anchors_of`, `BODY_KINDS`, `CATEGORIES`, `VERDICTS`, `STATUSES` | +| `claims` | References to paperclaim claims and their pins. | `parse_ref`, `parse_pin`, `format_ref`, `format_pin` | | `traces` | Traced citations. | `resolve`, `problems`, `disputed` | | `structures` | Chemical structures tied to words. | `scan`, `problems` | | `sequences` | Biological sequences tied to words. | `residues`, `problems` | @@ -64,6 +69,32 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru The `problems` functions of the four annotation modules return the rules an annotation breaks, as a list of `{"rule", "message"}` records in the order checked. +## Correcting a package + +`revision.revise(root, proposals, store)` validates the package folder at `root`, runs each proposed correction through the gate in order, and writes the corrected package in place, or to `out` when it is given; `revision.revise_zip` does the same for a zipped package. +Nothing is written unless every proposal passes and the package written validates. +The writer recomputes every node's values and `document.json`, and records the corrections, the retired IDs and the resolved references. +A leaf a correction changed is F1 when the correction's evidence includes a crop of the page and F0 otherwise, and is no longer corroborated; an `emend`, `relink` or `resolve` keeps its level ([spec/08 §8.1](https://paperspec.scient.ing/spec/08-fidelity.html#81-levels-of-fidelity)). +A merged node's boxes go to the node it merged into, suspects at removed nodes and those named in `answered` leave `suspects.jsonl`, and the lines of `derived/` files and the datasets that read corrected text are left out and reported, since only the tools that made them can rebuild them ([spec/12 §12.7](https://paperspec.scient.ing/spec/12-package.html#127-derived-files)). +With no proposals, the package written is the package read, byte for byte. + +```python +from paperspec import resources, revision +from paperspec.schema import Store + +result = revision.revise("paper/", [proposal], Store(resources.path("schemas"))) +result["rev"], result["package"], result["dropped"] +``` + +A proposal the gate refuses raises `revision.Refused`, with the proposal's `index` and the `rule` it breaks, and a package that does not validate raises `revision.Invalid` with its errors. +`revision.revise_files` works on a package held in memory, as `{path: bytes}`, and `revision.summarize` recomputes what `document.json` derives from the nodes, for a producer that writes packages itself. + +## Math forms + +`package.validate` reports `doc.math.form` for a source with `form` that is not the form's own spelling ([spec/03 §3.7](https://paperspec.scient.ing/spec/03-nodes.html#37-math-forms)). +Without help it knows only what `scienting-math/1` never holds, a `\tag`, `\label`, `\nonumber`, `\notag`, style command, `\dfrac` or `\tfrac`; with papermath installed it spells every source again and requires it back unchanged. +Pass `forms={}` to check only the first, or a mapping from a form to a function that spells a source in it. + ## The command `paperspec.cli` holds the subcommands of the `paperspec` command, one module each, with a `run(args)` function that returns the exit status; see [Command line](cli.md). diff --git a/docs/package/reference/cli.md b/docs/package/reference/cli.md index b61a7f7..0e563cb 100644 --- a/docs/package/reference/cli.md +++ b/docs/package/reference/cli.md @@ -1,12 +1,13 @@ # Command line -One command, `paperspec`, with six subcommands. +One command, `paperspec`, with seven subcommands. `python -m paperspec` runs the same program. `paperspec --version` prints the package version and the Unicode version it pins, and `paperspec COMMAND --help` prints a command's options, advice and examples. | Command | Works | Does | | --- | --- | --- | | [`validate`](#validate) | anywhere the package is installed with its `tools` extra | Validate packages, or run proposed corrections through the gate. | +| [`correct`](#correct) | anywhere the package is installed with its `tools` extra | Apply proposed corrections to a package and write the corrected package. | | [`load`](#load) | anywhere the package is installed with its `tools` extra | Validate packages and load them into a SQLite database. | | [`check`](#check) | in a checkout | Check the whole repository. | | [`build`](#build) | in a checkout | Rebuild `fixtures/` and `vectors/` from their sources. | @@ -31,6 +32,26 @@ paperspec validate [-h] [--correction PROPOSALS] [--max-size BYTES] PACKAGE [PAC Each failure is printed to standard error as `package: where: rule: message`. Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error. +## correct + +```text +paperspec correct [-h] --correction PROPOSALS [--out PATH] [--answered SUSPECT] [--asset FILE] [--max-size BYTES] PACKAGE +``` + +| Option | Meaning | +| --- | --- | +| `PACKAGE` | A package folder or a zipped package. It is validated first. | +| `--correction PROPOSALS` | A JSON file holding one proposed correction, or a JSON Lines file of several, applied in order. Nothing is written unless every one passes the gate. | +| `--out PATH` | Write the corrected package here, a folder, or a zip for a zipped package, which must not exist. By default `PACKAGE` is replaced. | +| `--answered SUSPECT` | The ID of a suspect the corrections answer, which leaves `suspects.jsonl`. Repeat for several. | +| `--asset FILE` | A file to add under `assets/` as `.`, such as a crop a correction's evidence names. Repeat for several. | +| `--max-size BYTES` | As for `validate`. | + +The corrected package is validated before it replaces anything. +Each record appended to `corrections.jsonl` is printed, one per line; standard error names the derived lines and datasets left out because they read corrected text, and ends with the document's new revision and package hash. +A refusal is printed as `validate` prints it, and nothing is written. +Exit status: 0 when the corrections were applied, 1 when the package or a proposal is refused, 2 for a usage error. + ## load ```text diff --git a/fixtures/README.md b/fixtures/README.md index b030ba2..7f2f8d5 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -25,12 +25,12 @@ A conforming reader accepts each one and recomputes every hash it contains. | Package | What it is for | | --- | --- | -| `packages/hale2026harmonic/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, derived MathML, and the derived reading of three formulas in the math form `scienting-math/1`, one of which the form's tree cannot read ([spec/09 section 9.4](../spec/09-figures-tables-equations.md#94-the-math-form)). | -| `packages/hale2026harmonic.checks-only/` | The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` ([spec/05](../spec/05-hashes.md), [spec/12 section 12.5](../spec/12-package.md#125-text-policies)). It shows how a team can share its checks on a paywalled paper without sharing the paper. | -| `packages/okafor2025open/` | An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness. | -| `packages/smithlab2027charges/` | A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#a4jrz2t`, a sentence of the main example. | -| `packages/tiny2026example/` | A small document, the starting point of the correction vectors (`vectors/corrections.jsonl`) and of every broken package below. | -| `packages/tiny2026example.sct.zip` | The same document as a zipped package, written the way a producer makes a byte-identical archive: entries under one folder named by the key, sorted by path, stored, dated 1980-01-01, with no extra fields ([spec/12 section 12.2](../spec/12-package.md#122-paths)). A reader unpacks it to exactly the folder beside it. | +| `packages/hale2026harmonic-damb/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, derived MathML, and the derived reading of three formulas in the math form `scienting-math/1`, one of which the form's tree cannot read ([spec/09 section 9.4](../spec/09-figures-tables-equations.md#94-the-math-form)). | +| `packages/hale2026harmonic-damb.checks-only/` | The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` ([spec/05](../spec/05-hashes.md), [spec/12 section 12.5](../spec/12-package.md#125-text-policies)). It shows how a team can share its checks on a paywalled paper without sharing the paper. | +| `packages/okafor2025solvent-acth/` | An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness. | +| `packages/writer2027fixed-ynr7.v3/` | A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic-damb#a4jrz2t`, a sentence of the main example. | +| `packages/tiny2026tiny-j4qm/` | A small document, the starting point of the correction vectors (`vectors/corrections.jsonl`) and of every broken package below. | +| `packages/tiny2026tiny-j4qm.sct.zip` | The same document as a zipped package, written the way a producer makes a byte-identical archive: entries under one folder named by the key, sorted by path, stored, dated 1980-01-01, with no extra fields ([spec/12 section 12.2](../spec/12-package.md#122-paths)). A reader unpacks it to exactly the folder beside it. | ## Records @@ -64,7 +64,7 @@ Each record is a single file, or a folder where a format needs several files. ## Broken packages -Each folder in `invalid/` holds a `package/` folder, or a `package.sct.zip` archive, that breaks exactly one rule, built from `packages/tiny2026example/` by one edit, and an `expect.json` that names the rule. +Each folder in `invalid/` holds a `package/` folder, or a `package.sct.zip` archive, that breaks exactly one rule, built from `packages/tiny2026tiny-j4qm/` by one edit, and an `expect.json` that names the rule. A conforming implementation refuses every one of them and should report that rule. `registry/rules.json` lists every rule ID with its meaning, and a validator reports each failure as `path:line: rule: message` ([spec/22 section 22.2](../spec/22-conformance.md#222-validating-a-package)). @@ -85,6 +85,7 @@ A conforming implementation refuses every one of them and should report that rul | `location-unknown-page/` | A location on a page no witness has. | `doc.location.page` | | `manifest-hash/` | A file changed after the manifest was written. | `package.manifest.hash` | | `manifest-key/` | The manifest names another key than document.json. | `package.manifest.key` | +| `math-form-label/` | A `scienting-math/1` source that still holds its `\tag`, which the form lifts out. | `doc.math.form` | | `missing-required-file/` | A required file is missing. | `package.file.required` | | `not-canonical/` | A line that is valid JSON but not canonical JSON (the fixed form of RFC 8785, which gives equal data identical bytes). | `package.file.canonical` | | `receipt-root/` | A receipt whose Merkle root (one hash that summarizes a list of hashes) does not recompute. | `doc.receipt.root` | @@ -95,6 +96,7 @@ A conforming implementation refuses every one of them and should report that rul | `root-mismatch/` | document.json's root is not the root node's node_hash. | `doc.hash.root` | | `signer-mismatch/` | A receipt whose signature was made with another key than the signer it names. | `doc.receipt.signature` | | `style-sup-sub/` | A run cannot be both a subscript and a superscript. | `doc.run.style` | +| `suspect-witness/` | A suspect whose second reading names a witness the package does not have. | `doc.suspect.witness` | | `text-edited/` | Text changed without its hashes. | `doc.hash.text-hash` | | `unknown-kind/` | Kinds outside the list are refused. | `package.schema` | | `unlisted-file/` | A file the manifest does not list. | `package.manifest.unlisted` | diff --git a/fixtures/invalid/math-form-label/expect.json b/fixtures/invalid/math-form-label/expect.json new file mode 100644 index 0000000..86ebf34 --- /dev/null +++ b/fixtures/invalid/math-form-label/expect.json @@ -0,0 +1 @@ +{"note":"A scienting-math/1 source that still holds its \\tag, which the form lifts out.","rule":"doc.math.form"} diff --git a/fixtures/invalid/math-form-label/package/MANIFEST.json b/fixtures/invalid/math-form-label/package/MANIFEST.json new file mode 100644 index 0000000..5853a3e --- /dev/null +++ b/fixtures/invalid/math-form-label/package/MANIFEST.json @@ -0,0 +1 @@ +{"doc":"scienting-doc/1","files":[{"path":"document.json","sha256":"2586b1713112b6c2dc297066c497e096128e7594179862098439c2300cf179d9","size":764},{"path":"nodes.jsonl","sha256":"0146621bd4b98c3ef9f658b3b8a9cfe940a6b996bf10d5452fa13c0886eb4c7b","size":11405},{"path":"provenance.json","sha256":"4bc7649bf6d3113f6c72b4016713654f57c8f03069632eabd1e6025c05cb9dac","size":226},{"path":"receipts.jsonl","sha256":"f745067d43480bbd467c5d8f11ff13eec198fac821664894ad8bd90b8b7cf0b3","size":2182},{"path":"references.jsonl","sha256":"59908a8aff6a2d5800747a1f45c6c2943ea1e31e7e1f10b6348967d30f16b87e","size":60},{"path":"roster.json","sha256":"3c71b0c007924cbacd3b4af02485214ab904db8ff8fca133ab32e67c5c76d481","size":6941},{"path":"witnesses.json","sha256":"83ac4411c2efe65e0f7fc4c9336abbef6977cd32b0f90de029a7f8516afc3e97","size":750}],"format":"scienting-package/1","key":"tiny2026tiny-j4qm"} diff --git a/fixtures/invalid/math-form-label/package/document.json b/fixtures/invalid/math-form-label/package/document.json new file mode 100644 index 0000000..1480f1d --- /dev/null +++ b/fixtures/invalid/math-form-label/package/document.json @@ -0,0 +1 @@ +{"counts":{"leaves":10,"nodes":22},"fidelity":{"counts":{"F0":0,"F1":0,"F2":2,"F3":8},"lowest":"F2"},"format":"scienting-doc/1","hashing":{"algorithm":"sha-256","scheme":"plain"},"history":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","key":"tiny2026tiny-j4qm","kind":"article","language":"en","license":{"spdx":"CC0-1.0"},"meta":{"author":[{"family":"Tiny","given":"T."}],"issued":{"date-parts":[[2026]]},"title":"A tiny example"},"meta_hash":"1062e6079e2871c390f4737f5354448b65fe524d8c09d1c82afaa8b71ba6b3e7","policy":"full","relations":[],"rev":0,"root":"842b9e18bcc50da04b4bb45285ec006a34ad369730cc9b14286cc73a9f74d197","version":{"type":"version-of-record"},"work":{"key":"tiny2026tiny-j4qm","same_as":["doi:10.5555/example.tiny.2026"]}} diff --git a/fixtures/invalid/math-form-label/package/nodes.jsonl b/fixtures/invalid/math-form-label/package/nodes.jsonl new file mode 100644 index 0000000..4bc8495 --- /dev/null +++ b/fixtures/invalid/math-form-label/package/nodes.jsonl @@ -0,0 +1,22 @@ +{"align_key":"65f9b52a7bd652256da439faf19fc25f04b7f2ff7201d6af3a623bbca9fa39e2","id":"s0fn2fj","kind":"document","match_hash":"96546166e7b438ec513007778509c5cbe982076768183ef59de55a21a22dfb92","node_hash":"842b9e18bcc50da04b4bb45285ec006a34ad369730cc9b14286cc73a9f74d197","ord":0,"own_hash":"34444fe0e6bdbf3927a579fe3286859d204ea0882a381b7769ffd4905551d24f","parent":null,"text_hash":"37409459e7e2b3a8184e60c6f7b7b21cff6c4efa07d04fdf0dcb4635c3e981d8"} +{"align_key":"c375c776c496f0e334acc55b000f3cecd226661b2e337d20d3efa082a6754459","attrs":{"level":1},"id":"b9wp3tt","kind":"section","label":"1","match_hash":"59c4b8702f7716ec47e32f1bb5b331c6611d9623f2744deb8f8b203f7f6a095b","node_hash":"d9673b45baffdd2a338e49bfa6184f026ae67224aafc136ce97766554671e0d5","ord":0,"own_hash":"65da097b48f8237e8d171bfbd0053ff0e3795c0532d5522d23a92b53430956f4","parent":"s0fn2fj","text_hash":"141698d1f118f50b107f017875c14fc24b529da752ed66f6c17a6332f032cd09"} +{"align_key":"c099142bc3186ded72786ba27e9ea6d2da240fb9fd3fe79b479ecf8e734b2850","id":"vet7k0e","kind":"heading","match_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd","node_hash":"a3c82ab1e44bf4cffa1a59507f8bddc39a1ad191321ee803d0d60d96609a44c7","ord":0,"own_hash":"e813ff86a8920cef741e548eb5df545774b520d2fadbdda6e2b872a6729b9d26","parent":"b9wp3tt","text_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd"} +{"align_key":"c099142bc3186ded72786ba27e9ea6d2da240fb9fd3fe79b479ecf8e734b2850","content":[{"kind":"text","text":"Results"}],"fidelity":"F2","id":"em5pd9v","kind":"sentence","match_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd","node_hash":"293fc03ff4e39d7532467e4417ffcdb15f470f0bd3658dc1c021645ad53c9bef","ord":0,"own_hash":"c55f401232b19ff1133d77eac6c72868975064f177fc2d31ab8a181303c30599","parent":"vet7k0e","text_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd"} +{"align_key":"610c235321e38145abff21e31eaab303d85a6ce0d1b9340a599dc821a04922f6","id":"ytp1e5n","kind":"paragraph","match_hash":"bcb4902ec69f2a7a724384f561ee9aa0e214c824c5882c1ad92777bee680aad4","node_hash":"7667a3a4bc2e8fec1dc7af53e19c414952f9ce926b1e37e8d9adca66dd8f8d1d","ord":1,"own_hash":"494e069be36a18ecaabf5a63be7c31617ade7a5460218ca83a2de3d8d8f964e0","parent":"b9wp3tt","text_hash":"f589ee914c46c4aea7ce2de49d925c433c914121f576b81a3eab692f17f179a3"} +{"align_key":"8f21603c39df123eb8680362184274aac9bdf1dbd485b3bba230960179071c95","content":[{"kind":"text","text":"The value is 42.0 at 300 K."}],"fidelity":"F2","id":"jvnys4s","kind":"sentence","match_hash":"3bcf9fa8412dca28be91d6423460233ddedd7add6c324d8a0f2b26038567d3d3","node_hash":"f5761f338b2878dff39a6ee724a75b13626dd625a0df96cf1cc6f9e8258cb7cc","ord":0,"own_hash":"7f0b9190d076cf01b51a8be863a51e9a8eeaab5d3ec8656202ff4d41578d4d82","parent":"ytp1e5n","text_hash":"3bcf9fa8412dca28be91d6423460233ddedd7add6c324d8a0f2b26038567d3d3"} +{"align_key":"8e7fe271cd1bab88d86c8900fe3ffae0ce95c55af6659df1fca278d359296e2d","content":[{"kind":"text","text":"See "},{"kind":"xref","target":"tb49qpc","text":"Table 1"},{"kind":"text","text":" and the note."},{"kind":"cite","refs":["r3e21zr"],"text":"1"}],"fidelity":"F2","id":"mzwwhxj","kind":"sentence","match_hash":"9894a48790a8561966e68672fa13da20541d25a5178653720648f62cbc9ad36b","node_hash":"8f3e5338ba3c97efca4cef5c60d849757aed6334ee27ab807fa71d969089e46e","ord":1,"own_hash":"02ed6d50b8ac607b59e3f7b5116ee6143ed7e69af25b19c4037aa799bc330a88","parent":"ytp1e5n","text_hash":"5b1de19c9eb1399e54bde7e96f051cac8b337be1f307fac320292b8db7e39da7"} +{"align_key":"d1dfb8303cf760f1c9b5869aabebd28e80d6730e8fc0663366951b3f86459c1c","id":"exq1ter","kind":"paragraph","match_hash":"1320b130813dce9cd6e0630dddfce86d62983de5eeb7e3ce3dc39002aa4d74bc","node_hash":"5d72bc43a31cfd65c71630cb61bd00ec17cce925f7a5d86294feea2de6574de2","ord":2,"own_hash":"494e069be36a18ecaabf5a63be7c31617ade7a5460218ca83a2de3d8d8f964e0","parent":"b9wp3tt","text_hash":"f567a0cebf7b8cec19220dfa2a7a41dc254e7676f2ee17ff96d3107d6d438db0"} +{"align_key":"b02a85962a2cecd80fc82093410195e78374b6147bd169fda37f737cb12601c0","content":[{"kind":"text","text":"A second paragraph, "},{"form":"scienting-math/1","kind":"math","notation":"tex","source":"x=1\\tag{2}"},{"kind":"text","text":"."}],"fidelity":"F2","id":"j663d4h","kind":"sentence","match_hash":"c57b6ddabc53ff4d46acd560a10853de9b25d9ae9f95efd75344a3611178d1a9","node_hash":"d24303fe6b04bcf7b942ce79f5db1743763fa4c529793a0f11669d62a16f6a6d","ord":0,"own_hash":"432e8e839c6c4294b560300905e1d40af2f58103ce4055d8b49bd68f828bc937","parent":"exq1ter","text_hash":"6db4ee9addc6f343d677e0b612a9ccaf45f747ddbb893c64b398c00f9b986a67"} +{"align_key":"437c101749b0481e615fc7eed8dcacb536b65400c7e0695ad4cbb2bf2dae488c","content":[{"kind":"text","text":"It ends here."}],"fidelity":"F2","id":"rsw24qc","kind":"sentence","match_hash":"2888dbfa0c08fd65c58c923b920860c76e13913d9fa93b03eccbbf7bdf53b882","node_hash":"caeb3ae57d0821b8ac961c9507debbcda2b118b16cdc010b83a143a782058d57","ord":1,"own_hash":"705f77e8520c8c5ce3a7a495a20d8a9c55baa9801e49a55c341f5de79eeede75","parent":"exq1ter","text_hash":"2888dbfa0c08fd65c58c923b920860c76e13913d9fa93b03eccbbf7bdf53b882"} +{"align_key":"cb0107aa0d562e8bd1188c33dc1ef5a1d07808e51edeabaa87465556d11b6ec8","attrs":{"columns":[{"header_path":["af5b51r"],"id":"pj5sk05"},{"header_path":["ajqxkgh"],"id":"nymcs5q"}]},"id":"tb49qpc","kind":"table","label":"Table 1","match_hash":"1951e0b61441d7b817b4967b36eee007f8e7525ac75fd359437eddb871e59f72","node_hash":"154c910857f33c152fdc78e42582510327055e62da9a25ab10c3891eeaf69627","ord":3,"own_hash":"fb4cc66080def0b42a06f7aa566a24957c7345c77b8d873ca5c790704a890f47","parent":"b9wp3tt","text_hash":"1951e0b61441d7b817b4967b36eee007f8e7525ac75fd359437eddb871e59f72"} +{"align_key":"769a4e6d0003189c7e96c5d9b7e810a0d11c3a12832527ec94b0f86d277f51ca","id":"t2ps2mp","kind":"table-head","match_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19","node_hash":"6d8056adfb50dc0b65a2a65b47eccc5f47d5482a8dfbefa0d8a06f6641a4be3a","ord":0,"own_hash":"c5749bee3a4b981fb79cf6db125ebb416dce5a83613a00feecd9d4cf75df9bb4","parent":"tb49qpc","text_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19"} +{"align_key":"769a4e6d0003189c7e96c5d9b7e810a0d11c3a12832527ec94b0f86d277f51ca","id":"rr9rtf5","kind":"row","match_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19","node_hash":"8bd1fa377ec390fdc36e17baa8c3a39d2df7ef2a53cd0c0dc745ae40e98f99f8","ord":0,"own_hash":"6c7c11bf1b6260162c4141f1468aa6869f5c33359900497e2cdb1a051488bd62","parent":"t2ps2mp","text_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19"} +{"align_key":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881","attrs":{"colspan":1,"column":"pj5sk05","role":"column-header","rowspan":1},"content":[{"kind":"text","text":"x"}],"fidelity":"F1","id":"af5b51r","kind":"cell","match_hash":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881","node_hash":"59d1c20766a77bacde2aae80a134c7156fbcc708c140a3bb2d2a67f78bd9c6f5","ord":0,"own_hash":"c68933b07bc7c5b806892ae36b3e4c3205a0a2066fd6f8255578f7d1ca86dc10","parent":"rr9rtf5","text_hash":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"} +{"align_key":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa","attrs":{"colspan":1,"column":"nymcs5q","role":"column-header","rowspan":1},"content":[{"kind":"text","text":"y"}],"fidelity":"F1","id":"ajqxkgh","kind":"cell","match_hash":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa","node_hash":"838e3fba38f0797d40522b7063cb5c2962e32a019c09c19f09849ced73079a87","ord":1,"own_hash":"28df42bf1dcd44b053275ed014bd1599914f17adabc0c7fdb3e4247dc648ac49","parent":"rr9rtf5","text_hash":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa"} +{"align_key":"b902cc4550838229a710bfec4c38cbc7eb11082367a409df9135e7f007a96bda","id":"dmsstkg","kind":"table-body","match_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738","node_hash":"baf17577660aa6f0ba9c8703d30816d77cfce7dbf0ea3257ed830fa9ab94ce9d","ord":1,"own_hash":"b229a7f92314227bf225ad27010a546e7c2373f4b30b7f6ec372e568cbacb6aa","parent":"tb49qpc","text_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738"} +{"align_key":"b902cc4550838229a710bfec4c38cbc7eb11082367a409df9135e7f007a96bda","attrs":{"header_path":["rdrqkv7"]},"id":"r3e7hfs","kind":"row","match_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738","node_hash":"97a1f5b27bbfe8c9d2cec7212b519b4993506fc8080be3eff02f3aa31515a19d","ord":0,"own_hash":"6c7c11bf1b6260162c4141f1468aa6869f5c33359900497e2cdb1a051488bd62","parent":"dmsstkg","text_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738"} +{"align_key":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b","attrs":{"colspan":1,"column":"pj5sk05","role":"row-header","rowspan":1},"content":[{"kind":"text","text":"1"}],"fidelity":"F1","id":"rdrqkv7","kind":"cell","match_hash":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b","node_hash":"87da3bd568286f3d0578d8a813496d15faafb62f4db44b07aa1c29e52cda34df","ord":0,"own_hash":"fb474321001657b59bccd3cae0987c544a4ef9a27214626d77695f4db237aa93","parent":"r3e7hfs","text_hash":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b"} +{"align_key":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61","attrs":{"colspan":1,"column":"nymcs5q","role":"data","rowspan":1},"content":[{"kind":"text","text":"2.5"}],"fidelity":"F1","id":"f463wj7","kind":"cell","match_hash":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61","node_hash":"f1cd6d5339c84d5c36f8e1ea484a8d837f3200f05d1b106f6119dfa30d846bfd","ord":1,"own_hash":"088eff622eb3086d68c4433cf5e1783dc9f2a6724c3536c31ec0bf80b5f89db5","parent":"r3e7hfs","text_hash":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61"} +{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","id":"aw97y35","kind":"back","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"fcb8027f7c89115f048b9c5d6770847248f4e4fbc61cbc8ea0b04079b4d58350","ord":1,"own_hash":"bbb972bdc024231b505f4d8ce5e711fbb75c8b5ffd1b5d42ee3fb8586b85f867","parent":"s0fn2fj","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"} +{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","id":"yd513mw","kind":"references","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"8b8a9000e984c3efeb27fc40c8fbadd7349dd64860aa276dcd2a25a57ce52e87","ord":0,"own_hash":"2a5302e2041d0e59458fea212dd51b89299901318bfceeeff227c1a07d3f5d1a","parent":"aw97y35","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"} +{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","content":[{"kind":"text","text":"1. An Example, 2020."}],"fidelity":"F2","id":"r3e21zr","kind":"reference","label":"1","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"71b1af6dd6a56f24cb3b338e0fe99df06b9154ecdf4b391b00b7a7e61e23a8ed","ord":0,"own_hash":"f7c9bbc8c7456e6dc435feeaaa02cbfdbe487523b020105793fb11b91d0894f5","parent":"yd513mw","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"} diff --git a/fixtures/invalid/math-form-label/package/provenance.json b/fixtures/invalid/math-form-label/package/provenance.json new file mode 100644 index 0000000..961e28b --- /dev/null +++ b/fixtures/invalid/math-form-label/package/provenance.json @@ -0,0 +1 @@ +{"note":"Built by hand for the conformance suite; every reading is synthetic.","stages":[{"processor":"local","stage":"hand","tool":"paperspec tools/build.py"}],"tools":[{"name":"paperspec tools/build.py","version":"0.1.0"}]} diff --git a/fixtures/invalid/math-form-label/package/receipts.jsonl b/fixtures/invalid/math-form-label/package/receipts.jsonl new file mode 100644 index 0000000..d0952d1 --- /dev/null +++ b/fixtures/invalid/math-form-label/package/receipts.jsonl @@ -0,0 +1 @@ +{"at":"2026-10-11T12:00:00Z","by":"ana","doc":"tiny2026tiny-j4qm","evidence":[{"kind":"crop","node":"af5b51r","sha256":"78b08276b5122feed3829e8dd19684fa8a4a41f79cb83a499d40341e09f5d658"},{"kind":"crop","node":"ajqxkgh","sha256":"398bab4177685c88eef399d814d34c81baa556d58cd508bd5ddb33ac53a5cbfa"},{"kind":"crop","node":"f463wj7","sha256":"88b1e820b569c3a0fd025a6b6c502271ed73bbce720dd8a2e416d6a5dd0e9d67"},{"kind":"crop","node":"j663d4h","sha256":"d6484c89588a560f64bbceca108f5446e47bb5cb85521fbffc9d6d37f252e5c7"},{"kind":"crop","node":"jvnys4s","sha256":"3139fb30b88062bd0c6bbb028c4c57fc748218ce1d7a612d282e292d6a02ffe9"},{"kind":"crop","node":"mzwwhxj","sha256":"83a7fcf09321f8a644c58120f80dede5d73d89f50d080ccbb143ec5b485e3088"},{"kind":"crop","node":"r3e21zr","sha256":"ff809e11101fa6ff5422e5232e83142c1a43dada0e56d63ec03fbce3474dd1aa"},{"kind":"crop","node":"rdrqkv7","sha256":"c0699aceb344812ebebb1414838641f0761184d5113dcc870d07c721fd289437"},{"kind":"crop","node":"rsw24qc","sha256":"b4c3b133c707b1a9a2cec2efe5c17731bada0d6bace5b1fb3590c5c5ab18bb92"},{"kind":"witness","sha256":"fda3f1e9bb3538cd85dcc42281e00fd045f726652664439ce5a15d8489425b21"}],"format":"scienting-receipt/1","id":"k4wt522d9zvjr","level":"F3","nodes":[["af5b51r","59d1c20766a77bacde2aae80a134c7156fbcc708c140a3bb2d2a67f78bd9c6f5"],["ajqxkgh","838e3fba38f0797d40522b7063cb5c2962e32a019c09c19f09849ced73079a87"],["f463wj7","f1cd6d5339c84d5c36f8e1ea484a8d837f3200f05d1b106f6119dfa30d846bfd"],["j663d4h","443539440dca73149ca75276492334ce9a5a6c50070137e8dc7bf414b75a27ab"],["jvnys4s","f5761f338b2878dff39a6ee724a75b13626dd625a0df96cf1cc6f9e8258cb7cc"],["mzwwhxj","8f3e5338ba3c97efca4cef5c60d849757aed6334ee27ab807fa71d969089e46e"],["r3e21zr","71b1af6dd6a56f24cb3b338e0fe99df06b9154ecdf4b391b00b7a7e61e23a8ed"],["rdrqkv7","87da3bd568286f3d0578d8a813496d15faafb62f4db44b07aa1c29e52cda34df"],["rsw24qc","caeb3ae57d0821b8ac961c9507debbcda2b118b16cdc010b83a143a782058d57"]],"rev":0,"root":"15d9c8bd5472afd7064578e5663d3b7fd21249adc6cdb40fcbe1b790e3fae801","scope":"all","signature":{"key":"ana","scheme":"ed25519","value":"/xOfCboOpgu29ztKi22IU9crgC5LYetrHOMpQtqj4QV5dVP73n/ze94hyPknYO5qNPgBmYRrbJZBIIHdwqxoAA=="}} diff --git a/fixtures/invalid/math-form-label/package/references.jsonl b/fixtures/invalid/math-form-label/package/references.jsonl new file mode 100644 index 0000000..af52739 --- /dev/null +++ b/fixtures/invalid/math-form-label/package/references.jsonl @@ -0,0 +1 @@ +{"label":"1","node":"r3e21zr","resolution":{"type":"none"}} diff --git a/fixtures/invalid/math-form-label/package/roster.json b/fixtures/invalid/math-form-label/package/roster.json new file mode 100644 index 0000000..36584ed --- /dev/null +++ b/fixtures/invalid/math-form-label/package/roster.json @@ -0,0 +1 @@ +{"format":"scienting-roster/1","keys":[{"did":"did:key:z6Mkj7iUX8QG7zkAtNaV78CUXcenhD87qWwQbc7NU5aVXyW9","id":"ana","person":{"name":"Ana Example"},"public":"RUiz5j+y0QwgOtsQpqXrlq+gy3c/JFnkxwvlWUNX4Ao=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"owner","scheme":"ed25519"},{"did":"did:key:z6MkhemKN7rMQcgt6wT2eWPyx3vFRS1RUoRziLMnLq2vZ9yi","id":"ben","person":{"name":"Ben Example"},"public":"L4WMLs7xLjBLRsxS/EYpD194puMBpADybcfFEWxBkWk=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"editor","scheme":"ed25519"},{"did":"did:key:z6MkrBf2w3MUroaNBiHTQJzH4e1w31jpcosd5vNxjikijndU","id":"ben-spot","person":{"name":"Ben Example"},"public":"rkzxigjCXg869PGSFAVr4EDXXRmwk+S/+wJ93lrG4dc=","purposes":["spot-check"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"editor","scheme":"ed25519"},{"did":"did:key:z6Mkgrc9HCeEUL5tz1bVfuuDeSm6mP6hyWUX5ivhgReSbEQM","id":"cara","person":{"name":"Cara Example"},"public":"I7J1GqG5CHOc0LHYZPNNto1Me2lJUzBVn2A7GrUGbZ4=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"reader","scheme":"ed25519"},{"did":"did:key:z4xcFfnTC6XJy4gzg9whfyMzcnxV7yM2XafMFynKXhAEvMgjfQCxSnjZaa53arz2iNo8v35ZN5SdAUcbFkbuvftcMczRT2CPhYsXkNYZhDzXWQVS5KZZkhnMDmwChWfgRHu4Bs6KHyEmvzwesTioW5yS3YtfR7mUPJAwWSRq9f23HRxzenGpVVBcVjpd8YDZ4Y1ZamcUfesnB6pGLrPtLDvGM7Te1cJWoRG4m3TetEF3i9V3W4CH43nouCwUPbjy7dpTNEWiCqaTPiPhV8UEWBD8b9e1729PwoUFyAFreuq6wvhphLKGHhFCFujrUtqDZPs6FFF1Q7Lpt9eFcjbxrZSAYsh9VTfxaUeQ4HmFqzrg28EEjJzxxADZyjTXDsfwskRbibHSms9bbN2xK6GQcUxE2mrA2ULfYwsk5RQsVULUTj3pMMXpxRqP4zerzui2ZRZ3SqeNUwqnhmF861pEtcemZPY6ivcX41QK3ekCtpc1HndtY9xua1Lrhs4jd6qEbxHWzBPHok6qTpyxnn7ubZssBqSMz3uNrou1ayZGDn1ayAXtzCJPjAcxdNX1dpm5vyDB8GYtNKsQHuiX5xUd2EMAVfq4cFrZ6ncY1zrG5exH6mhJYbZfW7BgXxq7GsnzAAGkzgHCRADe5EEm1Hj1YdmVBTiaGsPGYkF4Mcrw3T2j3AcPJCzPLRNjKBKXyHAcf3VPUmreaHd1GoA5zqvtANSPmppY2AhCAj8u6n92fiyBgrvJTh4hyvtMFcwaa9vKnY9rBS4n5XGpdwKQe11Tk5sah8VZc2PyH2WPBU3DmhZSwUkKHRQQQQMXjrLjuMtygzxWBgrJAcWRPzxmzHZEj5x9GgFZb4dPDwqVXsgR25KrpuyQKcxfbxzQoRprnhu1ec8SfX5zeUo282Si9dBZ3JVdxzYHzP6xSiuBeHHFMvMwb1MKUq2dMi9vWdg2eCDo5wTvSaTVcdYGtZsT7HMswWtxx9GLw1RafYRcNqMhBhCbRdJszUKjQKwNNZbaryRs4kTYicuE24PWbqfxdJnrzrLbXnVBDMNHjX8ESXSKTMnAoE1AiLgjdj3zbBrY39Pxc72EikA7GvP2qxKZBkJ6G2RRWCw9d3nZKmTmQqP5PP9HgdtSTnZhNTJqBkwL3iqfoWxtbkLjLLSQc3dbFpmhe4kAh7VUUDcLzjb4nGzCbr5U1YoqAoLnSC9BPijnfMHxBP6i8cR762ryxPwe7HdGk62DVYoqxbwz8ygrDPjdPi3t11PpHp51GxuhMkoR3GFxAdMEv8j4tVpJ6qjefULEFTtRPzZ5kTE57RTxC4wg62qcYwupWx4dvvB2BR9ZGnNwCojpBYByeeYETrgm4HJjjS3SH64sbpyqz7336YMAcWyQFc9BeqjQHHUHS8CJ82N88apbPPE8N9X76DQ8PgwACtR4D5ofE9o14kpdVZFV2YMtmnKCybNLn3atspK6T1ZPYB3vQ1c4DbzZFgCpm6izEzxsY17in79ieo3cY965Cr5mkxca29VxcTaEtovdu9sKAL4VgCZzWkps5nnimX5JBEdVKYxTtMb6CcTkdWGFtXBTBuAMTpRNLXG5nG4bq9XXLkFS4oXpVS7KayqAMtrrqttpgwdAPNuzd6E4JskxdenwMq6qzMTDKoMqGDVEKkwCxCcbUtNpHGMLWMMk9MDrsPvcU8u2C89UmJmsfS5gQqswkz6TfXvthM3wPcuKaX7TUJJRzYstWsXRKUaPqqT3ppftCbXPFgia3mXkA5bk9Y3GWcTADGTnzKkTzbnL6hsgQW1FyvsFwUiWUJd5weeHkfaHt8iV3PKy99S6d831iLCfFWM6bYZgEBvyx5seHSpiaXRfSv4A2yDsf4Tb4EVsyDUJPUmZLnbFkJmw46A6PoQHdwfBbXWV47DFM6w5mKXCPmKzuAG1Vmw7D36n4oHd91r1VzTynVnhD1EA5UVh4qhiw2CnWgo2uAPxEpLYnvrCLiVprnzdPzvZZqMUxXRFWHwfshJDARRLvP933693aWxa5PeDNgMjZYgKJBV5e66Z1Mk5rYegEuSq7uyz8Mw3iuidtJeqBugbNCYeNdmeBPgKux8HcuFNqQaVwhP9vW8vyaU7DvuKEeXs1uyAgtk9RCua2Mx8rQqUwQigWNjXaJoSYqMT9SVEGUSYAGQiLnMFENqG1r87uhxV58hTwR8QED9cQkStFXeuZAMY6FWTZCsD5juC48KdqoUQYr77iBy7fW2quQNdPvwEUdhh1mzSTX2PPubCbm9rcdBxBQ2CBZizn1EsbPMic1X39t17scRE5F4Yank18EF4aUmJwwNHx5m1WKDQMZ1kJQRyhtRvAHTPqBdJCw5SHaEqBVpEQHUZwBKsQoTRj4Gk1opv6XfczoK3MLuprTMSx2maw5Gv6LhF2SWUvEXMMqoeqVzyP9g11dumkKQLMiAQa2VGaa7EF7Foj2ZDUhPM6PPiffHMhDMwWhRBywg3SC6aP9zz2TdMXweHv8rags9chFZ217aAP3bcCf5i1VZqPT9b8Ea7ykjjUy6PNwiSforRXN46suuRFREuhx6DuWDojythjH73wMAeKKx23rF4QfdUxzkC2mmmS4dq8Xx7Bda4e31EGab4t81Jk4dAVGjSrZC9wx89CAjovhoejgDTKNsN3ri4GjW9UD","id":"dana","person":{"name":"Dana Example"},"public":"lQ6M2DulKY4RkQLGTPGZD+CHlOOib4kBkwlccVNWRDPco1k5kHtelT/yUP4+CoAQnHnLu96vpeWztaee/9RmjR3bstGj+P5tOSrwFlPuyNUPOWPNjwCQLrVEyeUGBaA4+HRvFwK7C4OHB8EJWYtC0OuH+tRPlCdLpxlVcZZcM7dsZYe2BDNenbWfp4TeX2Oa+dThlceDY0m9MewwqUrd5xdo9tc8PS9R9o0ZUWf0vDw7Fc5AcsWwbf76lDLvIbyd0aCaHJ7eaK6DFsBuZFKVdlZlH0inJciAMF8dPoRuE+A0VlZTEOsOivNWprM5iLHzXse3zSoHLldWr0t4hWxHH3MBQ57Ucee8RTU6hFRfqfFRl6zJQyRZOFZZoYD9QwXkR3NmUzV/FcsOMHpbgTcY7O0TAajPnfpSTp+XdyPVBR7TlTsturUCpryP0ZX1S3VJk2IaneB123thtcOtWsbqdT3Bb846H7yudI95WVTixIS/KIwoCbekSf8Y9+iW/xRVSWLRVc1gpigDxXfieSA7bI+qiUBjiKR7QU1KFYZA58VcOQhzeMLY6an34rsW4WHAwzzSv8Q6vrwpn8Ct9fTsIeiZsWZLueNkIUfjR0G1bHHoT2RHKton0AMRjOUGSKUUB4DNZ5xzXvC0jvUMWlgHB9z2wBCsIkIDVWqSA2DD0xeIY2wSUVPwPc1MnBAf/Fx3poweUlcPWglR2VaO9ematC4hfLxxkGldGQt3C6ZLPsm96S7o6j88rwOqkDNIR1pNJOepgciJH/w4Xwtg9gJ52CpdlCeSllRQD74l95ysmZ2D/gLweNoUt61P0RYQQelio8VnPUNB1e2uK7zW9joG1k59CjJGQNvuZr6JkQ9+Hxo5xKEOy9KcpT6t48wTzbz3YMFDxTNl0/DzL8qdTU2UrSW6/TFgPLokbAQN00yhw0lg8cR/e1m+ujFyBeesL3luwE6HNcIyK5Y/+To04vR3OD7Uz8cnABxWyOeSBm2R2bu2fxeN0Hlv+qhoOYtbZ1DKx/aES4z51jATynEIdMIMSNhmVDoDiGODh9ocenzg/DUarjIzWltPFRyQj+tnFGTXp7C87s3yAf248ZB0HHhK1GCppxB4hsv89nNtWH1JdWjsc243s3f0WOZtJ1f+9yeybViJCW47ez7HkwtTq2AVeg+lHrb4TPCL5pBKwUOj5GVe2DzMFBtjilcl/ED4gev7DK383iyB4tGVZa1ALPKUyiyKV3sCRV/mN5DSGeASafI6OOjjUKifYCJaxOHFQGr6zvfhVj9L2Cl/hthxplTWoLpo4I7+oLPQJIrOMPf8FHLtRLAYthsxT3K02mVtUMhrviaVh5/kJKMMd4iRc8YXloC0r8roivoEPr5SzDprSffh8a2N2KRAHd3yrp4z7RxlpWKPWr5whD56q279S70A9r/r5r0bVzwabhKkTSJxc2LN72QondANsGFDYPD3Rm+FwSO2JwwgIG8GQxBbMWPOEqD0qDmaebNfP/EpbeF8mvA8v4e58CIv1LaHZuEky6vc16woEYHm11UuH6ChHbWxCoDXJsVM/jPme1vPlic38i9XYsE2c199gP1ikBGQANlryeIWH8wXtPUXNflNecMqIQkWAWSK/8jujgtylBQ/CklRyeW0+vyxV5eo8S8TfnonyG2eiHN1VS3HGLSidcYRi54IsOMtuR773QoLPZeRXlsNWO5kc3xcHU7K6olsMNtQhaFnkf3GATh9H80GUvEMK9u5yMUrCoYWuUy7JQhjsl+B1QbAFftdDu8ESgTOIKmFOLg+F9CH4lJrCqnwUdCs93zbJem2++Y64HRFLSUXwlhEUuQVugX7BejmcMYWMjYXROlatnznqYXcWW2uxx6ZNxQv0eB/CN1i4q1G8y9XUh4wz/caY8ly+eaTU5J4wGHfnUDpssu+TUvE5PGUs1gW6FswHlf1IXEnvKDIXKajbJgYnjEekVG8FxcSUc/10Mw/vqk/GQqdMOguf3mShlUIbtX2JVrA5ClZ9qza4HEXC7RkmxJLyc5dDyX/CuFdoDzrVu+1+JqRsFHGeLCo9rNvIQjvG09GQX6uDUavi25hCcOgug2DI0PuVLjcESFDyaYNdKo9GsiCEBpLPglp9viA2vX2kG6P2pidgsGfcBgdGuYGQbd23Qu9QpOd8AAuyWsm0dX6hKO2ChoAPVQxDNcj9N6UjumggVQ6zOuC0+gOcao/aPs/eH9agchdtnWKuL1clo4FduP4sdDtTvnJLFYnXONM//LSnjqSyBc7rzjJ2cxzPWQtLFawErzHXukl2eeP/xpDvLU0G6JroeJqBitHIoK00FGdxTUnrJBgy1DnBoIgUhxOvb1wtnx/O4DDS5tJZBDbl9XVbRmSfcljfQ5mB38hQ51CuYodHE9q12Bk1FtSwbVKvPHGb+9EdkbktJKRgdJubAQ+ajOCyMjEDCp9xLeKnlFL+WesI0Ro+rnmQL0qXVzkhZj8LP4a8yFYieIXCApLveflfv2yRJf3a/EML7yIgoEuzejddFrcbupWxDRqwS1YcLDkvGBEzjp8zuVnr72GfaFW/nvU9V9P/Z0fFS5hZhG5MnEp0Qzjsy/eKRI=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"editor","scheme":"ml-dsa-65"},{"did":"did:key:z6MkiHaTrUvYfXFtTw39oiWrgan52dRBHYgm9WeoiEPwMotg","id":"wit","person":{"name":"Open Witness (fictional)"},"public":"OPPD6UEKZ7IObfmh/pzKS29cNjnz9WbKMQivMBXrlC0=","purposes":["witness"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"guest","scheme":"ed25519"}],"store":"smithlab"} diff --git a/fixtures/invalid/math-form-label/package/witnesses.json b/fixtures/invalid/math-form-label/package/witnesses.json new file mode 100644 index 0000000..a7b09bb --- /dev/null +++ b/fixtures/invalid/math-form-label/package/witnesses.json @@ -0,0 +1 @@ +{"witnesses":[{"id":"nmfryke","kind":"pdf","media_type":"application/pdf","origin":{"source":"author"},"pages":1,"rights":{"asserted_by":{"by":"ana","evidence":[{"kind":"statement","note":"The team's own draft."}]},"at":"2026-09-15T12:00:00Z","basis":"own-work","format":"scienting-rights/1","license":{"spdx":"CC-BY-4.0"},"uses":{"accessible-copy":true,"ai":{"external/local":true,"external/outside":true,"external/outside-no-retention":true,"external/tenancy":true,"internal/local":true,"internal/outside":true,"internal/outside-no-retention":true,"internal/tenancy":true},"quote-public":true,"read":true,"redistribute":true,"tdm":true},"valid_until":null},"sha256":"fda3f1e9bb3538cd85dcc42281e00fd045f726652664439ce5a15d8489425b21","size":1000}]} diff --git a/fixtures/invalid/suspect-witness/expect.json b/fixtures/invalid/suspect-witness/expect.json new file mode 100644 index 0000000..adb3ac3 --- /dev/null +++ b/fixtures/invalid/suspect-witness/expect.json @@ -0,0 +1 @@ +{"note":"A suspect whose second reading names a witness the package does not have.","rule":"doc.suspect.witness"} diff --git a/fixtures/invalid/suspect-witness/package/MANIFEST.json b/fixtures/invalid/suspect-witness/package/MANIFEST.json new file mode 100644 index 0000000..0c03fe3 --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/MANIFEST.json @@ -0,0 +1 @@ +{"doc":"scienting-doc/1","files":[{"path":"document.json","sha256":"e10c4aaab285d02d859ed2cf590fa9add9b77b0fab7a2b71b8c82f61b310a031","size":764},{"path":"nodes.jsonl","sha256":"6ce54e0d837dceb4a2a39c1b2618cc66e16684b93f71b93ab4e6e709a90b0e52","size":11295},{"path":"provenance.json","sha256":"4bc7649bf6d3113f6c72b4016713654f57c8f03069632eabd1e6025c05cb9dac","size":226},{"path":"receipts.jsonl","sha256":"f745067d43480bbd467c5d8f11ff13eec198fac821664894ad8bd90b8b7cf0b3","size":2182},{"path":"references.jsonl","sha256":"59908a8aff6a2d5800747a1f45c6c2943ea1e31e7e1f10b6348967d30f16b87e","size":60},{"path":"roster.json","sha256":"3c71b0c007924cbacd3b4af02485214ab904db8ff8fca133ab32e67c5c76d481","size":6941},{"path":"suspects.jsonl","sha256":"62c33a53ec529721a3d9514d6dd6401b3cd45f8446ab4b06f2f78051610e7cfa","size":129},{"path":"witnesses.json","sha256":"83ac4411c2efe65e0f7fc4c9336abbef6977cd32b0f90de029a7f8516afc3e97","size":750}],"format":"scienting-package/1","key":"tiny2026tiny-j4qm"} diff --git a/fixtures/invalid/suspect-witness/package/document.json b/fixtures/invalid/suspect-witness/package/document.json new file mode 100644 index 0000000..9a4e921 --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/document.json @@ -0,0 +1 @@ +{"counts":{"leaves":10,"nodes":22},"fidelity":{"counts":{"F0":0,"F1":0,"F2":1,"F3":9},"lowest":"F2"},"format":"scienting-doc/1","hashing":{"algorithm":"sha-256","scheme":"plain"},"history":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","key":"tiny2026tiny-j4qm","kind":"article","language":"en","license":{"spdx":"CC0-1.0"},"meta":{"author":[{"family":"Tiny","given":"T."}],"issued":{"date-parts":[[2026]]},"title":"A tiny example"},"meta_hash":"1062e6079e2871c390f4737f5354448b65fe524d8c09d1c82afaa8b71ba6b3e7","policy":"full","relations":[],"rev":0,"root":"a05328e6d13edba652599aceb58c018dbccf54abd1f878c9f61623017960e1fe","version":{"type":"version-of-record"},"work":{"key":"tiny2026tiny-j4qm","same_as":["doi:10.5555/example.tiny.2026"]}} diff --git a/fixtures/invalid/suspect-witness/package/nodes.jsonl b/fixtures/invalid/suspect-witness/package/nodes.jsonl new file mode 100644 index 0000000..f08057c --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/nodes.jsonl @@ -0,0 +1,22 @@ +{"align_key":"b3108c22ec836d883e5116f7ed8546fa75390eb4aa960d78f2936115f69b6826","id":"s0fn2fj","kind":"document","match_hash":"b6a4e12f700a6214bf02c9a437559e82fccdbecea7b575fa438a7d2e8c33071a","node_hash":"a05328e6d13edba652599aceb58c018dbccf54abd1f878c9f61623017960e1fe","ord":0,"own_hash":"34444fe0e6bdbf3927a579fe3286859d204ea0882a381b7769ffd4905551d24f","parent":null,"text_hash":"bd2abc746d51cf52d57a18632383245d585e29e265426967894eed99bc937561"} +{"align_key":"caf6aee3882b3890996037e35d0ffdfe30b5e66653d45321d7ea0fa9135f1ee9","attrs":{"level":1},"id":"b9wp3tt","kind":"section","label":"1","match_hash":"5a96c01f314c4d8a22fd5e105bebef45df4301886cccbc2538ecefddfcb1d405","node_hash":"ea9a1409bbf0f0f7bd8d508bfdbbf7d14a22ab1e3da011eafbfcd4e14ae7e4fd","ord":0,"own_hash":"65da097b48f8237e8d171bfbd0053ff0e3795c0532d5522d23a92b53430956f4","parent":"s0fn2fj","text_hash":"7a97628fe9643c0fca3614a077e2e631c3fee81d3733ba1564292df65a294573"} +{"align_key":"c099142bc3186ded72786ba27e9ea6d2da240fb9fd3fe79b479ecf8e734b2850","id":"vet7k0e","kind":"heading","match_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd","node_hash":"a3c82ab1e44bf4cffa1a59507f8bddc39a1ad191321ee803d0d60d96609a44c7","ord":0,"own_hash":"e813ff86a8920cef741e548eb5df545774b520d2fadbdda6e2b872a6729b9d26","parent":"b9wp3tt","text_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd"} +{"align_key":"c099142bc3186ded72786ba27e9ea6d2da240fb9fd3fe79b479ecf8e734b2850","content":[{"kind":"text","text":"Results"}],"fidelity":"F2","id":"em5pd9v","kind":"sentence","match_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd","node_hash":"293fc03ff4e39d7532467e4417ffcdb15f470f0bd3658dc1c021645ad53c9bef","ord":0,"own_hash":"c55f401232b19ff1133d77eac6c72868975064f177fc2d31ab8a181303c30599","parent":"vet7k0e","text_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd"} +{"align_key":"610c235321e38145abff21e31eaab303d85a6ce0d1b9340a599dc821a04922f6","id":"ytp1e5n","kind":"paragraph","match_hash":"bcb4902ec69f2a7a724384f561ee9aa0e214c824c5882c1ad92777bee680aad4","node_hash":"7667a3a4bc2e8fec1dc7af53e19c414952f9ce926b1e37e8d9adca66dd8f8d1d","ord":1,"own_hash":"494e069be36a18ecaabf5a63be7c31617ade7a5460218ca83a2de3d8d8f964e0","parent":"b9wp3tt","text_hash":"f589ee914c46c4aea7ce2de49d925c433c914121f576b81a3eab692f17f179a3"} +{"align_key":"8f21603c39df123eb8680362184274aac9bdf1dbd485b3bba230960179071c95","content":[{"kind":"text","text":"The value is 42.0 at 300 K."}],"fidelity":"F2","id":"jvnys4s","kind":"sentence","match_hash":"3bcf9fa8412dca28be91d6423460233ddedd7add6c324d8a0f2b26038567d3d3","node_hash":"f5761f338b2878dff39a6ee724a75b13626dd625a0df96cf1cc6f9e8258cb7cc","ord":0,"own_hash":"7f0b9190d076cf01b51a8be863a51e9a8eeaab5d3ec8656202ff4d41578d4d82","parent":"ytp1e5n","text_hash":"3bcf9fa8412dca28be91d6423460233ddedd7add6c324d8a0f2b26038567d3d3"} +{"align_key":"8e7fe271cd1bab88d86c8900fe3ffae0ce95c55af6659df1fca278d359296e2d","content":[{"kind":"text","text":"See "},{"kind":"xref","target":"tb49qpc","text":"Table 1"},{"kind":"text","text":" and the note."},{"kind":"cite","refs":["r3e21zr"],"text":"1"}],"fidelity":"F2","id":"mzwwhxj","kind":"sentence","match_hash":"9894a48790a8561966e68672fa13da20541d25a5178653720648f62cbc9ad36b","node_hash":"8f3e5338ba3c97efca4cef5c60d849757aed6334ee27ab807fa71d969089e46e","ord":1,"own_hash":"02ed6d50b8ac607b59e3f7b5116ee6143ed7e69af25b19c4037aa799bc330a88","parent":"ytp1e5n","text_hash":"5b1de19c9eb1399e54bde7e96f051cac8b337be1f307fac320292b8db7e39da7"} +{"align_key":"282eea00b316a6950fb7179f6289a8b98359019e11cb23a3ff426cdecad90165","id":"exq1ter","kind":"paragraph","match_hash":"da88f35d24faf0099610c720953c4919647a5dfb9003889a15c464cd9c540fb1","node_hash":"062dd26bfb226ca9811d0dfb013df211cb201dfe7b4b25f9aba000f3212298d5","ord":2,"own_hash":"494e069be36a18ecaabf5a63be7c31617ade7a5460218ca83a2de3d8d8f964e0","parent":"b9wp3tt","text_hash":"da88f35d24faf0099610c720953c4919647a5dfb9003889a15c464cd9c540fb1"} +{"align_key":"b8074a8bc013aa007023c4cce7334f1e1f22384d57a526ca22fd2f5589d5ab17","content":[{"kind":"text","text":"A second paragraph."}],"fidelity":"F2","id":"j663d4h","kind":"sentence","match_hash":"2e8313b033de5e17878ae778d6293bbd6e197a072f36acf7fe0840a690a7f31e","node_hash":"443539440dca73149ca75276492334ce9a5a6c50070137e8dc7bf414b75a27ab","ord":0,"own_hash":"acee4b6b1399bfde1dc67dea8d8c86b0db961d2af516ae4aaee26f294261456f","parent":"exq1ter","text_hash":"2e8313b033de5e17878ae778d6293bbd6e197a072f36acf7fe0840a690a7f31e"} +{"align_key":"437c101749b0481e615fc7eed8dcacb536b65400c7e0695ad4cbb2bf2dae488c","content":[{"kind":"text","text":"It ends here."}],"fidelity":"F2","id":"rsw24qc","kind":"sentence","match_hash":"2888dbfa0c08fd65c58c923b920860c76e13913d9fa93b03eccbbf7bdf53b882","node_hash":"caeb3ae57d0821b8ac961c9507debbcda2b118b16cdc010b83a143a782058d57","ord":1,"own_hash":"705f77e8520c8c5ce3a7a495a20d8a9c55baa9801e49a55c341f5de79eeede75","parent":"exq1ter","text_hash":"2888dbfa0c08fd65c58c923b920860c76e13913d9fa93b03eccbbf7bdf53b882"} +{"align_key":"cb0107aa0d562e8bd1188c33dc1ef5a1d07808e51edeabaa87465556d11b6ec8","attrs":{"columns":[{"header_path":["af5b51r"],"id":"pj5sk05"},{"header_path":["ajqxkgh"],"id":"nymcs5q"}]},"id":"tb49qpc","kind":"table","label":"Table 1","match_hash":"1951e0b61441d7b817b4967b36eee007f8e7525ac75fd359437eddb871e59f72","node_hash":"154c910857f33c152fdc78e42582510327055e62da9a25ab10c3891eeaf69627","ord":3,"own_hash":"fb4cc66080def0b42a06f7aa566a24957c7345c77b8d873ca5c790704a890f47","parent":"b9wp3tt","text_hash":"1951e0b61441d7b817b4967b36eee007f8e7525ac75fd359437eddb871e59f72"} +{"align_key":"769a4e6d0003189c7e96c5d9b7e810a0d11c3a12832527ec94b0f86d277f51ca","id":"t2ps2mp","kind":"table-head","match_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19","node_hash":"6d8056adfb50dc0b65a2a65b47eccc5f47d5482a8dfbefa0d8a06f6641a4be3a","ord":0,"own_hash":"c5749bee3a4b981fb79cf6db125ebb416dce5a83613a00feecd9d4cf75df9bb4","parent":"tb49qpc","text_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19"} +{"align_key":"769a4e6d0003189c7e96c5d9b7e810a0d11c3a12832527ec94b0f86d277f51ca","id":"rr9rtf5","kind":"row","match_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19","node_hash":"8bd1fa377ec390fdc36e17baa8c3a39d2df7ef2a53cd0c0dc745ae40e98f99f8","ord":0,"own_hash":"6c7c11bf1b6260162c4141f1468aa6869f5c33359900497e2cdb1a051488bd62","parent":"t2ps2mp","text_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19"} +{"align_key":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881","attrs":{"colspan":1,"column":"pj5sk05","role":"column-header","rowspan":1},"content":[{"kind":"text","text":"x"}],"fidelity":"F1","id":"af5b51r","kind":"cell","match_hash":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881","node_hash":"59d1c20766a77bacde2aae80a134c7156fbcc708c140a3bb2d2a67f78bd9c6f5","ord":0,"own_hash":"c68933b07bc7c5b806892ae36b3e4c3205a0a2066fd6f8255578f7d1ca86dc10","parent":"rr9rtf5","text_hash":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"} +{"align_key":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa","attrs":{"colspan":1,"column":"nymcs5q","role":"column-header","rowspan":1},"content":[{"kind":"text","text":"y"}],"fidelity":"F1","id":"ajqxkgh","kind":"cell","match_hash":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa","node_hash":"838e3fba38f0797d40522b7063cb5c2962e32a019c09c19f09849ced73079a87","ord":1,"own_hash":"28df42bf1dcd44b053275ed014bd1599914f17adabc0c7fdb3e4247dc648ac49","parent":"rr9rtf5","text_hash":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa"} +{"align_key":"b902cc4550838229a710bfec4c38cbc7eb11082367a409df9135e7f007a96bda","id":"dmsstkg","kind":"table-body","match_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738","node_hash":"baf17577660aa6f0ba9c8703d30816d77cfce7dbf0ea3257ed830fa9ab94ce9d","ord":1,"own_hash":"b229a7f92314227bf225ad27010a546e7c2373f4b30b7f6ec372e568cbacb6aa","parent":"tb49qpc","text_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738"} +{"align_key":"b902cc4550838229a710bfec4c38cbc7eb11082367a409df9135e7f007a96bda","attrs":{"header_path":["rdrqkv7"]},"id":"r3e7hfs","kind":"row","match_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738","node_hash":"97a1f5b27bbfe8c9d2cec7212b519b4993506fc8080be3eff02f3aa31515a19d","ord":0,"own_hash":"6c7c11bf1b6260162c4141f1468aa6869f5c33359900497e2cdb1a051488bd62","parent":"dmsstkg","text_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738"} +{"align_key":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b","attrs":{"colspan":1,"column":"pj5sk05","role":"row-header","rowspan":1},"content":[{"kind":"text","text":"1"}],"fidelity":"F1","id":"rdrqkv7","kind":"cell","match_hash":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b","node_hash":"87da3bd568286f3d0578d8a813496d15faafb62f4db44b07aa1c29e52cda34df","ord":0,"own_hash":"fb474321001657b59bccd3cae0987c544a4ef9a27214626d77695f4db237aa93","parent":"r3e7hfs","text_hash":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b"} +{"align_key":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61","attrs":{"colspan":1,"column":"nymcs5q","role":"data","rowspan":1},"content":[{"kind":"text","text":"2.5"}],"fidelity":"F1","id":"f463wj7","kind":"cell","match_hash":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61","node_hash":"f1cd6d5339c84d5c36f8e1ea484a8d837f3200f05d1b106f6119dfa30d846bfd","ord":1,"own_hash":"088eff622eb3086d68c4433cf5e1783dc9f2a6724c3536c31ec0bf80b5f89db5","parent":"r3e7hfs","text_hash":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61"} +{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","id":"aw97y35","kind":"back","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"fcb8027f7c89115f048b9c5d6770847248f4e4fbc61cbc8ea0b04079b4d58350","ord":1,"own_hash":"bbb972bdc024231b505f4d8ce5e711fbb75c8b5ffd1b5d42ee3fb8586b85f867","parent":"s0fn2fj","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"} +{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","id":"yd513mw","kind":"references","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"8b8a9000e984c3efeb27fc40c8fbadd7349dd64860aa276dcd2a25a57ce52e87","ord":0,"own_hash":"2a5302e2041d0e59458fea212dd51b89299901318bfceeeff227c1a07d3f5d1a","parent":"aw97y35","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"} +{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","content":[{"kind":"text","text":"1. An Example, 2020."}],"fidelity":"F2","id":"r3e21zr","kind":"reference","label":"1","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"71b1af6dd6a56f24cb3b338e0fe99df06b9154ecdf4b391b00b7a7e61e23a8ed","ord":0,"own_hash":"f7c9bbc8c7456e6dc435feeaaa02cbfdbe487523b020105793fb11b91d0894f5","parent":"yd513mw","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"} diff --git a/fixtures/invalid/suspect-witness/package/provenance.json b/fixtures/invalid/suspect-witness/package/provenance.json new file mode 100644 index 0000000..961e28b --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/provenance.json @@ -0,0 +1 @@ +{"note":"Built by hand for the conformance suite; every reading is synthetic.","stages":[{"processor":"local","stage":"hand","tool":"paperspec tools/build.py"}],"tools":[{"name":"paperspec tools/build.py","version":"0.1.0"}]} diff --git a/fixtures/invalid/suspect-witness/package/receipts.jsonl b/fixtures/invalid/suspect-witness/package/receipts.jsonl new file mode 100644 index 0000000..d0952d1 --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/receipts.jsonl @@ -0,0 +1 @@ +{"at":"2026-10-11T12:00:00Z","by":"ana","doc":"tiny2026tiny-j4qm","evidence":[{"kind":"crop","node":"af5b51r","sha256":"78b08276b5122feed3829e8dd19684fa8a4a41f79cb83a499d40341e09f5d658"},{"kind":"crop","node":"ajqxkgh","sha256":"398bab4177685c88eef399d814d34c81baa556d58cd508bd5ddb33ac53a5cbfa"},{"kind":"crop","node":"f463wj7","sha256":"88b1e820b569c3a0fd025a6b6c502271ed73bbce720dd8a2e416d6a5dd0e9d67"},{"kind":"crop","node":"j663d4h","sha256":"d6484c89588a560f64bbceca108f5446e47bb5cb85521fbffc9d6d37f252e5c7"},{"kind":"crop","node":"jvnys4s","sha256":"3139fb30b88062bd0c6bbb028c4c57fc748218ce1d7a612d282e292d6a02ffe9"},{"kind":"crop","node":"mzwwhxj","sha256":"83a7fcf09321f8a644c58120f80dede5d73d89f50d080ccbb143ec5b485e3088"},{"kind":"crop","node":"r3e21zr","sha256":"ff809e11101fa6ff5422e5232e83142c1a43dada0e56d63ec03fbce3474dd1aa"},{"kind":"crop","node":"rdrqkv7","sha256":"c0699aceb344812ebebb1414838641f0761184d5113dcc870d07c721fd289437"},{"kind":"crop","node":"rsw24qc","sha256":"b4c3b133c707b1a9a2cec2efe5c17731bada0d6bace5b1fb3590c5c5ab18bb92"},{"kind":"witness","sha256":"fda3f1e9bb3538cd85dcc42281e00fd045f726652664439ce5a15d8489425b21"}],"format":"scienting-receipt/1","id":"k4wt522d9zvjr","level":"F3","nodes":[["af5b51r","59d1c20766a77bacde2aae80a134c7156fbcc708c140a3bb2d2a67f78bd9c6f5"],["ajqxkgh","838e3fba38f0797d40522b7063cb5c2962e32a019c09c19f09849ced73079a87"],["f463wj7","f1cd6d5339c84d5c36f8e1ea484a8d837f3200f05d1b106f6119dfa30d846bfd"],["j663d4h","443539440dca73149ca75276492334ce9a5a6c50070137e8dc7bf414b75a27ab"],["jvnys4s","f5761f338b2878dff39a6ee724a75b13626dd625a0df96cf1cc6f9e8258cb7cc"],["mzwwhxj","8f3e5338ba3c97efca4cef5c60d849757aed6334ee27ab807fa71d969089e46e"],["r3e21zr","71b1af6dd6a56f24cb3b338e0fe99df06b9154ecdf4b391b00b7a7e61e23a8ed"],["rdrqkv7","87da3bd568286f3d0578d8a813496d15faafb62f4db44b07aa1c29e52cda34df"],["rsw24qc","caeb3ae57d0821b8ac961c9507debbcda2b118b16cdc010b83a143a782058d57"]],"rev":0,"root":"15d9c8bd5472afd7064578e5663d3b7fd21249adc6cdb40fcbe1b790e3fae801","scope":"all","signature":{"key":"ana","scheme":"ed25519","value":"/xOfCboOpgu29ztKi22IU9crgC5LYetrHOMpQtqj4QV5dVP73n/ze94hyPknYO5qNPgBmYRrbJZBIIHdwqxoAA=="}} diff --git a/fixtures/invalid/suspect-witness/package/references.jsonl b/fixtures/invalid/suspect-witness/package/references.jsonl new file mode 100644 index 0000000..af52739 --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/references.jsonl @@ -0,0 +1 @@ +{"label":"1","node":"r3e21zr","resolution":{"type":"none"}} diff --git a/fixtures/invalid/suspect-witness/package/roster.json b/fixtures/invalid/suspect-witness/package/roster.json new file mode 100644 index 0000000..36584ed --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/roster.json @@ -0,0 +1 @@ +{"format":"scienting-roster/1","keys":[{"did":"did:key:z6Mkj7iUX8QG7zkAtNaV78CUXcenhD87qWwQbc7NU5aVXyW9","id":"ana","person":{"name":"Ana Example"},"public":"RUiz5j+y0QwgOtsQpqXrlq+gy3c/JFnkxwvlWUNX4Ao=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"owner","scheme":"ed25519"},{"did":"did:key:z6MkhemKN7rMQcgt6wT2eWPyx3vFRS1RUoRziLMnLq2vZ9yi","id":"ben","person":{"name":"Ben Example"},"public":"L4WMLs7xLjBLRsxS/EYpD194puMBpADybcfFEWxBkWk=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"editor","scheme":"ed25519"},{"did":"did:key:z6MkrBf2w3MUroaNBiHTQJzH4e1w31jpcosd5vNxjikijndU","id":"ben-spot","person":{"name":"Ben Example"},"public":"rkzxigjCXg869PGSFAVr4EDXXRmwk+S/+wJ93lrG4dc=","purposes":["spot-check"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"editor","scheme":"ed25519"},{"did":"did:key:z6Mkgrc9HCeEUL5tz1bVfuuDeSm6mP6hyWUX5ivhgReSbEQM","id":"cara","person":{"name":"Cara Example"},"public":"I7J1GqG5CHOc0LHYZPNNto1Me2lJUzBVn2A7GrUGbZ4=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"reader","scheme":"ed25519"},{"did":"did:key:z4xcFfnTC6XJy4gzg9whfyMzcnxV7yM2XafMFynKXhAEvMgjfQCxSnjZaa53arz2iNo8v35ZN5SdAUcbFkbuvftcMczRT2CPhYsXkNYZhDzXWQVS5KZZkhnMDmwChWfgRHu4Bs6KHyEmvzwesTioW5yS3YtfR7mUPJAwWSRq9f23HRxzenGpVVBcVjpd8YDZ4Y1ZamcUfesnB6pGLrPtLDvGM7Te1cJWoRG4m3TetEF3i9V3W4CH43nouCwUPbjy7dpTNEWiCqaTPiPhV8UEWBD8b9e1729PwoUFyAFreuq6wvhphLKGHhFCFujrUtqDZPs6FFF1Q7Lpt9eFcjbxrZSAYsh9VTfxaUeQ4HmFqzrg28EEjJzxxADZyjTXDsfwskRbibHSms9bbN2xK6GQcUxE2mrA2ULfYwsk5RQsVULUTj3pMMXpxRqP4zerzui2ZRZ3SqeNUwqnhmF861pEtcemZPY6ivcX41QK3ekCtpc1HndtY9xua1Lrhs4jd6qEbxHWzBPHok6qTpyxnn7ubZssBqSMz3uNrou1ayZGDn1ayAXtzCJPjAcxdNX1dpm5vyDB8GYtNKsQHuiX5xUd2EMAVfq4cFrZ6ncY1zrG5exH6mhJYbZfW7BgXxq7GsnzAAGkzgHCRADe5EEm1Hj1YdmVBTiaGsPGYkF4Mcrw3T2j3AcPJCzPLRNjKBKXyHAcf3VPUmreaHd1GoA5zqvtANSPmppY2AhCAj8u6n92fiyBgrvJTh4hyvtMFcwaa9vKnY9rBS4n5XGpdwKQe11Tk5sah8VZc2PyH2WPBU3DmhZSwUkKHRQQQQMXjrLjuMtygzxWBgrJAcWRPzxmzHZEj5x9GgFZb4dPDwqVXsgR25KrpuyQKcxfbxzQoRprnhu1ec8SfX5zeUo282Si9dBZ3JVdxzYHzP6xSiuBeHHFMvMwb1MKUq2dMi9vWdg2eCDo5wTvSaTVcdYGtZsT7HMswWtxx9GLw1RafYRcNqMhBhCbRdJszUKjQKwNNZbaryRs4kTYicuE24PWbqfxdJnrzrLbXnVBDMNHjX8ESXSKTMnAoE1AiLgjdj3zbBrY39Pxc72EikA7GvP2qxKZBkJ6G2RRWCw9d3nZKmTmQqP5PP9HgdtSTnZhNTJqBkwL3iqfoWxtbkLjLLSQc3dbFpmhe4kAh7VUUDcLzjb4nGzCbr5U1YoqAoLnSC9BPijnfMHxBP6i8cR762ryxPwe7HdGk62DVYoqxbwz8ygrDPjdPi3t11PpHp51GxuhMkoR3GFxAdMEv8j4tVpJ6qjefULEFTtRPzZ5kTE57RTxC4wg62qcYwupWx4dvvB2BR9ZGnNwCojpBYByeeYETrgm4HJjjS3SH64sbpyqz7336YMAcWyQFc9BeqjQHHUHS8CJ82N88apbPPE8N9X76DQ8PgwACtR4D5ofE9o14kpdVZFV2YMtmnKCybNLn3atspK6T1ZPYB3vQ1c4DbzZFgCpm6izEzxsY17in79ieo3cY965Cr5mkxca29VxcTaEtovdu9sKAL4VgCZzWkps5nnimX5JBEdVKYxTtMb6CcTkdWGFtXBTBuAMTpRNLXG5nG4bq9XXLkFS4oXpVS7KayqAMtrrqttpgwdAPNuzd6E4JskxdenwMq6qzMTDKoMqGDVEKkwCxCcbUtNpHGMLWMMk9MDrsPvcU8u2C89UmJmsfS5gQqswkz6TfXvthM3wPcuKaX7TUJJRzYstWsXRKUaPqqT3ppftCbXPFgia3mXkA5bk9Y3GWcTADGTnzKkTzbnL6hsgQW1FyvsFwUiWUJd5weeHkfaHt8iV3PKy99S6d831iLCfFWM6bYZgEBvyx5seHSpiaXRfSv4A2yDsf4Tb4EVsyDUJPUmZLnbFkJmw46A6PoQHdwfBbXWV47DFM6w5mKXCPmKzuAG1Vmw7D36n4oHd91r1VzTynVnhD1EA5UVh4qhiw2CnWgo2uAPxEpLYnvrCLiVprnzdPzvZZqMUxXRFWHwfshJDARRLvP933693aWxa5PeDNgMjZYgKJBV5e66Z1Mk5rYegEuSq7uyz8Mw3iuidtJeqBugbNCYeNdmeBPgKux8HcuFNqQaVwhP9vW8vyaU7DvuKEeXs1uyAgtk9RCua2Mx8rQqUwQigWNjXaJoSYqMT9SVEGUSYAGQiLnMFENqG1r87uhxV58hTwR8QED9cQkStFXeuZAMY6FWTZCsD5juC48KdqoUQYr77iBy7fW2quQNdPvwEUdhh1mzSTX2PPubCbm9rcdBxBQ2CBZizn1EsbPMic1X39t17scRE5F4Yank18EF4aUmJwwNHx5m1WKDQMZ1kJQRyhtRvAHTPqBdJCw5SHaEqBVpEQHUZwBKsQoTRj4Gk1opv6XfczoK3MLuprTMSx2maw5Gv6LhF2SWUvEXMMqoeqVzyP9g11dumkKQLMiAQa2VGaa7EF7Foj2ZDUhPM6PPiffHMhDMwWhRBywg3SC6aP9zz2TdMXweHv8rags9chFZ217aAP3bcCf5i1VZqPT9b8Ea7ykjjUy6PNwiSforRXN46suuRFREuhx6DuWDojythjH73wMAeKKx23rF4QfdUxzkC2mmmS4dq8Xx7Bda4e31EGab4t81Jk4dAVGjSrZC9wx89CAjovhoejgDTKNsN3ri4GjW9UD","id":"dana","person":{"name":"Dana Example"},"public":"lQ6M2DulKY4RkQLGTPGZD+CHlOOib4kBkwlccVNWRDPco1k5kHtelT/yUP4+CoAQnHnLu96vpeWztaee/9RmjR3bstGj+P5tOSrwFlPuyNUPOWPNjwCQLrVEyeUGBaA4+HRvFwK7C4OHB8EJWYtC0OuH+tRPlCdLpxlVcZZcM7dsZYe2BDNenbWfp4TeX2Oa+dThlceDY0m9MewwqUrd5xdo9tc8PS9R9o0ZUWf0vDw7Fc5AcsWwbf76lDLvIbyd0aCaHJ7eaK6DFsBuZFKVdlZlH0inJciAMF8dPoRuE+A0VlZTEOsOivNWprM5iLHzXse3zSoHLldWr0t4hWxHH3MBQ57Ucee8RTU6hFRfqfFRl6zJQyRZOFZZoYD9QwXkR3NmUzV/FcsOMHpbgTcY7O0TAajPnfpSTp+XdyPVBR7TlTsturUCpryP0ZX1S3VJk2IaneB123thtcOtWsbqdT3Bb846H7yudI95WVTixIS/KIwoCbekSf8Y9+iW/xRVSWLRVc1gpigDxXfieSA7bI+qiUBjiKR7QU1KFYZA58VcOQhzeMLY6an34rsW4WHAwzzSv8Q6vrwpn8Ct9fTsIeiZsWZLueNkIUfjR0G1bHHoT2RHKton0AMRjOUGSKUUB4DNZ5xzXvC0jvUMWlgHB9z2wBCsIkIDVWqSA2DD0xeIY2wSUVPwPc1MnBAf/Fx3poweUlcPWglR2VaO9ematC4hfLxxkGldGQt3C6ZLPsm96S7o6j88rwOqkDNIR1pNJOepgciJH/w4Xwtg9gJ52CpdlCeSllRQD74l95ysmZ2D/gLweNoUt61P0RYQQelio8VnPUNB1e2uK7zW9joG1k59CjJGQNvuZr6JkQ9+Hxo5xKEOy9KcpT6t48wTzbz3YMFDxTNl0/DzL8qdTU2UrSW6/TFgPLokbAQN00yhw0lg8cR/e1m+ujFyBeesL3luwE6HNcIyK5Y/+To04vR3OD7Uz8cnABxWyOeSBm2R2bu2fxeN0Hlv+qhoOYtbZ1DKx/aES4z51jATynEIdMIMSNhmVDoDiGODh9ocenzg/DUarjIzWltPFRyQj+tnFGTXp7C87s3yAf248ZB0HHhK1GCppxB4hsv89nNtWH1JdWjsc243s3f0WOZtJ1f+9yeybViJCW47ez7HkwtTq2AVeg+lHrb4TPCL5pBKwUOj5GVe2DzMFBtjilcl/ED4gev7DK383iyB4tGVZa1ALPKUyiyKV3sCRV/mN5DSGeASafI6OOjjUKifYCJaxOHFQGr6zvfhVj9L2Cl/hthxplTWoLpo4I7+oLPQJIrOMPf8FHLtRLAYthsxT3K02mVtUMhrviaVh5/kJKMMd4iRc8YXloC0r8roivoEPr5SzDprSffh8a2N2KRAHd3yrp4z7RxlpWKPWr5whD56q279S70A9r/r5r0bVzwabhKkTSJxc2LN72QondANsGFDYPD3Rm+FwSO2JwwgIG8GQxBbMWPOEqD0qDmaebNfP/EpbeF8mvA8v4e58CIv1LaHZuEky6vc16woEYHm11UuH6ChHbWxCoDXJsVM/jPme1vPlic38i9XYsE2c199gP1ikBGQANlryeIWH8wXtPUXNflNecMqIQkWAWSK/8jujgtylBQ/CklRyeW0+vyxV5eo8S8TfnonyG2eiHN1VS3HGLSidcYRi54IsOMtuR773QoLPZeRXlsNWO5kc3xcHU7K6olsMNtQhaFnkf3GATh9H80GUvEMK9u5yMUrCoYWuUy7JQhjsl+B1QbAFftdDu8ESgTOIKmFOLg+F9CH4lJrCqnwUdCs93zbJem2++Y64HRFLSUXwlhEUuQVugX7BejmcMYWMjYXROlatnznqYXcWW2uxx6ZNxQv0eB/CN1i4q1G8y9XUh4wz/caY8ly+eaTU5J4wGHfnUDpssu+TUvE5PGUs1gW6FswHlf1IXEnvKDIXKajbJgYnjEekVG8FxcSUc/10Mw/vqk/GQqdMOguf3mShlUIbtX2JVrA5ClZ9qza4HEXC7RkmxJLyc5dDyX/CuFdoDzrVu+1+JqRsFHGeLCo9rNvIQjvG09GQX6uDUavi25hCcOgug2DI0PuVLjcESFDyaYNdKo9GsiCEBpLPglp9viA2vX2kG6P2pidgsGfcBgdGuYGQbd23Qu9QpOd8AAuyWsm0dX6hKO2ChoAPVQxDNcj9N6UjumggVQ6zOuC0+gOcao/aPs/eH9agchdtnWKuL1clo4FduP4sdDtTvnJLFYnXONM//LSnjqSyBc7rzjJ2cxzPWQtLFawErzHXukl2eeP/xpDvLU0G6JroeJqBitHIoK00FGdxTUnrJBgy1DnBoIgUhxOvb1wtnx/O4DDS5tJZBDbl9XVbRmSfcljfQ5mB38hQ51CuYodHE9q12Bk1FtSwbVKvPHGb+9EdkbktJKRgdJubAQ+ajOCyMjEDCp9xLeKnlFL+WesI0Ro+rnmQL0qXVzkhZj8LP4a8yFYieIXCApLveflfv2yRJf3a/EML7yIgoEuzejddFrcbupWxDRqwS1YcLDkvGBEzjp8zuVnr72GfaFW/nvU9V9P/Z0fFS5hZhG5MnEp0Qzjsy/eKRI=","purposes":["sign"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"editor","scheme":"ml-dsa-65"},{"did":"did:key:z6MkiHaTrUvYfXFtTw39oiWrgan52dRBHYgm9WeoiEPwMotg","id":"wit","person":{"name":"Open Witness (fictional)"},"public":"OPPD6UEKZ7IObfmh/pzKS29cNjnz9WbKMQivMBXrlC0=","purposes":["witness"],"registered_at":"2026-09-01T00:00:00Z","revoked_at":null,"role":"guest","scheme":"ed25519"}],"store":"smithlab"} diff --git a/fixtures/invalid/suspect-witness/package/suspects.jsonl b/fixtures/invalid/suspect-witness/package/suspects.jsonl new file mode 100644 index 0000000..861984c --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/suspects.jsonl @@ -0,0 +1 @@ +{"a":"42.0","b":"42.5","format":"scienting-suspect/1","id":"aaaaaaaaaaaaa","kind":"number","node":"jvnys4s","witness":"zzzzzzz"} diff --git a/fixtures/invalid/suspect-witness/package/witnesses.json b/fixtures/invalid/suspect-witness/package/witnesses.json new file mode 100644 index 0000000..a7b09bb --- /dev/null +++ b/fixtures/invalid/suspect-witness/package/witnesses.json @@ -0,0 +1 @@ +{"witnesses":[{"id":"nmfryke","kind":"pdf","media_type":"application/pdf","origin":{"source":"author"},"pages":1,"rights":{"asserted_by":{"by":"ana","evidence":[{"kind":"statement","note":"The team's own draft."}]},"at":"2026-09-15T12:00:00Z","basis":"own-work","format":"scienting-rights/1","license":{"spdx":"CC-BY-4.0"},"uses":{"accessible-copy":true,"ai":{"external/local":true,"external/outside":true,"external/outside-no-retention":true,"external/tenancy":true,"internal/local":true,"internal/outside":true,"internal/outside-no-retention":true,"internal/tenancy":true},"quote-public":true,"read":true,"redistribute":true,"tdm":true},"valid_until":null},"sha256":"fda3f1e9bb3538cd85dcc42281e00fd045f726652664439ce5a15d8489425b21","size":1000}]} diff --git a/pyproject.toml b/pyproject.toml index d397cb0..b98ddbd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -62,23 +62,39 @@ dev = [ "twine>=6.1", ] -# The code is `src/paperspec`. The schemas, SQL and vectors stay at the repository root, where the -# specification links to them and the website serves the schemas at their `$id` URLs, and travel with the -# package as data subpackages, so a consumer reads them with importlib.resources. That mapping is why the -# packages are listed rather than found. +# The code is `src/paperspec`. The schemas, SQL, vectors, registries and JSON-LD context stay at the +# repository root, where the specification links to them and the website serves the schemas at their `$id` +# URLs, and travel with the package as data subpackages, so a consumer reads them with importlib.resources. +# That mapping is why the packages are listed rather than found. [tool.setuptools] -packages = ["paperspec", "paperspec.cli", "paperspec.schemas", "paperspec.sql", "paperspec.vectors"] +packages = [ + "paperspec", + "paperspec.cli", + "paperspec.schemas", + "paperspec.sql", + "paperspec.vectors", + "paperspec.registry", + "paperspec.registry.stamps", + "paperspec.registry.stages", + "paperspec.context", +] [tool.setuptools.package-dir] "" = "src" "paperspec.schemas" = "schemas" "paperspec.sql" = "sql" "paperspec.vectors" = "vectors" +"paperspec.registry" = "registry" +"paperspec.context" = "context" [tool.setuptools.package-data] "paperspec.schemas" = ["*.json"] "paperspec.sql" = ["*.sql"] "paperspec.vectors" = ["*.jsonl"] +"paperspec.registry" = ["*.json"] +"paperspec.registry.stamps" = ["*.json"] +"paperspec.registry.stages" = ["*.json"] +"paperspec.context" = ["*.jsonld"] # The package version comes from git tags (v0.1.1 builds as 0.1.1, later commits as 0.1.2.devN); CONTRIBUTING.md # says how it relates to the format version that CHANGELOG.md and pixi.toml carry. A source tree without git diff --git a/registry/rules.json b/registry/rules.json index 89b8573..677f6ba 100644 --- a/registry/rules.json +++ b/registry/rules.json @@ -113,6 +113,41 @@ "description": "document.json's rev is not the number of corrections.", "chapter": "spec/22-conformance.md" }, + { + "id": "doc.dataset.anchor", + "description": "A line of a dataset's anchors.jsonl does not match its schema, names no value of its resource's CSV, or has an anchor that does not resolve as current.", + "chapter": "spec/15-datasets.md" + }, + { + "id": "doc.dataset.file", + "description": "A dataset has no datapackage.json, or one of its files cannot be read: JSON that does not parse, anchors.jsonl not in canonical form, a CSV that does not read, or a resource whose path names no file of the dataset.", + "chapter": "spec/15-datasets.md" + }, + { + "id": "doc.dataset.id", + "description": "A dataset's folder is not named by its ID.", + "chapter": "spec/15-datasets.md" + }, + { + "id": "doc.dataset.key", + "description": "The primary key of a dataset's CSV resource is not unique.", + "chapter": "spec/15-datasets.md" + }, + { + "id": "doc.dataset.schema", + "description": "A dataset's datapackage.json does not match dataset.schema.json.", + "chapter": "spec/15-datasets.md" + }, + { + "id": "doc.dataset.unanchored", + "description": "A numeric value of a reported resource has no line in anchors.jsonl tying it to what prints it.", + "chapter": "spec/15-datasets.md" + }, + { + "id": "doc.dataset.value", + "description": "A value of a dataset's CSV is not, character for character after normalization, the text its anchor resolves to.", + "chapter": "spec/15-datasets.md" + }, { "id": "doc.derived.math", "description": "A line of derived/math.jsonl names a formula that is not TeX, says it is complete when its tree has an opaque node or the reverse, or lists symbols that are not the sym keys of its tree.", @@ -220,7 +255,7 @@ }, { "id": "doc.math.form", - "description": "A source with a form is not that form's canonical spelling. Only a validator that implements the form can check it; the reference validator does not.", + "description": "A source with a form is not that form's canonical spelling. The reference validator reports a scienting-math/1 source that holds a label or style command, which the form never does, and checks every source with papermath when it is installed.", "chapter": "spec/03-nodes.md" }, { diff --git a/spec/03-nodes.md b/spec/03-nodes.md index 17ba64a..feb85a1 100644 --- a/spec/03-nodes.md +++ b/spec/03-nodes.md @@ -150,4 +150,5 @@ A consumer MAY rely on these properties of a source that carries `form`: A producer writes `form` only on a source it has spelled in that form. A correction that changes such a source writes the new source in the form too, since the gate does not implement the form and cannot check it. A validator that implements the form reports a source with `form` that is not its own canonical spelling as `doc.math.form`. +One that does not can still report a `scienting-math/1` source that holds `\tag`, `\label`, `\nonumber`, `\notag`, a style command such as `\displaystyle`, `\dfrac` or `\tfrac`, which the form never holds ([9.4](09-figures-tables-equations.md#94-the-math-form)); the reference validator does that, and checks every source with papermath when it is installed. `form` enters no content value, so adding it to a source already in the form changes no hash, ID or anchor, while respelling a source in the form changes them as any edit of the source does ([5.2](05-hashes.md#52-own-content)). diff --git a/spec/08-fidelity.md b/spec/08-fidelity.md index b3687e2..0587787 100644 --- a/spec/08-fidelity.md +++ b/spec/08-fidelity.md @@ -11,7 +11,8 @@ Levels apply to leaves. The pipeline records F0, F1 or F2 in a leaf's `fidelity`; F3 comes only from receipts. -A correction that changes a leaf drops it back: the gate's crop evidence makes it F1 at most, until it is corroborated or proofread again. +A correction that changes a leaf drops it back, until it is corroborated or proofread again: to F1 when the correction's evidence includes a crop of the page, which answers it against the page image, and to F0 otherwise. +The leaves it changes are those its record lists in `touched` and `created` ([11.3](11-corrections.md#113-applied-fields)), and a dropped leaf no longer names a witness in `corroborated_by`. An `emend`, `relink` or `resolve` ([11.1](11-corrections.md#111-operations)) never does, because a level measures the reading against its witnesses and the page. An emend corrects the authors, not the reading, and leaves the leaf as printed ([3.6](03-nodes.md#36-the-plain-projection)) as it was; a relink or a resolve changes no text, and the references a citation names are not text a witness corroborates. diff --git a/spec/15-datasets.md b/spec/15-datasets.md index be6e44d..edf2fdc 100644 --- a/spec/15-datasets.md +++ b/spec/15-datasets.md @@ -24,4 +24,10 @@ A digitized value never passes as reported, and its uncertainty is at least the A dataset's content hash is SHA-256 of the canonical JSON of `{"format": "scienting-dataset/1", "files": [[path, sha256, size], …]}` over its files, sorted by path. A dataset can be a source of kind `dataset`, so a claim can rest on a value read from a figure, with that fact visible in its evidence. +A validator checks each dataset of a package against the package. +It reports a descriptor that does not match `dataset.schema.json` (`doc.dataset.schema`), a folder not named by the dataset's ID (`doc.dataset.id`), a file that cannot be read (`doc.dataset.file`), and a CSV resource whose primary key is not unique (`doc.dataset.key`). +Each line of `anchors.jsonl` matches its schema and names a value of its resource, and its anchor resolves as current ([6.5](06-anchors.md#65-resolution)) (`doc.dataset.anchor`), to text that is the value as the CSV holds it once both are normalized ([1.4](01-conventions.md#14-text-normalization-for-hashing)) (`doc.dataset.value`). +Every numeric value of a `reported` resource is anchored (`doc.dataset.unanchored`); a resource's other fields, such as the labels that name its rows, need not be. +A correction that changes or removes an anchored cell makes these fail, so a producer that applies one rebuilds the dataset or leaves it out, as it does a derived file ([12.7](12-package.md#127-derived-files)). + `fixtures/packages/hale2026harmonic-damb/datasets/ds-hale-bond-params` reports Table 1, and the suite checks that every value resolves to the cell that prints it. diff --git a/spec/22-conformance.md b/spec/22-conformance.md index b9d880d..ae22dfd 100644 --- a/spec/22-conformance.md +++ b/spec/22-conformance.md @@ -41,6 +41,7 @@ Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is | `package.` | The manifest, paths, files and schemas. | | `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.`, `doc.math.` | The tree, node content and runs. | | `doc.derived.` | Derived files ([12.7](12-package.md#127-derived-files)). | +| `doc.dataset.` | Datasets ([15](15-datasets.md)). | | `doc.hash.` | The five values, the root and the metadata hash. | | `doc.id.`, `doc.order.` | The ID table and file order. | | `doc.policy.` | Text policies and keyed hashing. | diff --git a/src/paperspec/anchors.py b/src/paperspec/anchors.py index f58398d..9e75391 100644 --- a/src/paperspec/anchors.py +++ b/src/paperspec/anchors.py @@ -11,7 +11,7 @@ Non-normative; see spec/06-anchors.md. import math import re -from collections.abc import Mapping, Sequence +from collections.abc import Collection, Iterable, Mapping, Sequence from fractions import Fraction from typing import Any @@ -206,11 +206,13 @@ def _decimal(fr: Fraction) -> str: return f"{v // 10000}.{v % 10000:04d}" -def _follow(state: State, nid: str) -> tuple[str | None, list[str], str | None]: - """Follow aliases and retirements. Returns (live id or None, chain, via).""" - live = {n["id"] for n in state.nodes} - table = {e["id"]: e for e in state.ids} - chain, via = [], None +def _walk( + live: Collection[str], ids: Iterable[Record], nid: str +) -> tuple[str | None, list[str], str | None]: + """Walk the ID table from `nid` until a live ID, a deletion, a repeat or an unknown ID.""" + table = {e["id"]: e for e in ids} + chain: list[str] = [] + via = None seen = set() while nid not in live: if nid in seen or nid not in table: @@ -227,6 +229,84 @@ def _follow(state: State, nid: str) -> tuple[str | None, list[str], str | None]: return nid, chain, via +def follow(state: State, nid: str) -> tuple[str | None, list[str], str | None]: + """Follow aliases and retirements from a node ID (spec/06 §6.5, "Follow the ID"). + + Args: + state: The document state, with its nodes and ID table. + nid: A node ID or a legacy alias. + + Returns: + A tuple `(live, chain, via)`: the live node ID reached, or `None` when a + deletion ends the chain or an ID is unknown or repeats; the IDs followed, in + order; and the kind of the chain (`retired` when it passed a retirement, + else `alias` when it followed an alias, else `None`). + """ + return _walk({n["id"] for n in state.nodes}, state.ids, nid) + + +# paperdesk's cite.py uses the old private name until it moves to `follow`. +_follow = follow + + +def resolve_address( + nodes: Iterable[Record], + ids: Iterable[Record], + address: str, + values: Mapping[str, Record] | None = None, +) -> Record: + """Resolve a pinned address against nodes and an ID table alone (spec/06 §6.1, §6.5). + + This needs no `State` and no corrections: it follows aliases and retirements + through `ids`, then compares the address's pin, the first six hex digits of a + `node_hash`, with the live node's. It never returns `moved`, `reattached` or + `changed-below`, which need a quote, an anchor or the corrections. + + An empty span such as `[3:3]` is a valid address: the grammar of spec/06 §6.1 + requires only that `start` is not greater than `end`, and `addresses.jsonl` + has `[0:0]`. The node part is any `lowalnum` run, a node ID or an alias, so + a caller that wants only seven-character node IDs checks that itself. + + Args: + nodes: The node records, as `nodes.jsonl` gives them (`id` and `node_hash`). + ids: The ID table records, as `ids.jsonl` gives them. + address: An address `[store/]key#node[@pin][[start:end]]`. + values: The content values of every node, as `hashing.compute` gives them, + used in place of the stored `node_hash` when given. + + Returns: + A record with `result`, `node`, `node_hash`, `via` and `span` as for + `resolve`. `result` is `current` when the live node is reached without a + retirement and the address has no pin or the pin is the start of the + node's `node_hash`; otherwise `stale`, with a `reason`: `not an address`, + `unknown node`, `retired without a successor`, `retired` (the node is the + successor reached), or `the pin differs from the node hash`. + """ + parts = parse_address(address) + if parts is None: + return {"result": "stale", "node": None, "reason": "not an address"} + index = {n["id"]: n for n in nodes} + nid, chain, via = _walk(index, ids, parts["node"]) + out: Record = {"result": "stale", "node": nid} + hash_: str | None = None + if nid is not None: + hash_ = (values[nid] if values is not None else index[nid]).get("node_hash") + out["node_hash"] = hash_ + if chain: + out["via"] = chain + if nid is None: + out["reason"] = "retired without a successor" if chain else "unknown node" + elif via == "retired": + out["reason"] = "retired" + elif parts["pin"] is not None and not (hash_ or "").startswith(parts["pin"]): + out["reason"] = "the pin differs from the node hash" + else: + out["result"] = "current" + if parts["span"] is not None: + out["span"] = parts["span"] + return out + + def _relabel_reverted_own( state: State, doc: Doc, nid: str, anchor: Record ) -> str | None: @@ -282,7 +362,7 @@ def resolve( values = hashing.compute(doc) kind = anchor.get("kind") original = anchor["node"] - nid, chain, via = _follow(state, original) + nid, chain, via = follow(state, original) live_kind = doc.nodes[nid]["kind"] if nid else None table = {e["id"]: e for e in state.ids} # The kind falls back to the retired entry the chain reached (spec/06 §6.5); an diff --git a/src/paperspec/claims.py b/src/paperspec/claims.py new file mode 100644 index 0000000..6ed2917 --- /dev/null +++ b/src/paperspec/claims.py @@ -0,0 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""Claim references and pins. + +A claim reference is `claim:TAG` or `claim:HOST/TAG`, the tag being eight Crockford +base32 capitals; a pin is a reference, `@` and the first six lowercase hex digits of +the claim's statement hash. The patterns are those of `claimRef` and `claimPin` in +`schemas/common.schema.json`. Non-normative; see spec/18 §18.4, spec/02 §2.4 and +spec/20. +""" + +import re + +from paperspec.jsontypes import Record + +_HOST = r"(?P[a-z0-9.-]+\.[a-z]{2,})" +_TAG = r"(?P[0-9A-HJKMNP-TV-Z]{8})" +# `\Z`, not `$`: the schema's `$` (ECMAScript) does not match before a final newline. +CLAIM_REF_RE = re.compile(rf"^claim:(?:{_HOST}/)?{_TAG}\Z") +CLAIM_PIN_RE = re.compile( + rf"^(?Pclaim:(?:{_HOST}/)?{_TAG})@(?P[0-9a-f]{{6}})\Z" +) +PIN_LENGTH = 6 + + +def parse_ref(s: str) -> Record | None: + """Parse a claim reference `claim:[host/]TAG` (spec/18 §18.4). + + Args: + s: The text to parse. + + Returns: + A dict with `host` (`None` for the local store) and `tag`, or `None` when + `s` is not a claim reference. + """ + m = CLAIM_REF_RE.match(s) + if not m: + return None + return {"host": m["host"], "tag": m["tag"]} + + +def parse_pin(s: str) -> Record | None: + """Parse a claim pin `claim:[host/]TAG@pin` (spec/18 §18.4). + + Args: + s: The text to parse. + + Returns: + A dict with `ref` (the reference without the pin), `host`, `tag` and `pin` + (six lowercase hex digits), or `None` when `s` is not a claim pin. + """ + m = CLAIM_PIN_RE.match(s) + if not m: + return None + return {"ref": m["ref"], "host": m["host"], "tag": m["tag"], "pin": m["pin"]} + + +def format_ref(tag: str, host: str | None = None) -> str: + """Format a claim reference. + + Args: + tag: The eight-character claim tag. + host: The domain of another claim store, or `None` for the local one. + + Returns: + `claim:TAG`, or `claim:HOST/TAG` when a host is given. + + Raises: + ValueError: If the result is not a claim reference. + """ + s = f"claim:{host}/{tag}" if host else f"claim:{tag}" + if not CLAIM_REF_RE.match(s): + msg = f"not a claim reference: {s!r}" + raise ValueError(msg) + return s + + +def format_pin(ref: str, statement_hash: str) -> str: + """Pin a claim reference to its statement hash. + + Args: + ref: A claim reference. + statement_hash: The claim's statement hash, 64 lowercase hex digits. + + Returns: + The reference, `@` and the first six digits of the hash. + + Raises: + ValueError: If `ref` is not a claim reference or the hash is not 64 + lowercase hex digits. + """ + if not CLAIM_REF_RE.match(ref): + msg = f"not a claim reference: {ref!r}" + raise ValueError(msg) + if not re.fullmatch(r"[0-9a-f]{64}", statement_hash): + msg = "a statement hash is 64 lowercase hex digits" + raise ValueError(msg) + return f"{ref}@{statement_hash[:PIN_LENGTH]}" diff --git a/src/paperspec/cli/__init__.py b/src/paperspec/cli/__init__.py index 1269942..6deb863 100644 --- a/src/paperspec/cli/__init__.py +++ b/src/paperspec/cli/__init__.py @@ -6,10 +6,11 @@ """The `paperspec` command's subcommands, one module each; the parser and the help are in `main.py`. -`paperspec validate` and `paperspec load` work anywhere the package is installed with -`paperspec[tools]`. `paperspec check`, `build`, `stress` and `site` work on a checkout of the -paperspec repository: the one an editable install points at, or else the one around the current -folder. Each module has `run(args)`, which takes the parsed arguments and returns the exit status. +`paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is +installed with `paperspec[tools]`. `paperspec check`, `build`, `stress` and `site` work on a +checkout of the paperspec repository: the one an editable install points at, or else the one +around the current folder. Each module has `run(args)`, which takes the parsed arguments and +returns the exit status. """ import importlib.util diff --git a/src/paperspec/cli/build.py b/src/paperspec/cli/build.py index 0430513..1cef5d4 100644 --- a/src/paperspec/cli/build.py +++ b/src/paperspec/cli/build.py @@ -39,6 +39,7 @@ from paperspec import ( log, mldsa, receipts as receipts_mod, + revision, rights, sequences, structures, @@ -6698,6 +6699,71 @@ def invalid(out: Path, tiny: Record) -> None: {"document.json": _doc_edit(files, key=wk + ".si")}, "A document key whose suffix is not the version's.", ) + wit = json.loads(files["witnesses.json"])["witnesses"][0]["id"] + other = "zzzzzzz" if wit != "zzzzzzz" else "yyyyyyy" + case( + "suspect-witness", + "doc.suspect.witness", + { + "suspects.jsonl": jsonl( + [ + { + "format": "scienting-suspect/1", + "id": "aaaaaaaaaaaaa", + "kind": "number", + "node": nm["@a"], + "a": "42.0", + "b": "42.5", + "witness": other, + } + ] + ) + }, + "A suspect whose second reading names a witness the package does not have.", + ) + + def math_in_form(ns: list[Record]) -> None: + by(ns, "@c")["content"] = [ + {"kind": "text", "text": "A second paragraph, "}, + { + "kind": "math", + "notation": "tex", + "source": "x=1\\tag{2}", + "form": "scienting-math/1", + }, + {"kind": "text", "text": "."}, + ] + + case( + "math-form-label", + "doc.math.form", + _rehashed(files, edit_nodes(math_in_form)), + "A scienting-math/1 source that still holds its \\tag, which the form lifts out.", + ) + + +def _rehashed(files: Mapping[str, bytes], nodes_jsonl: bytes) -> dict[str, bytes]: + """nodes.jsonl and document.json for edited nodes, every value recomputed. + + So a broken package built from them breaks only the rule its edit is about. + """ + document = json.loads(files["document.json"]) + nodes = [ + {k: v for k, v in json.loads(x).items() if k not in revision.VALUES} + for x in nodes_jsonl.decode().splitlines() + ] + doc = Doc(document["key"], nodes) + vals = hashing.compute(doc) + by_id = {n["id"]: n for n in nodes} + recs = [revision.node_record(by_id[nid], vals[nid]) for nid in doc.preorder()] + roster_keys = ( + json.loads(files["roster.json"])["keys"] if "roster.json" in files else [] + ) + recs_of = [ + json.loads(x) for x in files.get("receipts.jsonl", b"").decode().splitlines() + ] + d = revision.summarize(document, recs, vals, [], recs_of, roster_keys) + return {"nodes.jsonl": jsonl(recs), "document.json": (jcs.dumps(d) + "\n").encode()} def _manifest_edit(d: Path, **kw: Any) -> None: diff --git a/src/paperspec/cli/check.py b/src/paperspec/cli/check.py index 2c0f27a..3e2452e 100644 --- a/src/paperspec/cli/check.py +++ b/src/paperspec/cli/check.py @@ -46,6 +46,7 @@ from paperspec import ( hashing, jcs, keys, + layers, log, mldsa, package, @@ -438,10 +439,7 @@ def check_records() -> None: } ok( "records", - a["id"].rsplit("/", 1)[1] - == record_id( - sha256_hex(jcs.dumpb({k: v for k, v in body.items() if k != "id"})) - ), + a["id"].rsplit("/", 1)[1] == layers.annotation_id(a), "annotation id does not derive: " + a["id"], ) if a["status"] == "confirmed": @@ -1883,7 +1881,7 @@ def check_traces() -> None: and any( a["creator"]["type"] == "Person" and a["status"] == "confirmed" - and traces._anchors(a["body"])[0]["kind"] == "table" + and layers.anchors_of(a["body"])[0]["kind"] == "table" for a in mine ), "the layer lacks a confirmed model trace, a rejected candidate or a person's trace " @@ -2903,35 +2901,11 @@ def check_sql() -> None: # Store records satisfy the store tables' CHECK constraints. records_dir = repo_path("fixtures", "records") for layer in ("layer", "traces", "chemistry", "biology", "data"): - lay = read_json(records_dir / layer / "layer.json") - conn.execute( - "INSERT INTO layers VALUES (?,?,?,?,?,?,?,?)", - ( - lay["id"], - lay.get("project"), - lay["label"], - lay["purpose"], - lay.get("audience"), - lay["visibility"], - jcs.dumps(lay["license"]), - jcs.dumps(lay), - ), + db.load_layer( + conn, + read_json(records_dir / layer / "layer.json"), + read_jsonl(records_dir / layer / "annotations.jsonl"), ) - for a in read_jsonl(records_dir / layer / "annotations.jsonl"): - conn.execute( - "INSERT INTO annotations VALUES (?,?,?,?,?,?,?,?,?)", - ( - a["id"], - a["layer"], - a["body"]["kind"], - a["body"].get("category"), - a["status"], - a.get("receipt"), - a["creator"]["id"], - a["created"], - jcs.dumps(a), - ), - ) kinds = { k for (k,) in conn.execute("SELECT DISTINCT body_kind FROM annotations") } diff --git a/src/paperspec/cli/correct.py b/src/paperspec/cli/correct.py new file mode 100644 index 0000000..a40ded2 --- /dev/null +++ b/src/paperspec/cli/correct.py @@ -0,0 +1,108 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""Apply proposed corrections to a package and write the package they make. + +Non-normative tooling. +Usage: paperspec correct --correction PROPOSALS [--out PATH] PACKAGE +PACKAGE is a folder or a zipped package (`.sct.zip`). It is validated, each proposal runs +through the gate (spec/11 §11.4) in order, and only when every one passes is the corrected +package written, to PATH or in place of PACKAGE, and validated again. The records appended to +corrections.jsonl are printed, one per line; what was left out of `derived/` and `datasets/` +is reported on standard error. +Exit status: 0 when the corrections were applied, 1 when the package or a proposal is refused, +2 for a usage error. +""" + +import argparse +import sys +from pathlib import Path + +from paperspec import __version__, jcs, package, resources, revision +from paperspec.cli import UsageError, require +from paperspec.cli.validate import read_proposals +from paperspec.schema import Store +from paperspec.tree import Refusal + +require("cryptography", "jsonschema") + + +def run(a: argparse.Namespace) -> int: + """Run `paperspec correct` on the parsed arguments. + + Args: + a: The parsed arguments: `package`, `correction`, `out`, `answered`, `asset` and + `max_size`. + + Returns: + The exit status: 0 when the corrections were applied, 1 when something is refused. + + Raises: + UsageError: If an asset file does not exist or has no extension. + """ + assets = [] + for name in a.asset or []: + p = Path(name) + if not p.is_file() or not p.suffix: + raise UsageError(f"--asset {name}: not a file with an extension") + assets.append((p.read_bytes(), p.suffix)) + store = Store(resources.path("schemas")) + proposals = read_proposals(a.correction) + answered: list[str] = a.answered or [] + tool = {"name": "paperspec", "version": __version__} + try: + if package.is_zip(a.package): + result = revision.revise_zip( + a.package, + proposals, + store, + a.out, + answered=answered, + assets=assets, + tool=tool, + limit=a.max_size, + ) + else: + result = revision.revise( + a.package, + proposals, + store, + a.out, + answered=answered, + assets=assets, + tool=tool, + ) + except revision.Invalid as e: + for err in e.errors: + print( + f"{a.package}: {err['where']}: {err['rule']}: {err['message']}", + file=sys.stderr, + ) + return 1 + except revision.Refused as e: + where = f"{a.correction}:{e.index}" + (" " + e.node if e.node else "") + print(f"{where}: {e.rule}: {e.message}", file=sys.stderr) + return 1 + except Refusal as e: + print(f"{a.package}: {e.rule}: {e.message}", file=sys.stderr) + return 1 + except ValueError as e: + print(f"{a.package}: {e}", file=sys.stderr) + return 1 + for record in result["records"]: + print(jcs.dumps(record)) + for path, n in sorted(result["dropped"].items()): + what = "files" if path.endswith("/") else "lines" + print( + f"{result['path']}: left out {n} {what} of {path}, which read corrected text", + file=sys.stderr, + ) + print( + f"{result['path']}: {result['key']} at revision {result['rev']}, " + f"package {result['package']}", + file=sys.stderr, + ) + return 0 diff --git a/src/paperspec/cli/main.py b/src/paperspec/cli/main.py index 19bb5b3..7af27bd 100644 --- a/src/paperspec/cli/main.py +++ b/src/paperspec/cli/main.py @@ -24,19 +24,21 @@ from paperspec.package import ZIP_LIMIT SITE = "https://paperspec.scient.ing" DESCRIPTION = """\ -The reference implementation of the scienting formats: validate packages, load them into SQLite, -and check, rebuild and stress the paperspec repository itself. +The reference implementation of the scienting formats: validate packages, apply corrections to +them, load them into SQLite, and check, rebuild and stress the paperspec repository itself. """ EPILOG = f"""\ examples: paperspec validate paper.sct.zip check a package against every rule paperspec validate --correction fix.json paper/ try a proposed correction without writing it + paperspec correct --correction fix.json paper/ apply it and write the corrected package paperspec load --sqlite lab.db paper/ other/ load packages into a SQLite file to query paperspec check --quick in a checkout: check everything but the rebuild where each command works: - validate and load work wherever paperspec is installed with its tools extra, paperspec[tools]; + validate, correct and load work wherever paperspec is installed with its tools extra, + paperspec[tools]; installing from the package registry: {SITE}/deploy/#install-from-the-registry check, build, stress and site work on a checkout of the paperspec repository, where pixi also runs them as tasks: pixi run check @@ -109,6 +111,80 @@ first refusal""", return p +def _correct(sub: _Sub) -> argparse.ArgumentParser: + p = sub.add_parser( + "correct", + help="apply proposed corrections to a package and write the package they make", + description="Apply proposed corrections to a package, and write the corrected " + "package.\n\n" + + PACKAGE_ADVICE + + """ +The package is validated first. PROPOSALS is a JSON file holding one proposed correction, or a +JSON Lines file of several, applied in order; each runs through the gate (spec/11 §11.4), and +only when every one passes is anything written. The corrected package goes to --out, or +replaces PACKAGE, and is validated again before it does. + +The writer recomputes every node's values and document.json, records the corrections and the +retired IDs, sets each changed leaf to F1 when its correction has crop evidence and to F0 +otherwise (an emend, relink or resolve keeps the level), moves a merged node's boxes to the +node it merged into, and records paperspec in provenance.json. Lines of derived/ files and +datasets that read corrected text are left out, since only the tools that made them can rebuild +them, and each is reported. Use `paperspec validate --correction` to try proposals first. + +The records appended to corrections.jsonl are printed, one per line. +Exit status: 0 when the corrections were applied, 1 when the package or a proposal is refused, +2 for a usage error.""", + epilog="""\ +examples: + paperspec correct --correction fix.json paper/ correct the folder in place + paperspec correct --correction fixes.jsonl --out fixed/ paper/ write a corrected copy + paperspec correct --correction fix.json --asset crop.png paper.sct.zip + add the crop the evidence names, then zip again + paperspec correct --correction fix.json --answered 9z6fh0m4fx3qd paper/ + the correction answers that suspect""", + ) + p.add_argument( + "--correction", + required=True, + metavar="PROPOSALS", + help="a JSON or JSON Lines file of proposed corrections, applied in order", + ) + p.add_argument( + "--out", + metavar="PATH", + help="write the corrected package here (a folder, or a zip for a zipped package), " + "which must not exist; by default PACKAGE is replaced", + ) + p.add_argument( + "--answered", + action="append", + metavar="SUSPECT", + help="the ID of a suspect the corrections answer, removed from suspects.jsonl; " + "repeat for several", + ) + p.add_argument( + "--asset", + action="append", + metavar="FILE", + help="a file to add under assets/ as ., such as a crop the " + "evidence names; repeat for several", + ) + p.add_argument( + "--max-size", + metavar="BYTES", + type=int, + default=ZIP_LIMIT, + help=( + "refuse a zipped package that expands past this many bytes " + f"(default {ZIP_LIMIT}, 1 GiB)" + ), + ) + p.add_argument( + "package", metavar="PACKAGE", help="a package folder or a zipped package" + ) + return p + + def _load(sub: _Sub) -> argparse.ArgumentParser: p = sub.add_parser( "load", @@ -299,6 +375,7 @@ examples: COMMANDS: dict[str, Callable[[_Sub], argparse.ArgumentParser]] = { "validate": _validate, + "correct": _correct, "load": _load, "check": _check, "build": _build, diff --git a/src/paperspec/corrections.py b/src/paperspec/corrections.py index 35b54de..b7b7f92 100644 --- a/src/paperspec/corrections.py +++ b/src/paperspec/corrections.py @@ -67,6 +67,23 @@ PROSE_RUNS = frozenset(["text", "link", "cite", "xref", "fnmark"]) # The operations that never lower a leaf's level: what a witness read stays what it was # (spec/08 §8.1-8.2). KEEPS_LEVEL = frozenset(["emend", "relink", "resolve"]) +# The operations of spec/11 §11.1, in the order the chapter and correction.schema.json list them. +OPS = ( + "replace", + "insert", + "delete", + "split", + "merge", + "move", + "reclassify", + "relabel", + "set", + "emend", + "relink", + "resolve", +) +# What an emend corrects (spec/11 §11.1). +EMEND_KINDS = ("typo", "spelling", "grammar", "punctuation", "wording", "other") class State: diff --git a/src/paperspec/datasets.py b/src/paperspec/datasets.py new file mode 100644 index 0000000..e6c306d --- /dev/null +++ b/src/paperspec/datasets.py @@ -0,0 +1,180 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""The datasets of a package, scienting-dataset/1, checked against the package (spec/15). + +Non-normative. +A dataset is a folder `datasets//` with a `datapackage.json`, its CSV resources and, for +reported values, an `anchors.jsonl` that ties each value to the cell or sentence that prints it. +`problems` checks the descriptor against its schema, that the folder is named by the dataset's +ID, that each resource's primary key is unique, that each value anchor resolves as current to a +node whose text is the value as the CSV holds it, and that every value of a reported resource +is anchored. +Units are compared as strings elsewhere (`paperspec.data`); a producer with a unit library +SHOULD also check that each unit parses. +""" + +import csv +import io +from collections.abc import Mapping +from typing import Any + +from paperspec import anchors, hashing, jcs +from paperspec.corrections import State +from paperspec.jsontypes import Record +from paperspec.schema import Store, validate as schema_validate +from paperspec.text import norm, sha256_hex +from paperspec.tree import Refusal + + +def folders(paths: Any) -> list[str]: + """The dataset IDs a package's file paths hold, as `datasets//...`, sorted.""" + return sorted( + { + p.split("/")[1] + for p in paths + if p.startswith("datasets/") and p.count("/") >= 2 + } + ) + + +def content_hash(files: Mapping[str, bytes]) -> str: + """A dataset's content hash (spec/15): over its files by path inside the dataset folder.""" + items = [ + [p, sha256_hex(files[p]), len(files[p])] for p in sorted(files, key=str.encode) + ] + return sha256_hex(jcs.dumpb({"format": "scienting-dataset/1", "files": items})) + + +def _canonical(data: bytes, where: str) -> Any: + from paperspec import package # noqa: PLC0415 - package imports this module + + return package.parse_jsonl(data, where) + + +def problems( + dataset: str, files: Mapping[str, bytes], state: State, store: Store +) -> list[Record]: + """The rules one dataset of a package breaks (spec/15), as `{rule, where, message}` records. + + Args: + dataset: The dataset folder's name, which is its ID. + files: The dataset's files by path inside its folder, such as `datapackage.json`. + state: The package's current state, which the value anchors resolve against. + store: The schema store. + """ + out: list[Record] = [] + where = f"datasets/{dataset}" + + def add(rule: str, at: str, message: str) -> None: + out.append({"rule": rule, "where": at, "message": message}) + + if "datapackage.json" not in files: + add("doc.dataset.file", where, "the dataset has no datapackage.json") + return out + try: + desc = jcs.loads(files["datapackage.json"].decode("utf-8")) + except (UnicodeDecodeError, ValueError) as e: + add("doc.dataset.file", f"{where}/datapackage.json", f"not JSON ({e})") + return out + bad = schema_validate(store, "dataset.schema.json", desc) + for e in bad: + add("doc.dataset.schema", f"{where}/datapackage.json", e) + if bad: + return out + if desc["id"] != dataset: + add( + "doc.dataset.id", + where, + f"the folder is not named by the dataset's ID {desc['id']}", + ) + lines: list[Record] = [] + if "anchors.jsonl" in files: + try: + lines = _canonical(files["anchors.jsonl"], f"{where}/anchors.jsonl") + except Refusal as e: + add("doc.dataset.file", f"{where}/anchors.jsonl", e.message) + doc = state.doc() + for res in desc["resources"]: + path = res.get("path") + if not isinstance(path, str) or path not in files: + add( + "doc.dataset.file", + where, + f"the resource {res['name']} names no file of the dataset", + ) + continue + if not path.endswith(".csv"): + continue + try: + table = list( + csv.DictReader(io.StringIO(files[path].decode("utf-8"), newline="")) + ) + except (UnicodeDecodeError, csv.Error) as e: + add("doc.dataset.file", f"{where}/{path}", f"the CSV cannot be read ({e})") + continue + key = res.get("schema", {}).get("primaryKey") + rows = {r.get(key): r for r in table} if isinstance(key, str) else {} + if isinstance(key, str) and len(rows) != len(table): + add( + "doc.dataset.key", + f"{where}/{path}", + f"the primary key {key} is not unique", + ) + anchored = set() + for i, va in enumerate(lines, 1): + at = f"{where}/anchors.jsonl:{i}" + if va.get("resource") != res["name"]: + continue + errors = schema_validate(store, "value-anchor.schema.json", va) + for e in errors: + add("doc.dataset.anchor", at, e) + if errors: + continue + row = rows.get(va["row"]) + if row is None or va["field"] not in row: + add( + "doc.dataset.anchor", + at, + f"{va['row']}/{va['field']} is not a value of {path}", + ) + continue + anchored.add((va["row"], va["field"])) + if "anchor" not in va: + continue + r = anchors.resolve(state, va["anchor"]) + if r["result"] != "current" or r.get("node") not in doc.nodes: + add( + "doc.dataset.anchor", + at, + f"the anchor of {va['row']}/{va['field']} resolves as {r['result']}", + ) + continue + text = hashing.plain(doc, r["node"]) + a, b = r.get("span", [0, len(text)]) + printed = norm(text[a:b]) + if row[va["field"]] != printed: + add( + "doc.dataset.value", + at, + f"{va['row']}/{va['field']} is {row[va['field']]!r} but its anchor reads " + f"{printed!r}", + ) + numeric = { + f["name"] + for f in res.get("schema", {}).get("fields", []) + if f.get("type") in ("number", "integer") + } + if res.get("origin", {}).get("kind") == "reported" and isinstance(key, str): + for k, row in rows.items(): + for field, value in row.items(): + if field in numeric and value and (k, field) not in anchored: + add( + "doc.dataset.unanchored", + f"{where}/{path}", + f"the reported value {k}/{field} is not anchored to what prints it", + ) + return out diff --git a/src/paperspec/db.py b/src/paperspec/db.py index 8fc02d9..9ac9790 100644 --- a/src/paperspec/db.py +++ b/src/paperspec/db.py @@ -13,11 +13,12 @@ Loading is a pure function of the package, and unloading gives back the same byt import json import shutil import sqlite3 -from collections.abc import Mapping, Sequence +from collections.abc import Iterable, Mapping, Sequence from pathlib import Path from typing import Any -from paperspec import hashing, jcs +from paperspec import hashing, jcs, layers +from paperspec.corrections import State from paperspec.jsontypes import Record, StrPath from paperspec.model import Doc from paperspec.package import ( @@ -31,6 +32,8 @@ from paperspec.package import ( from paperspec.text import norm, sha256_hex from paperspec.tree import Refusal +# The node values a state does not carry (spec/11 §11.4). +VALUES = ("text_hash", "match_hash", "align_key", "own_hash", "node_hash") SPEC_VERSION = "0.1.0" BUNDLE_FORMAT = "scienting-bundle/1" @@ -433,6 +436,113 @@ def _q(conn: sqlite3.Connection, sql: str, args: Sequence[Any]) -> list[Record]: return [dict(zip(cols, row, strict=True)) for row in cur.fetchall()] +def _node_records(conn: sqlite3.Connection, key: str) -> list[Record]: + """The records of `nodes.jsonl` for `key`, in document order, with their five values.""" + nodes = [] + for n in _q(conn, "SELECT * FROM nodes WHERE doc_key = ? ORDER BY seq", (key,)): + rec = {"id": n["id"], "kind": n["kind"], "parent": n["parent"], "ord": n["ord"]} + _opt(rec, "label", n["label"]) + _opt(rec, "attrs", _from_json(n["attrs"])) + _opt(rec, "content", _from_json(n["content"])) + if n["withheld"]: + rec["withheld"] = True + _opt(rec, "fidelity", n["fidelity"]) + _opt(rec, "prov", _from_json(n["prov"])) + for h in VALUES: + rec[h] = n[h] + nodes.append(rec) + return nodes + + +def _ref_records( + conn: sqlite3.Connection, key: str, seq: Mapping[str, int] +) -> list[Record]: + """The records of `references.jsonl` for `key`, in the document order `seq` gives.""" + refs = [] + for r in _q(conn, "SELECT * FROM refs WHERE doc_key = ?", (key,)): + rec = {"node": r["node"]} + _opt(rec, "label", r["label"]) + _opt(rec, "csl", _from_json(r["csl"])) + rec["resolution"] = _from_json(r["resolution"]) + refs.append(rec) + refs.sort(key=lambda r: seq[r["node"]]) + return refs + + +def _correction_records(conn: sqlite3.Connection, key: str) -> list[Record]: + """The records of `corrections.jsonl` for `key`, in index order.""" + cors = [] + for c in _q( + conn, "SELECT * FROM corrections WHERE doc_key = ? ORDER BY idx", (key,) + ): + rec = { + "format": "scienting-correction/1", + "index": c["idx"], + "id": c["id"], + "op": c["op"], + } + _opt(rec, "target", c["target"]) + rec.update( + { + "evidence": _from_json(c["evidence"]), + "author": _from_json(c["author"]), + "reason": c["reason"], + "at": c["made_at"], + "before": c["before_hash"], + "after": c["after_hash"], + "path": _from_json(c["path"]), + "touched": _from_json(c["touched"]), + } + ) + rec.update(_from_json(c["details"])) + cors.append(rec) + return cors + + +def _id_records(conn: sqlite3.Connection, key: str) -> list[Record]: + """The records of `ids.jsonl` for `key`: aliases first, then by revision and ID.""" + ids = [] + for e in _q(conn, "SELECT * FROM node_ids WHERE doc_key = ?", (key,)): + rec = {"id": e["id"], "kind": e["kind"], "successor": e["successor"]} + _opt(rec, "node_kind", e["node_kind"]) + _opt(rec, "rev", e["rev"]) + ids.append(rec) + ids.sort(key=lambda e: (0 if e["kind"] == "alias" else 1, e.get("rev", 0), e["id"])) + return ids + + +def read_state(conn: sqlite3.Connection, key: str) -> State: + """The state of the loaded document `key`, as `package.read_state` gives for its package. + + It is the nodes without their five hash values, the ID table, the corrections applied + and the reference records (spec/11 §11.4), read from the tables. + + Args: + conn: The database. + key: The document key. + + Returns: + The state. + + Raises: + KeyError: If `key` is not a loaded document. + """ + if ( + conn.execute("SELECT 1 FROM documents WHERE doc_key = ?", (key,)).fetchone() + is None + ): + raise KeyError(key) + nodes = _node_records(conn, key) + seq = {n["id"]: i for i, n in enumerate(nodes)} + return State( + key, + [{k: v for k, v in n.items() if k not in VALUES} for n in nodes], + _id_records(conn, key), + _correction_records(conn, key), + references=_ref_records(conn, key, seq), + ) + + def unload( conn: sqlite3.Connection, key: str, out: StrPath, asset_store: StrPath ) -> str: @@ -505,19 +615,7 @@ def unload( ) ws.append(rec) put("witnesses.json", {"witnesses": ws}) - nodes = [] - for n in _q(conn, "SELECT * FROM nodes WHERE doc_key = ? ORDER BY seq", (key,)): - rec = {"id": n["id"], "kind": n["kind"], "parent": n["parent"], "ord": n["ord"]} - _opt(rec, "label", n["label"]) - _opt(rec, "attrs", _from_json(n["attrs"])) - _opt(rec, "content", _from_json(n["content"])) - if n["withheld"]: - rec["withheld"] = True - _opt(rec, "fidelity", n["fidelity"]) - _opt(rec, "prov", _from_json(n["prov"])) - for h in ("text_hash", "match_hash", "align_key", "own_hash", "node_hash"): - rec[h] = n[h] - nodes.append(rec) + nodes = _node_records(conn, key) putl("nodes.jsonl", nodes) seq = {n["id"]: i for i, n in enumerate(nodes)} pages = [] @@ -545,49 +643,9 @@ def unload( locs.append(rec) locs.sort(key=lambda r: (seq[r["node"]], r["page"], r["page_fp"])) putl("locations.jsonl", locs) - refs = [] - for r in _q(conn, "SELECT * FROM refs WHERE doc_key = ?", (key,)): - rec = {"node": r["node"]} - _opt(rec, "label", r["label"]) - _opt(rec, "csl", _from_json(r["csl"])) - rec["resolution"] = _from_json(r["resolution"]) - refs.append(rec) - refs.sort(key=lambda r: seq[r["node"]]) - putl("references.jsonl", refs) - cors = [] - for c in _q( - conn, "SELECT * FROM corrections WHERE doc_key = ? ORDER BY idx", (key,) - ): - rec = { - "format": "scienting-correction/1", - "index": c["idx"], - "id": c["id"], - "op": c["op"], - } - _opt(rec, "target", c["target"]) - rec.update( - { - "evidence": _from_json(c["evidence"]), - "author": _from_json(c["author"]), - "reason": c["reason"], - "at": c["made_at"], - "before": c["before_hash"], - "after": c["after_hash"], - "path": _from_json(c["path"]), - "touched": _from_json(c["touched"]), - } - ) - rec.update(_from_json(c["details"])) - cors.append(rec) - putl("corrections.jsonl", cors) - ids = [] - for e in _q(conn, "SELECT * FROM node_ids WHERE doc_key = ?", (key,)): - rec = {"id": e["id"], "kind": e["kind"], "successor": e["successor"]} - _opt(rec, "node_kind", e["node_kind"]) - _opt(rec, "rev", e["rev"]) - ids.append(rec) - ids.sort(key=lambda e: (0 if e["kind"] == "alias" else 1, e.get("rev", 0), e["id"])) - putl("ids.jsonl", ids) + putl("references.jsonl", _ref_records(conn, key, seq)) + putl("corrections.jsonl", _correction_records(conn, key)) + putl("ids.jsonl", _id_records(conn, key)) recs = [] for r in _q(conn, "SELECT * FROM receipts WHERE doc_key = ?", (key,)): rec = { @@ -655,3 +713,253 @@ def unload( } ) return write(out, files, absent) + + +def _target_rows( + annotation: Record, +) -> list[tuple[str, int, str | None, str | None, str | None]]: + """The `annotation_targets` rows of one annotation, one for each target (spec/13 §13.2). + + A target of a document gives its bare document key and the node of its first + `AnchorSelector`, NULL when it has none. A target that is another annotation (a + reply) gives that annotation's ID as `target_annotation`. + """ + target = annotation["target"] + rows = [] + for i, t in enumerate(target if isinstance(target, list) else [target]): + source = t["source"] + if "selector" in t: + found = layers.anchors_of(t) + node = found[0]["node"] if found else None + rows.append((annotation["id"], i, source.rsplit("/", 1)[-1], node, None)) + else: + rows.append((annotation["id"], i, None, None, source)) + return rows + + +def load_layer( + conn: sqlite3.Connection, layer: Record, annotations: Iterable[Record] +) -> int: + """Insert a `scienting-layer/1` record and its annotations into the store tables. + + The rows are those `paperspec check` writes: one in `layers`, one in `annotations` for + each annotation, and one in `annotation_targets` for each target of each annotation + (spec/13 §13.2, spec/18). Nothing is validated beyond what the tables' CHECK + constraints and keys enforce, so validate the layer and its annotations first + (`schema.validate`, then `layers.annotation_id` against each `id`). + + Args: + conn: A database made by `create`. + layer: The layer record. + annotations: The layer's annotations. + + Returns: + The number of annotations inserted. + + Raises: + sqlite3.IntegrityError: If a layer or annotation of that ID is loaded, or a value + breaks a CHECK constraint. Rows inserted before the failure stay. + """ + conn.execute( + "INSERT INTO layers VALUES (?,?,?,?,?,?,?,?)", + ( + layer["id"], + layer.get("project"), + layer["label"], + layer["purpose"], + layer.get("audience"), + layer["visibility"], + jcs.dumps(layer["license"]), + jcs.dumps(layer), + ), + ) + count = 0 + for a in annotations: + conn.execute( + "INSERT INTO annotations VALUES (?,?,?,?,?,?,?,?,?)", + ( + a["id"], + a["layer"], + a["body"]["kind"], + a["body"].get("category"), + a["status"], + a.get("receipt"), + a["creator"]["id"], + a["created"], + jcs.dumps(a), + ), + ) + conn.executemany( + "INSERT INTO annotation_targets VALUES (?,?,?,?,?)", _target_rows(a) + ) + count += 1 + return count + + +def delete(conn: sqlite3.Connection, key: str) -> int: + """Forget the loaded document `key`: its rows in every table that has a `doc_key`. + + That is each table of `doc.sql` and `store.sql` with the column (and the bundle's + `search`), the values of the datasets of the document, and the document's `works` row + when no other document is of that work. Annotations stay: they belong to a layer, and + only their `annotation_targets` rows on the document go. Files in the asset store stay, + since another document may share them. Run it in a transaction to undo it. + + Args: + conn: The database. + key: The document key. + + Returns: + The number of rows deleted. + """ + n = 0 + work = conn.execute( + "SELECT work FROM documents WHERE doc_key = ?", (key,) + ).fetchone() + tables = [ + name + for (name,) in conn.execute( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'" + ).fetchall() + if any(c[1] == "doc_key" for c in conn.execute(f'PRAGMA table_info("{name}")')) + ] + if "dataset_values" in _tables(conn) and "datasets" in tables: + n += conn.execute( + "DELETE FROM dataset_values WHERE dataset IN " + "(SELECT id FROM datasets WHERE doc_key = ?)", + (key,), + ).rowcount + for name in tables: + n += conn.execute( + f'DELETE FROM "{name}" WHERE doc_key = ?', # noqa: S608 - names come from sqlite_master + (key,), + ).rowcount + if work is not None: + wk = json.loads(work[0])["key"] + n += conn.execute( + "DELETE FROM works WHERE work_key = ? AND NOT EXISTS " + "(SELECT 1 FROM documents WHERE json_extract(work, '$.key') = ?)", + (wk, wk), + ).rowcount + return n + + +def _tables(conn: sqlite3.Connection) -> set[str]: + """The names of the tables of the database.""" + return { + name + for (name,) in conn.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ) + } + + +def _shipped(sql_dir: StrPath, with_bundle: bool) -> dict[str, tuple[str, list[str]]]: + """The tables of the shipped DDL: name to its CREATE TABLE and its CREATE INDEXes.""" + scratch = sqlite3.connect(":memory:") + try: + names = ["doc.sql", "store.sql"] + ( + ["bundle-sqlite.sql"] if with_bundle else [] + ) + for name in names: + scratch.executescript((Path(sql_dir) / name).read_text(encoding="utf-8")) + out: dict[str, tuple[str, list[str]]] = {} + for name, sql in scratch.execute( + "SELECT name, sql FROM sqlite_master WHERE type = 'table'" + ).fetchall(): + # The full-text table and its shadow tables are rebuilt by `index`, not here. + if sql.startswith("CREATE VIRTUAL") or name.startswith("search_"): + continue + indexes = [ + s + for (s,) in scratch.execute( + "SELECT sql FROM sqlite_master WHERE type = 'index' AND tbl_name = ? " + "AND sql IS NOT NULL ORDER BY name", + (name,), + ) + ] + out[name] = (sql, indexes) + return out + finally: + scratch.close() + + +def upgrade(conn: sqlite3.Connection, sql_dir: StrPath) -> list[str]: + """Rebuild the tables whose stored CREATE differs from the shipped DDL in `sql_dir`. + + SQLite cannot alter a CHECK, so a store made before a CHECK widened (a `body_kind` + that lacks `trace`, a `corrections.op` that lacks `emend`) is rebuilt the way + gives: with foreign keys off and in one + transaction, copy the table's rows aside, drop it, create it from the shipped text, + copy back the columns both versions have and check foreign keys. Its indexes are made + again from the shipped text, and its triggers and other indexes as stored. The tables + are those of `doc.sql` and `store.sql`, and of `bundle-sqlite.sql` when the database + has `bundle_meta`; a table the database lacks is left alone, and so is the bundle's + virtual `search`. A second call finds nothing to do. + + Args: + conn: The database; it must not be in a transaction. + sql_dir: The folder of `doc.sql`, `store.sql` and `bundle-sqlite.sql`. + + Returns: + The names of the tables rebuilt, in the order of the shipped DDL. + + Raises: + ValueError: If `conn` is in a transaction. + sqlite3.IntegrityError: If a row breaks the new constraints or a foreign key check + fails. The database is then as it was. + """ + if conn.in_transaction: + raise ValueError("upgrade needs a connection that is not in a transaction") + shipped = _shipped(sql_dir, "bundle_meta" in _tables(conn)) + stored = dict( + conn.execute("SELECT name, sql FROM sqlite_master WHERE type = 'table'") + ) + todo = [t for t, (sql, _) in shipped.items() if t in stored and stored[t] != sql] + if not todo: + return [] + foreign_keys = conn.execute("PRAGMA foreign_keys").fetchone()[0] + conn.execute("PRAGMA foreign_keys = OFF") + try: + conn.execute("BEGIN") + for t in todo: + _rebuild(conn, t, *shipped[t]) + _check_foreign_keys(conn) + conn.commit() + except BaseException: + conn.rollback() + raise + finally: + conn.execute(f"PRAGMA foreign_keys = {int(foreign_keys)}") + return todo + + +def _check_foreign_keys(conn: sqlite3.Connection) -> None: + """Raise `sqlite3.IntegrityError` if a row breaks a foreign key the tables declare.""" + if conn.execute("PRAGMA foreign_key_check").fetchall(): + raise sqlite3.IntegrityError("foreign_key_check failed after the upgrade") + + +def _rebuild( + conn: sqlite3.Connection, table: str, create: str, indexes: Sequence[str] +) -> None: + """Make `table` again from `create`, keeping its rows, triggers and unshipped indexes.""" + shipped_index = sorted({s.split()[2] for s in indexes}) + marks = ",".join("?" * len(shipped_index)) + query = ( + "SELECT sql FROM sqlite_master WHERE tbl_name = ? AND sql IS NOT NULL " # noqa: S608 + f"AND type IN ('index', 'trigger') AND name NOT IN ({marks}) ORDER BY name" + ) + keep = [s for (s,) in conn.execute(query, (table, *shipped_index))] + conn.execute(f'CREATE TEMP TABLE "upgrade_{table}" AS SELECT * FROM "{table}"') # noqa: S608 + old = [c[1] for c in conn.execute(f'PRAGMA table_info("{table}")')] + conn.execute(f'DROP TABLE "{table}"') + conn.execute(create) + new = {c[1] for c in conn.execute(f'PRAGMA table_info("{table}")')} + cols = ", ".join(f'"{c}"' for c in old if c in new) + conn.execute( + f'INSERT INTO "{table}" ({cols}) SELECT {cols} FROM "upgrade_{table}"' # noqa: S608 + ) + conn.execute(f'DROP TABLE "upgrade_{table}"') + for sql in (*indexes, *keep): + conn.execute(sql) diff --git a/src/paperspec/fingerprint.py b/src/paperspec/fingerprint.py index 87007a0..7899a5a 100644 --- a/src/paperspec/fingerprint.py +++ b/src/paperspec/fingerprint.py @@ -14,7 +14,7 @@ import re import unicodedata from collections.abc import Sequence -from paperspec import jcs +from paperspec import jcs, resources from paperspec.jsontypes import Record from paperspec.notation import fold_plain_text from paperspec.text import is_ws, norm, sha256_hex @@ -71,6 +71,19 @@ def fp_text(text: str) -> str: return _strip(unicodedata.normalize("NFKC", fold_plain_text(_strip(text)))) +PUBLISHED = "stamps/scienting-stamps-1.json" + + +def published_rules() -> list[Record]: + """Return the published stamp rules, `registry/stamps/scienting-stamps-1.json` (spec/07 §7.2). + + They travel with the package, so a producer installed from a wheel fingerprints pages with + the same rules as a checkout. A team's learned rules (`learned: true`) are files of its own, + passed with these. + """ + return resources.registry(PUBLISHED)["rules"] + + def fingerprint(page: Record, rules: Sequence[Record]) -> tuple[str, list[str]]: """Fingerprint a page after removing the lines that the stamp rules match. diff --git a/src/paperspec/hashing.py b/src/paperspec/hashing.py index c154aa3..68e715e 100644 --- a/src/paperspec/hashing.py +++ b/src/paperspec/hashing.py @@ -9,8 +9,10 @@ Non-normative; see spec/05-hashes.md. """ +import functools import hashlib import hmac +import re import unicodedata from collections.abc import Callable, Sequence from typing import Self @@ -24,7 +26,7 @@ from paperspec.notation import ( fold_plain_text, fold_styled_script, ) -from paperspec.text import collapse_ws, is_ws, norm, remove_ws, trim +from paperspec.text import WHITE_SPACE, collapse_ws, norm, remove_ws, trim OBJ = "\ufffc" MARK_OPEN = "\ufff9" @@ -129,11 +131,13 @@ def _finish_strict(s: str) -> str: return trim(collapse_ws(s)) -def strict_runs_raw(runs: Sequence[Record], doc: Doc, complexes: list[str]) -> str: +def strict_runs_raw( + runs: Sequence[Record], doc: Doc | None, complexes: list[str] +) -> str: """The strict projection of runs before NFKC and whitespace (spec/05 §5.3). It is taken over the runs as printed. The complex keys of the runs are appended to - `complexes`. + `complexes`. `doc` may be None when no run is an `equation` run. """ out = [] for r in as_printed(runs): @@ -164,39 +168,66 @@ def strict_runs_raw(runs: Sequence[Record], doc: Doc, complexes: list[str]) -> s return "".join(out) +def strict_runs( + runs: Sequence[Record], doc: Doc | None = None +) -> tuple[str, list[str]]: + """The strict projection of runs that need not belong to a node (spec/05 §5.3). + + A producer compares a witness's reading of a leaf, which is runs and not yet a node, with + the leaf this way. `doc` is needed only when the runs hold an `equation` run, whose source + is its node's. + + Returns: + The strict projection and the runs' complex math keys, in order. + """ + complexes: list[str] = [] + return _finish_strict(strict_runs_raw(runs, doc, complexes)), complexes + + +# What the strict projection's scan stops at: a script's opening, a brace, or a citation mark. +_LOSSY_TOKEN = re.compile(r"[_^]\{|[{}]|" + re.escape(MARK_OPEN)) + + +@functools.cache +def _lossy_dropped() -> dict[int, None]: + """The characters the lossy projection drops: whitespace, dashes (Pd) and the soft hyphen. + + The table is built once, on first use, from the character data the package pins. + """ + pd = (c for c in range(0x110000) if unicodedata.category(chr(c)) == "Pd") + return dict.fromkeys([*WHITE_SPACE, *pd, 0x00AD]) + + def lossy_from_strict(s: str) -> str: - """The lossy projection (spec/05 §5.5) of a strict projection.""" + """The lossy projection (spec/05 §5.5) of a strict projection. + + A script's braces go with their script marker, a citation mark goes whole, and other + braces stay; then case folding, NFKC, and no whitespace, dash or soft hyphen. + """ out = [] stack = [] i, n = 0, len(s) while i < n: - ch = s[i] - if ch in "_^" and i + 1 < n and s[i + 1] == "{": + m = _LOSSY_TOKEN.search(s, i) + if m is None: + out.append(s[i:]) + break + out.append(s[i : m.start()]) + tok = m.group() + i = m.end() + if tok in ("_{", "^{"): stack.append("s") - i += 2 - continue - if ch == "{": + elif tok == "{": stack.append("t") - out.append(ch) - elif ch == "}": - if stack: - if stack.pop() == "t": - out.append(ch) - else: - out.append(ch) - elif ch == MARK_OPEN: - j = s.find(MARK_CLOSE, i) - i = n if j < 0 else j + 1 - continue + out.append(tok) + elif tok == "}": + if not stack or stack.pop() == "t": + out.append(tok) else: - out.append(ch) - i += 1 + j = s.find(MARK_CLOSE, m.start()) + i = n if j < 0 else j + 1 t = unicodedata.normalize("NFKC", "".join(out).casefold()) - return "".join( - c - for c in t - if not is_ws(c) and unicodedata.category(c) != "Pd" and c != "\u00ad" - ) + return t.translate(_lossy_dropped()) def prose_length(lossy: str) -> int: diff --git a/src/paperspec/jcs.py b/src/paperspec/jcs.py index 2c40630..2633540 100644 --- a/src/paperspec/jcs.py +++ b/src/paperspec/jcs.py @@ -17,7 +17,7 @@ UTF-16 code units rather than by code points. import json import math -from typing import Any +from typing import Any, NoReturn _C = json.JSONEncoder( ensure_ascii=False, separators=(",", ":"), sort_keys=True, allow_nan=False @@ -174,14 +174,48 @@ def dumpb(value: Any) -> bytes: return dumps(value).encode("utf-8") +def parse_int(text: str) -> int | float: + """Read a JSON integer as an ECMAScript engine would, for `json.loads(parse_int=...)`. + + That is exactly within 2^53, and as a double beyond (spec/01 §1.1), so the canonical form + of a whole double above 2^53, such as `100000000000000000000` for 1e20, reads back. + """ + v = int(text) + return v if abs(v) <= _EXACT_INT else float(v) + + +def refuse_constant(name: str) -> NoReturn: + """Refuse `NaN`, `Infinity` and `-Infinity`, for `json.loads(parse_constant=...)`. + + Raises: + ValueError: Always; JSON has no canonical form for them. + """ + raise ValueError("JSON has no canonical form for " + name) + + +def loads(text: str) -> Any: + """Parse JSON text with every number read as canonical JSON reads it (spec/01 §1.1). + + Integers beyond 2^53 become doubles and `NaN` and the infinities are refused; the text + need not be canonical. + + Raises: + ValueError: If `text` does not parse, or holds `NaN` or an infinity. + """ + return json.loads(text, parse_int=parse_int, parse_constant=refuse_constant) + + def loads_line(data: bytes) -> Any: """Parse one canonical JSON line of bytes, refusing a line that is not canonical. + Numbers are read as IEEE 754 doubles, as the canonical form writes them, so + `loads_line(dumpb(x))` returns a value equal to `x` for every JSON value `x`. + Raises: ValueError: If `data` does not parse, or parses to a value whose canonical form differs from `data`. """ - value = json.loads(data.decode("utf-8")) + value = loads(data.decode("utf-8")) if dumpb(value) != data: raise ValueError("not canonical JSON") return value diff --git a/src/paperspec/layers.py b/src/paperspec/layers.py new file mode 100644 index 0000000..1087668 --- /dev/null +++ b/src/paperspec/layers.py @@ -0,0 +1,88 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""Layers and their annotations: the record-ID rule and the enumerations (spec/18-layers.md). + +Non-normative; the normative text is spec/18 and `schemas/annotation.schema.json`. The +constants here repeat the schema's enumerations so a tool need not read the schema; a test +keeps them equal to it and to the CHECK lists of `sql/store.sql`. +""" + +from paperspec import jcs +from paperspec.jsontypes import Record +from paperspec.text import record_id, sha256_hex + +# The fields an annotation's ID does not cover: the ID itself, and the three that change +# while the annotation stands (spec/18 §18.2). +UNCOVERED = ("id", "status", "receipt", "modified") +# The body kinds, in the order of the schema's `body` alternatives (spec/18 §18.3). +BODY_KINDS = ( + "gloss", + "claim-link", + "symbol", + "comment", + "caveat", + "question", + "exercise", + "citation-context", + "prerequisite", + "crosslink", + "trace", + "structure", + "sequence", + "data", + "translation", + "description", + "generated", +) +# The categories of a body that takes one (spec/18 §18.4). +CATEGORIES = ( + "comment", + "context", + "critique", + "question", + "correction", + "feedback", + "resource", +) +# The verdicts of a `citation-context` or a `trace` body (spec/18 §18.5, spec/19 §19.2). +VERDICTS = ("supports", "partly", "does-not", "cannot-tell") +# The statuses of an annotation (spec/18 §18.2). +STATUSES = ("proposed", "confirmed", "rejected", "withdrawn") + + +def annotation_id(annotation: Record) -> str: + """The record ID an annotation's `id` must end with (spec/18 §18.2). + + It is 13 letter-first base32 characters of the SHA-256 of the annotation's canonical + JSON without `id`, `status`, `receipt` and `modified`. The `id` itself is + `tag:scient.ing,2026:annotation:/` and this. + + Args: + annotation: The annotation, with or without its `id`. + + Returns: + The record ID. + """ + body = {k: v for k, v in annotation.items() if k not in UNCOVERED} + return record_id(sha256_hex(jcs.dumpb(body))) + + +def anchors_of(body: object) -> list[Record]: + """The anchors of the `AnchorSelector`s of a body or a target, in order. + + Args: + body: A body or a target; anything without a `selector` list holds no anchor. + + Returns: + The `scienting-anchor/1` records, in selector order. + """ + sel = body.get("selector") if isinstance(body, dict) else None + return [ + s["anchor"] + for s in sel or [] + if isinstance(s, dict) and s.get("type") == "AnchorSelector" + ] diff --git a/src/paperspec/package.py b/src/paperspec/package.py index b6fea17..b9cc5c6 100644 --- a/src/paperspec/package.py +++ b/src/paperspec/package.py @@ -12,6 +12,7 @@ Non-normative; see spec/12-package.md and spec/22-conformance.md. import base64 import binascii import hashlib +import importlib import json import posixpath import re @@ -21,11 +22,11 @@ import tempfile import unicodedata import zipfile import zlib -from collections.abc import Iterable, Mapping, Sequence +from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence from pathlib import Path -from typing import Any, NoReturn +from typing import Any -from paperspec import crypto, hashing, jcs, keys as doc_keys, receipts, vrf +from paperspec import crypto, datasets, hashing, jcs, keys as doc_keys, receipts, vrf from paperspec.corrections import State from paperspec.jsontypes import Record, StrPath from paperspec.model import LEAVES, MATH_KINDS, RUN_KINDS, Doc @@ -135,30 +136,22 @@ def write( return sha256_hex(mb) -def _refuse_constant(name: str) -> NoReturn: - raise ValueError("JSON has no canonical form for " + name) - - -def _parse_int(text: str) -> int | float: - """Read a JSON integer as an ECMAScript engine would. - - That is exactly within 2^53, and as a double beyond (spec/01 §1.1). - """ - v = int(text) - return v if abs(v) <= 2**53 else float(v) - - -def _canonical_bytes(data: bytes, where: str) -> Any: +def parse_canonical(data: bytes, where: str) -> Any: """Parse one canonical JSON value given as bytes; refuse anything else (spec/01 §1.1, §1.9). Numbers are read as IEEE 754 doubles, as the canonical form writes them, and the comparison is on bytes, so a carriage return, a byte order mark, a lone surrogate, NaN or invalid UTF-8 is refused, not silently translated. + + Args: + data: The value's bytes, without a final newline. + where: What to name in a refusal, such as a path and line number. + + Raises: + Refusal: `data` is not canonical JSON (`package.file.canonical`). """ try: - rec = json.loads( - data.decode("utf-8"), parse_constant=_refuse_constant, parse_int=_parse_int - ) + rec = jcs.loads(data.decode("utf-8")) canon = jcs.dumps(rec).encode("utf-8") except ( UnicodeDecodeError, @@ -176,19 +169,56 @@ def _canonical_bytes(data: bytes, where: str) -> Any: return rec +# paperdesk's reader (reading.py) calls the old private name; it goes once paperdesk uses +# parse_canonical, parse_json or parse_jsonl. +_canonical_bytes = parse_canonical + + +def parse_jsonl(data: bytes, where: str) -> list[Record]: + """Parse the bytes of a canonical JSONL file, one JSON value per line (spec/01 §1.9). + + Args: + data: The file's bytes, as read from a folder or a zip entry. + where: The file's name in refusals; a line is named `where:N`. + + Raises: + Refusal: The bytes do not end with a newline, or a line is not canonical JSON. + """ + if data and not data.endswith(b"\n"): + raise Refusal( + "package.file.newline", f"{where}: every line ends with a newline" + ) + return [ + parse_canonical(ln, f"{where}:{i}") + for i, ln in enumerate(data.split(b"\n")[:-1], 1) + ] + + +def parse_json(data: bytes, where: str) -> Record: + """Parse the bytes of a canonical JSON file: one value and one final newline (spec/01 §1.9). + + Args: + data: The file's bytes, as read from a folder or a zip entry. + where: The file's name in refusals. + + Raises: + Refusal: The bytes do not end with one newline, or are not canonical JSON. + """ + if not data.endswith(b"\n"): + raise Refusal( + "package.file.canonical", + f"{where}: the file does not end with one newline", + ) + return parse_canonical(data[:-1], where) + + def read_jsonl(path: StrPath) -> list[Record]: """Read a canonical JSONL file, one JSON value per line. Raises: Refusal: The file does not end with a newline, or a line is not canonical JSON. """ - data = Path(path).read_bytes() - if data and not data.endswith(b"\n"): - raise Refusal("package.file.newline", f"{path}: every line ends with a newline") - return [ - _canonical_bytes(ln, f"{path}:{i}") - for i, ln in enumerate(data.split(b"\n")[:-1], 1) - ] + return parse_jsonl(Path(path).read_bytes(), str(path)) def read_json(path: StrPath) -> Record: @@ -197,13 +227,7 @@ def read_json(path: StrPath) -> Record: Raises: Refusal: The file does not end with one newline, or is not canonical JSON. """ - data = Path(path).read_bytes() - if not data.endswith(b"\n"): - raise Refusal( - "package.file.canonical", - f"{path}: the file does not end with one newline", - ) - return _canonical_bytes(data[:-1], str(path)) + return parse_json(Path(path).read_bytes(), str(path)) class Report: @@ -261,7 +285,12 @@ def _hash_file(full: StrPath, budget: int) -> tuple[str, int]: def validate( - root: StrPath, store: Store, key: bytes | None = None, limit: int = SIZE_LIMIT + root: StrPath, + store: Store, + key: bytes | None = None, + limit: int = SIZE_LIMIT, + *, + forms: Mapping[str, Callable[[str], str]] | None = None, ) -> Report: """Validate the package at `root`. @@ -270,20 +299,27 @@ def validate( store: The schema store that holds the package's schemas. key: The hashing key, for keyed packages. limit: The most bytes the validator reads before refusing. + forms: The math forms to check sources against (`check_forms`); the default is + `default_forms()`, which is papermath's `scienting-math/1` when it is installed. Returns: The report of every rule the package breaks; it is true when there is none. """ rep = Report() try: - _validate(root, store, key, rep, limit) + _validate(root, store, key, rep, limit, forms) except Refusal as e: rep.add(e.rule, e.node or "", e.message) return rep def _validate( - root: StrPath, store: Store, key: bytes | None, rep: Report, limit: int = SIZE_LIMIT + root: StrPath, + store: Store, + key: bytes | None, + rep: Report, + limit: int = SIZE_LIMIT, + forms: Mapping[str, Callable[[str], str]] | None = None, ) -> None: root = Path(root) mpath = root / "MANIFEST.json" @@ -489,6 +525,7 @@ def _validate( rep.add("doc.counts", "document.json", "counts do not recompute") for p in DERIVED: _check_derived(doc, p, tables[p], rep) + check_forms(doc, rep, forms) # Hashes. hasher = ( @@ -837,6 +874,112 @@ def _validate( "the fidelity distribution does not recompute", ) + # Datasets (spec/15): a checks-only package carries none, which the policy rules report. + present = [p for p, e in listed.items() if e.get("present", True) and p in actual] + if not withheld and datasets.folders(present): + values = ("text_hash", "match_hash", "align_key", "own_hash", "node_hash") + state = State( + document["key"], + [{k: v for k, v in n.items() if k not in values} for n in nodes], + ids, + cors, + references=tables["references.jsonl"], + ) + for ds in datasets.folders(present): + prefix = f"datasets/{ds}/" + files = { + p[len(prefix) :]: (root / p).read_bytes() + for p in present + if p.startswith(prefix) + } + for e in datasets.problems(ds, files, state, store): + rep.add(e["rule"], e["where"], e["message"]) + + +MATH_FORM = "scienting-math/1" +# Commands a source in scienting-math/1 never holds (spec/03 §3.7, spec/09 §9.4): the form lifts +# out `\tag` and drops `\label`, `\nonumber`, `\notag`, the style commands and `\dfrac` and +# `\tfrac`, so a source with `form` that holds one is not the form's spelling, whatever else the +# form says. A command is a backslash not itself escaped, then letters. +_NEVER_IN_FORM = re.compile( + r"(? dict[str, Callable[[str], str]]: + """The math forms this validator implements: `scienting-math/1` when papermath is installed. + + papermath is the form's reference implementation (spec/09 §9.4); with it, a source with + `form` is checked to be its own canonical spelling. paperspec does not depend on it. + + Returns: + By form, a function from a source to its canonical spelling, which raises when the + source does not read. + """ + try: + papermath = importlib.import_module("papermath") + except ImportError: + return {} + return {MATH_FORM: lambda source: papermath.canonical(source).tex} + + +def _formulas(doc: Doc) -> Iterator[tuple[str, int | None, Record]]: + """Every formula of a document that carries `form`: its node, its run index and itself.""" + for nid in doc.order: + n = doc.nodes[nid] + content = n.get("content") + if n["kind"] in MATH_KINDS and isinstance(content, dict) and "form" in content: + yield nid, None, content + elif n["kind"] in RUN_KINDS and isinstance(content, list): + for i, r in enumerate(content): + if r.get("kind") == "math" and "form" in r: + yield nid, i, r + + +def check_forms( + doc: Doc, rep: Report, forms: Mapping[str, Callable[[str], str]] | None = None +) -> None: + """Report each source with `form` that is not that form's spelling (`doc.math.form`). + + Whatever forms are implemented, a `scienting-math/1` source that holds a label or a style + command is reported, since the form never holds one (spec/03 §3.7). With an implementation + of the form, such as papermath's, every source is also spelled again and must come back + unchanged. + + Args: + doc: The document. + rep: The report the failures go to. + forms: By form, a function from a source to its canonical spelling; the default is + `default_forms()`. + """ + forms = default_forms() if forms is None else forms + for nid, run, f in _formulas(doc): + where = nid if run is None else f"{nid} run {run}" + if f["form"] == MATH_FORM and (m := _NEVER_IN_FORM.search(f["source"])): + rep.add( + "doc.math.form", + where, + f"a {MATH_FORM} source holds no \\{m.group(1)}", + ) + continue + spell = forms.get(f["form"]) + if spell is None: + continue + try: + again = spell(f["source"]) + except Exception as e: # noqa: BLE001 - any failure to read is the source's + rep.add( + "doc.math.form", where, f"the source does not read in {f['form']}: {e}" + ) + continue + if again != f["source"]: + rep.add( + "doc.math.form", + where, + f"the source is not its {f['form']} spelling, which is {again!r}", + ) + def derived_formula(doc: Doc, node: str, run: int | None) -> Record | None: """Return the formula a derived line names (spec/12 §12.7), as `{notation, source, form?}`. @@ -1015,8 +1158,12 @@ def write_zip(root: StrPath, path: StrPath) -> str: return sha256_hex(Path(path).read_bytes()) -def _zip_entries(zf: zipfile.ZipFile, limit: int) -> str: - """Check every entry of an open archive as spec/12 §12.2 requires. +def check_zip_entries(zf: zipfile.ZipFile, limit: int = ZIP_LIMIT) -> str: + """Check every entry of an open archive as spec/12 §12.2 requires, without reading any. + + Args: + zf: The open archive. + limit: The most bytes its entries may declare in all. Returns: The top-level folder every entry sits under. @@ -1071,6 +1218,56 @@ def _zip_entries(zf: zipfile.ZipFile, limit: int) -> str: return folder +# paperdesk's reader (reading.py) calls the old private name; it goes once paperdesk uses +# check_zip_entries or read_zip_files. +_zip_entries = check_zip_entries + + +def _open_zip(path: StrPath) -> zipfile.ZipFile: + try: + return zipfile.ZipFile(path) + except _ZIP_ERRORS as e: + raise Refusal( + "package.zip.entry", f"not a zip archive this reader can open: {e}" + ) from e + + +def read_zip_files( + path: StrPath, limit: int = ZIP_LIMIT +) -> tuple[str, dict[str, bytes]]: + """Read a zipped package into memory, refusing what spec/12 §12.2 refuses, as `read_zip` does. + + For a consumer that reads a package without unpacking it to a folder. Every byte read is + counted against each entry's declared size and against `limit`. + + Returns: + The top-level folder, which is the key, and the bytes of every file by its path inside + the package. + + Raises: + Refusal: The archive is damaged, or an entry or the total size breaks a rule. + """ + with _open_zip(path) as zf: + folder = check_zip_entries(zf, limit) + files, total = {}, 0 + for zi in zf.infolist(): + if zi.is_dir(): + continue + try: + with zf.open(zi) as src: + data = src.read(zi.file_size + 1) + except _ZIP_ERRORS as e: + raise Refusal("package.zip.entry", f"{zi.filename}: {e}") from e + total += len(data) + if len(data) > zi.file_size or total > limit: + raise Refusal( + "package.zip.size", + f"{zi.filename}: holds more than its header declares", + ) + files[zi.filename.partition("/")[2]] = data + return folder, files + + def read_zip(path: StrPath, dest: StrPath, limit: int = ZIP_LIMIT) -> str: """Unpack a zipped package into the empty or new folder `dest`. @@ -1083,14 +1280,8 @@ def read_zip(path: StrPath, dest: StrPath, limit: int = ZIP_LIMIT) -> str: Raises: Refusal: The archive is damaged, or an entry or the total size breaks a rule. """ - try: - zf = zipfile.ZipFile(path) - except _ZIP_ERRORS as e: - raise Refusal( - "package.zip.entry", f"not a zip archive this reader can open: {e}" - ) from e - with zf: - folder = _zip_entries(zf, limit) + with _open_zip(path) as zf: + folder = check_zip_entries(zf, limit) root = Path(dest) / folder root.mkdir(parents=True) written = 0 @@ -1121,11 +1312,16 @@ def read_zip(path: StrPath, dest: StrPath, limit: int = ZIP_LIMIT) -> str: def validate_zip( - path: StrPath, store: Store, key: bytes | None = None, limit: int = ZIP_LIMIT + path: StrPath, + store: Store, + key: bytes | None = None, + limit: int = ZIP_LIMIT, + *, + forms: Mapping[str, Callable[[str], str]] | None = None, ) -> Report: """Validate a zipped package: unpack it under spec/12 §12.2's rules, then validate the folder. - `limit` is the most bytes the archive may expand to. + `limit` is the most bytes the archive may expand to; `forms` is as for `validate`. """ with tempfile.TemporaryDirectory() as tmp: try: @@ -1134,7 +1330,7 @@ def validate_zip( rep = Report() rep.add(e.rule, e.node or "", e.message) return rep - rep = validate(root, store, key) + rep = validate(root, store, key, forms=forms) mpath = Path(root, "MANIFEST.json") if mpath.exists(): try: diff --git a/src/paperspec/resources.py b/src/paperspec/resources.py index 56dad79..81abbeb 100644 --- a/src/paperspec/resources.py +++ b/src/paperspec/resources.py @@ -4,25 +4,29 @@ # Source Code: https://git.scient.ing/papers/paperspec # See the LICENSE.md file for full license terms. -"""Where the schemas, SQL and vectors are, as filesystem paths. +"""Where the schemas, SQL, vectors, registries, JSON-LD context and fixtures are, as paths. -An installed wheel carries them inside the package (`paperspec/schemas`, `paperspec/sql`, -`paperspec/vectors`). An editable install or a checkout keeps them at the repository root, where -the specification links to them, so `importlib.resources.files("paperspec") / "sql"` finds nothing -there; `path("sql")` finds both. +An installed wheel carries all but the fixtures inside the package (`paperspec/schemas`, +`paperspec/sql`, `paperspec/vectors`, `paperspec/registry`, `paperspec/context`). An editable +install or a checkout keeps them at the repository root, where the specification links to them, +so `importlib.resources.files("paperspec") / "sql"` finds nothing there; `path("sql")` finds +both. The fixtures, which some vectors name, are found only in a checkout or an editable +install of one. """ +import json from pathlib import Path +from typing import Any -FOLDERS = ("schemas", "sql", "vectors") +FOLDERS = ("schemas", "sql", "vectors", "registry", "context", "fixtures") _PACKAGE = Path(__file__).absolute().parent def path(name: str) -> str: - """Return the folder `schemas`, `sql` or `vectors`. + """Return the folder `schemas`, `sql`, `vectors`, `registry`, `context` or `fixtures`. Raises: - ValueError: `name` is not one of the three folders. + ValueError: `name` is not one of `FOLDERS`. FileNotFoundError: The folder is neither in the package nor at the root of its checkout. """ if name not in FOLDERS: @@ -33,3 +37,20 @@ def path(name: str) -> str: raise FileNotFoundError( f"the {name} folder is neither in the package nor at the root of its checkout" ) + + +def registry(name: str) -> Any: + """Return a registry file as JSON, by its path under `registry/`. + + Args: + name: Such as `formats.json`, `rules.json`, `stamps/scienting-stamps-1.json` or + `stages/scienting-stages-default.json`. + + Raises: + ValueError: `name` is not a plain relative path to a JSON file. + FileNotFoundError: There is no such registry file. + """ + parts = name.split("/") + if not name.endswith(".json") or any(p in ("", ".", "..") for p in parts): + raise ValueError(f"no registry file {name!r}") + return json.loads((Path(path("registry")) / name).read_text(encoding="utf-8")) diff --git a/src/paperspec/revision.py b/src/paperspec/revision.py new file mode 100644 index 0000000..508e90e --- /dev/null +++ b/src/paperspec/revision.py @@ -0,0 +1,554 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""Apply corrections to a package and write the package they make (spec/11, spec/12). + +Non-normative. +`corrections.gate` says what a correction would do to a document's state; this module writes +the result back as a package. Every proposal runs through the gate in order, and the package is +written only when all of them pass: + +- `nodes.jsonl` in document order, with every value recomputed, and the leaves a correction + changed at the level its evidence gives them ([`level_after`](#level_after)); +- `corrections.jsonl`, `ids.jsonl` and `references.jsonl` from the state the gate returns; +- `locations.jsonl` without the nodes that are gone, a merged node keeping the boxes of the + node merged into it; +- `suspects.jsonl` without the suspects at nodes that are gone and those named as answered; +- `derived/` without the lines of formulas a correction touched, and `datasets/` without a + dataset whose anchored cells a correction touched or removed, which a producer that has the + tools rebuilds (spec/12 §12.7, spec/15); +- `document.json` with its root, revision, history root, counts and fidelity recomputed, and + `provenance.json` with the tool that applied the corrections, when one is named; +- and `MANIFEST.json` over all of it. Everything else is copied byte for byte. + +The package written is validated before it replaces anything. +""" + +import copy +import hashlib +import shutil +import tempfile +from collections.abc import Iterable, Mapping, Sequence +from pathlib import Path +from typing import Any + +from paperspec import corrections, crypto, hashing, jcs, package, receipts +from paperspec.jsontypes import Record, StrPath +from paperspec.model import LEAVES, Doc +from paperspec.schema import Store +from paperspec.tree import Refusal + +# The node keys a package stores besides the five values, in nodes.jsonl (spec/03 §3.1). +_OPTIONAL = ("label", "attrs", "content", "withheld", "fidelity", "prov") +VALUES = ("text_hash", "match_hash", "align_key", "own_hash", "node_hash") + + +class Refused(Exception): # noqa: N818 - a proposal the gate refused, as Refusal is a broken rule + """A proposal the gate refused; nothing was written. + + Attributes: + index: The proposal's position among those given, from 1. + rule: The rule it breaks (spec/22). + message: What the gate said. + node: The node the refusal names, if any. + """ + + def __init__(self, index: int, refusal: Refusal) -> None: + self.index = index + self.rule = refusal.rule + self.message = refusal.message + self.node = refusal.node + super().__init__(f"proposal {index}: {refusal.rule}: {refusal.message}") + + +class Invalid(Exception): # noqa: N818 - the package as it is, not an error of this module + """The package does not validate, so no correction is applied to it. + + Attributes: + errors: The validation errors, each `{rule, where, message}`. + """ + + def __init__(self, errors: Sequence[Record]) -> None: + self.errors = list(errors) + first = self.errors[0] if self.errors else {"rule": "?", "message": ""} + super().__init__( + f"the package is not valid ({len(self.errors)} errors; the first is " + f"{first['rule']}: {first['message']})" + ) + + +def has_crop(correction: Record) -> bool: + """Whether a correction's evidence includes a crop of the page (spec/11 §11.2).""" + return any(e.get("kind") == "crop" for e in correction.get("evidence", [])) + + +def level_after(correction: Record) -> str | None: + """The level a correction leaves the leaves it changes at, or None when it keeps their level. + + A correction that changes a leaf drops it back (spec/08 §8.1): a crop of the page answers + it against the page image, which makes the leaf F1, and without one the leaf is F0 until it + is corroborated or proofread again. An `emend`, `relink` or `resolve` keeps the level, + since the leaf as printed is unchanged. + """ + if correction["op"] in corrections.KEEPS_LEVEL: + return None + return "F1" if has_crop(correction) else "F0" + + +def changed_leaves(correction: Record) -> list[str]: + """The leaves a correction changed or created, as its record names them (spec/11 §11.3).""" + return [*correction.get("touched", []), *correction.get("created", [])] + + +def node_record(node: Record, values: Mapping[str, str]) -> Record: + """The `nodes.jsonl` record of a node: its stored keys, without empty ones, and its values.""" + rec = {k: node[k] for k in ("id", "kind", "parent", "ord")} + for k in _OPTIONAL: + if k in node and node[k] not in (None, {}, []): + rec[k] = node[k] + rec.update(values) + return rec + + +def summarize( + document: Record, + nodes: Sequence[Record], + values: Mapping[str, Record], + records: Sequence[Record], + receipt_records: Sequence[Record] = (), + roster: Sequence[Record] = (), +) -> Record: + """Return `document.json` with the values that follow from its nodes recomputed (spec/02 §2.2). + + They are the root's `node_hash`, the revision, the history root (spec/11 §11.5), the counts, + the fidelity distribution (spec/08 §8.2) and `meta_hash` (spec/05 §5.9). + + Args: + document: The document record; it is not changed. + nodes: The node records, the root first, each with its `fidelity` where it has one. + values: The five values of every node, by ID. + records: Every correction record, in order. + receipt_records: The package's receipts and revocations. + roster: The roster's keys; without them no receipt counts. + """ + root = nodes[0]["id"] + levels = receipts.effective_levels( + list(nodes), + dict(values), + list(receipt_records), + {k["id"]: k for k in roster}, + list(records), + ) + d = dict(document) + d.update( + { + "root": values[root]["node_hash"], + "rev": len(records), + "history": crypto.history_root(package.history_items(records)), + "counts": { + "nodes": len(nodes), + "leaves": sum(1 for n in nodes if n["kind"] in LEAVES), + }, + "fidelity": receipts.distribution(levels), + } + ) + d["meta_hash"] = hashing.meta_hash(d) + return d + + +def _jsonl(files: Mapping[str, bytes], name: str) -> list[Record]: + data = files.get(name) + return package.parse_jsonl(data, name) if data else [] + + +def _put(files: dict[str, bytes], name: str, rows: Sequence[Any]) -> None: + """Write a JSON Lines file, or leave it out when it has no records (spec/12 §12.1).""" + if rows: + files[name] = package.jsonl(rows) + else: + files.pop(name, None) + + +def _json(rec: Any) -> bytes: + return (jcs.dumps(rec) + "\n").encode("utf-8") + + +def _locations( + rows: Sequence[Record], + live: set[str], + successors: Mapping[str, str], + pos: Mapping[str, int], +) -> list[Record]: + """The locations of the live nodes; a node merged into another gives it its boxes. + + A live node's records are kept as they are. A retired node's go to its successor, joining + the successor's record for the same page when there is one, so that a merged sentence is + located wherever its two parts were. + """ + kept = [dict(r) for r in rows if r["node"] in live] + held = {(r["node"], r["page"], r["page_fp"]): r for r in kept} + for r in rows: + node = r["node"] + if node in live: + continue + while node not in live and node in successors: + node = successors[node] + if node not in live: + continue + k = (node, r["page"], r["page_fp"]) + if k in held: + h = held[k] + h["boxes"] = h["boxes"] + [b for b in r["boxes"] if b not in h["boxes"]] + if r.get("approx"): + h["approx"] = True + else: + held[k] = dict(r, node=node) + kept.append(held[k]) + # A stable sort, so records that tie keep the order the package gave them. + return sorted(kept, key=lambda r: (pos[r["node"]], r["page"], r["page_fp"])) + + +def _drop_stale( + files: dict[str, bytes], gone: set[str], live: set[str] +) -> dict[str, int]: + """Leave out derived lines and datasets that read text the corrections changed. + + Returns: + How many lines of each derived file, and how many files of each dataset, were left out. + """ + dropped: dict[str, int] = {} + for path in sorted( + p for p in files if p.startswith("derived/") and p.endswith(".jsonl") + ): + rows = _jsonl(files, path) + kept = [r for r in rows if r.get("node") in live and r.get("node") not in gone] + if len(kept) != len(rows): + dropped[path] = len(rows) - len(kept) + _put(files, path, kept) + datasets = sorted( + { + p.split("/")[1] + for p in files + if p.startswith("datasets/") and p.count("/") >= 2 + } + ) + for ds in datasets: + inside = [p for p in files if p.startswith(f"datasets/{ds}/")] + cells = { + ln["anchor"].get("node") + for p in inside + if p.endswith(".jsonl") + for ln in _jsonl(files, p) + if isinstance(ln.get("anchor"), dict) + } + if cells & gone or cells - live: + for p in inside: + del files[p] + dropped[f"datasets/{ds}/"] = len(inside) + return dropped + + +def revise_files( + files: Mapping[str, bytes], + absent: Sequence[Record], + proposals: Iterable[Record], + store: Store | None, + *, + answered: Iterable[str] = (), + assets: Iterable[tuple[bytes, str]] = (), + tool: Record | None = None, +) -> tuple[dict[str, bytes], list[Record], Record]: + """Apply proposals to a package held in memory and return the files it becomes. + + Args: + files: Every present file of a valid package by path, MANIFEST.json left out. + absent: The manifest entries of assets that stay in the asset store. + proposals: The proposed corrections, applied in order. + store: The schema store; with None the gate's schema checks are skipped, as + `corrections.apply` skips them, which only a caller that validated the proposals + should do. + answered: IDs of suspects the corrections answer, which leave `suspects.jsonl`. + assets: Files to add under `assets/`, each its bytes and its extension, such as the + crops the corrections' evidence names; each is named by its SHA-256. + tool: `{name, version}` of the program applying the corrections, recorded in + `provenance.json` with a `review` stage. + + Returns: + The package's files without MANIFEST.json, the records appended to `corrections.jsonl`, + and a summary: `rev`, `dropped` (derived lines and dataset files left out, by path) and + `answered`. + + Raises: + Refused: A proposal breaks a rule; nothing is returned. + ValueError: The package withholds its text or is keyed, or an answered suspect is not in + `suspects.jsonl`. + """ + files = dict(files) + document = package.parse_json(files["document.json"], "document.json") + if document.get("policy") != "full" or document["hashing"]["scheme"] != "plain": + raise ValueError( + "a package that withholds its text or is keyed takes no corrections (spec/12 §12.5)" + ) + stored = _jsonl(files, "nodes.jsonl") + state = corrections.State( + document["key"], + [{k: v for k, v in n.items() if k not in VALUES} for n in stored], + _jsonl(files, "ids.jsonl"), + _jsonl(files, "corrections.jsonl"), + references=_jsonl(files, "references.jsonl"), + ) + base = state.rev + for i, proposal in enumerate(proposals, 1): + try: + if store is None: + state, _ = corrections.apply(state, proposal) + else: + state, _ = corrections.gate(state, proposal, store) + except Refusal as e: + raise Refused(i, e) from e + added = state.corrections[base:] + + nodes = copy.deepcopy(state.nodes) + by_id = {n["id"]: n for n in nodes} + for c in added: + level = level_after(c) + if level is None: + continue + for x in changed_leaves(c): + n = by_id.get(x) + if n is None or n["kind"] not in LEAVES: + continue + n["fidelity"] = level + # The witness corroborated a reading the correction replaced. + prov = { + k: v for k, v in n.get("prov", {}).items() if k != "corroborated_by" + } + if prov: + n["prov"] = prov + else: + n.pop("prov", None) + doc = Doc(document["key"], nodes) + values = hashing.compute(doc) + order = doc.preorder() + pos = {nid: i for i, nid in enumerate(order)} + live = set(order) + records = [node_record(by_id[nid], values[nid]) for nid in order] + + gone = { + x + for c in added + for x in (c.get("target"), *c.get("touched", []), *c.get("created", [])) + if x + } + dropped = _drop_stale(files, gone, live) + successors = { + e["id"]: e["successor"] + for e in state.ids + if e["kind"] == "retired" and e.get("successor") + } + _put(files, "nodes.jsonl", records) + _put(files, "corrections.jsonl", state.corrections) + _put(files, "ids.jsonl", sorted(state.ids, key=corrections.id_order)) + _put( + files, + "references.jsonl", + sorted( + (r for r in state.references if r["node"] in live), + key=lambda r: pos[r["node"]], + ), + ) + _put( + files, + "locations.jsonl", + _locations(_jsonl(files, "locations.jsonl"), live, successors, pos), + ) + answered = set(answered) + suspects = _jsonl(files, "suspects.jsonl") + unknown = answered - {s["id"] for s in suspects} + if unknown: + raise ValueError(f"no suspect {', '.join(sorted(unknown))} in suspects.jsonl") + _put( + files, + "suspects.jsonl", + [ + s + for s in suspects + if s["id"] not in answered and (s.get("node") is None or s["node"] in live) + ], + ) + for data, ext in assets: + files[ + f"assets/{hashlib.sha256(data).hexdigest()}.{ext.lstrip('.').lower()}" + ] = data + if tool is not None: + prov = package.parse_json(files["provenance.json"], "provenance.json") + tools = prov.setdefault("tools", []) + if tool not in tools: + tools.append(dict(tool)) + stage = { + "stage": "review", + "processor": "local", + "tool": f"{tool['name']} {tool['version']}", + } + stages = prov.setdefault("stages", []) + if stage not in stages: + stages.append(stage) + files["provenance.json"] = _json(prov) + + roster = ( + package.parse_json(files["roster.json"], "roster.json").get("keys", []) + if "roster.json" in files + else [] + ) + files["document.json"] = _json( + summarize( + document, + records, + values, + state.corrections, + _jsonl(files, "receipts.jsonl"), + roster, + ) + ) + del absent # listed again by the caller as it was + return ( + files, + added, + {"rev": state.rev, "dropped": dropped, "answered": sorted(answered)}, + ) + + +def _read_folder(root: Path) -> tuple[dict[str, bytes], list[Record]]: + manifest = package.read_json(root / "MANIFEST.json") + files, absent = {}, [] + for e in manifest["files"]: + if e.get("present") is False: + absent.append(dict(e)) + else: + files[e["path"]] = (root / e["path"]).read_bytes() + return files, absent + + +def revise( + root: StrPath, + proposals: Iterable[Record], + store: Store, + out: StrPath | None = None, + *, + answered: Iterable[str] = (), + assets: Iterable[tuple[bytes, str]] = (), + tool: Record | None = None, +) -> Record: + """Apply proposed corrections to the package folder at `root` and write the package they make. + + The package is validated first, every proposal runs through the gate in order, and the + package written is validated before it replaces anything: a refusal leaves every file as it + was. + + Args: + root: The package folder. + proposals: The proposed corrections (spec/11 §11.2), applied in order. + store: The schema store. + out: A folder to write the corrected package into, which must not exist; with None the + package at `root` is replaced. + answered: IDs of suspects the corrections answer, which leave `suspects.jsonl`. + assets: Files to add under `assets/`, each its bytes and its extension. + tool: `{name, version}` of the program applying the corrections, for `provenance.json`. + + Returns: + `{key, rev, package, records, dropped, answered, path}`: the document key, its new + revision, the SHA-256 of the new MANIFEST.json, the records appended to + `corrections.jsonl`, the derived lines and dataset files left out, the answered + suspects, and the folder written. + + Raises: + Invalid: The package does not validate. + Refused: A proposal breaks a rule. + ValueError: The package takes no corrections, an answered suspect is unknown, or `out` + exists. + RuntimeError: The package written does not validate, which is a fault of this module. + """ + root = Path(root) + report = package.validate(root, store) + if report.errors: + raise Invalid(report.errors) + if out is not None and Path(out).exists(): + raise ValueError(f"{out} exists; give a folder that does not") + files, absent = _read_folder(root) + new, added, summary = revise_files( + files, absent, proposals, store, answered=answered, assets=assets, tool=tool + ) + dest = Path(out) if out is not None else root + dest.parent.mkdir(parents=True, exist_ok=True) + work = Path(tempfile.mkdtemp(prefix=f".{dest.name}.revise-", dir=dest.parent)) + try: + built = work / dest.name + sha = package.write(built, new, absent) + check = package.validate(built, store) + if check.errors: + e = check.errors[0] + raise RuntimeError( + f"the corrected package does not validate: {e['rule']}: {e['message']}" + ) + if out is None: + old = work / (dest.name + ".old") + root.rename(old) + built.rename(root) + else: + built.rename(dest) + finally: + shutil.rmtree(work, ignore_errors=True) + return { + "key": package.parse_json(new["document.json"], "document.json")["key"], + "rev": summary["rev"], + "package": sha, + "records": added, + "dropped": summary["dropped"], + "answered": summary["answered"], + "path": str(dest), + } + + +def revise_zip( + path: StrPath, + proposals: Iterable[Record], + store: Store, + out: StrPath | None = None, + *, + answered: Iterable[str] = (), + assets: Iterable[tuple[bytes, str]] = (), + tool: Record | None = None, + limit: int = package.ZIP_LIMIT, +) -> Record: + """Apply proposed corrections to a zipped package, as `revise` does to a folder. + + The archive is unpacked under spec/12 §12.2's rules, corrected, and zipped again as + `package.write_zip` writes, to `out` or in place of `path`. + + Returns: + What `revise` returns, with `path` the archive written and `zip` its SHA-256. + """ + path = Path(path) + dest = Path(out) if out is not None else path + if out is not None and dest.exists(): + raise ValueError(f"{out} exists; give a file that does not") + with tempfile.TemporaryDirectory() as tmp: + root = Path(package.read_zip(path, Path(tmp) / "in", limit)) + result = revise( + root, + proposals, + store, + Path(tmp) / "out" / root.name, + answered=answered, + assets=assets, + tool=tool, + ) + staged = Path(tmp) / dest.name + result["zip"] = package.write_zip(result["path"], staged) + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.move(staged, dest) + result["path"] = str(dest) + return result diff --git a/src/paperspec/text.py b/src/paperspec/text.py index ebae08c..b55e1a3 100644 --- a/src/paperspec/text.py +++ b/src/paperspec/text.py @@ -38,6 +38,14 @@ WHITE_SPACE = frozenset( ) +_WS_CHARS = "".join(chr(c) for c in sorted(WHITE_SPACE)) +_WS_RUN = re.compile("[" + re.escape(_WS_CHARS) + "]+") +_NO_WS = dict.fromkeys(WHITE_SPACE) +# In a str pattern, \d is exactly General Category Nd, from the same character data as +# unicodedata. +_DIGIT = re.compile(r"\d") + + def is_ws(ch: str) -> bool: """Whether the character is Unicode White_Space.""" return ord(ch) in WHITE_SPACE @@ -45,32 +53,17 @@ def is_ws(ch: str) -> bool: def collapse_ws(s: str) -> str: """Replace each run of whitespace in `s` by one U+0020, without trimming.""" - out = [] - prev_ws = False - for ch in s: - if is_ws(ch): - if not prev_ws: - out.append(" ") - prev_ws = True - else: - out.append(ch) - prev_ws = False - return "".join(out) + return _WS_RUN.sub(" ", s) def trim(s: str) -> str: """Strip Unicode White_Space from both ends of `s`.""" - a, b = 0, len(s) - while a < b and is_ws(s[a]): - a += 1 - while b > a and is_ws(s[b - 1]): - b -= 1 - return s[a:b] + return s.strip(_WS_CHARS) def remove_ws(s: str) -> str: """`s` without any whitespace character.""" - return "".join(ch for ch in s if not is_ws(ch)) + return s.translate(_NO_WS) def norm(s: str) -> str: @@ -92,7 +85,7 @@ def is_digit(ch: str) -> bool: def digits(s: str) -> str: """The characters of `s` with General Category Nd, in order.""" - return "".join(ch for ch in s if is_digit(ch)) + return "".join(_DIGIT.findall(s)) # Crockford base32, lowercase. diff --git a/src/paperspec/traces.py b/src/paperspec/traces.py index 638a7e1..22f7cbf 100644 --- a/src/paperspec/traces.py +++ b/src/paperspec/traces.py @@ -19,7 +19,7 @@ here needs more than the standard library. import re from collections.abc import Iterable, Sequence -from paperspec import anchors, hashing +from paperspec import anchors, hashing, layers from paperspec.corrections import State from paperspec.jsontypes import Record from paperspec.model import RUN_KINDS @@ -39,14 +39,8 @@ def doc_tag(store: str, key: str) -> str: return f"tag:scient.ing,2026:doc:{store}/{key}" -def _anchors(holder: object) -> list[Record]: - """The anchors of the `AnchorSelector`s of a body or a target, in order.""" - sel = holder.get("selector") if isinstance(holder, dict) else None - return [ - s["anchor"] - for s in sel or [] - if isinstance(s, dict) and s.get("type") == "AnchorSelector" - ] +# paperdesk's tracing.py uses the private name until it moves to layers.anchors_of. +_anchors = layers.anchors_of def _bare(doc: str) -> str: @@ -67,7 +61,7 @@ def _live(state: State, nid: str) -> str | None: None when the node was deleted or never was. """ - return anchors._follow(state, nid)[0] + return anchors.follow(state, nid)[0] def _record(state: State, nid: str) -> Record: @@ -163,7 +157,7 @@ def _sides( body = annotation.get("body", {}) if body.get("kind") != "trace": raise ValueError("the annotation is not a trace") - mine, theirs = _anchors(annotation["target"]), _anchors(body) + mine, theirs = layers.anchors_of(annotation["target"]), layers.anchors_of(body) if not mine or not theirs: raise ValueError( "a trace needs an AnchorSelector on its target and one in its body" @@ -321,7 +315,7 @@ def problems( found.append({"rule": rule, "message": message}) body, target = annotation["body"], annotation["target"] - mine, passage = _anchors(target), _anchors(body) + mine, passage = layers.anchors_of(target), layers.anchors_of(body) if citing_state is not None and mine and mine[0]["doc"] != citing_state.key: raise ValueError("citing_state is not the document the trace's target is on") if cited_state is not None and passage and passage[0]["doc"] != cited_state.key: diff --git a/src/paperspec/tree.py b/src/paperspec/tree.py index d776210..3808fd6 100644 --- a/src/paperspec/tree.py +++ b/src/paperspec/tree.py @@ -168,14 +168,45 @@ def _resolve_child_refs(tmp: list[Record]) -> None: n["content"] = canonical_runs(runs) +def flatten(spec: Record) -> list[Record]: + """Return the nodes of a nested spec in document order, without disturbing `spec`. + + Each node drops `children` and gains `_parent`, the index of its parent in the list, and + `ord`, its position among its siblings; every other key is kept, including a producer's own + private keys (those starting with `_`), which `mint_ids` then leaves out of the records. + """ + flat: list[Record] = [] + _flatten(copy.deepcopy(spec), None, 0, flat) + return flat + + def establish(doc_key: str, spec: Record, used: set[str] | None = None) -> list[Record]: """Turn a nested spec into flat node records with minted IDs, in document order. `used` holds IDs already taken and receives the minted ones. """ - flat = [] - _flatten(copy.deepcopy(spec), None, 0, flat) - return mint_ids(doc_key, flat, used if used is not None else set()) + return mint_ids(doc_key, flatten(spec), used if used is not None else set()) + + +def establish_with_side( + doc_key: str, spec: Record, used: set[str] | None = None +) -> tuple[list[Record], dict[str, Record]]: + """Establish a spec as `establish` does, keeping each spec node's private keys by its new ID. + + A producer marks spec nodes with keys of its own, such as `{"_name": "tab1"}`, to find out + which node each became: the records leave those keys out, and the side table returns them. + + Returns: + The node records in document order, and by node ID the private keys of the spec node it + was minted from (an empty record for a node that had none). + """ + flat = flatten(spec) + nodes = mint_ids(doc_key, flat, used if used is not None else set()) + side = { + n["id"]: {k: v for k, v in f.items() if k.startswith("_") and k != "_parent"} + for f, n in zip(flat, nodes, strict=True) + } + return nodes, side def validate_structure(doc: Doc) -> None: diff --git a/tests/test_anchors.py b/tests/test_anchors.py index 899e4f9..7cc7efc 100644 --- a/tests/test_anchors.py +++ b/tests/test_anchors.py @@ -6,11 +6,13 @@ """Addresses, anchors and their resolution (spec/06).""" +import json from fractions import Fraction import pytest from hypothesis import given, settings, strategies as st +from conftest import ROOT from helpers_lib1 import ( HALE, build_doc, @@ -820,3 +822,161 @@ class TestResolveContainers: got["reason"] == "the successor holds fewer than half of the original leaves" ) + + +# --- following IDs and resolving addresses ------------------------------------------ + +KEY = "hale2026harmonic-damb" + + +def _jsonl(name): + """The records of `nodes.jsonl` or `ids.jsonl` of the Hale package, as stored.""" + path = ROOT / HALE / f"{name}.jsonl" + return [json.loads(line) for line in path.read_text("utf-8").splitlines()] + + +class TestFollow: + def test_follow_is_the_old_private_name(self): + assert anchors._follow is anchors.follow + + def test_a_live_node_is_followed_to_itself(self, tree): + assert anchors.follow(tree["state"], tree["s1"]) == (tree["s1"], [], None) + + def test_an_alias_gives_a_chain_and_the_alias_kind(self, tree): + ids = [{"id": "aaaaaaa", "kind": "alias", "successor": tree["s1"]}] + got = anchors.follow(_edit(tree["state"], {}, ids=ids), "aaaaaaa") + assert got == (tree["s1"], ["aaaaaaa"], "alias") + + def test_a_retirement_makes_the_chain_retired_even_after_an_alias(self, tree): + ids = [ + {"id": "aaaaaaa", "kind": "alias", "successor": "bbbbbbb"}, + {"id": "bbbbbbb", "kind": "retired", "successor": tree["s1"]}, + ] + got = anchors.follow(_edit(tree["state"], {}, ids=ids), "aaaaaaa") + assert got == (tree["s1"], ["aaaaaaa", "bbbbbbb"], "retired") + + def test_a_deletion_ends_the_chain_without_a_node(self, tree): + ids = [{"id": "ddddddd", "kind": "retired", "successor": None}] + got = anchors.follow(_edit(tree["state"], {}, ids=ids), "ddddddd") + assert got == (None, ["ddddddd"], "retired") + + def test_an_unknown_id_gives_no_node_and_no_chain(self, tree): + assert anchors.follow(tree["state"], "zzzzzzz") == (None, [], None) + + +class TestResolveAddress: + @pytest.fixture + def hale(self): + nodes = _jsonl("nodes") + sentence_id = next(n["id"] for n in nodes if n["kind"] == "sentence") + return { + "nodes": nodes, + "ids": _jsonl("ids"), + "sid": sentence_id, + "hash": next(n["node_hash"] for n in nodes if n["id"] == sentence_id), + } + + def _resolve(self, hale, tail, **kw): + return anchors.resolve_address( + hale["nodes"], hale["ids"], f"{KEY}#{tail}", **kw + ) + + def test_an_unpinned_address_of_a_live_node_is_current(self, hale): + assert self._resolve(hale, hale["sid"]) == { + "result": "current", + "node": hale["sid"], + "node_hash": hale["hash"], + } + + def test_a_pin_that_starts_the_node_hash_is_current(self, hale): + got = self._resolve(hale, f"{hale['sid']}@{hale['hash'][:6]}") + assert got["result"] == "current" + + def test_a_pin_that_differs_is_stale(self, hale): + other = "000000" if not hale["hash"].startswith("000000") else "ffffff" + got = self._resolve(hale, f"{hale['sid']}@{other}") + assert got["result"] == "stale" + assert got["reason"] == "the pin differs from the node hash" + assert got["node"] == hale["sid"] + + def test_a_span_is_carried_to_a_current_result(self, hale): + got = self._resolve(hale, f"{hale['sid']}@{hale['hash'][:6]}[3:9]") + assert got["span"] == [3, 9] + + def test_an_empty_span_is_a_valid_address_and_resolves(self, hale): + assert anchors.parse_address(f"{KEY}#{hale['sid']}[4:4]")["span"] == [4, 4] + assert self._resolve(hale, f"{hale['sid']}[4:4]")["span"] == [4, 4] + + def test_a_reversed_span_is_not_an_address(self, hale): + assert self._resolve(hale, f"{hale['sid']}[9:3]") == { + "result": "stale", + "node": None, + "reason": "not an address", + } + + def test_the_grammar_takes_any_node_part_so_a_short_one_parses(self): + assert anchors.parse_address(f"{KEY}#n1")["node"] == "n1" + + def test_an_alias_is_followed_and_its_pin_judged(self, hale): + node = next(n for n in hale["nodes"] if n["id"] == "hca6dc5") + got = self._resolve(hale, f"b-p01-text-003@{node['node_hash'][:6]}") + assert got == { + "result": "current", + "node": "hca6dc5", + "node_hash": node["node_hash"], + "via": ["b-p01-text-003"], + } + + def test_a_retirement_with_a_successor_is_stale_and_names_the_successor(self, hale): + got = self._resolve(hale, "ayea71r") + assert got["result"] == "stale" + assert got["reason"] == "retired" + assert got["node"] == "xjns57z" + assert got["via"] == ["ayea71r"] + + def test_a_deletion_is_stale_without_a_node(self, hale): + assert self._resolve(hale, "v8e4z6z") == { + "result": "stale", + "node": None, + "via": ["v8e4z6z"], + "reason": "retired without a successor", + } + + def test_an_unknown_node_is_stale(self, hale): + assert self._resolve(hale, "zzzzzzz") == { + "result": "stale", + "node": None, + "reason": "unknown node", + } + + def test_text_that_is_not_an_address_is_stale(self, hale): + got = anchors.resolve_address(hale["nodes"], hale["ids"], "nonsense") + assert got["reason"] == "not an address" + + def test_given_values_replace_the_stored_node_hash(self, hale): + values = {hale["sid"]: {"node_hash": "abcdef" + "0" * 58}} + got = self._resolve(hale, f"{hale['sid']}@abcdef", values=values) + assert got["result"] == "current" + assert got["node_hash"] == "abcdef" + "0" * 58 + assert ( + self._resolve(hale, f"{hale['sid']}@{hale['hash'][:6]}", values=values)[ + "result" + ] + == "stale" + ) + + def test_a_node_without_a_hash_fails_a_pin_but_not_an_unpinned_address(self): + nodes = [{"id": "n1"}] + assert anchors.resolve_address(nodes, [], f"{KEY}#n1@abcdef")["result"] == ( + "stale" + ) + assert anchors.resolve_address(nodes, [], f"{KEY}#n1")["result"] == "current" + + def test_it_agrees_with_resolve_on_every_node_of_the_unchanged_document(self): + nodes, ids = _jsonl("nodes"), _jsonl("ids") + for n in nodes: + got = anchors.resolve_address( + nodes, ids, f"{KEY}#{n['id']}@{n['node_hash'][:6]}" + ) + assert got["result"] == "current" + assert got["node"] == n["id"] diff --git a/tests/test_claims.py b/tests/test_claims.py new file mode 100644 index 0000000..c8deb00 --- /dev/null +++ b/tests/test_claims.py @@ -0,0 +1,203 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""Claim references and pins (spec/18 §18.4).""" + +import json +import re + +import pytest +from hypothesis import given, settings, strategies as st + +from conftest import FIXTURES, ROOT +from paperspec import claims + +PROPERTIES = settings(derandomize=True, max_examples=200, database=None, deadline=None) + +SCHEMA = json.loads((ROOT / "schemas" / "common.schema.json").read_text("utf-8")) +# ECMAScript `$` matches only at the end; Python's also matches before a final newline. +REF_PATTERN = SCHEMA["$defs"]["claimRef"]["pattern"].removesuffix("$") + r"\Z" +PIN_PATTERN = SCHEMA["$defs"]["claimPin"]["pattern"].removesuffix("$") + r"\Z" + +TAGS = st.text(alphabet="0123456789ABCDEFGHJKMNPQRSTVWXYZ", min_size=8, max_size=8) +HOSTS = st.one_of( + st.none(), + st.from_regex(r"[a-z0-9][a-z0-9-]{0,8}\.[a-z]{2,6}", fullmatch=True), +) +HASHES = st.text(alphabet="0123456789abcdef", min_size=64, max_size=64) + + +def _fixture_texts() -> list[str]: + found = set() + for path in FIXTURES.rglob("*"): + if path.is_file(): + text = path.read_text("utf-8", errors="ignore") + found.update(re.findall(r"claim:[A-Za-z0-9./@_-]+", text)) + return sorted(found) + + +def test_the_patterns_are_those_of_the_schema(): + schema_ref, schema_pin = re.compile(REF_PATTERN), re.compile(PIN_PATTERN) + samples = [ + "claim:272XTX7C", + "claim:272XTX7C@02f47b", + "claim:example.org/272XTX7C", + "claim:example.org/272XTX7C@02f47b", + "claim:272XTX7I", + "claim:272xtx7c", + "claim:272XTX7", + "claim:272XTX7C@02F47B", + "claim:localhost/272XTX7C", + "claim:a.b/272XTX7C@02f47", + "claim:272XTX7C ", + "claim:272XTX7C\n", + ] + for s in samples: + assert bool(claims.CLAIM_REF_RE.match(s)) == bool(schema_ref.match(s)), s + assert bool(claims.CLAIM_PIN_RE.match(s)) == bool(schema_pin.match(s)), s + + +@given(st.from_regex(REF_PATTERN) | st.from_regex(PIN_PATTERN) | st.text(max_size=30)) +@PROPERTIES +def test_the_patterns_agree_with_the_schema_on_any_text(s): + assert bool(claims.CLAIM_REF_RE.match(s)) == bool(re.match(REF_PATTERN, s)) + assert bool(claims.CLAIM_PIN_RE.match(s)) == bool(re.match(PIN_PATTERN, s)) + + +def test_the_fixtures_hold_claim_references_and_pins(): + texts = _fixture_texts() + assert any("@" in t for t in texts) + assert any("@" not in t for t in texts) + + +@pytest.mark.parametrize("text", _fixture_texts()) +def test_a_fixture_claim_parses_and_formats_back(text): + pin = claims.parse_pin(text) + if "@" in text: + assert pin is not None + assert claims.parse_ref(text) is None + assert pin["ref"] + "@" + pin["pin"] == text + else: + ref = claims.parse_ref(text) + assert ref is not None + assert claims.format_ref(ref["tag"], ref["host"]) == text + assert pin is None + + +def test_a_reference_has_a_tag_and_no_host_for_the_local_store(): + assert claims.parse_ref("claim:272XTX7C") == {"host": None, "tag": "272XTX7C"} + + +def test_a_reference_to_another_store_has_its_host(): + assert claims.parse_ref("claim:claims.example.org/272XTX7C") == { + "host": "claims.example.org", + "tag": "272XTX7C", + } + + +def test_a_pin_splits_into_reference_tag_and_pin(): + assert claims.parse_pin("claim:a.org/272XTX7C@02f47b") == { + "ref": "claim:a.org/272XTX7C", + "host": "a.org", + "tag": "272XTX7C", + "pin": "02f47b", + } + + +@pytest.mark.parametrize( + "text", + [ + "", + "272XTX7C", + "claim:", + "claim:272XTX7", + "claim:272XTX7CC", + "claim:272xtx7c", + "claim:272XTX7U", + "claim:272XTX7L", + "claim:272XTX7I", + "claim:272XTX7O", + "claim:localhost/272XTX7C", + "claim:Example.org/272XTX7C", + "claim:amber/k-bond", + "claim:272XTX7C@02f47b ", + "claim:272XTX7C\n", + ], +) +def test_text_that_is_not_a_reference_does_not_parse(text): + assert claims.parse_ref(text) is None + + +@pytest.mark.parametrize( + "text", + [ + "claim:272XTX7C", + "claim:272XTX7C@", + "claim:272XTX7C@02f47", + "claim:272XTX7C@02f47b0", + "claim:272XTX7C@02F47B", + "claim:272XTX7C@02f47g", + "272XTX7C@02f47b", + "claim:272XTX7C@02f47b\n", + ], +) +def test_text_that_is_not_a_pin_does_not_parse(text): + assert claims.parse_pin(text) is None + + +def test_a_reference_is_not_a_pin_and_a_pin_is_not_a_reference(): + assert claims.parse_pin("claim:272XTX7C") is None + assert claims.parse_ref("claim:272XTX7C@02f47b") is None + + +def test_format_ref_writes_the_local_and_the_hosted_form(): + assert claims.format_ref("272XTX7C") == "claim:272XTX7C" + assert claims.format_ref("272XTX7C", "a.org") == "claim:a.org/272XTX7C" + assert claims.format_ref("272XTX7C", host="") == "claim:272XTX7C" + + +@pytest.mark.parametrize( + ("tag", "host"), [("short", None), ("272xtx7c", None), ("272XTX7C", "nodot")] +) +def test_format_ref_refuses_what_is_not_a_reference(tag, host): + with pytest.raises(ValueError, match="not a claim reference"): + claims.format_ref(tag, host) + + +def test_format_pin_keeps_the_first_six_digits_of_the_hash(): + digest = "02f47b" + "0" * 58 + assert claims.format_pin("claim:272XTX7C", digest) == "claim:272XTX7C@02f47b" + + +def test_format_pin_refuses_a_bad_reference_or_hash(): + with pytest.raises(ValueError, match="not a claim reference"): + claims.format_pin("claim:nope", "0" * 64) + with pytest.raises(ValueError, match="64 lowercase hex"): + claims.format_pin("claim:272XTX7C", "02f47b") + with pytest.raises(ValueError, match="64 lowercase hex"): + claims.format_pin("claim:272XTX7C", "A" * 64) + + +@given(TAGS, HOSTS) +@PROPERTIES +def test_a_formatted_reference_parses_back(tag, host): + ref = claims.format_ref(tag, host) + assert claims.parse_ref(ref) == {"host": host, "tag": tag} + assert re.match(REF_PATTERN, ref) + + +@given(TAGS, HOSTS, HASHES) +@PROPERTIES +def test_a_formatted_pin_parses_back(tag, host, digest): + ref = claims.format_ref(tag, host) + pin = claims.format_pin(ref, digest) + assert claims.parse_pin(pin) == { + "ref": ref, + "host": host, + "tag": tag, + "pin": digest[:6], + } + assert re.match(PIN_PATTERN, pin) diff --git a/tests/test_cli_correct.py b/tests/test_cli_correct.py new file mode 100644 index 0000000..7a0ac37 --- /dev/null +++ b/tests/test_cli_correct.py @@ -0,0 +1,194 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""`paperspec correct`: what it writes, prints and refuses.""" + +import hashlib +import json +import shutil + +import pytest + +from helpers_cli import CHECKS_ONLY, HALE, TINY_ZIP, edit_package, store +from paperspec import __version__, jcs, package +from paperspec.cli import main as cli_main +from test_revision import CROP, CROP_PNG, REPLACE, proposal, snapshot + + +def run_main(argv, capsys): + """Run `paperspec ARGV`, returning its exit status and the text it printed.""" + try: + status = cli_main.main([str(a) for a in argv]) + except SystemExit as stop: + status = stop.code + captured = capsys.readouterr() + return status, captured.out, captured.err + + +def write(path, proposals): + path.write_text("".join(json.dumps(p) + "\n" for p in proposals), encoding="utf-8") + return path + + +@pytest.fixture +def hale(tmp_path): + return shutil.copytree(HALE, tmp_path / HALE.name) + + +def test_it_corrects_a_folder_in_place_and_prints_the_records(hale, tmp_path, capsys): + fixes = write(tmp_path / "fix.jsonl", [REPLACE]) + status, out, err = run_main(["correct", "--correction", fixes, hale], capsys) + assert status == 0 + record = json.loads(out) + assert record["op"] == "replace" and record["index"] == 13 + assert out == jcs.dumps(record) + "\n" + sha = hashlib.sha256((hale / "MANIFEST.json").read_bytes()).hexdigest() + assert err == f"{hale}: hale2026harmonic-damb at revision 13, package {sha}\n" + assert not package.validate(hale, store()).errors + prov = package.read_json(hale / "provenance.json") + assert {"name": "paperspec", "version": __version__} in prov["tools"] + + +def test_out_writes_a_copy_and_reports_what_it_left_out(hale, tmp_path, capsys): + before = snapshot(hale) + fixes = write( + tmp_path / "fix.json", + [proposal("replace", target="hca6dc5", find="Water", replace="Liquid water")], + ) + out_dir = tmp_path / "fixed" / hale.name + status, _, err = run_main( + ["correct", "--correction", fixes, "--out", out_dir, hale], capsys + ) + assert status == 0 + assert snapshot(hale) == before + assert ( + f"{out_dir}: left out 1 lines of derived/math.jsonl, which read corrected text\n" + in err + ) + + +def test_assets_and_answered_suspects_are_passed_on(tmp_path, capsys): + suspect = { + "format": "scienting-suspect/1", + "id": "aaaaaaaaaaaaa", + "kind": "number", + "node": "c20cna6", + "a": "310.0", + "b": "310.5", + } + src = edit_package(tmp_path, HALE, {"suspects.jsonl": package.jsonl([suspect])}) + crop = tmp_path / "crop.PNG" + crop.write_bytes(CROP_PNG) + fixes = write( + tmp_path / "fix.json", + [ + proposal( + "replace", + evidence=CROP, + target="c20cna6", + find="310.0", + replace="310.5", + ) + ], + ) + status, _, err = run_main( + [ + "correct", + "--correction", + fixes, + "--asset", + crop, + "--answered", + "aaaaaaaaaaaaa", + src, + ], + capsys, + ) + assert status == 0, err + assert (src / "assets" / f"{CROP[0]['sha256']}.png").read_bytes() == CROP_PNG + assert not (src / "suspects.jsonl").exists() + assert "left out 3 files of datasets/ds-hale-bond-params/" in err + + +def test_a_refused_proposal_names_its_line_and_rule_and_writes_nothing( + hale, tmp_path, capsys +): + before = snapshot(hale) + bad = proposal("replace", target="mhp0at8", find="absent", replace="x") + fixes = write(tmp_path / "fix.jsonl", [REPLACE, bad]) + status, out, err = run_main(["correct", "--correction", fixes, hale], capsys) + assert status == 1 + assert out == "" + assert err == ( + f"{fixes}:2 mhp0at8: correction.find: find occurs 0 times, not exactly once\n" + ) + assert snapshot(hale) == before + + +def test_an_invalid_package_prints_its_errors(tmp_path, capsys): + broken = edit_package(tmp_path, HALE, {"document.json": lambda d: dict(d, rev=99)}) + fixes = write(tmp_path / "fix.json", [REPLACE]) + status, _, err = run_main(["correct", "--correction", fixes, broken], capsys) + assert status == 1 + assert f"{broken}: document.json: doc.correction.rev: " in err + + +def test_a_checks_only_package_is_refused(tmp_path, capsys): + src = shutil.copytree(CHECKS_ONLY, tmp_path / CHECKS_ONLY.name) + fixes = write(tmp_path / "fix.json", [REPLACE]) + status, _, err = run_main(["correct", "--correction", fixes, src], capsys) + # Without the team key its keyed hashes do not validate, so it is refused before that. + assert status == 1 + assert err.startswith(f"{src}: ") + + +def test_a_zipped_package_is_corrected_in_place(tmp_path, capsys): + src = shutil.copy(TINY_ZIP, tmp_path / TINY_ZIP.name) + unpacked = package.read_zip(TINY_ZIP, tmp_path / "unpacked") + leaf = next( + n + for n in package.read_jsonl(f"{unpacked}/nodes.jsonl") + if n["kind"] == "sentence" + ) + word = leaf["content"][0]["text"].split()[0] + fixes = write( + tmp_path / "fix.json", + [proposal("replace", target=leaf["id"], find=word, replace=word + "s")], + ) + status, _, err = run_main(["correct", "--correction", fixes, src], capsys) + assert status == 0, err + assert src.read_bytes() != TINY_ZIP.read_bytes() + status, _, err = run_main( + ["correct", "--correction", fixes, "--max-size", 10, src], capsys + ) + assert status == 1 + assert "package.zip.size" in err + + +def test_an_existing_out_is_refused(hale, tmp_path, capsys): + fixes = write(tmp_path / "fix.json", [REPLACE]) + status, _, err = run_main( + ["correct", "--correction", fixes, "--out", tmp_path, hale], capsys + ) + assert status == 1 + assert "exists; give a folder that does not" in err + + +def test_an_asset_without_an_extension_is_a_usage_error(hale, tmp_path, capsys): + fixes = write(tmp_path / "fix.json", [REPLACE]) + bare = tmp_path / "crop" + bare.write_bytes(CROP_PNG) + status, _, err = run_main( + ["correct", "--correction", fixes, "--asset", bare, hale], capsys + ) + assert status == 2 + assert "not a file with an extension" in err + + +def test_the_correction_file_is_required(hale, capsys): + status, _, err = run_main(["correct", hale], capsys) + assert status == 2 + assert "--correction" in err diff --git a/tests/test_cli_main.py b/tests/test_cli_main.py index f1e7fd9..b3526db 100644 --- a/tests/test_cli_main.py +++ b/tests/test_cli_main.py @@ -16,7 +16,7 @@ from paperspec import UNICODE_VERSION, __version__, cli from paperspec.cli import main as cli_main from paperspec.package import ZIP_LIMIT -COMMANDS = ["validate", "load", "check", "build", "stress", "site"] +COMMANDS = ["validate", "correct", "load", "check", "build", "stress", "site"] def run_main(argv, capsys): diff --git a/tests/test_datasets.py b/tests/test_datasets.py new file mode 100644 index 0000000..4f82c52 --- /dev/null +++ b/tests/test_datasets.py @@ -0,0 +1,190 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""A package's datasets, checked against the package (spec/15, paperspec.datasets).""" + +import json + +import pytest + +from conftest import FIXTURES +from helpers_cli import ( + CHECKS_ONLY, + HALE, + edit_package, + key_for, + read_package, + rules_of, + store, +) +from paperspec import datasets, package + +DS = "datasets/ds-hale-bond-params/" + + +def errors(root): + return [ + e + for e in package.validate(root, store()).errors + if e["rule"].startswith("doc.dataset") + ] + + +def edited(tmp_path, change): + """A copy of the main example with its dataset's files changed by `change(files)`.""" + files, absent = read_package(HALE) + change(files) + root = tmp_path / HALE.name + package.write(root, files, absent) + return root + + +def lines(files, name): + return [json.loads(x) for x in files[DS + name].decode().splitlines()] + + +def put_lines(files, name, recs): + files[DS + name] = package.jsonl(recs) + + +def test_the_example_dataset_breaks_no_rule(): + assert errors(HALE) == [] + + +def test_a_csv_value_that_is_not_its_cell_is_refused(tmp_path): + def change(files): + files[DS + "data.csv"] = files[DS + "data.csv"].replace(b"310.0", b"310.00") + + [e] = errors(edited(tmp_path, change)) + assert e["rule"] == "doc.dataset.value" + assert "'310.00' but its anchor reads '310.0'" in e["message"] + + +def test_a_numeric_value_without_an_anchor_is_refused(tmp_path): + def change(files): + put_lines(files, "anchors.jsonl", lines(files, "anchors.jsonl")[1:]) + + assert {e["rule"] for e in errors(edited(tmp_path, change))} == { + "doc.dataset.unanchored" + } + + +def test_an_anchor_to_a_row_the_csv_lacks_is_refused(tmp_path): + def change(files): + recs = lines(files, "anchors.jsonl") + recs[0] = dict(recs[0], row="no-such-row") + put_lines(files, "anchors.jsonl", recs) + + rules = {e["rule"] for e in errors(edited(tmp_path, change))} + assert "doc.dataset.anchor" in rules + + +def test_an_anchor_that_does_not_resolve_as_current_is_refused(tmp_path): + def change(files): + recs = lines(files, "anchors.jsonl") + recs[0]["anchor"] = dict( + recs[0]["anchor"], node_hash="0" * 64, own_hash="0" * 64 + ) + put_lines(files, "anchors.jsonl", recs) + + found = errors(edited(tmp_path, change)) + assert any( + e["rule"] == "doc.dataset.anchor" and "resolves as" in e["message"] + for e in found + ) + + +def test_an_anchor_off_its_schema_is_refused(tmp_path): + def change(files): + recs = lines(files, "anchors.jsonl") + recs[0] = dict(recs[0], page=3) + put_lines(files, "anchors.jsonl", recs) + + assert "doc.dataset.anchor" in {e["rule"] for e in errors(edited(tmp_path, change))} + + +def test_a_folder_not_named_by_the_id_is_refused(tmp_path): + def change(files): + for p in [p for p in files if p.startswith(DS)]: + files[p.replace("ds-hale-bond-params", "ds-other")] = files.pop(p) + + [e] = errors(edited(tmp_path, change)) + assert (e["rule"], e["where"]) == ("doc.dataset.id", "datasets/ds-other") + + +def test_a_repeated_primary_key_is_refused(tmp_path): + def change(files): + csv = files[DS + "data.csv"].decode().splitlines() + files[DS + "data.csv"] = ("\n".join([*csv, csv[1]]) + "\n").encode() + + assert "doc.dataset.key" in {e["rule"] for e in errors(edited(tmp_path, change))} + + +def test_a_descriptor_off_its_schema_is_refused(tmp_path): + def change(files): + desc = json.loads(files[DS + "datapackage.json"]) + desc["format"] = "scienting-dataset/9" + files[DS + "datapackage.json"] = package.line(desc).encode() + + assert {e["rule"] for e in errors(edited(tmp_path, change))} == { + "doc.dataset.schema" + } + + +@pytest.mark.parametrize( + ("change", "where"), + [ + (lambda f: f.pop(DS + "datapackage.json"), "datasets/ds-hale-bond-params"), + (lambda f: f.update({DS + "datapackage.json": b"{"}), DS + "datapackage.json"), + ( + lambda f: f.update({DS + "anchors.jsonl": b'{ "a": 1}\n'}), + DS + "anchors.jsonl", + ), + (lambda f: f.pop(DS + "data.csv"), "datasets/ds-hale-bond-params"), + (lambda f: f.update({DS + "data.csv": b"\xff\n"}), DS + "data.csv"), + ], + ids=[ + "no-descriptor", + "descriptor-not-json", + "anchors-not-canonical", + "no-csv", + "csv-not-utf8", + ], +) +def test_a_file_that_is_missing_or_unreadable_is_refused(tmp_path, change, where): + found = errors(edited(tmp_path, change)) + assert any(e["rule"] == "doc.dataset.file" and e["where"] == where for e in found) + + +def test_a_dataset_whose_cell_a_correction_changed_fails_until_rebuilt(tmp_path): + # Datasets read the text, so a corrected cell breaks the value rule; revision leaves it out. + def change(nodes): + for n in nodes: + if n["id"] == "c20cna6": + n["content"] = [{"kind": "text", "text": "310.5"}] + return nodes + + root = edit_package(tmp_path, HALE, {"nodes.jsonl": change}) + assert "doc.dataset.anchor" in {e["rule"] for e in errors(root)} + + +def test_the_content_hash_is_the_one_a_view_pins(): + folder = FIXTURES / "records" / "datasets" / "ds-hale-ctct-scan" + files = {p.name: p.read_bytes() for p in folder.iterdir()} + view = json.loads((FIXTURES / "records" / "view.vl.json").read_text()) + assert datasets.content_hash(files) in json.dumps(view) + + +def test_folders_are_the_dataset_ids_a_package_holds(): + files, _ = read_package(HALE) + assert datasets.folders(files) == ["ds-hale-bond-params"] + assert datasets.folders(["datasets/x", "nodes.jsonl"]) == [] + + +def test_a_checks_only_package_is_not_looked_at(): + # It carries no datasets, and the policy rules report one that does. + report = package.validate(CHECKS_ONLY, store(), key_for(CHECKS_ONLY)) + assert not [r for r in rules_of(report) if r.startswith("doc.dataset")] diff --git a/tests/test_db.py b/tests/test_db.py index 8e2e29c..a7235bd 100644 --- a/tests/test_db.py +++ b/tests/test_db.py @@ -11,6 +11,7 @@ import hashlib import json import shutil import sqlite3 +from pathlib import Path import pytest @@ -562,3 +563,377 @@ def test_flags_are_stored_as_one_zero_or_null(conn, tmp_path): "SELECT text_layer FROM pages WHERE doc_key = ?", (key,) ).fetchall() assert {row[0] for row in pages} <= {0, 1} + + +# --- layers ------------------------------------------------------------------------------------------------------------------ + +LAYER_DIRS = sorted(p.parent.name for p in (RECORDS).glob("*/annotations.jsonl")) + + +def read_layer(name): + """The layer record and the annotations of the fixture layer `name`.""" + folder = RECORDS / name + return ( + json.loads((folder / "layer.json").read_bytes()), + read_jsonl(folder / "annotations.jsonl"), + ) + + +def insert_as_check_does(conn, lay, annotations): + """The inserts `paperspec check` makes for a layer (cli/check.py), copied.""" + conn.execute( + "INSERT INTO layers VALUES (?,?,?,?,?,?,?,?)", + ( + lay["id"], + lay.get("project"), + lay["label"], + lay["purpose"], + lay.get("audience"), + lay["visibility"], + jcs.dumps(lay["license"]), + jcs.dumps(lay), + ), + ) + for a in annotations: + conn.execute( + "INSERT INTO annotations VALUES (?,?,?,?,?,?,?,?,?)", + ( + a["id"], + a["layer"], + a["body"]["kind"], + a["body"].get("category"), + a["status"], + a.get("receipt"), + a["creator"]["id"], + a["created"], + jcs.dumps(a), + ), + ) + + +def rows(conn, table): + """Every row of `table`, in a fixed order.""" + return conn.execute(f"SELECT * FROM {table} ORDER BY 1, 2").fetchall() # noqa: S608 + + +def snapshot(conn): + """Every row of every table of `conn`, by table name.""" + names = [ + n + for (n, _) in conn.execute( + "SELECT name, sql FROM sqlite_master WHERE type = 'table'" + ).fetchall() + if not n.startswith("search_") + ] + return {n: sorted(map(repr, conn.execute(f"SELECT * FROM {n}"))) for n in names} # noqa: S608 + + +def test_the_fixture_layers_are_found(): + assert {"layer", "traces", "chemistry", "biology", "data"} <= set(LAYER_DIRS) + + +@pytest.mark.parametrize("name", LAYER_DIRS) +def test_a_loaded_layer_has_the_rows_the_check_inserts(conn, name): + lay, annotations = read_layer(name) + other = sqlite3.connect(":memory:") + try: + db.create(other, SQL) + insert_as_check_does(other, lay, annotations) + assert db.load_layer(conn, lay, annotations) == len(annotations) + assert rows(conn, "layers") == rows(other, "layers") + assert rows(conn, "annotations") == rows(other, "annotations") + finally: + other.close() + + +@pytest.mark.parametrize("name", LAYER_DIRS) +def test_each_target_of_a_loaded_annotation_has_a_row_for_its_node_or_annotation( + conn, name +): + lay, annotations = read_layer(name) + db.load_layer(conn, lay, annotations) + expected = set() + for a in annotations: + targets = a["target"] if isinstance(a["target"], list) else [a["target"]] + for i, t in enumerate(targets): + if "selector" in t: + anchor = next( + s["anchor"] for s in t["selector"] if s["type"] == "AnchorSelector" + ) + expected.add( + (a["id"], i, t["source"].rsplit("/", 1)[1], anchor["node"], None) + ) + else: + expected.add((a["id"], i, None, None, t["source"])) + assert set(conn.execute("SELECT * FROM annotation_targets")) == expected + assert expected + + +def test_replies_and_annotations_on_a_node_are_found_through_the_target_rows(conn): + lay, annotations = read_layer("layer") + db.load_layer(conn, lay, annotations) + replies = column( + conn, + "SELECT annotation FROM annotation_targets WHERE target_annotation IS NOT NULL", + ) + on_nodes = column( + conn, "SELECT annotation FROM annotation_targets WHERE node IS NOT NULL" + ) + assert replies + assert on_nodes + assert not set(replies) & set(on_nodes) + + +def test_loading_a_layer_twice_is_refused(conn): + lay, annotations = read_layer("data") + db.load_layer(conn, lay, annotations) + with pytest.raises(sqlite3.IntegrityError): + db.load_layer(conn, lay, annotations) + + +def test_a_status_the_tables_do_not_allow_is_refused_by_their_check(conn): + lay, annotations = read_layer("data") + annotations[0]["status"] = "bogus" + with pytest.raises(sqlite3.IntegrityError): + db.load_layer(conn, lay, annotations) + + +# --- the state of a loaded document --------------------------------------------------------------------------------------- + + +@pytest.mark.parametrize("name", PACKAGES) +def test_the_state_read_from_the_tables_is_the_state_read_from_the_package( + conn, tmp_path, name +): + source = FIXTURES / "packages" / name + key = db.load(conn, str(source), str(tmp_path / "assets")) + want = package.read_state(str(source)) + got = db.read_state(conn, key) + assert got.key == want.key + assert got.nodes == want.nodes + assert got.ids == want.ids + assert got.corrections == want.corrections + assert got.references == want.references + assert got.rev == want.rev + + +def test_the_state_of_a_document_that_is_not_loaded_is_a_key_error(conn): + with pytest.raises(KeyError): + db.read_state(conn, "nobody2026nothing-aaaa") + + +# --- forgetting a document ---------------------------------------------------------------------------------------------------- + + +def test_loading_then_deleting_a_document_leaves_the_database_as_it_was(conn, tmp_path): + assets = str(tmp_path / "assets") + db.load(conn, str(FIXTURES / "packages" / TINY), assets) + before = snapshot(conn) + key = db.load(conn, str(FIXTURES / "packages" / HALE), assets) + db.index(conn, key) + added = sum(len(v) for v in snapshot(conn).values()) - sum( + len(v) for v in before.values() + ) + assert db.delete(conn, key) == added > 0 + assert snapshot(conn) == before + + +def test_a_deleted_document_can_be_loaded_again(conn, tmp_path): + assets = str(tmp_path / "assets") + source = str(FIXTURES / "packages" / HALE) + key = db.load(conn, source, assets) + full = snapshot(conn) + db.delete(conn, key) + assert db.load(conn, source, assets) == key + assert snapshot(conn) == full + + +def test_deleting_a_document_keeps_its_work_while_another_document_is_of_it( + conn, tmp_path +): + second = tmp_path / "second" + shutil.copytree(FIXTURES / "packages" / TINY, second) + document = json.loads((second / "document.json").read_bytes()) + document["key"] += ".v2" + (second / "document.json").write_bytes((jcs.dumps(document) + "\n").encode("utf-8")) + assets = str(tmp_path / "assets") + db.load(conn, str(FIXTURES / "packages" / TINY), assets) + db.load(conn, str(second), assets) + db.delete(conn, TINY + ".v2") + assert column(conn, "SELECT COUNT(*) FROM works") == [1] + db.delete(conn, TINY) + assert column(conn, "SELECT COUNT(*) FROM works") == [0] + + +def test_deleting_a_document_removes_the_target_rows_on_it_and_keeps_the_annotations( + conn, tmp_path +): + key = db.load(conn, str(FIXTURES / "packages" / HALE), str(tmp_path / "assets")) + lay, annotations = read_layer("layer") + db.load_layer(conn, lay, annotations) + assert column( + conn, "SELECT COUNT(*) FROM annotation_targets WHERE doc_key = ?", key + )[0] + db.delete(conn, key) + assert column( + conn, "SELECT COUNT(*) FROM annotation_targets WHERE doc_key = ?", key + ) == [0] + assert column(conn, "SELECT COUNT(*) FROM annotations") == [len(annotations)] + + +def test_deleting_a_document_removes_the_values_of_its_datasets(conn): + conn.execute("INSERT INTO datasets VALUES ('d1', 'k', ?, '{}')", ("0" * 64,)) + conn.execute("INSERT INTO datasets VALUES ('d2', 'other', ?, '{}')", ("0" * 64,)) + for ds in ("d1", "d2"): + conn.execute( + "INSERT INTO dataset_values VALUES (?, 'r', 'row', 'f', NULL, 'reported', NULL)", + (ds,), + ) + assert db.delete(conn, "k") == 2 + assert column(conn, "SELECT dataset FROM dataset_values") == ["d2"] + + +def test_deleting_a_document_that_is_not_loaded_deletes_nothing(conn): + assert db.delete(conn, "nobody2026nothing-aaaa") == 0 + + +# --- upgrading a store made before the CHECKs widened ----------------------------------------------------------------- + + +def old_ddl(): + """The DDL as it was before `trace`, `structure`, `sequence`, `data`, `category` and the three ops.""" + doc = (Path(SQL) / "doc.sql").read_text(encoding="utf-8") + store = (Path(SQL) / "store.sql").read_text(encoding="utf-8") + old_doc = doc.replace(", 'emend', 'relink', 'resolve'", "") + old_store = store.replace( + "'crosslink', 'trace', 'structure', 'sequence', 'data', ", "'crosslink', " + ).replace( + " category VARCHAR(12) CHECK (category IN ('comment', 'context', 'critique', " + "'question', 'correction', 'feedback', 'resource')),\n", + "", + ) + assert old_doc != doc + assert old_store != store + assert ( + "category" not in old_store.split("CREATE TABLE annotations")[1].split(");")[0] + ) + return old_doc, old_store + + +@pytest.fixture +def old(tmp_path): + """A database made from the old DDL, holding one correction and one gloss annotation.""" + connection = sqlite3.connect(":memory:") + for text in old_ddl(): + connection.executescript(text) + connection.execute( + "INSERT INTO corrections VALUES ('k', 1, 'c1', 'replace', 'n1', '[]', '{}', 'r', " + "'2026-01-01T00:00:00Z', NULL, NULL, NULL, '[]', '{}')" + ) + annotations = read_layer("layer")[1] + gloss = next(a for a in annotations if a["body"]["kind"] == "gloss") + connection.execute( + "INSERT INTO annotations VALUES (?,?,?,?,?,?,?,?)", + ( + gloss["id"], + gloss["layer"], + "gloss", + gloss["status"], + gloss.get("receipt"), + gloss["creator"]["id"], + gloss["created"], + jcs.dumps(gloss), + ), + ) + connection.commit() + yield connection + connection.close() + + +def test_an_old_store_cannot_hold_a_data_annotation(old): + lay, annotations = read_layer("data") + with pytest.raises(sqlite3.Error): + db.load_layer(old, lay, annotations) + + +def test_upgrading_an_old_store_rebuilds_the_tables_whose_check_changed(old): + assert sorted(db.upgrade(old, SQL)) == ["annotations", "corrections"] + + +def test_after_an_upgrade_a_data_annotation_loads_and_the_old_rows_survive(old): + db.upgrade(old, SQL) + lay, annotations = read_layer("data") + assert db.load_layer(old, lay, annotations) == len(annotations) + assert "data" in column(old, "SELECT body_kind FROM annotations") + assert column( + old, "SELECT category FROM annotations WHERE body_kind = 'gloss'" + ) == [None] + assert column(old, "SELECT op FROM corrections") == ["replace"] + old.execute( + "INSERT INTO corrections VALUES ('k', 2, 'c2', 'resolve', NULL, '[]', '{}', 'r', " + "'2026-01-01T00:00:00Z', NULL, NULL, NULL, '[]', '{}')" + ) + + +def test_an_upgrade_keeps_the_indexes_and_makes_the_stored_sql_the_shipped_sql(old): + db.upgrade(old, SQL) + fresh = sqlite3.connect(":memory:") + try: + db.create(fresh, SQL) + query = "SELECT type, name, sql FROM sqlite_master ORDER BY type, name" + assert old.execute(query).fetchall() == fresh.execute(query).fetchall() + finally: + fresh.close() + + +def test_a_second_upgrade_finds_nothing_to_do(old): + db.upgrade(old, SQL) + assert db.upgrade(old, SQL) == [] + + +def test_a_current_database_is_not_rebuilt(conn): + assert db.upgrade(conn, SQL) == [] + + +def test_an_upgrade_leaves_foreign_keys_as_it_found_them(old): + old.execute("PRAGMA foreign_keys = ON") + db.upgrade(old, SQL) + assert column(old, "PRAGMA foreign_keys") == [1] + + +def test_an_upgrade_of_a_bundle_also_checks_the_bundle_tables(conn): + conn.execute("DROP TABLE embeddings_meta") + conn.execute("CREATE TABLE embeddings_meta (model TEXT)") + assert db.upgrade(conn, SQL) == ["embeddings_meta"] + + +def test_a_row_the_new_check_refuses_makes_the_upgrade_raise_and_changes_nothing(old): + old.execute("DROP TABLE annotations") + old.execute( + "CREATE TABLE annotations (id VARCHAR(256) NOT NULL, layer VARCHAR(256) NOT NULL, " + "body_kind VARCHAR(24) NOT NULL, status VARCHAR(10) NOT NULL, receipt VARCHAR(32), " + "creator VARCHAR(256) NOT NULL, created VARCHAR(20) NOT NULL, record TEXT NOT NULL, " + "PRIMARY KEY (id))" + ) + old.execute( + "INSERT INTO annotations (id, layer, body_kind, status, creator, created, record) " + "VALUES ('a', 'l', 'legacy', 'proposed', 'c', 't', '{}')" + ) + old.commit() + with pytest.raises(sqlite3.IntegrityError): + db.upgrade(old, SQL) + assert column(old, "SELECT body_kind FROM annotations") == ["legacy"] + # The rebuild of corrections, made before the failure, was undone too. + assert ( + "emend" + not in column(old, "SELECT sql FROM sqlite_master WHERE name = 'corrections'")[ + 0 + ] + ) + assert not old.in_transaction + + +def test_an_upgrade_inside_a_transaction_is_refused(old): + old.execute("INSERT INTO works VALUES ('w', '[]')") + with pytest.raises(ValueError, match="transaction"): + db.upgrade(old, SQL) diff --git a/tests/test_fingerprint.py b/tests/test_fingerprint.py index 1ca9dbf..a1b4a81 100644 --- a/tests/test_fingerprint.py +++ b/tests/test_fingerprint.py @@ -228,3 +228,8 @@ def test_inserting_a_stamp_anywhere_leaves_the_fingerprint_unchanged(lines, posi got, removed = fp.fingerprint(page(stamped), RULES) assert got == expected assert "downloaded-via" in removed + + +def test_the_published_rules_travel_with_the_package_and_are_the_registry_file(): + assert fp.published_rules() == RULES + assert RULES[0]["id"] == "downloaded-via" diff --git a/tests/test_hashing.py b/tests/test_hashing.py index c0d359e..1c1ca4f 100644 --- a/tests/test_hashing.py +++ b/tests/test_hashing.py @@ -692,3 +692,66 @@ class TestMetaHash: other = dict(self.DOCUMENT, policy="withheld", key="another") assert meta_hash(other) == meta_hash(self.DOCUMENT) assert meta_hash(dict(self.DOCUMENT, language="fr")) != meta_hash(self.DOCUMENT) + + +def _lossy_by_hand(s): + """The lossy projection of a strict projection, character by character (spec/05 §5.5).""" + out, stack, i, n = [], [], 0, len(s) + while i < n: + ch = s[i] + if ch in "_^" and i + 1 < n and s[i + 1] == "{": + stack.append("s") + i += 2 + continue + if ch == "{": + stack.append("t") + out.append(ch) + elif ch == "}": + if not stack or stack.pop() == "t": + out.append(ch) + elif ch == hashing.MARK_OPEN: + j = s.find(hashing.MARK_CLOSE, i) + i = n if j < 0 else j + 1 + continue + else: + out.append(ch) + i += 1 + t = unicodedata.normalize("NFKC", "".join(out).casefold()) + return "".join( + c for c in t if not is_ws(c) and unicodedata.category(c) != "Pd" and c != "­" + ) + + +LOSSY_ALPHABET = [ + "_", + "^", + "{", + "}", + hashing.MARK_OPEN, + hashing.MARK_CLOSE, + hashing.OBJ, + "a", + "B", + "\u00df", # sharp s, which case folding expands + "\u00ad", # soft hyphen + "-", + "\u2013", # en dash + "\u2212", # minus sign + " ", + "\u00a0", # no-break space + "\u2009", # thin space + "\ufb01", # the fi ligature, which NFKC expands + "\u00b2", # superscript two + "1", +] + + +@given(st.text(alphabet=st.sampled_from(LOSSY_ALPHABET), max_size=40)) +@settings(derandomize=True, max_examples=500, database=None, deadline=None) +def test_the_lossy_projection_matches_its_character_by_character_definition(s): + assert hashing.lossy_from_strict(s) == _lossy_by_hand(s) + + +def test_the_lossy_projection_drops_every_dash_and_white_space_character(): + every = "".join(chr(c) for c in range(0x110000) if not 0xD800 <= c <= 0xDFFF) + assert hashing.lossy_from_strict(every) == _lossy_by_hand(every) diff --git a/tests/test_jcs.py b/tests/test_jcs.py index c2796da..4fc962c 100644 --- a/tests/test_jcs.py +++ b/tests/test_jcs.py @@ -230,6 +230,37 @@ def test_loads_line_refuses_a_line_that_is_not_canonical(line): jcs.loads_line(line) +@pytest.mark.parametrize("x", [1e20, -1e20, 2.0**60, 123456789012345680000.0]) +def test_loads_line_reads_back_a_whole_double_beyond_2_53(x): + # The canonical form of 1e20 is 100000000000000000000, an integer literal beyond 2^53, + # which is read as the double it is (spec/01 §1.1), not refused. + line = jcs.dumpb({"a": x}) + assert jcs.loads_line(line) == {"a": x} + + +def test_loads_line_refuses_an_integer_beyond_2_53_that_is_no_double(): + with pytest.raises(ValueError, match="not canonical JSON"): + jcs.loads_line(b"9007199254740993") + + +@pytest.mark.parametrize("text", ["NaN", "[Infinity]", '{"a":-Infinity}']) +def test_loads_refuses_nan_and_the_infinities(text): + with pytest.raises(ValueError, match="no canonical form"): + jcs.loads(text) + + +def test_loads_reads_numbers_as_doubles_without_requiring_canonical_text(): + assert jcs.loads('{ "b": 100000000000000000000, "a": 7 }') == {"a": 7, "b": 1e20} + assert type(jcs.loads("100000000000000000000")) is float + assert type(jcs.loads("9007199254740992")) is int + + +@given(JSON_VALUES) +@PROPERTIES +def test_loads_line_reads_back_every_canonical_line(value): + assert jcs.dumps(jcs.loads_line(jcs.dumpb(value))) == jcs.dumps(value) + + def test_loads_line_refuses_bytes_that_are_not_utf8(): with pytest.raises(UnicodeDecodeError): jcs.loads_line(b'"\xff"') diff --git a/tests/test_layers.py b/tests/test_layers.py new file mode 100644 index 0000000..df8f715 --- /dev/null +++ b/tests/test_layers.py @@ -0,0 +1,143 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""Layers and their annotations: the record-ID rule and the enumerations (paperspec.layers).""" + +import copy +import json +import re +from pathlib import Path + +import pytest + +from conftest import FIXTURES, read_jsonl +from paperspec import jcs, layers, resources +from paperspec.text import record_id, sha256_hex + +SCHEMAS = Path(resources.path("schemas")) +SQL = Path(resources.path("sql")) +LAYER_FILES = sorted((FIXTURES / "records").glob("*/annotations.jsonl")) +ANNOTATIONS = [a for f in LAYER_FILES for a in read_jsonl(f)] +SCHEMA = json.loads((SCHEMAS / "annotation.schema.json").read_text(encoding="utf-8")) + + +def sql_list(column): + """The values of the CHECK list `column ... IN (...)` of `annotations` in store.sql.""" + text = (SQL / "store.sql").read_text(encoding="utf-8") + match = re.search(rf"\b{column} VARCHAR\(\d+\)[^,]*?IN \(([^)]*)\)", text) + assert match, column + return tuple(re.findall(r"'([^']*)'", match.group(1))) + + +def check_py_rule(annotation): + """The rule as `paperspec check` computes it, copied from cli/check.py.""" + body = { + k: v + for k, v in annotation.items() + if k not in ("status", "receipt", "modified") + } + return record_id( + sha256_hex(jcs.dumpb({k: v for k, v in body.items() if k != "id"})) + ) + + +# --- the record ID ------------------------------------------------------------------------------------------------------ + + +def test_the_fixtures_hold_annotations_of_every_layer_file(): + assert len(LAYER_FILES) >= 5 + assert len(ANNOTATIONS) > 20 + + +@pytest.mark.parametrize("annotation", ANNOTATIONS, ids=lambda a: a["id"]) +def test_every_fixture_annotation_id_ends_with_its_computed_record_id(annotation): + assert annotation["id"].rsplit("/", 1)[1] == layers.annotation_id(annotation) + + +@pytest.mark.parametrize("annotation", ANNOTATIONS, ids=lambda a: a["id"]) +def test_the_record_id_is_the_one_the_check_computes(annotation): + assert layers.annotation_id(annotation) == check_py_rule(annotation) + + +def test_the_record_id_ignores_the_id_status_receipt_and_modified(): + a = copy.deepcopy(ANNOTATIONS[0]) + want = layers.annotation_id(a) + a.pop("id") + assert layers.annotation_id(a) == want + a["status"] = "withdrawn" + a["receipt"] = "zzzzzzzzzzzzz" + a["modified"] = "2030-01-01T00:00:00Z" + assert layers.annotation_id(a) == want + + +def test_the_record_id_changes_with_any_other_field(): + a = copy.deepcopy(ANNOTATIONS[0]) + changed = copy.deepcopy(a) + changed["created"] = "2030-01-01T00:00:00Z" + assert layers.annotation_id(changed) != layers.annotation_id(a) + + +def test_the_record_id_is_thirteen_letter_first_characters(): + assert re.fullmatch( + r"[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{12}", layers.annotation_id({}) + ) + + +# --- the enumerations ------------------------------------------------------------------------------------------------------ + + +def schema_body_kinds(): + """The `kind` of each alternative of the schema's `body`, in order.""" + return tuple( + b["properties"]["kind"]["const"] for b in SCHEMA["properties"]["body"]["oneOf"] + ) + + +def test_the_body_kinds_are_the_schemas_and_the_store_tables(): + assert schema_body_kinds() == layers.BODY_KINDS + assert set(layers.BODY_KINDS) == set(sql_list("body_kind")) + + +def test_the_categories_are_the_schemas_and_the_store_tables(): + assert tuple(SCHEMA["$defs"]["category"]["enum"]) == layers.CATEGORIES + assert sql_list("category") == layers.CATEGORIES + + +def test_the_statuses_are_the_schemas_and_the_store_tables(): + assert tuple(SCHEMA["properties"]["status"]["enum"]) == layers.STATUSES + assert sql_list("status") == layers.STATUSES + + +def test_the_verdicts_are_those_of_a_citation_context_and_of_a_trace(): + verdicts = { + b["properties"]["kind"]["const"]: tuple(b["properties"]["verdict"]["enum"]) + for b in SCHEMA["properties"]["body"]["oneOf"] + if "verdict" in b["properties"] + } + assert set(verdicts) == {"citation-context", "trace"} + assert set(verdicts.values()) == {layers.VERDICTS} + + +# --- the anchors of a selector --------------------------------------------------------------------------------------------- + + +def test_anchors_are_those_of_the_anchor_selectors_in_order(): + one, two = {"node": "a"}, {"node": "b"} + holder = { + "selector": [ + {"type": "AnchorSelector", "anchor": one}, + {"type": "TextQuoteSelector", "exact": "x"}, + {"type": "AnchorSelector", "anchor": two}, + ] + } + assert layers.anchors_of(holder) == [one, two] + + +@pytest.mark.parametrize( + "holder", [None, "text", [], {}, {"selector": None}, {"selector": ["x"]}] +) +def test_a_holder_without_a_selector_list_of_selectors_has_no_anchor(holder): + assert layers.anchors_of(holder) == [] diff --git a/tests/test_math_form.py b/tests/test_math_form.py new file mode 100644 index 0000000..d9e1c06 --- /dev/null +++ b/tests/test_math_form.py @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""`doc.math.form`: a source with `form` is that form's own spelling (spec/03 §3.7).""" + +import types + +import pytest + +from helpers_cli import HALE, edit_package, rules_of, store +from paperspec import package +from paperspec.model import Doc + +EQUATION = "tkxsmts" +SENTENCE = "rr20tev" +CANONICAL = r"E_{\text{bond}}=\frac{1}{2}K_{r}(r-r_{\text{eq}})^{2}" + + +def spell(source): + """A stand-in for the form: it knows one formula, and refuses what it cannot read.""" + if "{" in source and source.count("{") != source.count("}"): + raise ValueError("a brace does not balance") + return CANONICAL if "bond" in source else source + + +def with_form(tmp_path, equation=None, run_source=None): + """A copy of the main example whose equation and second math run carry `form`. + + `form` enters no content value, so the copy is valid but for what the sources hold. + """ + + def edit(nodes): + for n in nodes: + if n["id"] == EQUATION: + n["content"] = dict( + n["content"], + form=package.MATH_FORM, + **({"source": equation} if equation else {}), + ) + if n["id"] == SENTENCE and run_source is not None: + n["content"][3] = dict( + n["content"][3], form=package.MATH_FORM, source=run_source + ) + return nodes + + return edit_package(tmp_path, HALE, {"nodes.jsonl": edit}) + + +def rules(root, forms): + return rules_of(package.validate(root, store(), forms=forms)) + + +def test_a_canonical_source_with_form_breaks_no_form_rule(tmp_path): + # Respelling the source changes its hashes, which the copy does not recompute, so only + # the form rule is looked at here. + root = with_form(tmp_path, equation=CANONICAL) + assert "doc.math.form" not in rules(root, {package.MATH_FORM: spell}) + + +def test_adding_form_alone_changes_no_value(tmp_path): + root = with_form(tmp_path) + assert not package.validate(root, store(), forms={}).errors + + +def test_without_an_implementation_only_what_the_form_never_holds_is_reported(tmp_path): + root = with_form(tmp_path) + assert "doc.math.form" not in rules(root, {}) + + +@pytest.mark.parametrize( + "source", + [ + r"x=1\tag{2}", + r"x=1\label{eq:x}", + r"x=1\nonumber", + r"\notag x", + r"\displaystyle x", + r"\dfrac{1}{2}", + r"\tfrac{1}{2}", + ], +) +def test_a_label_or_style_command_is_never_in_the_form(tmp_path, source): + root = with_form(tmp_path, run_source=source) + report = package.validate(root, store(), forms={}) + assert any( + e["rule"] == "doc.math.form" and e["where"] == f"{SENTENCE} run 3" + for e in report.errors + ) + + +@pytest.mark.parametrize( + "source", [r"\\tag", r"\text{tagged}", r"\tagx", r"x\\\\notag"] +) +def test_an_escaped_backslash_or_a_longer_command_is_not_a_label_command( + tmp_path, source +): + root = with_form(tmp_path, run_source=source) + assert "doc.math.form" not in rules(root, {}) + + +def test_with_an_implementation_a_source_not_its_own_spelling_is_reported(tmp_path): + root = with_form(tmp_path) + report = package.validate(root, store(), forms={package.MATH_FORM: spell}) + [e] = [e for e in report.errors if e["rule"] == "doc.math.form"] + assert e["where"] == EQUATION + assert "is not its scienting-math/1 spelling, which is" in e["message"] + + +def test_a_source_the_form_cannot_read_is_reported(tmp_path): + root = with_form(tmp_path, run_source="x_{1") + report = package.validate(root, store(), forms={package.MATH_FORM: spell}) + assert any( + e["rule"] == "doc.math.form" and "does not read" in e["message"] + for e in report.errors + ) + + +def test_a_form_without_an_implementation_is_not_checked(): + doc = Doc( + "k", + [ + {"id": "r", "kind": "document", "parent": None, "ord": 0}, + { + "id": "e", + "kind": "equation", + "parent": "r", + "ord": 0, + "content": {"notation": "tex", "source": "x y", "form": "other/1"}, + }, + ], + ) + rep = package.Report() + package.check_forms(doc, rep, {package.MATH_FORM: spell}) + assert not rep.errors + + +def test_the_default_forms_are_papermaths_when_it_is_installed(): + papermath = pytest.importorskip("papermath") + forms = package.default_forms() + assert ( + forms[package.MATH_FORM](r"x_i^2 \le \dfrac12") + == papermath.canonical(r"x_i^2 \le \dfrac12").tex + ) + + +def test_the_default_forms_are_none_without_papermath(monkeypatch): + def no_papermath(name): + raise ImportError(name) + + monkeypatch.setattr(package.importlib, "import_module", no_papermath) + assert package.default_forms() == {} + + +def test_the_default_form_spells_with_papermaths_canonical(monkeypatch): + class Canon: + def __init__(self, tex): + self.tex = tex + + fake = types.SimpleNamespace(canonical=lambda s: Canon(s.replace(" ", ""))) + monkeypatch.setattr(package.importlib, "import_module", lambda name: fake) + assert package.default_forms()[package.MATH_FORM]("x + 1") == "x+1" + + +def test_validate_zip_passes_the_forms_on(tmp_path): + root = with_form(tmp_path) + archive = tmp_path / "form.sct.zip" + package.write_zip(root, archive) + report = package.validate_zip(archive, store(), forms={package.MATH_FORM: spell}) + assert "doc.math.form" in rules_of(report) diff --git a/tests/test_public_api.py b/tests/test_public_api.py new file mode 100644 index 0000000..0ce8695 --- /dev/null +++ b/tests/test_public_api.py @@ -0,0 +1,114 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""The public names other tools import instead of the private ones they once reached for.""" + +import zipfile + +import pytest + +from helpers_cli import HALE, TINY, TINY_ZIP, read_package +from paperspec import corrections, hashing, package, tree +from paperspec.model import Doc +from paperspec.tree import Refusal + + +def test_parse_canonical_reads_doubles_and_refuses_what_is_not_canonical(): + value = package.parse_canonical(b'{"a":100000000000000000000}', "x") + assert value == {"a": 1e20} and isinstance(value["a"], float) + assert package._canonical_bytes is package.parse_canonical + with pytest.raises(Refusal) as e: + package.parse_canonical(b'{ "a":1}', "x:1") + assert e.value.rule == "package.file.canonical" + + +def test_parse_json_needs_one_final_newline(): + assert package.parse_json(b'{"a":1}\n', "a.json") == {"a": 1} + for data in (b'{"a":1}', b'{"a":1}\n\n'): + with pytest.raises(Refusal): + package.parse_json(data, "a.json") + + +def test_parse_jsonl_names_the_line_at_fault(): + assert package.parse_jsonl(b"", "a.jsonl") == [] + assert package.parse_jsonl(b'{"a":1}\n{"b":2}\n', "a.jsonl") == [{"a": 1}, {"b": 2}] + with pytest.raises(Refusal) as e: + package.parse_jsonl(b'{"a":1}\n{"b": 2}\n', "a.jsonl") + assert "a.jsonl:2" in e.value.message + with pytest.raises(Refusal) as e: + package.parse_jsonl(b'{"a":1}', "a.jsonl") + assert e.value.rule == "package.file.newline" + + +def test_read_zip_files_holds_the_package_as_its_folder_does(): + key, files = package.read_zip_files(TINY_ZIP) + assert key == TINY.name + # read_package leaves the manifest out of the files it lists. + assert files.pop("MANIFEST.json") == (TINY / "MANIFEST.json").read_bytes() + assert files == read_package(TINY)[0] + + +def test_read_zip_files_counts_every_byte_against_the_limit(): + with pytest.raises(Refusal) as e: + package.read_zip_files(TINY_ZIP, limit=10) + assert e.value.rule == "package.zip.size" + + +def test_read_zip_files_refuses_what_is_not_a_zip(tmp_path): + bad = tmp_path / "bad.sct.zip" + bad.write_bytes(b"not a zip") + with pytest.raises(Refusal) as e: + package.read_zip_files(bad) + assert e.value.rule == "package.zip.entry" + + +def test_check_zip_entries_returns_the_folder_without_reading(): + with zipfile.ZipFile(TINY_ZIP) as zf: + assert package.check_zip_entries(zf) == TINY.name + assert package._zip_entries is package.check_zip_entries + + +def test_flatten_keeps_private_keys_and_leaves_the_spec_alone(): + spec = { + "kind": "document", + "_name": "root", + "children": [{"kind": "section", "children": [], "_name": "s"}], + } + flat = tree.flatten(spec) + assert [(f["kind"], f.get("_parent"), f["ord"], f["_name"]) for f in flat] == [ + ("document", None, 0, "root"), + ("section", 0, 0, "s"), + ] + assert "children" in spec and "_parent" not in spec + + +def test_establish_with_side_returns_the_private_keys_by_minted_id(): + spec = { + "kind": "document", + "children": [ + {"kind": "section", "_name": "intro", "children": []}, + {"kind": "section", "children": []}, + ], + } + nodes, side = tree.establish_with_side("doc2026x-abcd", spec) + assert nodes == tree.establish("doc2026x-abcd", spec) + assert not any(k.startswith("_") for n in nodes for k in n) + assert [side[n["id"]] for n in nodes] == [{}, {"_name": "intro"}, {}] + + +def test_strict_runs_is_the_strict_projection_of_a_leaf(): + nodes = package.parse_jsonl((HALE / "nodes.jsonl").read_bytes(), "nodes.jsonl") + doc = Doc(HALE.name, nodes) + leaves = [n for n in nodes if isinstance(n.get("content"), list)] + assert leaves + for n in leaves: + assert hashing.strict_runs(n["content"], doc) == hashing.strict(doc, n["id"]) + + +def test_the_operations_and_emend_kinds_are_the_schemas(): + assert len(set(corrections.OPS)) == len(corrections.OPS) == 12 + assert {"emend", "relink", "resolve"} <= set(corrections.OPS) + assert "typo" in corrections.EMEND_KINDS diff --git a/tests/test_resources.py b/tests/test_resources.py index 4a734f9..219434c 100644 --- a/tests/test_resources.py +++ b/tests/test_resources.py @@ -4,8 +4,9 @@ # Source Code: https://git.scient.ing/papers/paperspec # See the LICENSE.md file for full license terms. -"""Where the schemas, SQL and vectors are on disk (paperspec.resources).""" +"""Where the schemas, SQL, vectors, registries and context are on disk (paperspec.resources).""" +import json import pathlib import pytest @@ -16,11 +17,21 @@ MARKERS = { "schemas": "annotation.schema.json", "sql": "doc.sql", "vectors": "proofs.jsonl", + "registry": "stamps/scienting-stamps-1.json", + "context": "scienting-1.jsonld", + "fixtures": "packages/hale2026harmonic-damb/MANIFEST.json", } -def test_the_folders_are_the_three_the_package_names(): - assert resources.FOLDERS == ("schemas", "sql", "vectors") +def test_the_folders_are_the_six_the_package_names(): + assert resources.FOLDERS == ( + "schemas", + "sql", + "vectors", + "registry", + "context", + "fixtures", + ) @pytest.mark.parametrize("name", resources.FOLDERS) @@ -32,7 +43,9 @@ def test_each_folder_is_found_and_holds_its_files(name): @pytest.mark.parametrize("name", ["tests", "", "SCHEMAS", "schemas/", "../schemas"]) def test_any_other_name_is_refused_and_the_folders_are_listed(name): - with pytest.raises(ValueError, match="there are schemas, sql, vectors"): + with pytest.raises( + ValueError, match="there are schemas, sql, vectors, registry, context, fixtures" + ): resources.path(name) @@ -71,3 +84,34 @@ def test_a_file_in_place_of_a_folder_is_not_a_folder(layout): (layout / "schemas").write_text("not a folder", encoding="utf-8") with pytest.raises(FileNotFoundError): resources.path("schemas") + + +@pytest.mark.parametrize( + ("name", "fmt"), + [ + ("stamps/scienting-stamps-1.json", "scienting-stamps/1"), + ("stages/scienting-stages-default.json", "scienting-stages/1"), + ], +) +def test_a_registry_file_is_read_by_its_path_under_registry(name, fmt): + assert resources.registry(name)["format"] == fmt + + +def test_the_rules_registry_lists_every_rule_by_id(): + rules = resources.registry("rules.json") + assert "doc.suspect.witness" in json.dumps(rules) + + +@pytest.mark.parametrize( + "name", + ["", "README.md", "../pyproject.toml", "stamps/../rules.json", "/rules.json"], +) +def test_a_registry_name_that_is_not_a_plain_json_path_is_refused(name): + with pytest.raises(ValueError, match="no registry file"): + resources.registry(name) + + +def test_a_missing_registry_file_is_an_error(layout): + (layout / "registry").mkdir() + with pytest.raises(FileNotFoundError): + resources.registry("rules.json") diff --git a/tests/test_revision.py b/tests/test_revision.py new file mode 100644 index 0000000..86d630a --- /dev/null +++ b/tests/test_revision.py @@ -0,0 +1,378 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 Scienting +# +# Source Code: https://git.scient.ing/papers/paperspec +# See the LICENSE.md file for full license terms. + +"""Applying corrections to a package and writing the package they make (paperspec.revision).""" + +import hashlib +import json +import shutil + +import pytest + +from helpers_cli import CHECKS_ONLY, HALE, TINY_ZIP, edit_package, package_dirs, store +from paperspec import corrections, jcs, package, revision + +PERSON = [{"kind": "person", "by": "ana", "note": "Read against the page."}] +CROP_PNG = b"\x89PNG\r\n\x1a\n not really a crop" +CROP = [{"kind": "crop", "sha256": hashlib.sha256(CROP_PNG).hexdigest()}] + + +def proposal(op, evidence=PERSON, **fields): + """A proposed correction by Ana.""" + return { + "format": "scienting-correction/1", + "evidence": evidence, + "author": {"key": "ana"}, + "reason": "Test.", + "at": "2026-10-20T09:00:00Z", + "status": "proposed", + "op": op, + **fields, + } + + +REPLACE = proposal("replace", target="mhp0at8", find="harmonic", replace="quadratic") + + +def snapshot(folder): + return { + str(p.relative_to(folder)): p.read_bytes() + for p in sorted(folder.rglob("*")) + if p.is_file() + } + + +def records(folder, name): + path = folder / name + return package.read_jsonl(path) if path.exists() else [] + + +def node(folder, nid): + return next((n for n in records(folder, "nodes.jsonl") if n["id"] == nid), None) + + +@pytest.fixture +def hale(tmp_path): + """A copy of the main example package to correct.""" + return shutil.copytree(HALE, tmp_path / HALE.name) + + +@pytest.mark.parametrize( + "source", + [p for p in package_dirs() if not p.name.endswith("checks-only")], + ids=lambda p: p.name, +) +def test_no_corrections_rewrite_every_full_package_byte_for_byte(tmp_path, source): + # Everything the writer recomputes recomputes to what the package holds, so with no + # proposals the package it writes is the package it read. + out = tmp_path / "out" / source.name + result = revision.revise(source, [], store(), out) + assert snapshot(out) == snapshot(source) + assert result["records"] == [] + assert ( + result["package"] + == hashlib.sha256((source / "MANIFEST.json").read_bytes()).hexdigest() + ) + + +def test_a_correction_writes_a_valid_package_one_revision_on(hale): + before = json.loads((hale / "document.json").read_text()) + result = revision.revise(hale, [REPLACE], store()) + after = package.read_json(hale / "document.json") + assert not package.validate(hale, store()).errors + assert result["rev"] == after["rev"] == before["rev"] + 1 + assert after["history"] != before["history"] + assert result["path"] == str(hale) + assert result["key"] == "hale2026harmonic-damb" + assert ( + result["package"] + == hashlib.sha256((hale / "MANIFEST.json").read_bytes()).hexdigest() + ) + assert records(hale, "corrections.jsonl")[-1] == result["records"][0] + + +def test_the_records_are_those_the_gate_returns(hale): + state = package.read_state(hale) + _, expected = corrections.gate(state, REPLACE, store()) + assert revision.revise(hale, [REPLACE], store())["records"] == [expected] + + +def test_the_text_and_every_value_are_the_corrected_ones(hale): + revision.revise(hale, [REPLACE], store()) + n = node(hale, "mhp0at8") + assert n["content"][0]["text"].startswith("We fit quadratic bond constants") + # validate recomputed every value; the record's after is the leaf's new hash + assert records(hale, "corrections.jsonl")[-1]["after"] == n["node_hash"] + + +def test_a_leaf_changed_without_a_crop_is_f0_and_no_longer_corroborated(hale): + assert node(HALE, "mhp0at8")["fidelity"] == "F2" + revision.revise(hale, [REPLACE], store()) + n = node(hale, "mhp0at8") + assert n["fidelity"] == "F0" + assert "corroborated_by" not in n["prov"] + assert n["prov"]["witnesses"] == ["qptsanv"] + + +def test_a_leaf_changed_with_a_crop_is_f1(hale): + crop = proposal( + "replace", evidence=CROP, target="mhp0at8", find="twelve", replace="eleven" + ) + revision.revise(hale, [crop], store(), assets=[(CROP_PNG, "png")]) + assert node(hale, "mhp0at8")["fidelity"] == "F1" + name = f"assets/{CROP[0]['sha256']}.png" + assert (hale / name).read_bytes() == CROP_PNG + assert name in { + e["path"] for e in package.read_json(hale / "MANIFEST.json")["files"] + } + + +def test_an_emend_keeps_the_level(hale): + emend = proposal( + "emend", target="mhp0at8", find="harmonic", replace="harmonics", kind="typo" + ) + revision.revise(hale, [emend], store()) + assert node(hale, "mhp0at8")["fidelity"] == "F2" + assert node(hale, "mhp0at8")["prov"]["corroborated_by"] == ["qptsanv"] + + +def test_the_fidelity_report_counts_the_changed_leaf(hale): + before = package.read_json(HALE / "document.json")["fidelity"] + revision.revise(hale, [REPLACE], store()) + after = package.read_json(hale / "document.json")["fidelity"] + assert after != before + assert after["counts"].get("F0", 0) == before["counts"].get("F0", 0) + 1 + + +def test_a_merge_gives_the_merged_node_the_boxes_of_the_node_merged_into_it(hale): + locs = records(HALE, "locations.jsonl") + a = [loc for loc in locs if loc["node"] == "mhp0at8"] + b = [loc for loc in locs if loc["node"] == "v3063qv"] + revision.revise( + hale, [proposal("merge", target="mhp0at8", **{"with": "v3063qv"})], store() + ) + after = records(hale, "locations.jsonl") + assert not [loc for loc in after if loc["node"] == "v3063qv"] + merged = [loc for loc in after if loc["node"] == "mhp0at8"] + pages = {(loc["page"], loc["page_fp"]) for loc in a + b} + assert {(loc["page"], loc["page_fp"]) for loc in merged} == pages + boxes = [box for loc in a + b for box in loc["boxes"]] + assert sorted(box for loc in merged for box in loc["boxes"]) == sorted(boxes) + retired = {e["id"]: e for e in records(hale, "ids.jsonl")} + assert retired["v3063qv"]["successor"] == "mhp0at8" + + +def test_a_delete_drops_the_locations_of_what_it_removed(hale): + assert "nms5qjg" in {loc["node"] for loc in records(HALE, "locations.jsonl")} + revision.revise(hale, [proposal("delete", target="nms5qjg")], store()) + assert "nms5qjg" not in {loc["node"] for loc in records(hale, "locations.jsonl")} + assert node(hale, "nms5qjg") is None + retired = {e["id"]: e for e in records(hale, "ids.jsonl")} + assert retired["nms5qjg"].get("successor") is None + + +def test_a_cited_reference_cannot_be_deleted(hale): + with pytest.raises(revision.Refused) as e: + revision.revise(hale, [proposal("delete", target="vxsdvzq")], store()) + assert e.value.rule == "doc.run.cite" + + +def test_derived_lines_of_a_touched_formula_are_left_out_and_reported(hale): + fix = proposal("replace", target="hca6dc5", find="Water", replace="Liquid water") + result = revision.revise(hale, [fix], store()) + math = records(hale, "derived/math.jsonl") + assert "hca6dc5" not in {r["node"] for r in math} + assert {r["node"] for r in math} == {"tkxsmts", "rr20tev"} + assert result["dropped"] == {"derived/math.jsonl": 1} + + +def test_a_dataset_whose_cell_was_corrected_is_left_out_and_reported(hale): + fix = proposal( + "replace", evidence=CROP, target="c20cna6", find="310.0", replace="310.5" + ) + result = revision.revise(hale, [fix], store(), assets=[(CROP_PNG, "png")]) + assert not (hale / "datasets" / "ds-hale-bond-params").exists() + assert result["dropped"]["datasets/ds-hale-bond-params/"] == 3 + + +def test_answered_suspects_leave_and_an_unknown_one_is_refused(tmp_path): + suspect = { + "format": "scienting-suspect/1", + "id": "aaaaaaaaaaaaa", + "kind": "word", + "node": "mhp0at8", + "a": "harmonic", + "b": "quadratic", + } + gone = dict(suspect, id="bbbbbbbbbbbbb", node="v3063qv") + src = edit_package( + tmp_path, HALE, {"suspects.jsonl": package.jsonl([suspect, gone])} + ) + merge = proposal("merge", target="mhp0at8", **{"with": "v3063qv"}) + with pytest.raises(ValueError, match="no suspect zzzzzzzzzzzzz"): + revision.revise(src, [merge], store(), answered=["zzzzzzzzzzzzz"]) + revision.revise(src, [merge], store(), answered=["aaaaaaaaaaaaa"]) + # the answered one goes, and so does the one at the node the merge retired + assert not (src / "suspects.jsonl").exists() + + +def test_a_tool_is_recorded_in_the_provenance_once(hale): + tool = {"name": "paperdesk", "version": "0.3.0"} + revision.revise(hale, [REPLACE], store(), tool=tool) + again = proposal("replace", target="mhp0at8", find="quadratic", replace="harmonic") + revision.revise(hale, [again], store(), tool=tool) + prov = package.read_json(hale / "provenance.json") + assert prov["tools"].count(tool) == 1 + stage = {"stage": "review", "processor": "local", "tool": "paperdesk 0.3.0"} + assert prov["stages"].count(stage) == 1 + + +def test_a_refused_proposal_writes_nothing_and_names_its_place(hale): + before = snapshot(hale) + bad = proposal("replace", target="mhp0at8", find="absent", replace="x") + with pytest.raises(revision.Refused) as e: + revision.revise(hale, [REPLACE, bad], store()) + assert (e.value.index, e.value.rule) == (2, "correction.find") + assert str(e.value).startswith("proposal 2: correction.find: ") + assert snapshot(hale) == before + + +def test_a_proposal_off_the_schema_is_refused(hale): + with pytest.raises(revision.Refused) as e: + revision.revise(hale, [{"op": "replace"}], store()) + assert e.value.rule == "correction.schema" + + +def test_an_invalid_package_is_refused_with_its_errors(tmp_path): + broken = edit_package(tmp_path, HALE, {"document.json": lambda d: dict(d, rev=99)}) + with pytest.raises(revision.Invalid) as e: + revision.revise(broken, [REPLACE], store()) + assert "doc.correction.rev" in {err["rule"] for err in e.value.errors} + assert "the package is not valid" in str(e.value) + + +def test_a_checks_only_package_takes_no_corrections(tmp_path): + src = shutil.copytree(CHECKS_ONLY, tmp_path / CHECKS_ONLY.name) + files, absent = revision._read_folder(src) # validating it needs the team key + with pytest.raises(ValueError, match="withholds its text or is keyed"): + revision.revise_files(files, absent, [REPLACE], store()) + + +def test_an_existing_out_folder_is_refused(hale, tmp_path): + (tmp_path / "taken").mkdir() + with pytest.raises(ValueError, match="exists"): + revision.revise(hale, [REPLACE], store(), tmp_path / "taken") + + +def test_out_leaves_the_package_as_it_was(hale, tmp_path): + before = snapshot(hale) + result = revision.revise(hale, [REPLACE], store(), tmp_path / "fixed" / hale.name) + assert snapshot(hale) == before + assert not package.validate(result["path"], store()).errors + + +def test_a_package_that_would_not_validate_is_not_written(hale, monkeypatch): + before = snapshot(hale) + real = revision.summarize + monkeypatch.setattr( + revision, "summarize", lambda *a, **k: dict(real(*a, **k), rev=0) + ) + with pytest.raises(RuntimeError, match=r"does not validate: doc\.correction\.rev"): + revision.revise(hale, [REPLACE], store()) + assert snapshot(hale) == before + + +def test_two_runs_write_byte_identical_packages(tmp_path): + a = revision.revise(HALE, [REPLACE], store(), tmp_path / "a" / HALE.name) + b = revision.revise(HALE, [REPLACE], store(), tmp_path / "b" / HALE.name) + assert snapshot(tmp_path / "a" / HALE.name) == snapshot(tmp_path / "b" / HALE.name) + assert a["package"] == b["package"] + + +def test_several_corrections_apply_in_order(hale): + second = proposal("replace", target="mhp0at8", find="quadratic", replace="Morse") + result = revision.revise(hale, [REPLACE, second], store()) + assert [r["index"] for r in result["records"]] == [13, 14] + assert "Morse bond" in node(hale, "mhp0at8")["content"][0]["text"] + + +def test_without_a_store_the_gate_skips_its_schema_checks(tmp_path): + files, absent = revision._read_folder(HALE) + new, added, summary = revision.revise_files(files, absent, [REPLACE], None) + assert summary == {"rev": 13, "dropped": {}, "answered": []} + assert added[0]["op"] == "replace" + package.write(tmp_path / HALE.name, new, absent) + assert not package.validate(tmp_path / HALE.name, store()).errors + + +def test_a_zipped_package_is_corrected_and_zipped_again(tmp_path): + src = shutil.copy(TINY_ZIP, tmp_path / TINY_ZIP.name) + with_tiny = package.read_zip(TINY_ZIP, tmp_path / "unpacked") + leaf = next( + n + for n in package.read_jsonl(f"{with_tiny}/nodes.jsonl") + if n["kind"] == "sentence" + ) + word = leaf["content"][0]["text"].split()[0] + fix = proposal("replace", target=leaf["id"], find=word, replace=word + "s") + result = revision.revise_zip(src, [fix], store()) + assert result["path"] == str(src) + assert result["zip"] == hashlib.sha256(src.read_bytes()).hexdigest() + assert not package.validate_zip(src, store()).errors + out = tmp_path / "copy.sct.zip" + revision.revise_zip(TINY_ZIP, [fix], store(), out) + assert out.read_bytes() == src.read_bytes() + with pytest.raises(ValueError, match="exists"): + revision.revise_zip(TINY_ZIP, [fix], store(), out) + + +@pytest.mark.parametrize( + ("op", "level"), + [ + ("replace", "F0"), + ("move", "F0"), + ("emend", None), + ("relink", None), + ("resolve", None), + ], +) +def test_the_level_after_a_correction_follows_its_op_and_evidence(op, level): + assert revision.level_after({"op": op, "evidence": PERSON}) == level + if level is not None: + assert revision.level_after({"op": op, "evidence": PERSON + CROP}) == "F1" + + +def test_summarize_recomputes_what_document_json_derives_from_the_nodes(): + document = package.read_json(HALE / "document.json") + nodes = package.read_jsonl(HALE / "nodes.jsonl") + values = {n["id"]: {k: n[k] for k in revision.VALUES} for n in nodes} + roster = package.read_json(HALE / "roster.json")["keys"] + again = revision.summarize( + document, + nodes, + values, + package.read_jsonl(HALE / "corrections.jsonl"), + package.read_jsonl(HALE / "receipts.jsonl"), + roster, + ) + assert jcs.dumps(again) == jcs.dumps(document) + + +def test_a_live_nodes_records_for_one_page_are_kept_as_the_package_gave_them(tmp_path): + # paperform writes a block's approximate box and its glyph boxes as two records of one + # page; the writer leaves what it was not asked to change alone. + locs = records(HALE, "locations.jsonl") + first = next(loc for loc in locs if loc["node"] == "mhp0at8") + extra = dict(first, approx=True, boxes=[[1, 2, 3, 4]]) + at = locs.index(first) + src = edit_package( + tmp_path, + HALE, + {"locations.jsonl": lambda rows: [*rows[: at + 1], extra, *rows[at + 1 :]]}, + ) + out = tmp_path / "out" / src.name + revision.revise(src, [], store(), out) + assert records(out, "locations.jsonl") == records(src, "locations.jsonl") diff --git a/tests/test_text.py b/tests/test_text.py index 00ee79e..f1814c1 100644 --- a/tests/test_text.py +++ b/tests/test_text.py @@ -352,3 +352,51 @@ def test_cp_to_utf16_counts_astral_characters_twice(): assert cp_to_utf16("a\U0001d465b", 0) == 0 assert cp_to_utf16("a\U0001d465b", 2) == 3 assert cp_to_utf16("a\U0001d465b", 3) == 4 + + +# The whitespace and digit helpers are written with regular expressions and str methods for +# speed; these hold them to the character-by-character definitions over every code point. +EVERY_CHARACTER = "".join(chr(c) for c in range(0x110000) if not 0xD800 <= c <= 0xDFFF) + + +def test_digits_is_general_category_nd_over_every_code_point(): + assert text.digits(EVERY_CHARACTER) == "".join( + ch for ch in EVERY_CHARACTER if unicodedata.category(ch) == "Nd" + ) + + +def test_remove_ws_removes_exactly_white_space_over_every_code_point(): + assert text.remove_ws(EVERY_CHARACTER) == "".join( + ch for ch in EVERY_CHARACTER if ord(ch) not in text.WHITE_SPACE + ) + + +def _collapse_by_hand(s): + out, prev = [], False + for ch in s: + if text.is_ws(ch): + if not prev: + out.append(" ") + prev = True + else: + out.append(ch) + prev = False + return "".join(out) + + +@given( + st.text( + alphabet=st.sampled_from( + [*map(chr, sorted(text.WHITE_SPACE)), "a", "é", "\x00", "1"] + ) + ) +) +@settings(derandomize=True, max_examples=300, database=None, deadline=None) +def test_collapse_ws_and_trim_match_their_definitions(s): + assert text.collapse_ws(s) == _collapse_by_hand(s) + a, b = 0, len(s) + while a < b and text.is_ws(s[a]): + a += 1 + while b > a and text.is_ws(s[b - 1]): + b -= 1 + assert text.trim(s) == s[a:b] diff --git a/tools/README.md b/tools/README.md index 5cf82ba..98de355 100644 --- a/tools/README.md +++ b/tools/README.md @@ -6,7 +6,7 @@ Implementers get the commands and the traps. These commands are the repository's reference implementation. They apply every rule of the specification, so the rest of the repository can be checked against them. -Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website. +Two build and check the repository, one validates packages and proposed corrections, one applies corrections to a package, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website. They are subcommands of one command, `paperspec`, which the Python package `paperspec` installs; its code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`. `paperspec --help` lists them and says where each works, and `paperspec COMMAND --help` gives a command's options, advice and examples. They need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), except `paperspec site`, which needs only the standard library, and they are licensed Apache-2.0. @@ -20,11 +20,12 @@ A fixture is an example file or package under `fixtures/`. | `paperspec check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. | | `paperspec build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. | | `paperspec validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). | +| `paperspec correct` | Applies proposed corrections to a package, folder or zipped, and writes the package they make: `paperspec correct --correction PROPOSALS [--out PATH] PACKAGE`. Every proposal runs through the gate first and nothing is written unless all pass; the writer recomputes the nodes' values and `document.json`, records the corrections and retired IDs, lowers the changed leaves' levels, and leaves out the derived lines and datasets that read corrected text ([spec/11](../spec/11-corrections.md), [spec/12 §12.7](../spec/12-package.md#127-derived-files)). | | `paperspec load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). | | `paperspec site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). | | `paperspec stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. | -`paperspec validate` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries. +`paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is installed with its `tools` extra, using the schemas and SQL it carries. The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder. ## When to run what @@ -75,12 +76,12 @@ A node's five values depend only on its own subtree, so after a correction the o `paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree. Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code. -The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema`, `rights`, `traces`, `structures`, `sequences` and `data`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike. -The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository. +paperstack, paperform, paperdesk and paperclaim import it, and paperchem, paperbio and paperdata test their annotation bodies against it. +The library surface they use is `keys`, `text`, `jcs`, `hashing`, `tree`, `anchors`, `package`, `revision`, `corrections`, `db`, `schema`, `fingerprint`, `rights`, `layers`, `claims`, `traces`, `structures`, `sequences` and `data`, with the schemas, SQL, vectors, registries and JSON-LD context as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike. +The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository, where `paperspec.resources.path("fixtures")` finds them in a checkout or an editable install. The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other. Importing any of its modules needs only the standard library. Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`. `paperspec[tools]` installs both, and is what a program that validates packages needs. The package version comes from git tags through setuptools-scm, and `pixi run -e release dist` builds the sdist and the wheel into `dist/`. -When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing. Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).