Files
paperspec/spec/22-conformance.md
T
alexmandClaude 4fb8b4f5a7 Add paperspec correct, check math forms and datasets, and give the tools public names
The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
2026-10-20 22:10:17 -04:00

63 lines
5.9 KiB
Markdown

# 22. Conformance
## 22.1 The suite
| Path | Holds | A conforming implementation |
| --- | --- | --- |
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document, as a folder and as the zip of [12.2](12-package.md#122-paths). | Accepts each, and recomputes every value it contains. |
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own, and four more layers: traced citations ([18.5](18-layers.md#185-traced-citations)), whose traces resolve as `vectors/traces.jsonl` says, chemical structures ([18.6](18-layers.md#186-chemical-structures)), sequences ([18.7](18-layers.md#187-sequences)) and data ([18.8](18-layers.md#188-data)), whose rules `vectors/structures.jsonl`, `vectors/sequences.jsonl` and `vectors/data.jsonl` pin; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `paperspec load --bundle`. | Accepts each. |
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |
All fixture text is synthetic and dedicated to the public domain; license fields inside fixtures describe the fictional works.
Each part's CI runs this suite against its own implementation, so the parts cannot drift apart silently.
## 22.2 Validating a package
A validator checks, in order, and reports each failure as `path:line: rule: message`:
1. The manifest: present, valid, sorted, every listed file present with its hash and size, nothing unlisted, paths safe and no symbolic links, required files present.
2. Every file in canonical form, byte for byte, and every record against its schema.
3. The tree ([3](03-nodes.md)), runs and references between nodes, the counts, and the keys ([2.4](02-model.md#24-work-and-document-keys)).
4. The policy and hashing scheme ([12.5](12-package.md#125-text-policies)).
5. The five values of every node, the root and `meta_hash`, when the text is present and the key, if any, is known.
6. The ID table, the corrections and the history root, the witnesses, pages, locations and references, the formulas derived files name, and every file's order.
7. Receipts: IDs, roots, documents, node order, signatures against the package's roster, spot-check keys, seeds and samples, and the fidelity distribution.
## 22.3 Schemas
Schemas use draft 2020-12, restricted to the keywords `$schema`, `$id`, `$ref`, `$defs`, `$comment`, `title`, `description`, `examples`, `default`, `deprecated`, `type`, `enum`, `const`, `properties`, `required`, `additionalProperties`, `patternProperties`, `propertyNames`, `minProperties`, `maxProperties`, `dependentRequired`, `items`, `prefixItems`, `minItems`, `maxItems`, `uniqueItems`, `contains`, `minLength`, `maxLength`, `pattern`, `format`, `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then` and `else`.
Patterns avoid `\d`, `\w`, `\s`, `\b` and lookbehind, whose meaning differs between regular-expression dialects.
A schema checks shape; the chapters' rules, such as the five values, are checked by validators.
## 22.4 Rules
[registry/rules.json](../registry/rules.json) lists every rule ID with its meaning.
Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is never reused.
| Prefix | Covers |
| --- | --- |
| `package.` | The manifest, paths, files and schemas. |
| `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.`, `doc.math.` | The tree, node content and runs. |
| `doc.derived.` | Derived files ([12.7](12-package.md#127-derived-files)). |
| `doc.dataset.` | Datasets ([15](15-datasets.md)). |
| `doc.hash.` | The five values, the root and the metadata hash. |
| `doc.id.`, `doc.order.` | The ID table and file order. |
| `doc.policy.` | Text policies and keyed hashing. |
| `doc.key.`, `doc.counts`, `doc.history` | The keys, the counts and the history root of `document.json`. |
| `doc.correction.`, `correction.` | A package's corrections, and the gate. |
| `doc.receipt.`, `doc.qa.`, `doc.fidelity.` | Receipts, spot-checks and levels. |
| `doc.witness.`, `doc.page.`, `doc.location.`, `doc.reference.` | Witnesses and geometry. |
| `grade.` | Grading. |
| `layer.` | Annotations: `layer.trace.source` and `layer.trace.target`, which refer to documents ([18.5](18-layers.md#185-traced-citations)); `layer.structure.value`, `layer.structure.identity` and `layer.structure.term` ([18.6](18-layers.md#186-chemical-structures)); `layer.sequence.value`, `layer.sequence.identity`, `layer.sequence.region` and `layer.sequence.variant` ([18.7](18-layers.md#187-sequences)); and `layer.data.type`, `layer.data.quantity`, `layer.data.subject` and `layer.data.dataset` ([18.8](18-layers.md#188-data)). |
A package is validated by [22.2](#222-validating-a-package), but a layer is not part of one.
The `layer.` rules are reported for an annotation, in the form `annotation: rule: message`, by a validator that has the annotation and, for a trace, the states of the documents it names; the reference implementation does so in `paperspec.traces.problems`, `paperspec.structures.problems`, `paperspec.sequences.problems` and `paperspec.data.problems`.
## 22.5 Stability
Within a major version of a format, a valid file stays valid, and every value the chapters define is computed the same way.
Rule IDs, enumeration values and format identifiers are never reused with another meaning.
A `candidate` format changes only with a recorded reason and a new or changed vector; a `provisional` one may change more freely, and says so in [registry/formats.json](../registry/formats.json).