paperspec/CHANGELOG.md

# Changelog

A change to any computed value is a breaking change and needs a new major version of the format it touches; see [spec/00-introduction.md](spec/00-introduction.md).

## 0.1.0

The first release.

- The specification, in chapters 0 to 23, covering every format: 32 formats, of which 19 are `candidate` and 13 `provisional`.
- 43 JSON Schemas, using a keyword subset that independent validators read alike.
- Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search.
- The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements.
- Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3.
- Example packages (a converted chemistry paper with corrections and receipts, its checks-only export, an open-access paper, a Typst draft and a tiny document, also as a zip), a record of every other format, and 31 broken packages, each breaking one named rule.
- Work keys minted from a Scholar-style citation key and a hash of the work's identity, with document keys for versions; `tag:` URIs for layers, annotations and documents.
- Spot-check seeds as VRF proofs (RFC 9381), RFC 9162 inclusion and consistency proofs, a history root over every document's corrections, and a provisional transparency log with signed checkpoints.
- Signed QA reports, a conformance checklist covered by a receipt, and a `description` annotation for the figure descriptions an accessible export needs.
- ML-DSA-65 (FIPS 204) as a signature scheme, with a reference implementation checked against NIST's vectors; `cose` reserved; `did:key` identifiers for roster keys; the hash algorithm named in every document; model provenance by weights, prompt and parameters; witnesses and a static tile layout for the log; and the character data pinned to Unicode 16.0.
- Guides for newcomers: an overview, the concepts, a walkthrough of the example paper, runnable examples, the rationale, a FAQ and a glossary.
- Tools that build and check everything, validate packages and zipped packages and run proposed corrections through the gate, load packages into SQLite or a bundle, and stress the reference implementation with random inputs.
- A pixi workspace pinned to Python 3.14, with tasks for checking, building, linting, validating and loading.

### 0.1.1

Packaging, commands and lazy imports only; the formats stay at 0.1.0, and no computed value, vector, fixture, schema or SQL file changed.

- The oracle moves from `tools/oracle/` to `src/paperspec/` and installs as the Python package `paperspec` (`pyproject.toml`, setuptools), with `schemas/`, `sql/` and `vectors/` as the data subpackages `paperspec.schemas`, `paperspec.sql` and `paperspec.vectors`, so other repositories import the reference implementation instead of reimplementing it.
- The package version comes from git tags through setuptools-scm.
- The base package has no dependencies and requires Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import under any other Unicode version; the extras `paperspec[schema]` (`jsonschema`, `referencing`), `paperspec[crypto]` (`cryptography`) and `paperspec[tools]` (both) cover `schema` and the HKDF and Ed25519 functions of `crypto`, which now import their packages on first use.
- `paperspec.resources.path` finds the schemas, SQL and vectors in an installed wheel and in an editable install alike.
- The programs in `tools/` become subcommands of one command, `paperspec`, in `src/paperspec/cli/`: `paperspec validate`, `load`, `check`, `build`, `stress` and `site`, which the pixi tasks and the workflows run, and which `python -m paperspec` also runs.
  The scripts `tools/*.py` are gone, so `python3 tools/check.py` and the like no longer work; `tools/` keeps the page that documents the commands.
- `paperspec --help` says which commands work anywhere and which need a checkout, each command's `--help` gives advice and examples, a misspelled command gets a suggestion, and `paperspec --version` prints the package version and the Unicode version it pins.
- `paperspec load` refuses `--built-from` and `--store` without `--bundle`, where it used to ignore them.
- `src/` is Apache-2.0, as `tools/` was, and the package metadata states `Apache-2.0 AND CC0-1.0` for the code and the data it carries.
- The pixi workspace installs the package editable, and a `dev` environment builds the sdist and the wheel with `pixi run -e dev dist`.
- Pushing a version tag publishes the package to the Gitea package registry through `.gitea/workflows/publish.yml` ([deploy/](deploy/README.md#publish-the-python-package)); a release tag versions the package, and [CONTRIBUTING.md](CONTRIBUTING.md#releasing) says how that relates to the format version.

### Unreleased

Two changes asked for by the first producer that compares a converted reading with a second witness (paperform with a JATS file).
Both are minor changes in the sense of [spec/00](spec/00-introduction.md): no valid file becomes invalid, and no hash, ID or key changes; the next release numbers them.

- Alignment has a substitution step ([spec/05 §5.6](spec/05-hashes.md#56-alignment), step 3): in each stretch between pairs, an unpaired sentence of the first reading is paired with the first unpaired sentence of the second whose similarity (2*L* / (|*a*| + |*b*|) over code points of the lossy projections, *L* their longest common subsequence) is at least 0.8, both of prose length 20 or more.
  A sentence one reading misread is now one `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one, so a producer re-aligning the same two readings can get different suspects than before; the pairs and suspects of readings with no such sentence are unchanged, as the first alignment vector is.
  `paperspec.align` gains `lcs_length` (bit-parallel, Hyyrö 2004) and `similarity`, and `alignment.jsonl` a second case, `substitutions`: a misread number, a misread word, a sentence too short to be paired this way and two sentences too unlike to be one.
- A suspect may name the witness its second reading came from (`witness`, a witness ID; [spec/05 §5.7](spec/05-hashes.md#57-suspects), `suspect.schema.json`), and a package whose suspect names a witness it does not have breaks `doc.suspect.witness`.
  No broken package exercises the rule yet: adding one changes the count of broken packages that the 0.1.0 entry above states, so it waits for the release.

Four more, asked for by editors who correct and discuss papers rather than readings.
They are minor changes too: new operations, enumeration values and optional properties, with no valid file made invalid, no value of an existing file computed differently, and no existing vector case or package changed.

- Annotation categories ([spec/18 §18.4](spec/18-layers.md#184-bodies), `annotation.schema.json`, the `category` term of the JSON-LD context): a `comment`, `caveat` or `question` body may carry a `category` from the fixed list `comment`, `context`, `critique`, `question`, `correction`, `feedback` and `resource`, which a producer that supports categories SHOULD set on every such body.
  The store's `annotations` table gains a nullable `category` column with its CHECK ([sql/store.sql](sql/store.sql)).
  The example layer's comment, question and reply gain categories, which changes their record IDs, and it gains a caveat; no other fixture changes.
- `emend`, a correction of the authors' own typo, spelling, grammar, punctuation or wording, where the reading matches the page and the page is wrong ([spec/11 §11.6](spec/11-corrections.md#116-emendations)).
  The emended text keeps what was printed in a new optional run property, `printed` ([spec/03 §3.5](spec/03-nodes.md#35-inline-runs), `runs.schema.json`), so a leaf as printed is recovered from the package alone ([3.6](spec/03-nodes.md#36-the-plain-projection), `paperspec.hashing.printed` and `as_printed`), and runs with `printed` are never joined.
  The strict and lossy projections are of the runs as printed, so an emend changes the leaf's `text_hash`, `own_hash` and `node_hash` but not its `match_hash` or `align_key`, and alignment, corroboration and suspects never report an emendation ([spec/05 §5.1](spec/05-hashes.md#51-the-five-values)).
  The gate refuses an emend in math, across runs, or that changes a digit or edits beside one, and refuses a `replace`, text `insert` or `split` inside an emended run (`correction.emend`).
- `relink` makes a run a citation of other references, a text run a cite run, or a cite run text, without changing its text, and `resolve` sets a reference's resolution in `references.jsonl`, keeping the one before in `from` ([spec/11 §11.7](spec/11-corrections.md#117-citations-and-references)); a package whose last resolve of a reference is not its resolution breaks `doc.correction.resolution`.
  Like an emend, neither lowers a leaf's level, and a receipt keeps counting across all three ([spec/08 §8.1–8.2](spec/08-fidelity.md#82-levels), `receipts.carried_hashes`).
- The rules `correction.emend`, `correction.relink`, `correction.resolve` and `doc.correction.resolution` are registered, the `corrections` table's CHECK takes the three operations ([sql/doc.sql](sql/doc.sql)), `corrections.State` carries the reference records, and `paperspec stress` walks the three operations with their invariants.
  `corrections.jsonl` gains thirteen cases and `projections.jsonl` and `projection-groups.jsonl` two each; no existing case changes.

Four more, asked for by papermath, which defines the math form `scienting-math/1`, and by paperclaim, the claim store whose claims and symbols layers point at.
The first two are minor changes: an optional property, a derived file and its schema, with no hash, ID or key of an existing file changed.
The last two make some files that were valid invalid, which [spec/00](spec/00-introduction.md) calls breaking; they are made now because no release of `scienting-doc/1` or `scienting-layer/1` has been tagged, and a reference worth changing is changed before one is ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), item 4).

- Math forms ([spec/03 §3.7](spec/03-nodes.md#37-math-forms), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `node.schema.json`, `runs.schema.json`, `mathForm` in `common.schema.json`): an equation's content and a `math` run may carry `form`, whose one value, `scienting-math/1`, says that the source is canonical TeX; the notation is then `tex`.
  A consumer may rely on the source being canonical and idempotent, carrying no label, and on everything else being derived from it.
  `form` enters no content value ([spec/05 §5.2](spec/05-hashes.md#52-own-content)): own content leaves it out of math runs (`paperspec.hashing`), an equation's own content never held more than its notation and source, and no projection reads it, so adding `form` to a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does.
  Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first.
  `hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator does not.
- `derived/math.jsonl` ([spec/12 §12.7](spec/12-package.md#127-derived-files), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `math.schema.json`): one line per formula, `{node, run, tool, form, tree, symbols, complete, issues, label?}`, papermath's reading without the canonical TeX, which is the source; `run` is null for an equation, as in `derived/mathml.jsonl`, and `label` is omitted when the source has no `\tag`.
  A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 49 kinds and six codes of version 1.
  The validator now checks derived files: each line against its schema, that it names a formula of the package (`doc.derived.node`), that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`), and that a checks-only package carries none (`doc.policy.withheld`), since a tree or MathML reads the withheld text.
  The main example gains `derived/math.jsonl`, papermath 0.1.0's reading of its equation and of two math runs, copied by hand because the oracle does not implement the form; `paperspec check` validates fourteen variants of it, and no fixture's hashes change.
  There are now 43 schemas, which the 0.1.0 entry above, held to the repository by `paperspec check`, states too.
- Claim references ([spec/18 §18.4](spec/18-layers.md#184-bodies), [spec/02 §2.4](spec/02-model.md#24-work-and-document-keys), [spec/20](spec/20-projects.md), `claimRef` and `claimPin` in `common.schema.json`): the `source` of a `claim-link` or `symbol` body, a `claim:` identifier in `same_as`, and a project's claim step are paperclaim references, `claim:<tag>` or `claim:<host>/<tag>` with the tag eight Crockford base32 capitals, and a `pin` is the reference, `@` and six lowercase hex digits of the claim's statement hash, as paperclaim writes them; a project names its claim namespace as `claims.paperclaim`.
  A reference by slug, such as `amber/k-bond`, which paperclaim may rename, or under any other prefix, was valid before and is not now.
  The fixtures' slug reference and `illustrative-pin` become `claim:272XTX7C` and `claim:272XTX7C@02f47b`, the claim `amber/k-bond` of paperclaim's example store, which changes the claim link's and the symbol's record IDs, the layer receipt and `project.json`.
  The main example's two targeted receipts take the scope `targeted:paperclaim`, which changes their IDs and signatures, Ben's spot-check of the second (its seed and its sample of ten, still without errors), the QA report, the inclusion proof, the transparency log's checkpoint and tiles, and the signature, spot-check, VRF and checkpoint vectors built from them; no node's level or the report's counts change.
- Symbol keys ([spec/18 §18.4](spec/18-layers.md#184-bodies), `annotation.schema.json`, the `key` term of the JSON-LD context, `sct:symbolKey`): a `symbol` body names its symbol by `key`, its `scienting-math/1` key, besides the span its target selects, so a symbol is found across papers and matched to its claim by key; a `symbol` body without `key` was valid before and is not now.
  The example layer gains a proposed symbol annotation on `$K_r$`, with the key `K_{r}`.
- Three failures of `paperspec stress` with the seeds 1, 2 and 7, in code the earlier changes did not touch.
  The zip reader let `zipfile`'s own errors escape on an archive it cannot read, such as `NotImplementedError` for an entry that needs zip version 9.8; `paperspec.package.read_zip` now refuses every such archive with `package.zip.entry` ([spec/12 §12.2](spec/12-package.md#122-paths)), and `paperspec check` crafts one of each kind.
  A checkpoint note was split into lines with `str.splitlines`, which also breaks at U+000B and other separators and so accepted a note whose last newline was replaced by one; [spec/23 §23.2](spec/23-log.md#232-checkpoints) now says every line ends in U+000A alone, `paperspec.log.parse_checkpoint` splits on it, and `paperspec check` tries ten other separators.
  The third was the stress harness's own: a truncation could cut a file at its full length, leaving the package unchanged and valid, so it now always shortens the file.
  So was a fourth, with 1000 rounds: the random walk of corrections could delete the body section, leaving nothing to correct and too few corrections accepted, so a delete now leaves at least eight nodes.