paperspec/CHANGELOG.md

Changelog

A change to any computed value is a breaking change and needs a new major version of the format it touches; see spec/00-introduction.md.

0.1.0

The first release.

• The specification, in chapters 0 to 23, covering every format: 32 formats, of which 19 are candidate and 13 provisional.
• 43 JSON Schemas, using a keyword subset that independent validators read alike.
• Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search.
• The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements.
• Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3.
• Example packages (a converted chemistry paper with corrections and receipts, its checks-only export, an open-access paper, a Typst draft and a tiny document, also as a zip), a record of every other format, and 31 broken packages, each breaking one named rule.
• Work keys minted from a Scholar-style citation key and a hash of the work's identity, with document keys for versions; tag: URIs for layers, annotations and documents.
• Spot-check seeds as VRF proofs (RFC 9381), RFC 9162 inclusion and consistency proofs, a history root over every document's corrections, and a provisional transparency log with signed checkpoints.
• Signed QA reports, a conformance checklist covered by a receipt, and a description annotation for the figure descriptions an accessible export needs.
• ML-DSA-65 (FIPS 204) as a signature scheme, with a reference implementation checked against NIST's vectors; cose reserved; did:key identifiers for roster keys; the hash algorithm named in every document; model provenance by weights, prompt and parameters; witnesses and a static tile layout for the log; and the character data pinned to Unicode 16.0.
• Guides for newcomers: an overview, the concepts, a walkthrough of the example paper, runnable examples, the rationale, a FAQ and a glossary.
• Tools that build and check everything, validate packages and zipped packages and run proposed corrections through the gate, load packages into SQLite or a bundle, and stress the reference implementation with random inputs.
• A pixi workspace pinned to Python 3.14, with tasks for checking, building, linting, validating and loading.

0.1.1

Packaging, commands and lazy imports only; the formats stay at 0.1.0, and no computed value, vector, fixture, schema or SQL file changed.

• The oracle moves from tools/oracle/ to src/paperspec/ and installs as the Python package paperspec (pyproject.toml, setuptools), with schemas/, sql/ and vectors/ as the data subpackages paperspec.schemas, paperspec.sql and paperspec.vectors, so other repositories import the reference implementation instead of reimplementing it.
• The package version comes from git tags through setuptools-scm.
• The base package has no dependencies and requires Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import under any other Unicode version; the extras paperspec[schema] (jsonschema, referencing), paperspec[crypto] (cryptography) and paperspec[tools] (both) cover schema and the HKDF and Ed25519 functions of crypto, which now import their packages on first use.
• paperspec.resources.path finds the schemas, SQL and vectors in an installed wheel and in an editable install alike.
• The programs in tools/ become subcommands of one command, paperspec, in src/paperspec/cli/: paperspec validate, load, check, build, stress and site, which the pixi tasks and the workflows run, and which python -m paperspec also runs. The scripts tools/*.py are gone, so python3 tools/check.py and the like no longer work; tools/ keeps the page that documents the commands.
• paperspec --help says which commands work anywhere and which need a checkout, each command's --help gives advice and examples, a misspelled command gets a suggestion, and paperspec --version prints the package version and the Unicode version it pins.
• paperspec load refuses --built-from and --store without --bundle, where it used to ignore them.
• src/ is Apache-2.0, as tools/ was, and the package metadata states Apache-2.0 AND CC0-1.0 for the code and the data it carries.
• The pixi workspace installs the package editable, and a dev environment builds the sdist and the wheel with pixi run -e dev dist.
• Pushing a version tag publishes the package to the Gitea package registry through .gitea/workflows/publish.yml (deploy/); a release tag versions the package, and CONTRIBUTING.md says how that relates to the format version.

Unreleased

Two changes asked for by the first producer that compares a converted reading with a second witness (paperform with a JATS file). Both are minor changes in the sense of spec/00: no valid file becomes invalid, and no hash, ID or key changes; the next release numbers them.

• Alignment has a substitution step (spec/05 §5.6, step 3): in each stretch between pairs, an unpaired sentence of the first reading is paired with the first unpaired sentence of the second whose similarity (2L / (|a| + |b|) over code points of the lossy projections, L their longest common subsequence) is at least 0.8, both of prose length 20 or more. A sentence one reading misread is now one number, equation or word suspect rather than a dropped and an added one, so a producer re-aligning the same two readings can get different suspects than before; the pairs and suspects of readings with no such sentence are unchanged, as the first alignment vector is. paperspec.align gains lcs_length (bit-parallel, Hyyrö 2004) and similarity, and alignment.jsonl a second case, substitutions: a misread number, a misread word, a sentence too short to be paired this way and two sentences too unlike to be one.
• A suspect may name the witness its second reading came from (witness, a witness ID; spec/05 §5.7, suspect.schema.json), and a package whose suspect names a witness it does not have breaks doc.suspect.witness. No broken package exercises the rule yet: adding one changes the count of broken packages that the 0.1.0 entry above states, so it waits for the release.

Four more, asked for by editors who correct and discuss papers rather than readings. They are minor changes too: new operations, enumeration values and optional properties, with no valid file made invalid, no value of an existing file computed differently, and no existing vector case or package changed.

• Annotation categories (spec/18 §18.4, annotation.schema.json, the category term of the JSON-LD context): a comment, caveat or question body may carry a category from the fixed list comment, context, critique, question, correction, feedback and resource, which a producer that supports categories SHOULD set on every such body. The store's annotations table gains a nullable category column with its CHECK (sql/store.sql). The example layer's comment, question and reply gain categories, which changes their record IDs, and it gains a caveat; no other fixture changes.
• emend, a correction of the authors' own typo, spelling, grammar, punctuation or wording, where the reading matches the page and the page is wrong (spec/11 §11.6). The emended text keeps what was printed in a new optional run property, printed (spec/03 §3.5, runs.schema.json), so a leaf as printed is recovered from the package alone (3.6, paperspec.hashing.printed and as_printed), and runs with printed are never joined. The strict and lossy projections are of the runs as printed, so an emend changes the leaf's text_hash, own_hash and node_hash but not its match_hash or align_key, and alignment, corroboration and suspects never report an emendation (spec/05 §5.1). The gate refuses an emend in math, across runs, or that changes a digit or edits beside one, and refuses a replace, text insert or split inside an emended run (correction.emend).
• relink makes a run a citation of other references, a text run a cite run, or a cite run text, without changing its text, and resolve sets a reference's resolution in references.jsonl, keeping the one before in from (spec/11 §11.7); a package whose last resolve of a reference is not its resolution breaks doc.correction.resolution. Like an emend, neither lowers a leaf's level, and a receipt keeps counting across all three (spec/08 §8.1–8.2, receipts.carried_hashes).
• The rules correction.emend, correction.relink, correction.resolve and doc.correction.resolution are registered, the corrections table's CHECK takes the three operations (sql/doc.sql), corrections.State carries the reference records, and paperspec stress walks the three operations with their invariants. corrections.jsonl gains thirteen cases and projections.jsonl and projection-groups.jsonl two each; no existing case changes.

Four more, asked for by papermath, which defines the math form scienting-math/1, and by paperclaim, the claim store whose claims and symbols layers point at. The first two are minor changes: an optional property, a derived file and its schema, with no hash, ID or key of an existing file changed. The last two make some files that were valid invalid, which spec/00 calls breaking; they are made now because no release of scienting-doc/1 or scienting-layer/1 has been tagged, and a reference worth changing is changed before one is (CONTRIBUTING.md, item 4).

• Math forms (spec/03 §3.7, spec/09 §9.4, node.schema.json, runs.schema.json, mathForm in common.schema.json): an equation's content and a math run may carry form, whose one value, scienting-math/1, says that the source is canonical TeX; the notation is then tex. A consumer may rely on the source being canonical and idempotent, carrying no label, and on everything else being derived from it. form enters no content value (spec/05 §5.2): own content leaves it out of math runs (paperspec.hashing), an equation's own content never held more than its notation and source, and no projection reads it, so adding form to a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does. Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first. hashes.jsonl gains a case, math-form, whose values paperspec check also computes without form; the rule doc.math.form is registered for validators that implement the form, which the reference validator does not.
• derived/math.jsonl (spec/12 §12.7, spec/09 §9.4, math.schema.json): one line per formula, {node, run, tool, form, tree, symbols, complete, issues, label?}, papermath's reading without the canonical TeX, which is the source; run is null for an equation, as in derived/mathml.jsonl, and label is omitted when the source has no \tag. A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 49 kinds and six codes of version 1. The validator now checks derived files: each line against its schema, that it names a formula of the package (doc.derived.node), that a math line's formula is TeX and its complete and symbols agree with its tree (doc.derived.math), and that a checks-only package carries none (doc.policy.withheld), since a tree or MathML reads the withheld text. The main example gains derived/math.jsonl, papermath 0.1.0's reading of its equation and of two math runs, copied by hand because the oracle does not implement the form; paperspec check validates fourteen variants of it, and no fixture's hashes change. There are now 43 schemas, which the 0.1.0 entry above, held to the repository by paperspec check, states too.
• Claim references (spec/18 §18.4, spec/02 §2.4, spec/20, claimRef and claimPin in common.schema.json): the source of a claim-link or symbol body, a claim: identifier in same_as, and a project's claim step are paperclaim references, claim:<tag> or claim:<host>/<tag> with the tag eight Crockford base32 capitals, and a pin is the reference, @ and six lowercase hex digits of the claim's statement hash, as paperclaim writes them; a project names its claim namespace as claims.paperclaim. A reference by slug, such as amber/k-bond, which paperclaim may rename, or under any other prefix, was valid before and is not now. The fixtures' slug reference and illustrative-pin become claim:272XTX7C and claim:272XTX7C@02f47b, the claim amber/k-bond of paperclaim's example store, which changes the claim link's and the symbol's record IDs, the layer receipt and project.json. The main example's two targeted receipts take the scope targeted:paperclaim, which changes their IDs and signatures, Ben's spot-check of the second (its seed and its sample of ten, still without errors), the QA report, the inclusion proof, the transparency log's checkpoint and tiles, and the signature, spot-check, VRF and checkpoint vectors built from them; no node's level or the report's counts change.
• Symbol keys (spec/18 §18.4, annotation.schema.json, the key term of the JSON-LD context, sct:symbolKey): a symbol body names its symbol by key, its scienting-math/1 key, besides the span its target selects, so a symbol is found across papers and matched to its claim by key; a symbol body without key was valid before and is not now. The example layer gains a proposed symbol annotation on $K_r$, with the key K_{r}.
• Three failures of paperspec stress with the seeds 1, 2 and 7, in code the earlier changes did not touch. The zip reader let zipfile's own errors escape on an archive it cannot read, such as NotImplementedError for an entry that needs zip version 9.8; paperspec.package.read_zip now refuses every such archive with package.zip.entry (spec/12 §12.2), and paperspec check crafts one of each kind. A checkpoint note was split into lines with str.splitlines, which also breaks at U+000B and other separators and so accepted a note whose last newline was replaced by one; spec/23 §23.2 now says every line ends in U+000A alone, paperspec.log.parse_checkpoint splits on it, and paperspec check tries ten other separators. The third was the stress harness's own: a truncation could cut a file at its full length, leaving the package unchanged and valid, so it now always shortens the file. So was a fourth, with 1000 rounds: the random walk of corrections could delete the body section, leaving nothing to correct and too few corrections accepted, so a delete now leaves at least eight nodes.