package._canonical_bytes, package._zip_entries, anchors._follow and traces._anchors are gone; paperdesk and paperforms use the public names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YCVvg4F3wpYQjq1NQUSmxN
31 KiB
Changelog
A change to any computed value is a breaking change and needs a new major version of the format it touches; see spec/00-introduction.md.
0.1.0
The first release.
- The specification, in chapters 0 to 23, covering every format: 32 formats, of which 19 are
candidateand 13provisional. - 43 JSON Schemas, using a keyword subset that independent validators read alike.
- Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search.
- The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements.
- Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3.
- Example packages (a converted chemistry paper with corrections and receipts, its checks-only export, an open-access paper, a Typst draft and a tiny document, also as a zip), a record of every other format, and 33 broken packages, each breaking one named rule.
- Work keys minted from a Scholar-style citation key and a hash of the work's identity, with document keys for versions;
tag:URIs for layers, annotations and documents. - Spot-check seeds as VRF proofs (RFC 9381), RFC 9162 inclusion and consistency proofs, a history root over every document's corrections, and a provisional transparency log with signed checkpoints.
- Signed QA reports, a conformance checklist covered by a receipt, and a
descriptionannotation for the figure descriptions an accessible export needs. - ML-DSA-65 (FIPS 204) as a signature scheme, with a reference implementation checked against NIST's vectors;
cosereserved;did:keyidentifiers for roster keys; the hash algorithm named in every document; model provenance by weights, prompt and parameters; witnesses and a static tile layout for the log; and the character data pinned to Unicode 16.0. - Guides for newcomers: an overview, the concepts, a walkthrough of the example paper, runnable examples, the rationale, a FAQ and a glossary.
- Tools that build and check everything, validate packages and zipped packages and run proposed corrections through the gate, load packages into SQLite or a bundle, and stress the reference implementation with random inputs.
- A pixi workspace pinned to Python 3.14, with tasks for checking, building, linting, validating and loading.
0.1.1
Packaging, commands and lazy imports only; the formats stay at 0.1.0, and no computed value, vector, fixture, schema or SQL file changed.
- The oracle moves from
tools/oracle/tosrc/paperspec/and installs as the Python packagepaperspec(pyproject.toml, setuptools), withschemas/,sql/andvectors/as the data subpackagespaperspec.schemas,paperspec.sqlandpaperspec.vectors, so other repositories import the reference implementation instead of reimplementing it. - The package version comes from git tags through setuptools-scm.
- The base package has no dependencies and requires Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import under any other Unicode version; the extras
paperspec[schema](jsonschema,referencing),paperspec[crypto](cryptography) andpaperspec[tools](both) coverschemaand the HKDF and Ed25519 functions ofcrypto, which now import their packages on first use. paperspec.resources.pathfinds the schemas, SQL and vectors in an installed wheel and in an editable install alike.- The programs in
tools/become subcommands of one command,paperspec, insrc/paperspec/cli/:paperspec validate,load,check,build,stressandsite, which the pixi tasks and the workflows run, and whichpython -m paperspecalso runs. The scriptstools/*.pyare gone, sopython3 tools/check.pyand the like no longer work;tools/keeps the page that documents the commands. paperspec --helpsays which commands work anywhere and which need a checkout, each command's--helpgives advice and examples, a misspelled command gets a suggestion, andpaperspec --versionprints the package version and the Unicode version it pins.paperspec loadrefuses--built-fromand--storewithout--bundle, where it used to ignore them.src/is Apache-2.0, astools/was, and the package metadata statesApache-2.0 AND CC0-1.0for the code and the data it carries.- The pixi workspace installs the package editable, and a
devenvironment builds the sdist and the wheel withpixi run -e dev dist. - Pushing a version tag publishes the package to the Gitea package registry through
.gitea/workflows/publish.yml(deploy/); a release tag versions the package, and CONTRIBUTING.md says how that relates to the format version.
Unreleased
Two changes asked for by the first producer that compares a converted reading with a second witness (paperforms with a JATS file). Both are minor changes in the sense of spec/00: no valid file becomes invalid, and no hash, ID or key changes; the next release numbers them.
- Alignment has a substitution step (spec/05 §5.6, step 3): in each stretch between pairs, an unpaired sentence of the first reading is paired with the first unpaired sentence of the second whose similarity (2L / (|a| + |b|) over code points of the lossy projections, L their longest common subsequence) is at least 0.8, both of prose length 20 or more.
A sentence one reading misread is now one
number,equationorwordsuspect rather than adroppedand anaddedone, so a producer re-aligning the same two readings can get different suspects than before; the pairs and suspects of readings with no such sentence are unchanged, as the first alignment vector is.paperspec.aligngainslcs_length(bit-parallel, Hyyrö 2004) andsimilarity, andalignment.jsonla second case,substitutions: a misread number, a misread word, a sentence too short to be paired this way and two sentences too unlike to be one. - A suspect may name the witness its second reading came from (
witness, a witness ID; spec/05 §5.7,suspect.schema.json), and a package whose suspect names a witness it does not have breaksdoc.suspect.witness.fixtures/invalid/suspect-witness/exercises the rule (see the last section).
Four more, asked for by editors who correct and discuss papers rather than readings. They are minor changes too: new operations, enumeration values and optional properties, with no valid file made invalid, no value of an existing file computed differently, and no existing vector case or package changed.
- Annotation categories (spec/18 §18.4,
annotation.schema.json, thecategoryterm of the JSON-LD context): acomment,caveatorquestionbody may carry acategoryfrom the fixed listcomment,context,critique,question,correction,feedbackandresource, which a producer that supports categories SHOULD set on every such body. The store'sannotationstable gains a nullablecategorycolumn with its CHECK (sql/store.sql). The example layer's comment, question and reply gain categories, which changes their record IDs, and it gains a caveat; no other fixture changes. emend, a correction of the authors' own typo, spelling, grammar, punctuation or wording, where the reading matches the page and the page is wrong (spec/11 §11.6). The emended text keeps what was printed in a new optional run property,printed(spec/03 §3.5,runs.schema.json), so a leaf as printed is recovered from the package alone (3.6,paperspec.hashing.printedandas_printed), and runs withprintedare never joined. The strict and lossy projections are of the runs as printed, so an emend changes the leaf'stext_hash,own_hashandnode_hashbut not itsmatch_hashoralign_key, and alignment, corroboration and suspects never report an emendation (spec/05 §5.1). The gate refuses an emend in math, across runs, or that changes a digit or edits beside one, and refuses areplace, textinsertorsplitinside an emended run (correction.emend).relinkmakes a run a citation of other references, a text run a cite run, or a cite run text, without changing its text, andresolvesets a reference's resolution inreferences.jsonl, keeping the one before infrom(spec/11 §11.7); a package whose last resolve of a reference is not its resolution breaksdoc.correction.resolution. Like an emend, neither lowers a leaf's level, and a receipt keeps counting across all three (spec/08 §8.1–8.2,receipts.carried_hashes).- The rules
correction.emend,correction.relink,correction.resolveanddoc.correction.resolutionare registered, thecorrectionstable's CHECK takes the three operations (sql/doc.sql),corrections.Statecarries the reference records, andpaperspec stresswalks the three operations with their invariants.corrections.jsonlgains thirteen cases andprojections.jsonlandprojection-groups.jsonltwo each; no existing case changes.
Four more, asked for by papermath, which defines the math form scienting-math/1, and by paperclaim, the claim store whose claims and symbols layers point at.
The first two are minor changes: an optional property, a derived file and its schema, with no hash, ID or key of an existing file changed.
The last two make some files that were valid invalid, which spec/00 calls breaking; they are made now because no release of scienting-doc/1 or scienting-layer/1 has been tagged, and a reference worth changing is changed before one is (CONTRIBUTING.md, item 4).
- Math forms (spec/03 §3.7, spec/09 §9.4,
node.schema.json,runs.schema.json,mathFormincommon.schema.json): an equation's content and amathrun may carryform, whose one value,scienting-math/1, says that the source is canonical TeX; the notation is thentex. A consumer may rely on the source being canonical and idempotent, carrying no label, and on everything else being derived from it.formenters no content value (spec/05 §5.2): own content leaves it out of math runs (paperspec.hashing), an equation's own content never held more than its notation and source, and no projection reads it, so addingformto a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does. Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first.hashes.jsonlgains a case,math-form, whose valuespaperspec checkalso computes withoutform; the ruledoc.math.formis registered for validators that implement the form, which the reference validator now does in part (see the last section). derived/math.jsonl(spec/12 §12.7, spec/09 §9.4,math.schema.json): one line per formula,{node, run, tool, form, tree, symbols, complete, issues, label?}, papermath's reading without the canonical TeX, which is the source;runis null for an equation, as inderived/mathml.jsonl, andlabelis omitted when the source has no\tag. A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 49 kinds and six codes of version 1. The validator now checks derived files: each line against its schema, that it names a formula of the package (doc.derived.node), that a math line's formula is TeX and itscompleteandsymbolsagree with its tree (doc.derived.math), and that a checks-only package carries none (doc.policy.withheld), since a tree or MathML reads the withheld text. The main example gainsderived/math.jsonl, papermath 0.1.0's reading of its equation and of two math runs, copied by hand because the oracle does not implement the form;paperspec checkvalidates fourteen variants of it, and no fixture's hashes change. There are now 43 schemas, which the 0.1.0 entry above, held to the repository bypaperspec check, states too.- Claim references (spec/18 §18.4, spec/02 §2.4, spec/20,
claimRefandclaimPinincommon.schema.json): thesourceof aclaim-linkorsymbolbody, aclaim:identifier insame_as, and a project's claim step are paperclaim references,claim:<tag>orclaim:<host>/<tag>with the tag eight Crockford base32 capitals, and apinis the reference,@and six lowercase hex digits of the claim's statement hash, as paperclaim writes them; a project names its claim namespace asclaims.paperclaim. A reference by slug, such asamber/k-bond, which paperclaim may rename, or under any other prefix, was valid before and is not now. The fixtures' slug reference andillustrative-pinbecomeclaim:272XTX7Candclaim:272XTX7C@02f47b, the claimamber/k-bondof paperclaim's example store, which changes the claim link's and the symbol's record IDs, the layer receipt andproject.json. The main example's two targeted receipts take the scopetargeted:paperclaim, which changes their IDs and signatures, Ben's spot-check of the second (its seed and its sample of ten, still without errors), the QA report, the inclusion proof, the transparency log's checkpoint and tiles, and the signature, spot-check, VRF and checkpoint vectors built from them; no node's level or the report's counts change. - Symbol keys (spec/18 §18.4,
annotation.schema.json, thekeyterm of the JSON-LD context,sct:symbolKey): asymbolbody names its symbol bykey, itsscienting-math/1key, besides the span its target selects, so a symbol is found across papers and matched to its claim by key; asymbolbody withoutkeywas valid before and is not now. The example layer gains a proposed symbol annotation on$K_r$, with the keyK_{r}. - Three failures of
paperspec stresswith the seeds 1, 2 and 7, in code the earlier changes did not touch. The zip reader letzipfile's own errors escape on an archive it cannot read, such asNotImplementedErrorfor an entry that needs zip version 9.8;paperspec.package.read_zipnow refuses every such archive withpackage.zip.entry(spec/12 §12.2), andpaperspec checkcrafts one of each kind. A checkpoint note was split into lines withstr.splitlines, which also breaks at U+000B and other separators and so accepted a note whose last newline was replaced by one; spec/23 §23.2 now says every line ends in U+000A alone,paperspec.log.parse_checkpointsplits on it, andpaperspec checktries ten other separators. The third was the stress harness's own: a truncation could cut a file at its full length, leaving the package unchanged and valid, so it now always shortens the file. So was a fourth, with 1000 rounds: the random walk of corrections could delete the body section, leaving nothing to correct and too few corrections accepted, so a delete now leaves at least eight nodes.
One more, asked for by paperdesk, which lets a reader follow a citation to the passage it rests on. It is a minor change in the sense of spec/00: a new body kind, optional properties and two rules, with no valid file made invalid, no computed value changed and no existing vector case or package changed.
- Traced citations (spec/18 §18.5,
annotation.schema.json, thenoteandproposerterms of the JSON-LD context): atracebody follows one citation of a sentence of a citing paper to a passage of the cited paper. The target is a leaf or span anchor on the citing sentence, which must cite the body'sreference; the body names the cited document insource, the passage inselector(anAnchorSelectorwith its pin and span, then aTextQuoteSelectorwhen the quote is plain), and optionally anote, averdictand aproposer({model, run, score?, rank?}, required when the creator isSoftware). Acrosslinkstays what it was, a symmetric correspondence of two spans with no room for a note, the reference meant or a proposer's provenance. The state of a trace is the annotation'sstatus, and a trace is disputed while anassessingreply to it stands, which is computed from the replies. Thebody_kindCHECK of the store'sannotationstable takestrace(sql/store.sql), andpaperspec checkholds it to the schema's list of body kinds. paperspec.tracesand the ruleslayer.trace.sourceandlayer.trace.target(spec/22 §22.4).resolve(annotation, citing_state, cited_state, same_as)resolves both anchors by spec/06 §6.5 and returnscurrent,changed,stale(with the side),unlinked(arelinkremoved the citation) ormisresolved(aresolvepoints the reference at another work), withopen, the passage's current address, andcites, the citations inside it, so a chain of citations is followed one trace at a time.problemsreports the two rules, anddisputesanddisputedread the replies.fixtures/records/traces/is a second layer, on the Typst draft, with a model's confirmed trace to the main example's abstract and its receipt, a rejected candidate, a person's trace to Table 1 and a reply that disputes the first, andvectors/traces.jsonlhas 29 cases that correct either paper and show every outcome and both rules.paperspec checkruns both layers through the record checks and the store. The example project holds the new layer, which changesproject.json; no other fixture changes.
One more, asked for by paperchem and paperbio, which tie molecules and sequences to the words of a paper. It is a minor change in the sense of spec/00: two new body kinds, optional properties and seven rules, with no valid file made invalid and no computed value changed; the example layers, the traces and every receipt are as they were.
- Chemical structures (spec/18 §18.6,
annotation.schema.json): astructurebody ties a SMILES or SMARTS to a span, with arole(names,depicts,parametrizesoruses), anidentitythat finds the same structure however it was written (the InChIKey of a SMILES, elsesmiles:orsmarts:and twelve hex digits of the SHA-256 of the value), the paper's ownwrittennotation, alabel, outsideaccessionsandterms: the bonds, angles, torsions and impropers a paper gives constants for. A group, a fragment with open positions (R-OHis[*:1]-[OX2H]) and a pattern of atom types are SMARTS, and SMILES text a toolkit would refuse is kept as one rather than dropped. A term cites atoms by number, from 0 in the order the value writes them and with no number for a hydrogen that is not written, so the numbers are a rule of the text and a producer needs no toolkit to check them. - Sequences (spec/18 §18.7,
annotation.schema.json): asequencebody ties protein, DNA or RNA residues to a span, with analphabet, arole(names,depicts,usesorreports), anidentity(the alphabet and sixteen hex digits of the SHA-256 of the residues without gaps), astartso a fragment is numbered as the whole molecule is, alabel, anorganism,accessions,regionsandvariants. The schema holds each alphabet's letters; a stop (*) belongs to protein only. paperspec.structuresandpaperspec.sequencesand the ruleslayer.structure.value,layer.structure.identity,layer.structure.term,layer.sequence.value,layer.sequence.identity,layer.sequence.regionandlayer.sequence.variant(spec/22 §22.4).structures.scanreads the atoms and bonds of a SMILES or SMARTS with the standard library, so a term's atoms are checked to exist and to be joined as the kind needs (a chain for a bond, angle or torsion, a star for an improper); it gave RDKit's atom count and bonds for a corpus of fifty-five SMILES and SMARTS and for three thousand random SMILES, and refused the malformed texts RDKit refuses. Valence, aromaticity and the InChIKey need a toolkit, which a producer SHOULD use for them.body_kindof the store'sannotationstable takesstructureandsequence(sql/store.sql). A store made before this change has the old CHECK, so a consumer migrates it by rebuilding the table; the table's columns do not change.- The JSON-LD context gains nine terms:
identity,written,alphabet,organism,start, andaccessions,terms(assct:structureTerms),regionsandvariantsas JSON literals. fixtures/records/chemistry/, on the main example, andfixtures/records/biology/, on the draft, withvectors/structures.jsonl(64 cases) andvectors/sequences.jsonl(31): scans, identities, annotations that break no rule and one that breaks each. The example project holds the two layers, which changesproject.json. The draft gains a last sentence, which the biology layer is about, so the draft package, its draft IDs and the one traces vector that anchors its paragraph change; the traces layer's anchors, its receipt and every other vector are as they were.- The prose check's UK spelling pattern no longer takes
organismfor a form oforganise.
One more, asked for by paperdata, which finds data by its kind and by the molecules and sequences it is about. It is a minor change in the sense of spec/00: one new body kind, optional properties and four rules, with no valid file made invalid and no computed value changed; the other layers, the traces and every receipt are as they were.
- Data (spec/18 §18.8,
annotation.schema.json): adatabody says what kind of data a span presents, with adata_typefrom a fixed list of fifteen, arole(reportsoruses), thedatasetthat holds the values, thequantitiesits fields measure (a field, a name and the dataset's unit), thesubjectsit is about (the identities ofstructureandsequencebodies), amethod, alabeland outsideaccessions, so that a term of an ontology can say more than the list does. A dataset'soriginalready says whether its values were reported, digitized, computed or measured, so the body does not. paperspec.dataand the ruleslayer.data.type,layer.data.quantity,layer.data.subjectandlayer.data.dataset(spec/22 §22.4).data.problemscompares a quantity's unit with its dataset's as a string and parses no unit; a producer with a unit library SHOULD also check that each unit parses and has the dimension its quantity needs.body_kindof the store'sannotationstable takesdata(sql/store.sql). A store made before this change has the old CHECK, so a consumer migrates it by rebuilding the table; the table's columns do not change.- The JSON-LD context gains five terms:
data_type,dataset,method, andquantitiesandsubjectsas JSON literals. fixtures/records/data/, on the main example, withvectors/data.jsonl(35 cases): a valid case for each kind of data, annotations that break no rule and one or more that break each. The layer ties the caption of Table 1 to its dataset as force-field parameters for ethane, water and the CT–CT pattern, the caption of Figure 1a to the dataset read from it as an energy profile of ethane, and the sentence that introduces Table 1 to the same dataset as a model's proposal. The example project holds the layer, which changesproject.json; no other fixture or vector changes.
Tooling and code quality only; no computed value, vector, fixture, schema or SQL file changes.
- The Python code meets Scienting's standard:
ruff format,ruff checkwith the full rule set, andty checkpass onsrc/paperspec, every function is annotated and documented, andtests/holds a pytest suite, driven by the vectors and fixtures, with branch coverage that fails under 90%.paperspec check(with the byte-for-byte rebuild) andpaperspec stressgive the same results as before. - Fixed:
structures.scanraised an unrelatedValueError("unsupported format character") for a malformed ring closure such asC%1, because its message was an invalid format string; the message is now "% needs two digits at N". - Fixed:
paperspec site --out /passed the check that the output folder does not contain the repository and went on to delete the folder it was given; it is now refused as a usage error. A site built into a folder inside the checkout also no longer copies that folder's subfolders into itself. - Fixed:
receipts.verify_signature, documented never to raise, raisedKeyErrorfor a record without aformat; it now returns false. - Fixed:
traces.DOC_TAGmatched a document tag followed by a line feed, because it ended in$; it now ends at the end of the string. - Fixed:
paperspec loadandpaperspec checkleft their SQLite connections open; they now close them. - The pixi tasks follow the standard (
fmt,fmt-check,lintfor ruff,typecheck,test,cov,hooks); the Markdown lint islint-md,paperspec checkis alsoverify, andcheckruns them all. The build environment isrelease(wasdev), anddocsbuilds the package documentation. - The package documentation, for the Python package and its command, is built with Zensical from
docs/package/and published at https://docs.scient.ing/paperspec/; the format documentation stays indocs/and on https://paperspec.scient.ing. - The repository lives in the Gitea organization
papers: the repository and registry addresses change frominfratopapers. - The publish workflow also creates the Gitea release, with the files attached and the notes taken from this changelog; CI runs Scienting's checks (
.gitea/workflows/ci.yml) onmain,devand pull requests. - Python files open with an
SPDX-License-Identifier: Apache-2.0header, which ruff and a test check. - The checker and the site builder skip
work/, caches and build output.
One more, the work the other paper tools planned for paperspec (paperdesk's plan, item M5.4, and paperforms's, §4.2): a command that applies corrections, two checks the validator had left to producers, and public names for what the tools had been importing as private ones. It is a minor change in the sense of spec/00: new rules, functions and a command, with no fixture package made invalid and no computed value changed.
paperspec correctandpaperspec.revisionapply proposed corrections to a package, a folder or a zip, through the gate (spec/11), and write the package they make, in place or to--out, only once it validates. A changed leaf becomes F1 when a crop of it is the evidence and F0 otherwise (spec/08 §8.1), and anemend,relinkorresolvekeeps its level. A retired node's locations move to its successor, suspects at removed nodes and those named by--answeredare dropped, derived lines and datasets that read the corrected text are left out and reported, and provenance gains areviewstage. With no corrections it writes every example package and every valid package of paperforms's bench back byte for byte (test_revision.py,test_cli_correct.py).doc.math.formis checked (spec/03 §3.7). The validator refuses a source in the form that holds a label or style command the form never holds, such as\tag,\label,\nonumber,\displaystyleor\dfrac. When papermath is installed, or a caller passesforms=, it also refuses a source that is not the form's own spelling or that the form cannot read (test_math_form.py).- The validator checks a package's datasets (spec/15,
paperspec.datasets): the descriptor against its schema, the folder named by the ID, a unique primary key, each value anchor resolving as current to the text the CSV holds, and every numeric value of a reported resource anchored. These are the rulesdoc.dataset.schema,.id,.file,.key,.anchor,.valueand.unanchored(test_datasets.py). fixtures/invalid/gainssuspect-witnessandmath-form-label, so there are 33 broken packages, which the 0.1.0 entry above states too (paperspec check).- Canonical JSON reads numbers as doubles, so
1e20is read back as it was written (test_jcs.py). - The registries and the JSON-LD context ship in the wheel as
paperspec.registryandpaperspec.context, withpaperspec.resources.registryandpaperspec.fingerprint.published_rulesfor the download-stamp rules (test_resources.py,test_fingerprint.py). - Public names, with the old private names kept as aliases:
package.parse_canonical,parse_json,parse_jsonl,check_zip_entriesandread_zip_files;tree.flattenandestablish_with_side, which keeps a producer's private keys;hashing.strict_runs; andcorrections.OPSandEMEND_KINDS(test_public_api.py). Alsoanchors.followandresolve_address,claims.parse_ref,parse_pin,format_refandformat_pin,layers.annotation_id,anchors_ofand the lists of body kinds, categories, verdicts and statuses, anddb.load_layer,read_state,deleteandupgrade(test_anchors.py,test_claims.py,test_layers.py,test_db.py). text.collapse_ws,trim,remove_wsanddigitsandhashing.lossy_from_strictare rewritten for speed, so a correction goes through the gate about three times faster, and are tested equal to the old code on every code point (test_text.py,test_hashing.py).- The private aliases kept for the siblings are gone, now that paperdesk and paperforms use the public names:
package._canonical_bytes,package._zip_entries,anchors._followandtraces._anchors(tests/test_public_api.py::test_parse_canonical_reads_doubles_and_refuses_what_is_not_canonical,::test_check_zip_entries_returns_the_folder_without_reading,tests/test_anchors.py::TestFollow::test_follow_has_no_alias_under_its_old_private_name).