diff --git a/CHANGELOG.md b/CHANGELOG.md index 593621b..d045fd2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ A change to any computed value is a breaking change and needs a new major versio The first release. - The specification, in chapters 0 to 23, covering every format: 32 formats, of which 19 are `candidate` and 13 `provisional`. -- 42 JSON Schemas, using a keyword subset that independent validators read alike. +- 43 JSON Schemas, using a keyword subset that independent validators read alike. - Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search. - The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements. - Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3. @@ -61,3 +61,28 @@ They are minor changes too: new operations, enumeration values and optional prop Like an emend, neither lowers a leaf's level, and a receipt keeps counting across all three ([spec/08 §8.1–8.2](spec/08-fidelity.md#82-levels), `receipts.carried_hashes`). - The rules `correction.emend`, `correction.relink`, `correction.resolve` and `doc.correction.resolution` are registered, the `corrections` table's CHECK takes the three operations ([sql/doc.sql](sql/doc.sql)), `corrections.State` carries the reference records, and `paperspec stress` walks the three operations with their invariants. `corrections.jsonl` gains thirteen cases and `projections.jsonl` and `projection-groups.jsonl` two each; no existing case changes. + +Four more, asked for by papermath, which defines the math form `scienting-math/1`, and by lode, the claim store whose claims and symbols layers point at. +The first two are minor changes: an optional property, a derived file and its schema, with no hash, ID or key of an existing file changed. +The last two make some files that were valid invalid, which [spec/00](spec/00-introduction.md) calls breaking; they are made now because no release of `scienting-doc/1` or `scienting-layer/1` has been tagged, and a reference worth changing is changed before one is ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), item 4). + +- Math forms ([spec/03 §3.7](spec/03-nodes.md#37-math-forms), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `node.schema.json`, `runs.schema.json`, `mathForm` in `common.schema.json`): an equation's content and a `math` run may carry `form`, whose one value, `scienting-math/1`, says that the source is canonical TeX; the notation is then `tex`. + A consumer may rely on the source being canonical and idempotent, carrying no label, and on everything else being derived from it. + `form` enters no content value ([spec/05 §5.2](spec/05-hashes.md#52-own-content)): own content leaves it out of math runs (`paperspec.hashing`), an equation's own content never held more than its notation and source, and no projection reads it, so adding `form` to a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does. + Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first. + `hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator does not. +- `derived/math.jsonl` ([spec/12 §12.7](spec/12-package.md#127-derived-files), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `math.schema.json`): one line per formula, `{node, run, tool, form, tree, symbols, complete, issues, label?}`, papermath's reading without the canonical TeX, which is the source; `run` is null for an equation, as in `derived/mathml.jsonl`, and `label` is omitted when the source has no `\tag`. + A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 47 kinds and six codes of version 1. + The validator now checks derived files: each line against its schema, that it names a formula of the package (`doc.derived.node`), that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`), and that a checks-only package carries none (`doc.policy.withheld`), since a tree or MathML reads the withheld text. + The main example gains `derived/math.jsonl`, papermath 0.1.0's reading of its equation and of two math runs, copied by hand because the oracle does not implement the form; `paperspec check` validates fourteen variants of it, and no fixture's hashes change. + There are now 43 schemas, which the 0.1.0 entry above, held to the repository by `paperspec check`, states too. +- Claim references ([spec/18 §18.4](spec/18-layers.md#184-bodies), [spec/02 §2.4](spec/02-model.md#24-work-and-document-keys), [spec/20](spec/20-projects.md), `claimRef` and `claimPin` in `common.schema.json`): the `source` of a `claim-link` or `symbol` body, a `lode:` identifier in `same_as`, and a project's claim step are lode references, `lode:` or `lode:/` with the tag eight Crockford base32 capitals, and a `pin` is the reference, `@` and six lowercase hex digits of the claim's statement hash, as lode writes them. + A slug such as `lode:amber/k-bond`, which lode may rename, was valid before and is not now. + The fixtures' `lode:amber/k-bond` and `illustrative-pin` become `lode:272XTX7C` and `lode:272XTX7C@02f47b`, the claim `amber/k-bond` of lode's example store, which changes the claim link's record ID, the layer receipt and `project.json`. +- Symbol keys ([spec/18 §18.4](spec/18-layers.md#184-bodies), `annotation.schema.json`, the `key` term of the JSON-LD context, `sct:symbolKey`): a `symbol` body names its symbol by `key`, its `scienting-math/1` key, besides the span its target selects, so a symbol is found across papers and matched to its claim by key; a `symbol` body without `key` was valid before and is not now. + The example layer gains a proposed symbol annotation on `$K_r$`, with the key `K_{r}`. +- Three failures of `paperspec stress` with the seeds 1, 2 and 7, in code the earlier changes did not touch. + The zip reader let `zipfile`'s own errors escape on an archive it cannot read, such as `NotImplementedError` for an entry that needs zip version 9.8; `paperspec.package.read_zip` now refuses every such archive with `package.zip.entry` ([spec/12 §12.2](spec/12-package.md#122-paths)), and `paperspec check` crafts one of each kind. + A checkpoint note was split into lines with `str.splitlines`, which also breaks at U+000B and other separators and so accepted a note whose last newline was replaced by one; [spec/23 §23.2](spec/23-log.md#232-checkpoints) now says every line ends in U+000A alone, `paperspec.log.parse_checkpoint` splits on it, and `paperspec check` tries ten other separators. + The third was the stress harness's own: a truncation could cut a file at its full length, leaving the package unchanged and valid, so it now always shortens the file. + So was a fourth, with 1000 rounds: the random walk of corrections could delete the body section, leaving nothing to correct and too few corrections accepted, so a delete now leaves at least eight nodes. diff --git a/context/README.md b/context/README.md index 268ec80..e3cc38c 100644 --- a/context/README.md +++ b/context/README.md @@ -33,7 +33,7 @@ The repository's name, paperspec, is a separate matter. ## An example from the fixtures -[`fixtures/records/layer/annotations.jsonl`](../fixtures/records/layer/annotations.jsonl) holds eleven annotations on the example paper `hale2026harmonic-damb`. +[`fixtures/records/layer/annotations.jsonl`](../fixtures/records/layer/annotations.jsonl) holds twelve annotations on the example paper `hale2026harmonic-damb`. The first is a `gloss`, an explanation for a novice, of the phrase "bond force constant". Shortened, it reads: @@ -57,10 +57,10 @@ This context defines `kind`, `status`, `audience`, `AnchorSelector` and `anchor` ## What it defines -The file defines 24 terms. -Twenty-three map to IRIs under `https://paperspec.scient.ing/vocab#`: `format`, `kind`, `status`, `receipt`, `layer`, `audience`, `visibility`, `purpose`, `project`, `docs`, `pin`, `role`, `symbol`, `verdict`, `reason`, `reference`, `for`, `category`, `generator`, `exercise`, `anchor`, `rights_use` and the class `AnchorSelector`. -Three are renamed: `for` is `sct:questionFor`, `generator` is `sct:generatedBy`, and `rights_use` is `sct:rightsUse`. -The twenty-fourth, `total`, maps to `totalItems` in the ActivityStreams namespace. +The file defines 25 terms. +Twenty-four map to IRIs under `https://paperspec.scient.ing/vocab#`: `format`, `kind`, `status`, `receipt`, `layer`, `audience`, `visibility`, `purpose`, `project`, `docs`, `pin`, `role`, `symbol`, `key`, `verdict`, `reason`, `reference`, `for`, `category`, `generator`, `exercise`, `anchor`, `rights_use` and the class `AnchorSelector`. +Four are renamed: `key`, the key of a `symbol` body's symbol, is `sct:symbolKey`, `for` is `sct:questionFor`, `generator` is `sct:generatedBy`, and `rights_use` is `sct:rightsUse`. +The twenty-fifth, `total`, maps to `totalItems` in the ActivityStreams namespace. Two terms carry a type rule. A processor reads `layer` as an IRI, and it always reads `docs` as a set, so a one-element list stays a list. diff --git a/context/scienting-1.jsonld b/context/scienting-1.jsonld index 169f1ca..766a4d8 100644 --- a/context/scienting-1.jsonld +++ b/context/scienting-1.jsonld @@ -21,6 +21,7 @@ "pin": "sct:pin", "role": "sct:role", "symbol": "sct:symbol", + "key": "sct:symbolKey", "verdict": "sct:verdict", "reason": "sct:reason", "reference": "sct:reference", diff --git a/docs/comparison.md b/docs/comparison.md index e103377..1216845 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -428,7 +428,7 @@ Keep using it as a reading and distribution format. *Status.* The specification is version 0.1.0. `registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)). -The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`. +The repository holds 24 chapters, 43 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`. Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; the code in `src/` Apache-2.0 ([README.md](../README.md#licenses)). *No adoption claim.* diff --git a/docs/concepts.md b/docs/concepts.md index 497e93a..dbf2a9a 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -434,7 +434,7 @@ hale2026harmonic-damb/ provenance.json tools, versions and where each stage ran assets/ images, by SHA-256 datasets/ datasets reported from the document - derived/ MathML and other renderings, rebuildable + derived/ MathML, math trees and other renderings, rebuildable ``` Only `MANIFEST.json`, `document.json`, `nodes.jsonl`, `witnesses.json` and `provenance.json` are required, and an optional file with no records is left out; the example package omits `suspects.jsonl`. diff --git a/docs/glossary.md b/docs/glossary.md index ed8ad26..dc0827a 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -100,6 +100,11 @@ Match hash Two readings agree only when their match hashes are equal and their complex-notation keys are equal. A complex-notation key is `notation:source` for a piece of math too complex for the formats to normalize, with the source's whitespace removed ([5.3](../spec/05-hashes.md#53-the-strict-projection)). ([5.1](../spec/05-hashes.md#51-the-five-values), [5.3](../spec/05-hashes.md#53-the-strict-projection)) +Math form +: A standard spelling of formulas, so that one formula is written alike in every paper and claim. +`scienting-math/1` is canonical TeX, with a tree read from it and a key for each symbol; papermath implements it. +An equation or math run in the form says so with `form`, which enters no hash, and its tree lives in `derived/math.jsonl`. ([3.7](../spec/03-nodes.md#37-math-forms), [9.4](../spec/09-figures-tables-equations.md#94-the-math-form)) + ML-DSA-65 : Outside term. The lattice-based signature scheme of FIPS 204, believed to resist quantum computers. @@ -172,6 +177,10 @@ Strict projection Suspect : A doubtful reading raised for a person or a review session to answer. ([5.7](../spec/05-hashes.md#57-suspects)) +Symbol key +: The name of a symbol in a formula, its canonical TeX with `\mathrm` spelled `\text`: `K_{r}` for `K_r`. +A paper's symbols and a lode claim's have one key, so a `symbol` annotation names its symbol by it. ([9.4](../spec/09-figures-tables-equations.md#94-the-math-form), [18.4](../spec/18-layers.md#184-bodies)) + Targeted F3 : Proofreading the nodes a team's claims, quotations and datasets rest on, rather than every node. The target set also includes the leaves of every anchored container, every equation and, under the project's digit rule (`all`, `anchored-sections` or `none`), sentences and cells with a digit in a text or math run (an inline stretch of plain text, or of TeX or Typst math, within the node; [3.5](../spec/03-nodes.md#35-inline-runs)). ([8.10](../spec/08-fidelity.md#810-targeted-f3)) diff --git a/docs/overview.md b/docs/overview.md index abdba08..33a909c 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -52,7 +52,7 @@ It appears inside hashed data and test vectors, and it is separate from the repo | Folder | Holds | | --- | --- | | [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. | -| [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. | +| [schemas/](../schemas/README.md) | 43 JSON (plain-text data format) Schemas for the files and records. | | [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. | | [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. | | [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. | diff --git a/fixtures/README.md b/fixtures/README.md index 79b3e62..70319e8 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -25,7 +25,7 @@ A conforming reader accepts each one and recomputes every hash it contains. | Package | What it is for | | --- | --- | -| `packages/hale2026harmonic/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, and derived MathML. | +| `packages/hale2026harmonic/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, derived MathML, and the derived reading of three formulas in the math form `scienting-math/1`, one of which the form's tree cannot read ([spec/09 section 9.4](../spec/09-figures-tables-equations.md#94-the-math-form)). | | `packages/hale2026harmonic.checks-only/` | The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` ([spec/05](../spec/05-hashes.md), [spec/12 section 12.5](../spec/12-package.md#125-text-policies)). It shows how a team can share its checks on a paywalled paper without sharing the paper. | | `packages/okafor2025open/` | An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness. | | `packages/smithlab2027charges/` | A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#a4jrz2t`, a sentence of the main example. | @@ -47,7 +47,7 @@ Each record is a single file, or a folder where a format needs several files. | `backend-docling.json`, `backend-examplevendor.json` | The code and weight licenses of a conversion tool and the conditions attached to them. | | `profile-acme.json`, `profile-smithlab.json` | What those conditions ask about a team: its purpose, its funding and whom it competes with. | | `roster.json` | The keys that may sign, the role of each key's holder in the team, and what each key may sign. Ana is the owner, so she may revoke anyone's receipt. | -| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). | +| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). Its claim link and its symbol name the claim `amber/k-bond` of lode's example store as lode references it, `lode:272XTX7C`, pinned at `lode:272XTX7C@02f47b`; the symbol names `K_r` by its key, `K_{r}`. | | `submissions/`, `grades/` | A student's answers to exercises in the layer, and the grader's result for each. | | `suspect.json` | A doubtful reading raised for review: the converter's reading of an exponent in the main example, before the first correction answered it. | | `qa-report.json` | Reviewer error bounds for a delivery that covers two of the main example's receipts, signed by the team's owner key. | diff --git a/fixtures/packages/hale2026harmonic-damb/MANIFEST.json b/fixtures/packages/hale2026harmonic-damb/MANIFEST.json index 8b3095b..5efc0b3 100644 --- a/fixtures/packages/hale2026harmonic-damb/MANIFEST.json +++ b/fixtures/packages/hale2026harmonic-damb/MANIFEST.json @@ -1 +1 @@ -{"doc":"scienting-doc/1","files":[{"path":"assets/586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f.png","present":false,"sha256":"586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f","size":18213},{"path":"assets/a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75.png","sha256":"a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75","size":73},{"path":"corrections.jsonl","sha256":"026b493baaecba29c15a834e29933b840f306956d377fe85f350372313cab18f","size":8310},{"path":"datasets/ds-hale-bond-params/anchors.jsonl","sha256":"cfacea27f25c0ca912b82882c1f9221b693ae6bcf91c888628b780243ba7e625","size":6046},{"path":"datasets/ds-hale-bond-params/data.csv","sha256":"dbde72e4526e469e1535b87272eb50683160b2fe7c40b7f064e6592341f754a8","size":206},{"path":"datasets/ds-hale-bond-params/datapackage.json","sha256":"d8bbf8bd7c1b0207ada817d10580e835bda3a8691ad3e14fc1910519f4341974","size":574},{"path":"derived/mathml.jsonl","sha256":"47981f878391f3c733072081a0f6d1ad59ba8fa37507c8a4d774feb780c7df6c","size":348},{"path":"document.json","sha256":"f9d3fd85dec8b32378be5b045d0734ec05f3745ec88737bd0554749290e2fbbd","size":1088},{"path":"ids.jsonl","sha256":"d19c4716a605352d4cd84f81381d99a5e2db6b59c88e926611dd13a5a16e4155","size":852},{"path":"locations.jsonl","sha256":"37a3f73ff3dc81082e98b4a02afff535fbdf0f1439cf56fdb39566d28076c6e2","size":7396},{"path":"nodes.jsonl","sha256":"87eeac4b781f67872da36f44f74b859fe5a7c528e3c46276ec0b7dd4f5e2cb71","size":45649},{"path":"pages.jsonl","sha256":"948177c6b7e5e361ac1da0ecff6b52406d3513f14d58cd27cd84f17e6f1c054e","size":438},{"path":"provenance.json","sha256":"4bc7649bf6d3113f6c72b4016713654f57c8f03069632eabd1e6025c05cb9dac","size":226},{"path":"receipts.jsonl","sha256":"da6a4d9cf8af852dff7a175585867d6f2a224c664bd2935f4d47733cb76d1f5d","size":13150},{"path":"references.jsonl","sha256":"25c7a54848e0c3167475236846b85bcea1a35698f6ae5b24a31b6a3690078697","size":567},{"path":"roster.json","sha256":"3c71b0c007924cbacd3b4af02485214ab904db8ff8fca133ab32e67c5c76d481","size":6941},{"path":"witnesses.json","sha256":"99ae654df4fb461a424bb51edd476a89041182c7df9c98653a81b78031f92ada","size":1006}],"format":"scienting-package/1","key":"hale2026harmonic-damb"} +{"doc":"scienting-doc/1","files":[{"path":"assets/586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f.png","present":false,"sha256":"586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f","size":18213},{"path":"assets/a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75.png","sha256":"a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75","size":73},{"path":"corrections.jsonl","sha256":"026b493baaecba29c15a834e29933b840f306956d377fe85f350372313cab18f","size":8310},{"path":"datasets/ds-hale-bond-params/anchors.jsonl","sha256":"cfacea27f25c0ca912b82882c1f9221b693ae6bcf91c888628b780243ba7e625","size":6046},{"path":"datasets/ds-hale-bond-params/data.csv","sha256":"dbde72e4526e469e1535b87272eb50683160b2fe7c40b7f064e6592341f754a8","size":206},{"path":"datasets/ds-hale-bond-params/datapackage.json","sha256":"d8bbf8bd7c1b0207ada817d10580e835bda3a8691ad3e14fc1910519f4341974","size":574},{"path":"derived/math.jsonl","sha256":"7c2e892825f2eb503187d6ac683d4a5f50b99e1d186ddd9ce64597eb0ba3493c","size":774},{"path":"derived/mathml.jsonl","sha256":"47981f878391f3c733072081a0f6d1ad59ba8fa37507c8a4d774feb780c7df6c","size":348},{"path":"document.json","sha256":"f9d3fd85dec8b32378be5b045d0734ec05f3745ec88737bd0554749290e2fbbd","size":1088},{"path":"ids.jsonl","sha256":"d19c4716a605352d4cd84f81381d99a5e2db6b59c88e926611dd13a5a16e4155","size":852},{"path":"locations.jsonl","sha256":"37a3f73ff3dc81082e98b4a02afff535fbdf0f1439cf56fdb39566d28076c6e2","size":7396},{"path":"nodes.jsonl","sha256":"87eeac4b781f67872da36f44f74b859fe5a7c528e3c46276ec0b7dd4f5e2cb71","size":45649},{"path":"pages.jsonl","sha256":"948177c6b7e5e361ac1da0ecff6b52406d3513f14d58cd27cd84f17e6f1c054e","size":438},{"path":"provenance.json","sha256":"4bc7649bf6d3113f6c72b4016713654f57c8f03069632eabd1e6025c05cb9dac","size":226},{"path":"receipts.jsonl","sha256":"da6a4d9cf8af852dff7a175585867d6f2a224c664bd2935f4d47733cb76d1f5d","size":13150},{"path":"references.jsonl","sha256":"25c7a54848e0c3167475236846b85bcea1a35698f6ae5b24a31b6a3690078697","size":567},{"path":"roster.json","sha256":"3c71b0c007924cbacd3b4af02485214ab904db8ff8fca133ab32e67c5c76d481","size":6941},{"path":"witnesses.json","sha256":"99ae654df4fb461a424bb51edd476a89041182c7df9c98653a81b78031f92ada","size":1006}],"format":"scienting-package/1","key":"hale2026harmonic-damb"} diff --git a/fixtures/packages/hale2026harmonic-damb/derived/math.jsonl b/fixtures/packages/hale2026harmonic-damb/derived/math.jsonl new file mode 100644 index 0000000..de57d65 --- /dev/null +++ b/fixtures/packages/hale2026harmonic-damb/derived/math.jsonl @@ -0,0 +1,3 @@ +{"complete":true,"form":"scienting-math/1","issues":[],"node":"tkxsmts","run":null,"symbols":["E_{\\text{bond}}","K_{r}","r","r_{\\text{eq}}"],"tool":{"name":"papermath","version":"0.1.0"},"tree":["rel",["sym","E_{\\text{bond}}"],"=",["mul",["frac",["num","1"],["num","2"]],["sym","K_{r}"],["pow",["add",["sym","r"],["neg",["sym","r_{\\text{eq}}"]]],["num","2"]]]]} +{"complete":true,"form":"scienting-math/1","issues":[],"node":"rr20tev","run":3,"symbols":["K_{r}"],"tool":{"name":"papermath","version":"0.1.0"},"tree":["sym","K_{r}"]} +{"complete":false,"form":"scienting-math/1","issues":[{"at":"_{2}","code":"opaque","message":"a part after the formula's end"}],"node":"hca6dc5","run":1,"symbols":[],"tool":{"name":"papermath","version":"0.1.0"},"tree":["opaque","_{2}"]} diff --git a/fixtures/records/layer/annotations.jsonl b/fixtures/records/layer/annotations.jsonl index 0f661aa..1ee204f 100644 --- a/fixtures/records/layer/annotations.jsonl +++ b/fixtures/records/layer/annotations.jsonl @@ -1,11 +1,12 @@ -{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"audience":"novice","format":"text/markdown","kind":"gloss","language":"en","type":"TextualBody","value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/f4s8pc1hdhd9d","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib"},"rev":12,"span":[111,130],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} -{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"claim-link","pin":"illustrative-pin","role":"states","source":"lode:amber/k-bond","type":"SpecificResource"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/nzwtq23khdx4s","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"linking","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length."},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length.","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} +{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"audience":"novice","format":"text/markdown","kind":"gloss","language":"en","type":"TextualBody","value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/f4s8pc1hdhd9d","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"y0d9dhj1k9zx0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib"},"rev":12,"span":[111,130],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} +{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"claim-link","pin":"lode:272XTX7C@02f47b","role":"states","source":"lode:272XTX7C","type":"SpecificResource"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/p5shwykrcv73a","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"linking","receipt":"y0d9dhj1k9zx0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length."},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length.","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"critique","kind":"comment","type":"TextualBody","value":"The step size matters more than the text admits."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/sxj4eq8415s92","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"commenting","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"xjns57z","node_hash":"42b1de7852cdc68730984fff789fe248ecd5b43198c0e7d671931ff3fe561dc3","own_hash":"4dc28d1ad57c74d044f87264dfbc744ae9d5e9644638db976dbc346361442486","quote":{"exact":"steps of 0.01","prefix":"Each bond was scanned in ","suffix":" Å around its minimum, and the e"},"rev":12,"span":[25,38],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"steps of 0.01","prefix":"Each bond was scanned in ","suffix":" Å around its minimum, and the e","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"question","for":"group","kind":"question","type":"TextualBody","value":"Why fit to a scan first and to frequencies second?"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/fsg960szdyy6q","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"questioning","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"section","node":"d400rz3","node_hash":"e5a8eeb3c112f2dc3341da8f433540a3fc5a7e7f631e72c4dd36c6933aae808d","own_hash":"c3d50f266b05d7889c001aef16662cecd4e99ac4a2747ec331c1044e6adc8e3e","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"citation-context","reason":"value","reference":"vxsdvzq","type":"TextualBody","value":"Cited for the band position; the reference is not in the library.","verdict":"cannot-tell"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/gbj7nn87ndfd8","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"hca6dc5","node_hash":"8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34","own_hash":"c270571517825ceed31ca912b80ac76203effe47f1c84f8ee8962800d0afc92f","quote":{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2"},"rev":12,"span":[71,75],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"generator":{"model":{"name":"example-model","parameters":"998b5161e39e01c8bc43ef81ec89141e15b7ea53b2e936707920a6097dbcb2c5","processor":"local","prompt":"9cc338fba7c62db7266f530cb555a5ee36910d41a0120a3fe7fd7de1944d0931","provider":"Example Models (illustrative)","version":"2026-09","weights":"504b2d340045b92c22682b60020c21103aae7ff41172bb632fbe305eb4af9b5e"},"run":"illustrative-run-1"},"kind":"generated","type":"TextualBody","value":"Two panels: an energy scan and its fit residual."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:software:example-model","name":"A model (illustrative)","type":"Software"},"id":"tag:scient.ing,2026:annotation:smithlab/szb6k3fj9z7aw","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"figure","node":"v7cx6qs","node_hash":"f76864779843485fa9e2f3cc3cd1c76ad222e1649c56f4cface33bf20b8f33b6","own_hash":"a1e46cb40201a17903dfd6f128db1e707b0e9f5bf45fcc32ec7ca772011edeae","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} -{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"anchor","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"context":{"column_path":["Set A","$K_r$ (kcal mol⁻¹ Å⁻²)"],"row_path":["Bond parameters","CT–CT"],"table_label":"Table 1"},"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"cell","node":"c20cna6","node_hash":"c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272","own_hash":"45bf6ab427d435f28b82cb2b64d93c502968921507fc9397abd5d5170731c6ab","quote":{"exact":"310.0"},"rev":12,"store":"smithlab"}],"feedback":{"hit":"That is the cell.","miss":"Look in Table 1, row CT–CT."},"points":"1"}]},"prompt":"Find the table cell that gives the Set A bond constant for the CT–CT bond.","rule":{"match":"same-node"}},"kind":"exercise","type":"TextualBody","value":"Find the table cell that gives the Set A bond constant for the CT–CT bond."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/whcypxav2cqbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"table","node":"hj4s7pb","node_hash":"6c6fba200faf41cd06fa8b64e06e2aa29ff8848568b859ae42d1902596348643","own_hash":"66310a9ab1c92568f4c6cae012e35d4245d4ca4b89d2542b7c2113a31ad379ed","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} -{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"check","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"the most common description","prefix":"Harmonic bond terms remain ","suffix":" of covalent stretching in class"},"rev":12,"span":[27,54],"store":"smithlab"}],"points":"1"}],"verdict":{"points":"1","value":"supports"}},"prompt":"Does the cited passage support the claim that harmonic terms are the most common choice?","rule":{"match":"overlap","min_overlap":"0.5"}},"kind":"exercise","type":"TextualBody","value":"Does the cited passage support the claim that harmonic terms are the most common choice?"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/y7bd3he0jwjbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1"},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} -{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"description","language":"en","role":"alt","type":"TextualBody","value":"Energy along the C–C bond stretch (panel a) and the fit residual (panel b)."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/gb5tmv3hq08vz","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"figure","node":"v7cx6qs","node_hash":"f76864779843485fa9e2f3cc3cd1c76ad222e1649c56f4cface33bf20b8f33b6","own_hash":"a1e46cb40201a17903dfd6f128db1e707b0e9f5bf45fcc32ec7ca772011edeae","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} +{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"anchor","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"context":{"column_path":["Set A","$K_r$ (kcal mol⁻¹ Å⁻²)"],"row_path":["Bond parameters","CT–CT"],"table_label":"Table 1"},"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"cell","node":"c20cna6","node_hash":"c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272","own_hash":"45bf6ab427d435f28b82cb2b64d93c502968921507fc9397abd5d5170731c6ab","quote":{"exact":"310.0"},"rev":12,"store":"smithlab"}],"feedback":{"hit":"That is the cell.","miss":"Look in Table 1, row CT–CT."},"points":"1"}]},"prompt":"Find the table cell that gives the Set A bond constant for the CT–CT bond.","rule":{"match":"same-node"}},"kind":"exercise","type":"TextualBody","value":"Find the table cell that gives the Set A bond constant for the CT–CT bond."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/whcypxav2cqbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"y0d9dhj1k9zx0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"table","node":"hj4s7pb","node_hash":"6c6fba200faf41cd06fa8b64e06e2aa29ff8848568b859ae42d1902596348643","own_hash":"66310a9ab1c92568f4c6cae012e35d4245d4ca4b89d2542b7c2113a31ad379ed","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} +{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"check","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"the most common description","prefix":"Harmonic bond terms remain ","suffix":" of covalent stretching in class"},"rev":12,"span":[27,54],"store":"smithlab"}],"points":"1"}],"verdict":{"points":"1","value":"supports"}},"prompt":"Does the cited passage support the claim that harmonic terms are the most common choice?","rule":{"match":"overlap","min_overlap":"0.5"}},"kind":"exercise","type":"TextualBody","value":"Does the cited passage support the claim that harmonic terms are the most common choice?"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/y7bd3he0jwjbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"y0d9dhj1k9zx0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1"},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} +{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"description","language":"en","role":"alt","type":"TextualBody","value":"Energy along the C–C bond stretch (panel a) and the fit residual (panel b)."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/gb5tmv3hq08vz","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"y0d9dhj1k9zx0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"figure","node":"v7cx6qs","node_hash":"f76864779843485fa9e2f3cc3cd1c76ad222e1649c56f4cface33bf20b8f33b6","own_hash":"a1e46cb40201a17903dfd6f128db1e707b0e9f5bf45fcc32ec7ca772011edeae","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"comment","kind":"comment","type":"TextualBody","value":"Agreed; see the residual panel."},"created":"2026-10-11T19:05:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/t5cv4rec7j0mt","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"replying","status":"proposed","target":{"source":"tag:scient.ing,2026:annotation:smithlab/sxj4eq8415s92","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"context","kind":"caveat","type":"TextualBody","value":"The band position is cited, not measured here; reference 2 is not in the library."},"created":"2026-10-11T19:10:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/g7ngzk3jkdy40","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"commenting","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"hca6dc5","node_hash":"8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34","own_hash":"c270571517825ceed31ca912b80ac76203effe47f1c84f8ee8962800d0afc92f","quote":{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2"},"rev":12,"span":[71,75],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} +{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"key":"K_{r}","kind":"symbol","pin":"lode:272XTX7C@02f47b","source":"lode:272XTX7C","symbol":"K_r","type":"SpecificResource"},"created":"2026-10-11T19:15:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/a2ehr6sv7nqff","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"identifying","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"$K_r$","prefix":"(r - r_{\\mathrm{eq}})^2$$ where ","suffix":" is the bond force constant and "},"rev":12,"span":[98,103],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"$K_r$","prefix":"(r - r_{\\mathrm{eq}})^2$$ where ","suffix":" is the bond force constant and ","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} diff --git a/fixtures/records/layer/layer.json b/fixtures/records/layer/layer.json index 5b5b2e3..12e6587 100644 --- a/fixtures/records/layer/layer.json +++ b/fixtures/records/layer/layer.json @@ -1 +1 @@ -{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"audience":"student","created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"docs":["smithlab/hale2026harmonic-damb"],"format":"scienting-layer/1","id":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","label":"Smith lab reading of Hale and Osei","license":{"spdx":"CC-BY-4.0"},"project":"smithlab-charges","purpose":"Glosses, claim links and exercises for new students.","total":11,"type":"AnnotationCollection","visibility":"team"} +{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"audience":"student","created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"docs":["smithlab/hale2026harmonic-damb"],"format":"scienting-layer/1","id":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","label":"Smith lab reading of Hale and Osei","license":{"spdx":"CC-BY-4.0"},"project":"smithlab-charges","purpose":"Glosses, claim links and exercises for new students.","total":12,"type":"AnnotationCollection","visibility":"team"} diff --git a/fixtures/records/layer/receipt.json b/fixtures/records/layer/receipt.json index 962ccac..ce73033 100644 --- a/fixtures/records/layer/receipt.json +++ b/fixtures/records/layer/receipt.json @@ -1 +1 @@ -{"at":"2026-10-11T19:30:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[],"format":"scienting-receipt/1","id":"r10e5mzvw6tja","level":"F3","nodes":[],"records":[["annotation","f4s8pc1hdhd9d","9db553d2150a912e4ba3027d002706e08ccaded843d3e49c79b95a2e0581b6f1"],["annotation","gb5tmv3hq08vz","3b776d37af21d1bdfd3a0130c9bfa26affa484fb4d76e41d83cbd8bb02b4975c"],["annotation","nzwtq23khdx4s","8ea94b7a3d266f51e11dd31399784019acd1b1b53ed093e15c199417327992ab"],["annotation","whcypxav2cqbx","11845db3ef685b0b9bd13895cc5b8ade5b345ee7d72737a630be60d690a2ba5d"],["annotation","y7bd3he0jwjbx","0286ede04d8bf548432e34dd7beeee0fb044bd43022a8661a292018dd2ef5578"]],"rev":12,"root":"246581b0926273458b78ece15b465f0567c150a1e071f72322ce41895f9502e1","scope":"targeted:layer","signature":{"key":"ana","scheme":"ed25519","value":"WMDdARYwB0ETltF57ks1MxJ6ny/c4XziLvYUBd8PoW7MwcfhPKmDTqNGxRNg4z5g4c/bHnWS4ftZLJiONTL/Cg=="}} +{"at":"2026-10-11T19:30:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[],"format":"scienting-receipt/1","id":"y0d9dhj1k9zx0","level":"F3","nodes":[],"records":[["annotation","f4s8pc1hdhd9d","9db553d2150a912e4ba3027d002706e08ccaded843d3e49c79b95a2e0581b6f1"],["annotation","gb5tmv3hq08vz","3b776d37af21d1bdfd3a0130c9bfa26affa484fb4d76e41d83cbd8bb02b4975c"],["annotation","p5shwykrcv73a","f657ba315ce9114e2bfd5a767ce16cc85481c64424eb610277e13a7315cd07b3"],["annotation","whcypxav2cqbx","11845db3ef685b0b9bd13895cc5b8ade5b345ee7d72737a630be60d690a2ba5d"],["annotation","y7bd3he0jwjbx","0286ede04d8bf548432e34dd7beeee0fb044bd43022a8661a292018dd2ef5578"]],"rev":12,"root":"2990f8d3ed6afde9036c5c5939af41b051c5647ecea35447a82ae20311ecbc0a","scope":"targeted:layer","signature":{"key":"ana","scheme":"ed25519","value":"79tCSFA56N+qXZAaBSwEK1ibHS6np4haqv/cA5QDoXNunpc9bitfEXLi1ak3POT+xokZfp7DutBAs2cGSAbHAg=="}} diff --git a/fixtures/records/project.json b/fixtures/records/project.json index 87f9794..a0b4185 100644 --- a/fixtures/records/project.json +++ b/fixtures/records/project.json @@ -1 +1 @@ -{"claims":{"lode":"amber"},"datasets":["ds-hale-bond-params","ds-hale-ctct-scan"],"documents":[{"doc":"smithlab/hale2026harmonic-damb","reason":"The bond constants we start from.","status":{"Ben Example":"read","Cara Example":"reading"},"tags":["parameters"]},{"doc":"okafor2025solvent-acth","reason":"Solvent shifts for the validation set.","tags":["spectra"]}],"drafts":["smithlab/writer2027fixed-ynr7.v3"],"format":"scienting-project/1","key":"smithlab-charges","layers":["tag:scient.ing,2026:layer:smithlab/kp4m2x7q"],"members":[{"keys":["ana"],"person":"Ana Example","role":"owner"},{"keys":["ben","ben-spot"],"person":"Ben Example","role":"editor"},{"keys":["cara"],"person":"Cara Example","role":"reader"}],"name":"Charges and bonds","paths":[{"id":"onboarding","steps":[{"note":"Read the introduction.","ref":"smithlab/hale2026harmonic-damb","type":"document"},{"note":"Then the table.","ref":"smithlab/hale2026harmonic-damb#hj4s7pb","type":"node"},{"ref":"lode:amber/k-bond","type":"claim"}],"title":"First week"}],"settings":{"digit_rule":"anchored-sections","drift_threshold":"0.25","log":{"origin":"scient.ing/log/smithlab","threshold":1,"witnesses":["wit"]},"rights_unknown":"refuse","spot_check":{"minimum":10,"rate":"0.05"},"stages":"scienting-stages-default"},"store":"smithlab","views":["ctct-scan"]} +{"claims":{"lode":"amber"},"datasets":["ds-hale-bond-params","ds-hale-ctct-scan"],"documents":[{"doc":"smithlab/hale2026harmonic-damb","reason":"The bond constants we start from.","status":{"Ben Example":"read","Cara Example":"reading"},"tags":["parameters"]},{"doc":"okafor2025solvent-acth","reason":"Solvent shifts for the validation set.","tags":["spectra"]}],"drafts":["smithlab/writer2027fixed-ynr7.v3"],"format":"scienting-project/1","key":"smithlab-charges","layers":["tag:scient.ing,2026:layer:smithlab/kp4m2x7q"],"members":[{"keys":["ana"],"person":"Ana Example","role":"owner"},{"keys":["ben","ben-spot"],"person":"Ben Example","role":"editor"},{"keys":["cara"],"person":"Cara Example","role":"reader"}],"name":"Charges and bonds","paths":[{"id":"onboarding","steps":[{"note":"Read the introduction.","ref":"smithlab/hale2026harmonic-damb","type":"document"},{"note":"Then the table.","ref":"smithlab/hale2026harmonic-damb#hj4s7pb","type":"node"},{"note":"The bond force constant, amber/k-bond.","ref":"lode:272XTX7C","type":"claim"}],"title":"First week"}],"settings":{"digit_rule":"anchored-sections","drift_threshold":"0.25","log":{"origin":"scient.ing/log/smithlab","threshold":1,"witnesses":["wit"]},"rights_unknown":"refuse","spot_check":{"minimum":10,"rate":"0.05"},"stages":"scienting-stages-default"},"store":"smithlab","views":["ctct-scan"]} diff --git a/registry/rules.json b/registry/rules.json index 239fa7c..ca522ee 100644 --- a/registry/rules.json +++ b/registry/rules.json @@ -113,6 +113,16 @@ "description": "document.json's rev is not the number of corrections.", "chapter": "spec/22-conformance.md" }, + { + "id": "doc.derived.math", + "description": "A line of derived/math.jsonl names a formula that is not TeX, says it is complete when its tree has an opaque node or the reverse, or lists symbols that are not the sym keys of its tree.", + "chapter": "spec/09-figures-tables-equations.md" + }, + { + "id": "doc.derived.node", + "description": "A line of a derived file names no equation, equation line or math run of the package.", + "chapter": "spec/12-package.md" + }, { "id": "doc.fidelity.report", "description": "document.json's fidelity distribution does not recompute.", @@ -208,6 +218,11 @@ "description": "A location's page_fp is not a page of any witness.", "chapter": "spec/22-conformance.md" }, + { + "id": "doc.math.form", + "description": "A source with a form is not that form's canonical spelling. Only a validator that implements the form can check it; the reference validator does not.", + "chapter": "spec/03-nodes.md" + }, { "id": "doc.node.attrs", "description": "A section has no level.", diff --git a/schemas/README.md b/schemas/README.md index f07f386..90132bd 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -1,6 +1,6 @@ # Schemas -This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the paperspec formats (paperspec is the specification this repository holds) define: 42 schema files. +This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the paperspec formats (paperspec is the specification this repository holds) define: 43 schema files. A JSON Schema is a file, itself written in JSON, that states what a valid JSON file looks like. Newcomers can read it as a form: it lists the fields a record must have and what kind of value each field takes. Implementers will find the loading recipe and the traps under "Validating" and "Writing schemas". @@ -20,7 +20,7 @@ The number after each name is the chapter of `spec/` that defines the file. | Topic | Schemas (chapter) | | --- | --- | | Shared definitions | `common.schema.json` (1): hashes, IDs, keys, addresses, times, decimals, boxes, licenses, signatures, evidence, quotes. | -| A package | `manifest.schema.json` (12), `document.schema.json` (2, 12), `node.schema.json` (3), `runs.schema.json` (3), `witnesses.schema.json` (2), `provenance.schema.json` (12), `page.schema.json` (7), `location.schema.json` (7), `reference.schema.json` (10), `id-entry.schema.json` (4), `correction.schema.json` (11), `receipt.schema.json` (8), `revocation.schema.json` (8), `roster.schema.json` (8), `suspect.schema.json` (5), `mathml.schema.json` (12) | +| A package | `manifest.schema.json` (12), `document.schema.json` (2, 12), `node.schema.json` (3), `runs.schema.json` (3), `witnesses.schema.json` (2), `provenance.schema.json` (12), `page.schema.json` (7), `location.schema.json` (7), `reference.schema.json` (10), `id-entry.schema.json` (4), `correction.schema.json` (11), `receipt.schema.json` (8), `revocation.schema.json` (8), `roster.schema.json` (8), `suspect.schema.json` (5), `mathml.schema.json` (12), `math.schema.json` (9, 12) | | Pointing into documents | `anchor.schema.json` (6) | | Rights | `rights.schema.json` (14), `policy.schema.json` (14), `stages.schema.json` (14), `backend.schema.json` (14), `profile.schema.json` (14), `stamps.schema.json` (7) | | Review | `qa-report.schema.json` (8), `spot-check.schema.json` (8), `checklist.schema.json` (8), `proof.schema.json` (8, 11) | @@ -37,7 +37,8 @@ The Data schemas for datasets, digitizations and views, the Teams and drafts sch The rest belong to `candidate` formats or to no format of their own. The bundle has no JSON Schema: its registry entry points to `sql/bundle-sqlite.sql`. -Of the 42 schemas, 11 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `annotation` and `value-anchor`. +Of the 43 schemas, 12 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `math`, `annotation` and `value-anchor`. +A line of `derived/math.jsonl` carries `form`, the math form `scienting-math/1` it reads the formula in; that names a spelling of TeX, which papermath implements ([spec/09 §9.4](../spec/09-figures-tables-equations.md#94-the-math-form)), not a file format, so the registry does not list it. Of the schemas that do have one, 27 require a `format` property; `spot-check.schema.json` describes the input a VRF proof is made over rather than a file. The others (`page`, `draft-ids`, `view` and `log-entry`) describe a line of a file, a Vega-Lite specification (a JSON description of a chart) or an item of a tree, and the record itself does not need to carry the identifier. diff --git a/schemas/annotation.schema.json b/schemas/annotation.schema.json index 8b90121..0cc4201 100644 --- a/schemas/annotation.schema.json +++ b/schemas/annotation.schema.json @@ -111,15 +111,13 @@ "const": "SpecificResource" }, "source": { - "type": "string", - "minLength": 1 + "$ref": "common.schema.json#/$defs/claimRef" }, "kind": { "const": "claim-link" }, "pin": { - "type": "string", - "minLength": 1 + "$ref": "common.schema.json#/$defs/claimPin" }, "role": { "enum": [ @@ -144,17 +142,21 @@ "const": "SpecificResource" }, "source": { - "type": "string", - "minLength": 1 + "$ref": "common.schema.json#/$defs/claimRef" }, "kind": { "const": "symbol" }, "pin": { + "$ref": "common.schema.json#/$defs/claimPin" + }, + "key": { + "description": "The symbol's scienting-math/1 key, which the symbol claim's notation has too (spec/09 §9.4, spec/18 §18.4).", "type": "string", "minLength": 1 }, "symbol": { + "description": "The symbol as the span writes it.", "type": "string", "minLength": 1 } @@ -162,7 +164,8 @@ "required": [ "type", "kind", - "source" + "source", + "key" ], "additionalProperties": false }, diff --git a/schemas/common.schema.json b/schemas/common.schema.json index 6defcd0..d9862a9 100644 --- a/schemas/common.schema.json +++ b/schemas/common.schema.json @@ -71,6 +71,22 @@ "type": "string", "pattern": "^([a-z0-9][a-z0-9._-]*/)?[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\\.[a-z0-9][a-z0-9-]*)*#[a-z0-9][a-z0-9._-]*(@[0-9a-f]{6})?(\\[(0|[1-9][0-9]*):(0|[1-9][0-9]*)\\])?$" }, + "claimRef": { + "type": "string", + "pattern": "^lode:([a-z0-9.-]+\\.[a-z]{2,}/)?[0-9A-HJKMNP-TV-Z]{8}$", + "description": "A claim in a lode store: lode:TAG, or lode:host/TAG for another store, TAG eight Crockford base32 capitals (spec/18 §18.4)." + }, + "claimPin": { + "type": "string", + "pattern": "^lode:([a-z0-9.-]+\\.[a-z]{2,}/)?[0-9A-HJKMNP-TV-Z]{8}@[0-9a-f]{6}$", + "description": "A claim pinned to its statement: the claim's reference, @ and the first six hex digits of its statement hash (spec/18 §18.4)." + }, + "mathForm": { + "enum": [ + "scienting-math/1" + ], + "description": "The form a math source is spelled in; scienting-math/1 is canonical TeX (spec/03 §3.7, spec/09 §9.4)." + }, "time": { "type": "string", "pattern": "^[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]Z$" diff --git a/schemas/document.schema.json b/schemas/document.schema.json index e03bae6..d6024c5 100644 --- a/schemas/document.schema.json +++ b/schemas/document.schema.json @@ -47,8 +47,15 @@ "same_as": { "type": "array", "items": { - "type": "string", - "pattern": "^(doi|arxiv|isbn|pmid|pmcid|openalex|url|lode):[^ ]+$" + "anyOf": [ + { + "type": "string", + "pattern": "^(doi|arxiv|isbn|pmid|pmcid|openalex|url):[^ ]+$" + }, + { + "$ref": "common.schema.json#/$defs/claimRef" + } + ] }, "uniqueItems": true } diff --git a/schemas/math.schema.json b/schemas/math.schema.json new file mode 100644 index 0000000..6fa4a91 --- /dev/null +++ b/schemas/math.schema.json @@ -0,0 +1,172 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://paperspec.scient.ing/schemas/math.schema.json", + "title": "Derived math", + "description": "One line of derived/math.jsonl: a formula of the package read in the form scienting-math/1 by a pinned tool, its tree, symbol keys and issues; never authoritative; spec/09-figures-tables-equations.md §9.4, spec/12-package.md §12.7.", + "$defs": { + "kind": { + "description": "What a tree node is. The kinds of scienting-math/1 are num, sym, text, const, oper, dots, empty, add, neg, pm, mp, mul, times, div, op, frac, binom, sqrt, root, pow, sub, prime, fact, apply, lim, sum, prod, coprod, int, iint, iiint, oint, bigcup, bigcap, abs, norm, angle, floor, ceil, braces, interval, rel, list, lines, cases, matrix and opaque (spec/09 §9.4). The schema takes any lowercase name, so that a later minor version of the form may add a kind; a consumer that meets a kind it does not know treats that node as it treats opaque.", + "type": "string", + "pattern": "^[a-z][a-z0-9-]*$", + "examples": [ + "sym", + "frac", + "rel", + "opaque" + ] + }, + "node": { + "description": "A tree node: an array whose first item is its kind and whose other items are its parts.", + "type": "array", + "minItems": 1, + "prefixItems": [ + { + "$ref": "#/$defs/kind" + } + ], + "items": { + "$ref": "#/$defs/part" + } + }, + "group": { + "description": "Parts that are not a node: a case's value and condition, or a matrix's rows and cells. Its first item, when it has one, is a node, a group or null, never a string, so it is never mistaken for a node.", + "type": "array", + "prefixItems": [ + { + "anyOf": [ + { + "type": "array" + }, + { + "type": "null" + } + ] + } + ], + "items": { + "$ref": "#/$defs/part" + } + }, + "part": { + "description": "A part of a node: a string (digits, a symbol key, words, a command, a relation such as =), null (a script or condition that is absent), a node, or a group.", + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + }, + { + "$ref": "#/$defs/node" + }, + { + "$ref": "#/$defs/group" + } + ] + }, + "issue": { + "description": "Something a reader of the formula should look at; the formula is read all the same.", + "type": "object", + "properties": { + "code": { + "description": "unbalanced, unknown-command, label-or-power, script-on-group, opaque or infix-fraction in scienting-math/1 (spec/09 §9.4); a later minor version may add a code.", + "type": "string", + "pattern": "^[a-z][a-z0-9-]*$" + }, + "message": { + "type": "string", + "minLength": 1 + }, + "at": { + "description": "The TeX the issue is about, in canonical form; empty when it is about the whole formula.", + "type": "string" + } + }, + "required": [ + "code", + "message", + "at" + ], + "additionalProperties": false + } + }, + "type": "object", + "properties": { + "node": { + "$ref": "common.schema.json#/$defs/nodeId" + }, + "run": { + "description": "The index of the math run in the node's runs, or null for an equation or equation-line node, as in derived/mathml.jsonl.", + "anyOf": [ + { + "type": "integer", + "minimum": 0 + }, + { + "type": "null" + } + ] + }, + "tool": { + "type": "object", + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "version": { + "type": "string", + "minLength": 1 + } + }, + "required": [ + "name", + "version" + ], + "additionalProperties": false + }, + "form": { + "enum": [ + "scienting-math/1" + ] + }, + "tree": { + "$ref": "#/$defs/node" + }, + "symbols": { + "description": "The key of every symbol in the tree, each once, in the order the formula first writes it.", + "type": "array", + "items": { + "type": "string", + "minLength": 1 + }, + "uniqueItems": true + }, + "complete": { + "description": "True exactly when the tree has no opaque node.", + "type": "boolean" + }, + "issues": { + "type": "array", + "items": { + "$ref": "#/$defs/issue" + } + }, + "label": { + "description": "The printed number the tool lifted out of a \\tag in the source; omitted when the source has none.", + "type": "string", + "minLength": 1 + } + }, + "required": [ + "node", + "run", + "tool", + "form", + "tree", + "symbols", + "complete", + "issues" + ], + "additionalProperties": false +} diff --git a/schemas/node.schema.json b/schemas/node.schema.json index 3a5aa92..e3e9858 100644 --- a/schemas/node.schema.json +++ b/schemas/node.schema.json @@ -329,13 +329,28 @@ "source": { "type": "string", "minLength": 1 + }, + "form": { + "$ref": "common.schema.json#/$defs/mathForm" } }, "required": [ "notation", "source" ], - "additionalProperties": false + "additionalProperties": false, + "if": { + "required": [ + "form" + ] + }, + "then": { + "properties": { + "notation": { + "const": "tex" + } + } + } } } } diff --git a/schemas/project.schema.json b/schemas/project.schema.json index ef8a284..55b436e 100644 --- a/schemas/project.schema.json +++ b/schemas/project.schema.json @@ -131,7 +131,21 @@ "type", "ref" ], - "additionalProperties": false + "additionalProperties": false, + "if": { + "properties": { + "type": { + "const": "claim" + } + } + }, + "then": { + "properties": { + "ref": { + "$ref": "common.schema.json#/$defs/claimRef" + } + } + } }, "minItems": 1 } diff --git a/schemas/runs.schema.json b/schemas/runs.schema.json index a806816..6f57bb8 100644 --- a/schemas/runs.schema.json +++ b/schemas/runs.schema.json @@ -64,6 +64,9 @@ "source": { "type": "string", "minLength": 1 + }, + "form": { + "$ref": "common.schema.json#/$defs/mathForm" } }, "required": [ @@ -71,7 +74,19 @@ "notation", "source" ], - "additionalProperties": false + "additionalProperties": false, + "if": { + "required": [ + "form" + ] + }, + "then": { + "properties": { + "notation": { + "const": "tex" + } + } + } }, { "type": "object", diff --git a/spec/02-model.md b/spec/02-model.md index 2b6ed3b..739e677 100644 --- a/spec/02-model.md +++ b/spec/02-model.md @@ -84,6 +84,7 @@ The stop words are `a an the of on in for to and or at by with from as into onto **The identity** is the work's first identifier among `doi`, `arxiv`, `isbn`, `pmid`, `pmcid` and `openalex` in `same_as`, in that order, written `:` with a DOI's ASCII letters lower-cased. A work with none has the identity `meta:|<year>|<family>` from the same folded parts. `url:` and `lode:` identifiers name copies and claims, not the work, and never enter the identity. +A `lode:` identifier is a claim reference as a layer writes one, `lode:<tag>` or `lode:<host>/<tag>` ([18.4](18-layers.md#184-bodies)). **The hash part** is the leading bits of SHA-256 of `scienting-key/1:` followed by the identity, in UTF-8, written in lowercase Crockford base32, five bits a character: four characters, or, when another work in the store already holds that key, six, eight and so on until the key is free. A key is minted once, when the work enters a store, and never recomputed: a corrected title or a year revised later leaves it alone, while the hash of an identifier still shows which work the key names. diff --git a/spec/03-nodes.md b/spec/03-nodes.md index d83694a..17ba64a 100644 --- a/spec/03-nodes.md +++ b/spec/03-nodes.md @@ -79,11 +79,11 @@ Page breaks, running heads and columns are not nodes: they are geometry ([7](07- | Content | Kinds | Shape | | --- | --- | --- | | Runs | `sentence`, `cell`, `reference`, `authors`, `keywords` | A non-empty list of inline runs ([3.5](#35-inline-runs)). | -| Math | `equation`, `equation-line` | `{notation, source}`: `tex` or `typst`, and the source as written. | +| Math | `equation`, `equation-line` | `{notation, source, form?}`: `tex` or `typst`, the source as written or as a form spells it, and that form ([3.7](#37-math-forms)). | | Verbatim | `code`, `algorithm` | `{language?, text}`. | Other kinds have no `content`. -Derived renderings, such as MathML, are not content; they live in `derived/` ([12.7](12-package.md#127-derived-files)). +Derived renderings and readings, such as MathML and a formula's tree, are not content; they live in `derived/` ([12.7](12-package.md#127-derived-files)). ## 3.4 Attributes @@ -105,7 +105,7 @@ No other kind carries `attrs`, and each kind carries only its own attributes. | Run | Properties | Plain projection | | --- | --- | --- | | `text` | `text`, and optional `style`: a sorted list of distinct values from `em`, `strong`, `sup`, `sub`, `sc`, `mono`, never both `sup` and `sub`. | The text. | -| `math` | `notation` (`tex` or `typst`), `source`. | `$` + source + `$` | +| `math` | `notation` (`tex` or `typst`), `source`, and optional `form` ([3.7](#37-math-forms)). | `$` + source + `$` | | `cite` | `refs`: reference node IDs; `text`: the printed form, such as `23`, `[1,2]` or `(Smith, 2019)`. | The printed form. | | `xref` | `target`: a node ID in this document, an address in another, or null when not yet resolved; `text`. | The printed form. | | `fnmark` | `footnote`: a footnote's ID, or null; `text`: the mark. | The mark. | @@ -133,3 +133,21 @@ A node's runs as printed are its runs with each run's `printed` in place of its Its projection as printed is the plain projection of its runs as printed; an equation's or a verbatim node's is its plain projection, and any other node's joins its text leaves' projections as printed with one U+0020. It is the text a witness of the paper holds, recovered from `nodes.jsonl` alone; for a node no emend touched it is the plain projection. The strict and lossy projections are taken over the runs as printed ([5.3](05-hashes.md#53-the-strict-projection)), and `projections.jsonl` gives an emended sentence's projections with its projection as printed. + +## 3.7 Math forms + +A math form is a standard spelling of formulas, so that one formula is written alike whichever paper, converter or claim it comes from. +An equation's or equation line's content and a `math` run MAY carry `form`, which names the form its `source` is written in; a source without `form` is as the witness wrote it. +The one form is `scienting-math/1`, canonical TeX, which [9.4](09-figures-tables-equations.md#94-the-math-form) describes; a node or run with `form` has the notation `tex`. + +A consumer MAY rely on these properties of a source that carries `form`: + +1. It is canonical: it is the form's own spelling of the formula the witness wrote, which renders as that formula did, apart from presentation such as spacing, `\displaystyle` and delimiter sizes. +2. It is idempotent: spelling it in the form again gives it back unchanged, so a tool that rewrites it to check it, or a converter run twice, changes nothing. +3. It carries no label: the form lifts a `\tag` out of the source, and an equation keeps it as its `label`; `\label`, `\nonumber` and `\notag` are gone. +4. Everything else is derived from it: the tree, the symbol keys and the issues of [9.4](09-figures-tables-equations.md#94-the-math-form) are read from the source by a pinned tool into `derived/math.jsonl` ([12.7](12-package.md#127-derived-files)), are never content, and are the same for the same source under the same form. + +A producer writes `form` only on a source it has spelled in that form. +A correction that changes such a source writes the new source in the form too, since the gate does not implement the form and cannot check it. +A validator that implements the form reports a source with `form` that is not its own canonical spelling as `doc.math.form`. +`form` enters no content value, so adding it to a source already in the form changes no hash, ID or anchor, while respelling a source in the form changes them as any edit of the source does ([5.2](05-hashes.md#52-own-content)). diff --git a/spec/05-hashes.md b/spec/05-hashes.md index 28a02de..87e12d8 100644 --- a/spec/05-hashes.md +++ b/spec/05-hashes.md @@ -30,8 +30,8 @@ A node's own content is an object with its `kind` and these members, and nothing | Kinds | Members | | --- | --- | | Kinds that take a label | `label`: the label, or null. | -| `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content. | -| `equation`, `equation-line` | `notation`, `source`. | +| `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content, and each `math` run is without its `form`. | +| `equation`, `equation-line` | `notation`, `source`; never `form`. | | `code`, `algorithm` | `language` (or null), `text`. | | `section` | `level`, and `heading`: the `text_hash` of its heading, or null. | | `table` | `columns`, as stored. | @@ -42,6 +42,14 @@ So paragraphs, rows, lists, items and the other containers without content have Own content covers stored values exactly; it is not normalized. A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table. +A math form ([3.7](03-nodes.md#37-math-forms)) enters no content value. +`form` says how the source is spelled, which the source itself shows, so it is left out of own content, and no projection reads it: a source in canonical TeX is projected as the TeX it is, by the rules of [5.3](#53-the-strict-projection) and [5.4](#54-simple-notation). +So a producer that adds `form` to sources already in the form leaves every hash, ID, pin and anchor as it was, and a reading with `form` and one without agree whenever their sources do. +Respelling a source in the form is an edit of the source: it changes the `text_hash`, `own_hash` and `node_hash` of the node, and, for a complex source, its complex key ([5.3](#53-the-strict-projection)). +Simple sources such as `K_r` and `K_{r}` have one canonical linear form, so their `match_hash` and `align_key` stay; a respelling that turns a simple character into a command [5.4](#54-simple-notation) does not read, as `ħ` becomes `\hbar`, makes the source complex and changes them. +Because the complex key compares sources as stored, a producer that aligns a reading in a form with a second reading ([5.6](#56-alignment)) spells the second reading's math in the same form first, or each complex formula it respelled becomes an `equation` suspect. +`hashes.jsonl` gives the values of a small document whose equation and math runs carry `form`, which are those of the same document without it. + ## 5.3 The strict projection The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement. diff --git a/spec/09-figures-tables-equations.md b/spec/09-figures-tables-equations.md index b38a54c..1abff4e 100644 --- a/spec/09-figures-tables-equations.md +++ b/spec/09-figures-tables-equations.md @@ -34,4 +34,81 @@ For people, a cell can be shown as "Table 2, row CT–CT, column K_r" from its h An equation keeps its source as written, `tex` from a conversion or `typst` from a draft, and its printed number as its `label`, so numbering never collides with a renderer's own. An equation that the source numbers line by line (Eq. 3a) has `equation-line` children, each with its own label and source. -MathML is derived by a pinned tool and recorded, with the tool's name and version, in `derived/mathml.jsonl`. +A converter that respells TeX in the standard form of [9.4](#94-the-math-form) stores the respelled source and says so with `form` beside it ([3.7](03-nodes.md#37-math-forms)); the source as written stays in the witness. +MathML is derived by a pinned tool and recorded, with the tool's name and version, in `derived/mathml.jsonl`, and the form's reading of each formula in `derived/math.jsonl` ([12.7](12-package.md#127-derived-files)). + +## 9.4 The math form + +`scienting-math/1` is the standard form of a formula in a paper: one canonical TeX spelling, a tree read from it, a key for each symbol, and the issues a reader should look at. +A converter writes a package's equations and math runs in it, a reader renders and searches it, and a claim store reads a paper's equations and its own claims through it, so the two can be matched. +papermath is its reference implementation, and what papermath computes is what the form says; this section states what the specification relies on. +The identifier names a spelling of formulas, not a file format, so [registry/formats.json](../registry/formats.json) does not list it; its readings travel in a package's `derived/math.jsonl`. +The form reads TeX only. + +**Canonical TeX** respells and never reinterprets, so it renders as the source did apart from presentation: + +1. Unicode NFC, and a math character a converter handed over becomes its command: `−` is `-`, `≤` is `\leq`, `α` is `\alpha`. +2. Space is dropped, except inside text and where a command needs one before a letter (`\sin x`). +3. One spelling per meaning: `\le` is `\leq`, `\dfrac` and `\tfrac` are `\frac`, `\textrm` is `\text`, `x^{\prime}` is `x'`. +4. Presentation is dropped: `\displaystyle` and the other styles, `\big` and its sizes, `\color`, and `\left` and `\right` around ordinary delimiters. +5. Every argument and script is braced, and on one base the subscript comes first: `x_i^2` is `x_{i}^{2}`, `\frac12` is `\frac{1}{2}`. +6. `\tag` is lifted out as the label, and `\label`, `\nonumber` and `\notag` are dropped. +7. An environment goes by one name: `align`, `split` and `multline` are `aligned`, `equation` and `gather` are `gathered`. +8. A command outside the form's vocabulary is kept as written and reported. + +Spelling a canonical source again gives it back unchanged. + +**The tree** is the canonical TeX read as JSON. +A node is an array whose first item is a string, its kind; its other items are its parts: strings, null for a part that is absent, nodes, and groups, which are arrays whose first item is not a string. +Brackets group and leave no node. + +| Kind | Parts | Is | +| --- | --- | --- | +| `num` | The digits | A number as printed, a decimal comma read as `.`. | +| `sym` | The key | A symbol. | +| `text` | The words | Words in the formula. | +| `const`, `oper` | The command | A constant, such as `\infty`, or an operator, such as `\partial`. | +| `dots`, `empty` | None | An ellipsis; nothing. | +| `add` | The terms | A sum, in which a subtracted term is a `neg` node, ± a `pm` and ∓ an `mp`, each with one part. | +| `mul` | The factors | A product, by juxtaposition or `\cdot`. | +| `times`, `div`, `op` | Two operands, after the operator's command for `op` | `\times`; `/` or `\div`; any other binary operator. | +| `frac`, `binom`, `sqrt`, `root` | Numerator and denominator; the radicand, after the index for `root` | Fractions, binomials and roots. | +| `pow`, `sub`, `prime`, `fact` | The base, and the script for `pow` and `sub` | Scripts, primes and factorials on what is not a symbol. | +| `apply` | The function's command, its argument, and its subscript if any | A named function: `\cos(x)`, `\log_{2} n`, `\max_{i} x_{i}`. | +| `lim` | The command, what is under it, the body | `\lim`, `\liminf` and `\limsup`. | +| `sum`, `prod`, `coprod`, `int`, `iint`, `iiint`, `oint`, `bigcup`, `bigcap` | The lower limit, the upper limit, the body | A big operator, whose body runs to the next sign outside brackets. | +| `abs`, `norm`, `angle`, `floor`, `ceil`, `braces` | The inside | A fence. | +| `interval` | The opening and closing brackets, the inside | Brackets that do not match, as in `[0, 1)`. | +| `rel` | Operands with a relation between each two | A relation or a chain; a continued line starts with null. | +| `list` | The formulas | Formulas side by side, at `,`, `;`, `\quad` or `\qquad` outside brackets. | +| `lines` | The lines | The lines of an `aligned` or `gathered` block. | +| `cases` | A group of value and condition for each case | Cases, a condition being null when there is none. | +| `matrix` | The environment, then a group of rows, each a group of cells | A matrix or array. | +| `opaque` | Canonical TeX | What the tree does not read. | + +A tree with no `opaque` node is complete. +A later minor version of the form may add a kind, and a consumer that meets a kind it does not know reads that node as it reads `opaque`; `math.schema.json` therefore takes any lowercase kind, and its description lists these. + +**A symbol** is a base, which is a letter, a Greek letter, or a font or accent command on a group (`\mathbf{F}`, `\hat{H}`), with any subscript, any superscript that names rather than raises (`R^{*}`, `A^{\dagger}`, `E^{\text{LJ}}`), and primes. +A text group alone is a symbol too, unless it holds several words, is a condition in `cases`, or is the set under a big operator. +A symbol's key is its canonical TeX with `\mathrm` spelled `\text`, `\ast` spelled `*`, `\prime` spelled `'`, and no space inside text: `K_r` and `K_{r}` have the key `K_{r}`, and `r_{\mathrm{eq}}` has the key `r_{\text{eq}}`. +The key is the one a lode claim store gives a symbol, so a symbol in a paper and the same symbol in a claim have one key, and a `symbol` annotation names its symbol by it ([18.4](18-layers.md#184-bodies)). + +**Issues** never stop the reading; each is `{code, message, at}`, with `at` the canonical TeX it is about. + +| Code | Means | +| --- | --- | +| `unbalanced` | A brace, bracket or fence does not close; with a brace, the whole formula is one `opaque` node. | +| `unknown-command` | A command outside the vocabulary, kept as written. | +| `label-or-power` | A superscript that may be a label written without `\text`, such as `E^{LJ}`, read as a power. | +| `script-on-group` | A script on a braced group rather than on a symbol. | +| `opaque` | A part the tree does not read. | +| `infix-fraction` | `\over`, `\choose` or `\atop`, rewritten as `\frac` or `\binom`. | + +A later minor version may add a code. + +**In a package**, the canonical source is the equation's or the math run's `source`, with `form` beside it ([3.7](03-nodes.md#37-math-forms)), and the rest is derived: one line of `derived/math.jsonl` per formula a pinned tool read, `{node, run, tool, form, tree, symbols, complete, issues, label?}`. +`run` is the index of the math run among the node's runs, or null for an equation or equation line, as in `derived/mathml.jsonl`; `symbols` lists the keys in the order the formula first writes them; `label` is the label the tool lifted from a `\tag`, omitted when the source has none, as a canonical source never does. +The schema is `math.schema.json`. +A validator checks what it can without implementing the form: that each line names a formula of the package (`doc.derived.node`), and that the formula is TeX, `complete` is true exactly when the tree has no `opaque` node, and `symbols` holds each `sym` key of the tree once (`doc.derived.math`). +`fixtures/packages/hale2026harmonic-damb/derived/math.jsonl` holds papermath's reading of the example's equation, of a math run that is one symbol, and of the run `_2` of "H₂O", which the tree cannot read alone. diff --git a/spec/12-package.md b/spec/12-package.md index b7d6cf0..dc66705 100644 --- a/spec/12-package.md +++ b/spec/12-package.md @@ -32,6 +32,7 @@ Paths use `/`, are relative, have no empty, `.` or `..` segment, no backslash an No two paths are the same once case-folded, so a package reads alike on every file system. A package folder carries no symbolic links, and a consumer refuses one that does, as it refuses a package whose files exceed its own size limit. In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit. +It refuses in the same way (`package.zip.entry`) an archive it cannot read, such as one whose entries need a later version of the zip format than it implements, or whose deflated data does not inflate. A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields; `fixtures/packages/tiny2026tiny-j4qm.sct.zip` is such an archive. ## 12.3 The manifest @@ -50,7 +51,7 @@ Every stage that uses a model or an outside service records where it ran ([14](1 | Policy | Means | | --- | --- | | `full` | Every leaf keeps its content. | -| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). | +| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects, datasets or derived files; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). | A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper. Its text lives in the team's private asset store, and only key holders can check its values. @@ -63,8 +64,17 @@ Bytes that live only in the asset store are listed in the manifest with `present ## 12.7 Derived files -`derived/` holds what a pinned tool makes from the package, such as `derived/mathml.jsonl`, whose lines are `{node, run, tool, mathml}` with `run` null for an equation node. -Nothing derived is content, and a consumer may delete and rebuild it. +`derived/` holds what a pinned tool makes from the package. +Each line of a derived file names a formula by `node` and `run`, the index of a math run among the node's runs, or null for an equation or equation line, and the `tool` (`{name, version}`) that made it. + +| Path | Lines | Schema | +| --- | --- | --- | +| `derived/mathml.jsonl` | `{node, run, tool, mathml}`: the formula rendered as MathML. | `mathml.schema.json` | +| `derived/math.jsonl` | `{node, run, tool, form, tree, symbols, complete, issues, label?}`: the formula read in a math form ([9.4](09-figures-tables-equations.md#94-the-math-form)). | `math.schema.json` | + +Nothing derived is content, and a consumer may delete and rebuild it; a validator does not recompute a derived file, since it does not run the tool, but checks its lines against their schema and that each names a formula of the package (`doc.derived.node`), and that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`). +A derived file reads the text, so a checks-only package carries none ([12.5](#125-text-policies)). +A correction can delete a formula, change it or move its run, so a producer that applies one rebuilds the derived files or leaves them out. ## 12.8 Media type and extension diff --git a/spec/18-layers.md b/spec/18-layers.md index 7561e51..ed06dc3 100644 --- a/spec/18-layers.md +++ b/spec/18-layers.md @@ -43,8 +43,8 @@ Replies are annotations that target annotations, so discussions are threads anch | `kind` | W3C type | Adds | Says | | --- | --- | --- | --- | | `gloss` | `TextualBody` | `audience` | An explanation at `novice`, `student` or `expert` level. | -| `claim-link` | `SpecificResource` | `source` (`lode:<tag>`), `pin`, `role` | That the span `states`, `uses` or `disputes` a claim. | -| `symbol` | `SpecificResource` | `source`, `pin`, `symbol` | That this occurrence of a symbol is a symbol claim. | +| `claim-link` | `SpecificResource` | `source`, `pin`, `role` | That the span `states`, `uses` or `disputes` a claim. | +| `symbol` | `SpecificResource` | `source`, `pin`, `key`, `symbol` | That this occurrence of a symbol, the one whose key is `key`, is the symbol a symbol claim defines. | | `comment` | `TextualBody` | `category` | A remark, opinion or interpretation. | | `caveat` | `TextualBody` | `category` | A warning a reader should see. | | `question` | `TextualBody` | `for`, `category` | A question for a `reader`, a `group` or the `authors`. | @@ -58,6 +58,15 @@ Replies are annotations that target annotations, so discussions are threads anch A textual body has a `value`, and optionally a `format` (`text/plain` or `text/markdown`) and a `language`. +The `source` of a `claim-link` or `symbol` body is a claim in a lode claim store, written as lode writes a reference: `lode:` and the claim's tag, eight Crockford base32 capitals (the digits and the letters without I, L, O and U), as in `lode:272XTX7C`, or `lode:<host>/<tag>` for a claim in another store, `<host>` being a domain name. +A tag never changes, while a claim's slug, such as `amber/k-bond`, may be renamed, so a slug is never a reference. +The optional `pin` fixes the claim as the annotator read it: the same reference, `@`, and the first six lowercase hex digits of the claim's statement hash, as in `lode:272XTX7C@02f47b`. +`common.schema.json` defines both, as `claimRef` and `claimPin`. + +A `symbol` body names its symbol twice, as the target's span and by `key`, the symbol's key in the math form `scienting-math/1` ([9.4](09-figures-tables-equations.md#94-the-math-form)), which a claim store gives the symbol a symbol claim defines: `K_{r}` for a span `$K_r$`. +So a reader finds every occurrence of a symbol across a library by its key, however its source spaced or braced it, and a claim store matches the annotation to its claim without parsing the span. +The span is still required, since the key says which symbol and the span says where; `symbol`, optional, is the symbol as the span writes it. + A `comment`, `caveat` or `question` body MAY carry a `category`, which says what it is about, from a fixed list: | `category` | Says | @@ -77,4 +86,4 @@ The store's `annotations` table keeps the category beside the body's kind ([13.4 Nothing a model writes is ever stored as source text. A model may propose a `description`, and like any annotation it stays `proposed` until a person confirms it; an accessible export uses confirmed descriptions only, and a figure with none has no description. Other text a model writes about a span, such as a summary, is a `generated` annotation, which names its `generator`. -`fixtures/records/layer/` holds a layer with one annotation of each common kind, each comment, caveat and question with its category, and the receipt that confirms some of them. +`fixtures/records/layer/` holds a layer with one annotation of each common kind, each comment, caveat and question with its category, a claim link and a symbol that reference a lode claim and pin it, and the receipt that confirms some of them. diff --git a/spec/20-projects.md b/spec/20-projects.md index c2ce452..371d4e8 100644 --- a/spec/20-projects.md +++ b/spec/20-projects.md @@ -7,7 +7,7 @@ A project, `scienting-project/1`, is the unit a team works in. | `key`, `store`, `name`, `description` | Its identity. | | `members` | People with a `role` (`owner`, `editor`, `reader` or `guest`) and the roster keys they sign with. | | `documents` | The project's documents, each with the `reason` it is included, `tags`, and a reading `status` per member: `unread`, `reading` or `read`. | -| `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group. | +| `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group; a claim step's `ref` is a claim reference, `lode:<tag>` ([18.4](18-layers.md#184-bodies)). | | `layers`, `datasets`, `views`, `drafts` | What the project holds. | | `claims` | The claim namespace or bundle it uses, as the `lode` setting. | | `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), the stage requirements in use, as the `id` of their stage file ([14](14-rights.md)), and the project's transparency `log`: its `origin`, the roster keys that `witness` it, and the `threshold` of witnesses a checkpoint needs ([23](23-log.md)). | diff --git a/spec/22-conformance.md b/spec/22-conformance.md index 79bfa62..b8669cd 100644 --- a/spec/22-conformance.md +++ b/spec/22-conformance.md @@ -22,7 +22,7 @@ A validator checks, in order, and reports each failure as `path:line: rule: mess 3. The tree ([3](03-nodes.md)), runs and references between nodes, the counts, and the keys ([2.4](02-model.md#24-work-and-document-keys)). 4. The policy and hashing scheme ([12.5](12-package.md#125-text-policies)). 5. The five values of every node, the root and `meta_hash`, when the text is present and the key, if any, is known. -6. The ID table, the corrections and the history root, the witnesses, pages, locations and references, and every file's order. +6. The ID table, the corrections and the history root, the witnesses, pages, locations and references, the formulas derived files name, and every file's order. 7. Receipts: IDs, roots, documents, node order, signatures against the package's roster, spot-check keys, seeds and samples, and the fidelity distribution. ## 22.3 Schemas @@ -39,7 +39,8 @@ Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is | Prefix | Covers | | --- | --- | | `package.` | The manifest, paths, files and schemas. | -| `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.` | The tree, node content and runs. | +| `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.`, `doc.math.` | The tree, node content and runs. | +| `doc.derived.` | Derived files ([12.7](12-package.md#127-derived-files)). | | `doc.hash.` | The five values, the root and the metadata hash. | | `doc.id.`, `doc.order.` | The ID table and file order. | | `doc.policy.` | Text policies and keyed hashing. | diff --git a/spec/23-log.md b/spec/23-log.md index 2b5d288..7f1ce0e 100644 --- a/spec/23-log.md +++ b/spec/23-log.md @@ -35,6 +35,7 @@ Q2xlYXIgdGV4dCBpcyBub3QgYSByb290IGhhc2guLi4uLi4uLi4= The origin names the log, and a team's logs are named `scient.ing/log/<store>` or under a domain the team controls. A signature line is an em dash (U+2014), a space, the signer's roster key, a space, and Base64 of the signer's four-byte key hash followed by the signature. The key hash is the first four bytes of SHA-256 of the signer's name, a newline, the byte 0x01 and the Ed25519 public key; the signature is Ed25519 over the note's first three lines, each ending in a newline. +Every line of a note, the last signature line included, ends in one U+000A LINE FEED; a note that ends a line with anything else, such as U+000B, U+0085 or U+2028, which some libraries also take for line breaks, or that lacks its final U+000A, is malformed. A note parsed into JSON has `format`, `origin`, `size`, `root` as hex, `signatures` as `{name, key_hash, value}` and the `note` text itself; the schema is `checkpoint.schema.json`. `checkpoint.jsonl` gives a checkpoint over the example paper's receipts and its verification. diff --git a/src/paperspec/cli/build.py b/src/paperspec/cli/build.py index 47cf728..9ca2633 100644 --- a/src/paperspec/cli/build.py +++ b/src/paperspec/cli/build.py @@ -633,11 +633,29 @@ def build_hale(out): mathml = ('<math display="block"><msub><mi>E</mi><mtext>bond</mtext></msub><mo>=</mo><mfrac><mn>1</mn><mn>2' '</mn></mfrac><msub><mi>K</mi><mi>r</mi></msub><msup><mrow><mo>(</mo><mi>r</mi><mo>−</mo><msub><mi>r' '</mi><mtext>eq</mtext></msub><mo>)</mo></mrow><mn>2</mn></msup></math>') + # The scienting-math/1 reading of three formulas (spec/09 §9.4): what papermath 0.1.0 prints for each source, + # copied here because the oracle does not implement the form, and checked by the validator only for what it + # can check without it (spec/12 §12.7). The run `_2` of "H₂O" has no base, so its tree is one opaque node. + math_tool = {"name": "papermath", "version": "0.1.0"} + s2_runs, s3_runs = final_doc.nodes[names.ids["@s2"]]["content"], final_doc.nodes[names.ids["@s3"]]["content"] + math = [ + {"node": eq, "run": None, "tree": [ + "rel", ["sym", "E_{\\text{bond}}"], "=", + ["mul", ["frac", ["num", "1"], ["num", "2"]], ["sym", "K_{r}"], + ["pow", ["add", ["sym", "r"], ["neg", ["sym", "r_{\\text{eq}}"]]], ["num", "2"]]]], + "symbols": ["E_{\\text{bond}}", "K_{r}", "r", "r_{\\text{eq}}"], "complete": True, "issues": []}, + {"node": names.ids["@s2"], "run": next(i for i, r in enumerate(s2_runs) if r.get("source") == "K_r"), + "tree": ["sym", "K_{r}"], "symbols": ["K_{r}"], "complete": True, "issues": []}, + {"node": names.ids["@s3"], "run": next(i for i, r in enumerate(s3_runs) if r.get("source") == "_2"), + "tree": ["opaque", "_{2}"], "symbols": [], "complete": False, + "issues": [{"code": "opaque", "message": "a part after the formula's end", "at": "_{2}"}]}, + ] extra = { "assets/%s.png" % image_sha: image, "datasets/%s/datapackage.json" % ds_id: (jcs.dumps(datapackage) + "\n").encode(), "datasets/%s/data.csv" % ds_id: ("\n".join(csv) + "\n").encode(), "datasets/%s/anchors.jsonl" % ds_id: jsonl(anchors), + "derived/math.jsonl": jsonl([dict(m, tool=math_tool, form="scienting-math/1") for m in math]), "derived/mathml.jsonl": jsonl([{"node": eq, "run": None, "tool": {"name": "hand", "version": "0"}, "mathml": mathml}]), } @@ -667,7 +685,7 @@ def build_checks_only(out, hale): n["attrs"].pop("figure_text", None) nodes.append(node_record(n, keyed)) f = dict(hale["files"]) - for p in ("corrections.jsonl", "receipts.jsonl", "roster.json", "derived/mathml.jsonl"): + for p in ("corrections.jsonl", "receipts.jsonl", "roster.json", "derived/math.jsonl", "derived/mathml.jsonl"): f.pop(p, None) for p in [p for p in f if p.startswith("datasets/")]: f.pop(p) @@ -935,8 +953,23 @@ def vectors(out, hale, tiny): tdoc = Doc(tiny["key"], tnodes) tv = hashing.compute(tdoc) own = {nid: hashing.own_content(tdoc, nid, lambda x: tv[x]["text_hash"]) for nid in tdoc.order} + # A sentence whose display equation and math run carry the form their canonical TeX is spelled in (spec/03 §3.7): + # `form` is in no own content and no projection, so the values are those of the same nodes without it. + form = "scienting-math/1" + fnodes = [{"id": "aaaaaaa", "kind": "document", "parent": None, "ord": 0}, + {"id": "bbbbbbb", "kind": "section", "parent": "aaaaaaa", "ord": 0, "attrs": {"level": 1}}, + {"id": "ccccccc", "kind": "paragraph", "parent": "bbbbbbb", "ord": 0}, + {"id": "ddddddd", "kind": "sentence", "parent": "ccccccc", "ord": 0, "content": [ + T("The energy is "), {"kind": "equation", "node": "eeeeeee"}, T(" where "), + dict(M("K_{r}"), form=form), T(" is the force constant.")]}, + {"id": "eeeeeee", "kind": "equation", "parent": "ddddddd", "ord": 0, + "content": {"notation": "tex", "source": "E=\\frac{1}{2}K_{r}x^{2}", "form": form}}] + fdoc = Doc("vectors", fnodes) + fv = hashing.compute(fdoc) + fown = {nid: hashing.own_content(fdoc, nid, lambda x: fv[x]["text_hash"]) for nid in fdoc.order} wv(out, "hashes.jsonl", [{"id": "tiny2026example", "doc": tiny["key"], "nodes": tnodes, - "own_content": own, "values": tv}]) + "own_content": own, "values": tv}, + {"id": "math-form", "doc": "vectors", "nodes": fnodes, "own_content": fown, "values": fv}]) # Addresses. The example keys are real work keys: Cornell et al. (1995) by its DOI, a fictional preprint. cornell = "cornell1995second-" + K.suffix("doi:10.1021/ja00124a002") dong = "dong2026machine-" + K.suffix("doi:10.5555/example.dong.2026") @@ -1359,6 +1392,8 @@ def ann_hash(a): TAG = "tag:scient.ing,2026:" +# The claim amber/k-bond of lode's example store, by its tag; lode prints its pin as lode:272XTX7C@02f47b. +LODE_K_BOND = "lode:272XTX7C" # A model's provenance: enough to name the weights and the instructions it ran with (all illustrative). MODEL = {"name": "example-model", "provider": "Example Models (illustrative)", "version": "2026-09", "weights": sha256_hex("illustrative weights"), "prompt": sha256_hex("illustrative review prompt"), @@ -1477,7 +1512,8 @@ def records(out, hale): ann({"type": "TextualBody", "kind": "gloss", "audience": "novice", "format": "text/markdown", "language": "en", "value": "A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."}, target(anchor("@s2", "bond force constant")), "describing", "confirmed"), - ann({"type": "SpecificResource", "kind": "claim-link", "source": "lode:amber/k-bond", "pin": "illustrative-pin", + # lode's references (spec/18 §18.4): the claim amber/k-bond by its tag, pinned to its statement hash. + ann({"type": "SpecificResource", "kind": "claim-link", "source": LODE_K_BOND, "pin": LODE_K_BOND + "@02f47b", "role": "states"}, target(anchor("@s2")), "linking", "confirmed"), # Comments, caveats and questions carry the category a producer that supports categories always sets # (spec/18 §18.4). @@ -1508,6 +1544,10 @@ def records(out, hale): anns.append(ann({"type": "TextualBody", "kind": "caveat", "category": "context", "value": "The band position is cited, not measured here; reference 2 is not in the library."}, target(anchor("@s3", "1595")), "commenting", created="2026-10-11T19:10:00Z")) + # A symbol: the span $K_r$ is the symbol the claim defines, named by its scienting-math/1 key (spec/09 §9.4). + anns.append(ann({"type": "SpecificResource", "kind": "symbol", "source": LODE_K_BOND, "pin": LODE_K_BOND + "@02f47b", + "key": "K_{r}", "symbol": "K_r"}, + target(anchor("@s2", "$K_r$")), "identifying", created="2026-10-11T19:15:00Z")) confirmed = [a for a in anns if a["status"] == "confirmed"] lr = RC.seal({"format": "scienting-receipt/1", "doc": HALE, "rev": rev, "level": "F3", "scope": "targeted:layer", "nodes": [], @@ -1559,7 +1599,7 @@ def records(out, hale): "paths": [{"id": "onboarding", "title": "First week", "steps": [ {"type": "document", "ref": QHALE, "note": "Read the introduction."}, {"type": "node", "ref": QHALE + "#" + I["@tab1"], "note": "Then the table."}, - {"type": "claim", "ref": "lode:amber/k-bond"}]}], + {"type": "claim", "ref": LODE_K_BOND, "note": "The bond force constant, amber/k-bond."}]}], "layers": [layer_id], "datasets": ["ds-hale-bond-params", "ds-hale-ctct-scan"], "views": ["ctct-scan"], "drafts": [STORE + "/" + DRAFT], "claims": {"lode": "amber"}, "settings": {"digit_rule": "anchored-sections", "drift_threshold": "0.25", diff --git a/src/paperspec/cli/check.py b/src/paperspec/cli/check.py index 5c00954..d8e7323 100644 --- a/src/paperspec/cli/check.py +++ b/src/paperspec/cli/check.py @@ -225,6 +225,23 @@ def check_records(): # The fixtures' producer supports categories, so it sets one on every comment, caveat and question (spec/18 §18.4). ok("records", all("category" in a["body"] for a in anns if a["body"]["kind"] in ("comment", "caveat", "question")), "a comment, caveat or question of the layer has no category") + # A claim's pin names the claim its source does (spec/18 §18.4), and a symbol names its symbol by its key. + claims = [a["body"] for a in anns if a["body"]["kind"] in ("claim-link", "symbol")] + ok("records", {b["kind"] for b in claims} == {"claim-link", "symbol"}, "the layer lacks a claim link or a symbol") + for b in claims: + ok("records", "pin" not in b or b["pin"].startswith(b["source"] + "@"), + "a %s pins another claim than its source: %s, %s" % (b["kind"], b["source"], b.get("pin"))) + common = STORE.load("common.schema.json")["$id"] + + def errors(defn, value): + return list(Draft202012Validator({"$ref": common + "#/$defs/" + defn}, registry=STORE.registry()).iter_errors(value)) + for bad in ("lode:amber/k-bond", "lode:272xtx7c", "lode:272XTXIC", "lode:example/272XTX7C", "lode:272XTX7C@02f47b"): + ok("records", errors("claimRef", bad) != [], "%s is taken for a claim reference" % bad) + for bad in ("illustrative-pin", "lode:272XTX7C", "lode:272XTX7C@02F47B", "lode:272XTX7C@02f47"): + ok("records", errors("claimPin", bad) != [], "%s is taken for a claim pin" % bad) + for good in ("lode:272XTX7C", "lode:claims.example.org/272XTX7C"): + ok("records", errors("claimRef", good) == [] and errors("claimPin", good + "@02f47b") == [], + "%s is refused as a claim reference" % good) def state_of(pkg): @@ -265,6 +282,63 @@ def check_packages(): "CSV value differs from its cell: %s/%s" % (va["row"], va["field"])) for m in jl(os.path.join(hp, "derived", "mathml.jsonl")): valid("packages", "mathml.schema.json", m, "derived/mathml.jsonl") + for m in jl(os.path.join(hp, "derived", "math.jsonl")): + valid("packages", "math.schema.json", m, "derived/math.jsonl") + + +def check_derived(): + """Derived files are checked, never recomputed (spec/12 §12.7): a line names a formula of the package, and a math + line's formula is TeX and its `complete` and `symbols` agree with its tree (spec/09 §9.4); a tree kind the + schema does not list is accepted, so the form can add kinds; a checks-only package carries no derived file. + + No broken package carries these, so each is a variant of the main example's derived/math.jsonl. + """ + hp = p("fixtures", "packages", HALE) + manifest = js(os.path.join(hp, "MANIFEST.json")) + files, absent = {}, [f for f in manifest["files"] if f.get("present", True) is False] + for f in manifest["files"]: + if f.get("present", True): + with open(os.path.join(hp, f["path"]), "rb") as fh: + files[f["path"]] = fh.read() + math = jl(os.path.join(hp, "derived", "math.jsonl")) + eq, run, opaque = math[0], next(m for m in math if m["run"] is not None), next(m for m in math if not m["complete"]) + sentence = next(n for n in jl(os.path.join(hp, "nodes.jsonl")) if n["kind"] == "sentence") + cases = [ + ("as built", math, set()), + ("a new tree kind", [dict(eq, tree=["rel", ["sym", "x"], "=", ["tensor", ["sym", "y"]]], symbols=["x", "y"])] + + math[1:], set()), + ("unknown node", [dict(eq, node="zzzzzzz")], {"doc.derived.node"}), + ("not a math run", [dict(run, run=0)], {"doc.derived.node"}), + ("run past the end", [dict(run, run=99)], {"doc.derived.node"}), + ("run null on a sentence", [dict(run, node=sentence["id"], run=None)], {"doc.derived.node"}), + ("complete with an opaque node", [dict(opaque, complete=True)], {"doc.derived.math"}), + ("incomplete without one", [dict(eq, complete=False)], {"doc.derived.math"}), + ("a symbol missing", [dict(eq, symbols=eq["symbols"][1:])], {"doc.derived.math"}), + ("a symbol not in the tree", [dict(eq, symbols=eq["symbols"] + ["q"])], {"doc.derived.math"}), + ("a tree that is not a node", [dict(eq, tree=[["sym", "x"]])], {"package.schema"}), + ("a kind that is not a name", [dict(eq, tree=["Sym", "x"])], {"package.schema"}), + ("another form", [dict(eq, form="scienting-math/2")], {"package.schema"}), + ("an issue without its text", [dict(opaque, issues=[{"code": "opaque", "message": "x"}])], {"package.schema"}), + ] + for name, lines, want in cases: + files["derived/math.jsonl"] = PK.jsonl(lines) + with tempfile.TemporaryDirectory() as tmp: + PK.write(tmp, files, absent) + rules = {e["rule"] for e in PK.validate(tmp, STORE).errors} + ok("packages", rules == want, "derived math, %s: expected %s, got %s" % (name, sorted(want), sorted(rules))) + # A checks-only package withholds the text, and a derived file reads it (spec/12 §12.5). + co = p("fixtures", "packages", HALE + ".checks-only") + team_key = crypto.team_key(bytes.fromhex(open(p("fixtures", "keys", "smithlab-team-secret.hex")).read().strip())) + with tempfile.TemporaryDirectory() as tmp: + pkg = os.path.join(tmp, "pkg") + shutil.copytree(co, pkg) + cm = js(os.path.join(pkg, "MANIFEST.json")) + cfiles = {f["path"]: open(os.path.join(pkg, f["path"]), "rb").read() for f in cm["files"] if f.get("present", True)} + cfiles["derived/math.jsonl"] = PK.jsonl([math[0]]) + shutil.rmtree(pkg) + PK.write(pkg, cfiles, [f for f in cm["files"] if f.get("present", True) is False]) + rules = {e["rule"] for e in PK.validate(pkg, STORE, team_key).errors} + ok("packages", "doc.policy.withheld" in rules, "a checks-only package with a derived file: %s" % sorted(rules)) def _zip(path, entries): @@ -342,6 +416,29 @@ def check_zips(): _zip(os.path.join(tmp, "empty.zip"), []) ok("zips", {e["rule"] for e in PK.validate_zip(os.path.join(tmp, "empty.zip"), STORE).errors} == {"package.zip.entry"}, "an empty archive is refused") + # Archives the reader cannot read are refused with the rule, never with the zipfile module's own error: an + # entry that needs a later zip version (NotImplementedError), and deflated bytes that do not inflate + # (zlib.error), as `paperspec stress --seed 1` found. + with zipfile.ZipFile(os.path.join(tmp, "zip-version.zip"), "w") as zf: + for name, data, _ in entries: + zi = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0)) + zi.extract_version = 98 + zf.writestr(zi, data) + with zipfile.ZipFile(os.path.join(tmp, "bad-deflate.zip"), "w", zipfile.ZIP_DEFLATED) as zf: + for name, data, _ in entries: + zf.writestr(zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0)), data, zipfile.ZIP_DEFLATED) + with zipfile.ZipFile(os.path.join(tmp, "bad-deflate.zip")) as zf: + zi = next(i for i in zf.infolist() if i.compress_size > 8) + start = zi.header_offset + 30 + len(zi.filename.encode("utf-8")) + len(zi.extra) + with open(os.path.join(tmp, "bad-deflate.zip"), "r+b") as fh: + fh.seek(start) + fh.write(b"\xff" * 8) + for name in ("zip-version", "bad-deflate"): + try: + rules = {e["rule"] for e in PK.validate_zip(os.path.join(tmp, name + ".zip"), STORE).errors} + except Exception as e: # noqa: BLE001 + rules = {"raised %r" % e} + ok("zips", rules == {"package.zip.entry"}, "%s: expected package.zip.entry, got %s" % (name, sorted(rules))) outside = [f for f in os.listdir(tmp) if not f.endswith(".zip")] ok("zips", not outside, "reading archives wrote outside their folders: %s" % outside) @@ -553,6 +650,24 @@ def check_vectors(): ok("vectors", vals == c["values"], "hashes " + c["id"]) for nid, o in c["own_content"].items(): ok("vectors", sha256_hex(jcs.dumpb(o)) == vals[nid]["own_hash"], "own content " + nid) + # A math form enters no content value (spec/05 §5.2): the values of the nodes with `form` are those without it, + # and the nodes validate; the schema takes `form` only beside the notation tex. + fc = next(c for c in V("hashes.jsonl") if c["id"] == "math-form") + def unformed(n): + c = n.get("content") + if isinstance(c, dict): + c = {k: v for k, v in c.items() if k != "form"} + elif isinstance(c, list): + c = [{k: v for k, v in r.items() if k != "form"} for r in c] + return dict(n, content=c) if "content" in n else n + formed = sum(json.dumps(n).count('"form"') for n in fc["nodes"]) + ok("vectors", formed == 2 and hashing.compute(Doc(fc["doc"], [unformed(n) for n in fc["nodes"]])) == fc["values"], + "a math form changes a content value") + for n in fc["nodes"]: + valid("vectors", "node.schema.json", dict(n, **fc["values"][n["id"]]), "hashes math-form " + n["id"]) + eq = next(n for n in fc["nodes"] if n["kind"] == "equation") + typst = dict(eq, content=dict(eq["content"], notation="typst"), **fc["values"][eq["id"]]) + ok("vectors", schema_errors("node.schema.json", typst) != [], "the schema takes a form beside the notation typst") for c in V("addresses.jsonl"): ok("vectors", AN.parse_address(c["address"]) == c["parsed"], "address " + c["id"]) if c["parsed"]: @@ -1055,6 +1170,16 @@ def check_log(): ok("log", not LG.accept_checkpoint(note, roster, dict(settings, origin="other"))[0], "another origin was accepted") alone = LG.sign_checkpoint(settings["origin"], entries, [("wit", __import__("paperspec.cli.build", fromlist=["secret"]).secret("wit"))]) ok("log", not LG.accept_checkpoint(alone, roster, settings)[0], "a checkpoint without the team's key was accepted") + # Every line of a note ends in U+000A and nothing else (spec/23 §23.2): a note whose last line break, or the break + # between its signature lines, is another line separator is malformed, as `paperspec stress --seed 2` found. + head, _, sigs = note.partition("\n\n") + lines = sigs.split("\n")[:-1] + ok("log", len(lines) >= 2 and LG.parse_checkpoint(head + "\n\n" + "\n".join(lines) + "\n") is not None, + "the checkpoint does not have two signature lines to separate") + for sep in ("\x0b", "\x0c", "\x1c", "\x1d", "\x1e", "\x85", "
", "
", "\r", ""): + ok("log", LG.parse_checkpoint(note[:-1] + sep) is None, "a note ending in %r instead of a newline parses" % sep) + ok("log", LG.parse_checkpoint(head + "\n\n" + sep.join(lines) + "\n") is None, + "a note whose signature lines are separated by %r parses" % sep) # The ML-DSA-65 receipt verifies under dana's key, and fails under a tampered payload. rec = js(os.path.join(R, "receipt-ml-dsa.json")) ok("log", rec["signature"]["scheme"] == "ml-dsa-65" and RC.verify_signature(rec, roster), "the ML-DSA receipt does not verify") @@ -1187,6 +1312,7 @@ def run(args): check_packages() check_zips() check_owner_revocation() + check_derived() check_unsafe_paths() check_incremental() check_gate() diff --git a/src/paperspec/cli/stress.py b/src/paperspec/cli/stress.py index 041c6fa..d486b9f 100644 --- a/src/paperspec/cli/stress.py +++ b/src/paperspec/cli/stress.py @@ -189,7 +189,12 @@ def random_correction(rng, st, doc): c.update(target=rng.choice(cells), attr=rng.choice(["colspan", "rowspan", "role", "column", "level"]), value=rng.choice([1, 2, 3, "2", -1, "data", "row-header", "boss", None])) elif op == "delete" and len(doc.order) > 8: - c.update(target=rng.choice(doc.order[1:])) + # Never below eight nodes: deleting the body section would leave a document with nothing to correct, and the + # rest of the walk would only be refused. + targets = [nid for nid in doc.order[1:] if len(doc.order) - len(doc.preorder(nid)) >= 8] + if not targets: + return None + c.update(target=rng.choice(targets)) elif op == "reclassify": c.update(target=rng.choice(doc.order), kind=rng.choice(["footnote", "quote", "paragraph", "caption", "section"])) elif op == "insert-text" and sentences: @@ -481,8 +486,11 @@ def stress_packages(rng, rounds): data[k] = (data[k] + rng.randrange(1, 256)) % 256 open(full, "wb").write(data) elif mode == "truncate": + # Strictly shorter: a cut at the full length would leave the file as it was, and the package valid. data = open(full, "rb").read() - open(full, "wb").write(data[:rng.randrange(len(data) + 1)]) + if not data: + continue + open(full, "wb").write(data[:rng.randrange(len(data))]) elif mode == "delete": os.remove(full) elif mode == "add": diff --git a/src/paperspec/hashing.py b/src/paperspec/hashing.py index 883d79b..ff7a2b6 100644 --- a/src/paperspec/hashing.py +++ b/src/paperspec/hashing.py @@ -209,10 +209,14 @@ def lossy(doc, nid): def _runs_for_own(runs): + """The runs as own content (spec/05 §5.2): an equation run without the ID it names, a math run without its + `form`, which says how the source is spelled and adds nothing to it (spec/03 §3.7).""" out = [] for r in runs: if r["kind"] == "equation": out.append({"kind": "equation"}) + elif r["kind"] == "math" and "form" in r: + out.append({k: v for k, v in r.items() if k != "form"}) else: out.append(r) return out diff --git a/src/paperspec/log.py b/src/paperspec/log.py index 32b0fd3..5cf1801 100644 --- a/src/paperspec/log.py +++ b/src/paperspec/log.py @@ -63,15 +63,22 @@ def sign_checkpoint(origin, entries, signers): def parse_checkpoint(text): - """The origin, size, root (hex) and [(name, key hash, signature)] of a note, or None when malformed.""" + """The origin, size, root (hex) and [(name, key hash, signature)] of a note, or None when malformed. + + Every line ends in one U+000A and nothing else (spec/23 §23.2): a note is split on U+000A alone, never with + `str.splitlines`, which also ends a line at U+000B, U+000C, U+001C to U+001E, U+0085, U+2028 and U+2029 and + would accept a note whose last line break was replaced by one of them. + """ try: body, _, sigs = text.partition("\n\n") origin, size, root_b64 = body.split("\n") rt = _b64(root_b64) if rt is None or len(rt) != 32 or not size.isdigit() or str(int(size)) != size or not origin: return None + if sigs and not sigs.endswith("\n"): + return None out = [] - for ln in sigs.splitlines(): + for ln in sigs.split("\n")[:-1]: if not ln.startswith("— "): return None _, name, blob = ln.split(" ", 2) diff --git a/src/paperspec/package.py b/src/paperspec/package.py index ec6c932..02f589c 100644 --- a/src/paperspec/package.py +++ b/src/paperspec/package.py @@ -11,9 +11,11 @@ import os import posixpath import re import stat +import struct import tempfile import unicodedata import zipfile +import zlib from . import crypto, hashing, jcs, keys as K, receipts as R, vrf from .anchors import ADDRESS_RE @@ -42,6 +44,11 @@ JSON = { "roster.json": "roster.schema.json", } REQUIRED = ["document.json", "nodes.jsonl", "witnesses.json", "provenance.json"] +# Derived files (spec/12 §12.7): checked against their schemas and the formulas they name, never recomputed. +DERIVED = { + "derived/math.jsonl": "math.schema.json", + "derived/mathml.jsonl": "mathml.schema.json", +} def line(rec): @@ -253,6 +260,13 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT): rep.add("package.file.empty", p, "an optional JSONL file with no records is omitted") else: data[p] = [] + for p, sch in DERIVED.items(): + data[p] = read_jsonl(os.path.join(root, p)) if p in listed else [] + for i, rec in enumerate(data[p], 1): + for e in schema_validate(store, sch, rec): + rep.add("package.schema", "%s:%d" % (p, i), e) + if p in listed and not data[p]: + rep.add("package.file.empty", p, "an optional JSONL file with no records is omitted") if not rep: return @@ -290,6 +304,8 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT): rep.add("doc.policy.withheld", "nodes.jsonl", "a checks-only package withholds every leaf's content") if scheme == "plain": rep.add("doc.policy.keyed", "document.json", "a checks-only package carries keyed hashes") + for p in sorted(x for x in listed if x.startswith("derived/")): + rep.add("doc.policy.withheld", p, "a checks-only package carries no derived files, which read the text") if scheme == "hmac-team" and key is not None and document["hashing"]["key_id"] != crypto.key_id(key): rep.add("doc.hash.key", "document.json", "key_id does not match the key supplied") @@ -302,6 +318,8 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT): leaves_n = sum(1 for n in nodes if n["kind"] in LEAVES) if document.get("counts") != {"nodes": len(nodes), "leaves": leaves_n}: rep.add("doc.counts", "document.json", "counts do not recompute") + for p in DERIVED: + _check_derived(doc, p, data[p], rep) # Hashes. hasher = hashing.PLAIN if scheme == "plain" else (hashing.Hasher(key) if key else None) @@ -482,6 +500,62 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT): rep.add("doc.fidelity.report", "document.json", "the fidelity distribution does not recompute") +def derived_formula(doc, node, run): + """The formula a derived line names (spec/12 §12.7), as `{notation, source, form?}`, or None when it names none: + the content of an equation or equation line when `run` is null, else the math run at index `run` of a node + with runs.""" + n = doc.nodes.get(node) + if n is None or n.get("withheld"): + return None + if run is None: + return n.get("content") if n["kind"] in MATH_KINDS else None + runs = n.get("content") if n["kind"] in RUN_KINDS else None + if not isinstance(runs, list) or run >= len(runs) or runs[run]["kind"] != "math": + return None + return runs[run] + + +def tree_facts(tree): + """The `sym` keys of a scienting-math/1 tree in document order, and whether it has an `opaque` node (spec/09 + §9.4). A node is an array whose first item is its kind; an array whose first item is not a string is a group of + parts. Walked without recursion, so a deep tree cannot exhaust the stack.""" + keys, opaque, stack = [], False, [tree] + while stack: + x = stack.pop() + if not isinstance(x, list) or not x: + continue + if isinstance(x[0], str): + if x[0] == "opaque": + opaque = True + elif x[0] == "sym" and len(x) > 1 and isinstance(x[1], str): + keys.append(x[1]) + stack.extend(reversed(x[1:])) + else: + stack.extend(reversed(x)) + return keys, opaque + + +def _check_derived(doc, path, lines, rep): + """Each line of a derived file names a formula of the package; a math line's formula is TeX, and its `complete` + and `symbols` agree with its tree (spec/12 §12.7, spec/09 §9.4). The tree itself is the tool's and not + recomputed.""" + for i, rec in enumerate(lines, 1): + where = "%s:%d" % (path, i) + formula = derived_formula(doc, rec["node"], rec["run"]) + if formula is None: + rep.add("doc.derived.node", where, "names no equation, equation line or math run of the package") + continue + if path != "derived/math.jsonl": + continue + if formula["notation"] != "tex": + rep.add("doc.derived.math", where, "the math form reads TeX, and the formula is %s" % formula["notation"]) + keys, opaque = tree_facts(rec["tree"]) + if rec["complete"] == opaque: + rep.add("doc.derived.math", where, "complete is true exactly when the tree has no opaque node") + if sorted(set(keys)) != sorted(rec["symbols"]): + rep.add("doc.derived.math", where, "symbols are not the sym keys of the tree, each once") + + def read_state(root): """The current state of the package at `root`, for the gate (spec/11 §11.4): nodes without their values, the ID table and the corrections applied so far. Validate the package first.""" @@ -514,6 +588,11 @@ ZIP_LIMIT = 1 << 30 _ZIP_DATE = (1980, 1, 1, 0, 0, 0) _FLAG_ENCRYPTED = 0x1 _FLAG_UTF8 = 0x800 +# What `zipfile` raises on an archive it cannot read: a damaged structure (BadZipFile, struct.error, EOFError), a +# version or feature it does not implement (NotImplementedError, as for "zip file version 9.8"), a name that is not +# the UTF-8 its flag claims (UnicodeDecodeError, a ValueError), or deflated bytes that do not inflate (zlib.error). +_ZIP_ERRORS = (zipfile.BadZipFile, NotImplementedError, EOFError, ValueError, RuntimeError, OSError, struct.error, + zlib.error) def is_zip(path): @@ -589,8 +668,8 @@ def read_zip(path, dest, limit=ZIP_LIMIT): """ try: zf = zipfile.ZipFile(path) - except (zipfile.BadZipFile, OSError) as e: - raise Refusal("package.zip.entry", "not a zip archive: %s" % e) + except _ZIP_ERRORS as e: + raise Refusal("package.zip.entry", "not a zip archive this reader can open: %s" % e) with zf: folder = _zip_entries(zf, limit) root = os.path.join(dest, folder) @@ -614,7 +693,7 @@ def read_zip(path, dest, limit=ZIP_LIMIT): if n > zi.file_size or written > limit: raise Refusal("package.zip.size", "%s: holds more than its header declares" % zi.filename) dst.write(chunk) - except (zipfile.BadZipFile, RuntimeError, OSError) as e: + except _ZIP_ERRORS as e: raise Refusal("package.zip.entry", "%s: %s" % (zi.filename, e)) return root diff --git a/vectors/README.md b/vectors/README.md index ea3a4c8..80bdb48 100644 --- a/vectors/README.md +++ b/vectors/README.md @@ -109,7 +109,7 @@ The table below uses a few terms that the chapters define: | `projections.jsonl` | One-sentence runs: plain, normalized, strict and lossy projections, complex keys and the three text values, and the projection as printed (`printed`) of an emended sentence. | [5.3](../spec/05-hashes.md#53-the-strict-projection), [3.6](../spec/03-nodes.md#36-the-plain-projection) | | `projection-groups.jsonl` | Which cases must share, or must differ in, a value: H₂O in six spellings, cm⁻¹, 10⁻³, K_r, ligatures, the micro sign, "103" against 10³, and an emended sentence against its printed text. | [5.3](../spec/05-hashes.md#53-the-strict-projection) | | `offsets.jsonl` | Code-point spans and their UTF-16 equivalents. | [1.5](../spec/01-conventions.md#15-character-offsets) | -| `hashes.jsonl` | Every node of a small document, its own content and its five values. | [5.1](../spec/05-hashes.md#51-the-five-values) | +| `hashes.jsonl` | Every node of a small document, its own content and its five values; and of a sentence whose equation and math run carry a math form, which enters no value. | [5.1](../spec/05-hashes.md#51-the-five-values), [5.2](../spec/05-hashes.md#52-own-content) | | `addresses.jsonl` | How addresses parse; null for invalid ones. | [6.1](../spec/06-anchors.md#61-addresses) | | `keyed.jsonl` | The team key and its ID, an HMAC, page hashes, a page-derived key, and what someone without the paper cannot check. | [5.8](../spec/05-hashes.md#58-keyed-hashes) | | `fingerprints.jsonl` | Pages as text lines, the stamp rules that fired, and the fingerprint; two downloads that must agree, and a change that must not. | [7.1](../spec/07-geometry.md#71-page-fingerprints) | @@ -138,7 +138,7 @@ A candidate format changes only with a recorded reason and a new or changed vect A passing run shows one thing: for these inputs, your implementation computes the same values as the reference implementation in `src/paperspec/`. That code computed the expected outputs, and `pixi run build` regenerates them ([tools/](../tools/README.md)). -The vectors hold 329 cases in all, some of them small: `hashes.jsonl` has one document, `alignment.jsonl` two pairs of readings, `offsets.jsonl` two cases. +The vectors hold 330 cases in all, some of them small: `hashes.jsonl` has two documents, `alignment.jsonl` two pairs of readings, `offsets.jsonl` two cases. The vectors do not show the following. diff --git a/vectors/hashes.jsonl b/vectors/hashes.jsonl index c67013a..5432bfa 100644 --- a/vectors/hashes.jsonl +++ b/vectors/hashes.jsonl @@ -1 +1,2 @@ {"doc":"tiny2026tiny-j4qm","id":"tiny2026example","nodes":[{"id":"s0fn2fj","kind":"document","ord":0,"parent":null},{"attrs":{"level":1},"id":"b9wp3tt","kind":"section","label":"1","ord":0,"parent":"s0fn2fj"},{"id":"vet7k0e","kind":"heading","ord":0,"parent":"b9wp3tt"},{"content":[{"kind":"text","text":"Results"}],"fidelity":"F2","id":"em5pd9v","kind":"sentence","ord":0,"parent":"vet7k0e"},{"id":"ytp1e5n","kind":"paragraph","ord":1,"parent":"b9wp3tt"},{"content":[{"kind":"text","text":"The value is 42.0 at 300 K."}],"fidelity":"F2","id":"jvnys4s","kind":"sentence","ord":0,"parent":"ytp1e5n"},{"content":[{"kind":"text","text":"See "},{"kind":"xref","target":"tb49qpc","text":"Table 1"},{"kind":"text","text":" and the note."},{"kind":"cite","refs":["r3e21zr"],"text":"1"}],"fidelity":"F2","id":"mzwwhxj","kind":"sentence","ord":1,"parent":"ytp1e5n"},{"id":"exq1ter","kind":"paragraph","ord":2,"parent":"b9wp3tt"},{"content":[{"kind":"text","text":"A second paragraph."}],"fidelity":"F2","id":"j663d4h","kind":"sentence","ord":0,"parent":"exq1ter"},{"content":[{"kind":"text","text":"It ends here."}],"fidelity":"F2","id":"rsw24qc","kind":"sentence","ord":1,"parent":"exq1ter"},{"attrs":{"columns":[{"header_path":["af5b51r"],"id":"pj5sk05"},{"header_path":["ajqxkgh"],"id":"nymcs5q"}]},"id":"tb49qpc","kind":"table","label":"Table 1","ord":3,"parent":"b9wp3tt"},{"id":"t2ps2mp","kind":"table-head","ord":0,"parent":"tb49qpc"},{"id":"rr9rtf5","kind":"row","ord":0,"parent":"t2ps2mp"},{"attrs":{"colspan":1,"column":"pj5sk05","role":"column-header","rowspan":1},"content":[{"kind":"text","text":"x"}],"fidelity":"F1","id":"af5b51r","kind":"cell","ord":0,"parent":"rr9rtf5"},{"attrs":{"colspan":1,"column":"nymcs5q","role":"column-header","rowspan":1},"content":[{"kind":"text","text":"y"}],"fidelity":"F1","id":"ajqxkgh","kind":"cell","ord":1,"parent":"rr9rtf5"},{"id":"dmsstkg","kind":"table-body","ord":1,"parent":"tb49qpc"},{"attrs":{"header_path":["rdrqkv7"]},"id":"r3e7hfs","kind":"row","ord":0,"parent":"dmsstkg"},{"attrs":{"colspan":1,"column":"pj5sk05","role":"row-header","rowspan":1},"content":[{"kind":"text","text":"1"}],"fidelity":"F1","id":"rdrqkv7","kind":"cell","ord":0,"parent":"r3e7hfs"},{"attrs":{"colspan":1,"column":"nymcs5q","role":"data","rowspan":1},"content":[{"kind":"text","text":"2.5"}],"fidelity":"F1","id":"f463wj7","kind":"cell","ord":1,"parent":"r3e7hfs"},{"id":"aw97y35","kind":"back","ord":1,"parent":"s0fn2fj"},{"id":"yd513mw","kind":"references","ord":0,"parent":"aw97y35"},{"content":[{"kind":"text","text":"1. An Example, 2020."}],"fidelity":"F2","id":"r3e21zr","kind":"reference","label":"1","ord":0,"parent":"yd513mw"}],"own_content":{"af5b51r":{"colspan":1,"column":"pj5sk05","kind":"cell","role":"column-header","rowspan":1,"runs":[{"kind":"text","text":"x"}]},"ajqxkgh":{"colspan":1,"column":"nymcs5q","kind":"cell","role":"column-header","rowspan":1,"runs":[{"kind":"text","text":"y"}]},"aw97y35":{"kind":"back"},"b9wp3tt":{"heading":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd","kind":"section","label":"1","level":1},"dmsstkg":{"kind":"table-body"},"em5pd9v":{"kind":"sentence","runs":[{"kind":"text","text":"Results"}]},"exq1ter":{"kind":"paragraph"},"f463wj7":{"colspan":1,"column":"nymcs5q","kind":"cell","role":"data","rowspan":1,"runs":[{"kind":"text","text":"2.5"}]},"j663d4h":{"kind":"sentence","runs":[{"kind":"text","text":"A second paragraph."}]},"jvnys4s":{"kind":"sentence","runs":[{"kind":"text","text":"The value is 42.0 at 300 K."}]},"mzwwhxj":{"kind":"sentence","runs":[{"kind":"text","text":"See "},{"kind":"xref","target":"tb49qpc","text":"Table 1"},{"kind":"text","text":" and the note."},{"kind":"cite","refs":["r3e21zr"],"text":"1"}]},"r3e21zr":{"kind":"reference","label":"1","runs":[{"kind":"text","text":"1. An Example, 2020."}]},"r3e7hfs":{"kind":"row"},"rdrqkv7":{"colspan":1,"column":"pj5sk05","kind":"cell","role":"row-header","rowspan":1,"runs":[{"kind":"text","text":"1"}]},"rr9rtf5":{"kind":"row"},"rsw24qc":{"kind":"sentence","runs":[{"kind":"text","text":"It ends here."}]},"s0fn2fj":{"kind":"document"},"t2ps2mp":{"kind":"table-head"},"tb49qpc":{"columns":[{"header_path":["af5b51r"],"id":"pj5sk05"},{"header_path":["ajqxkgh"],"id":"nymcs5q"}],"kind":"table","label":"Table 1"},"vet7k0e":{"kind":"heading"},"yd513mw":{"kind":"references"},"ytp1e5n":{"kind":"paragraph"}},"values":{"af5b51r":{"align_key":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881","match_hash":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881","node_hash":"59d1c20766a77bacde2aae80a134c7156fbcc708c140a3bb2d2a67f78bd9c6f5","own_hash":"c68933b07bc7c5b806892ae36b3e4c3205a0a2066fd6f8255578f7d1ca86dc10","text_hash":"2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"},"ajqxkgh":{"align_key":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa","match_hash":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa","node_hash":"838e3fba38f0797d40522b7063cb5c2962e32a019c09c19f09849ced73079a87","own_hash":"28df42bf1dcd44b053275ed014bd1599914f17adabc0c7fdb3e4247dc648ac49","text_hash":"a1fce4363854ff888cff4b8e7875d600c2682390412a8cf79b37d0b11148b0fa"},"aw97y35":{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"fcb8027f7c89115f048b9c5d6770847248f4e4fbc61cbc8ea0b04079b4d58350","own_hash":"bbb972bdc024231b505f4d8ce5e711fbb75c8b5ffd1b5d42ee3fb8586b85f867","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"},"b9wp3tt":{"align_key":"caf6aee3882b3890996037e35d0ffdfe30b5e66653d45321d7ea0fa9135f1ee9","match_hash":"5a96c01f314c4d8a22fd5e105bebef45df4301886cccbc2538ecefddfcb1d405","node_hash":"ea9a1409bbf0f0f7bd8d508bfdbbf7d14a22ab1e3da011eafbfcd4e14ae7e4fd","own_hash":"65da097b48f8237e8d171bfbd0053ff0e3795c0532d5522d23a92b53430956f4","text_hash":"7a97628fe9643c0fca3614a077e2e631c3fee81d3733ba1564292df65a294573"},"dmsstkg":{"align_key":"b902cc4550838229a710bfec4c38cbc7eb11082367a409df9135e7f007a96bda","match_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738","node_hash":"baf17577660aa6f0ba9c8703d30816d77cfce7dbf0ea3257ed830fa9ab94ce9d","own_hash":"b229a7f92314227bf225ad27010a546e7c2373f4b30b7f6ec372e568cbacb6aa","text_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738"},"em5pd9v":{"align_key":"c099142bc3186ded72786ba27e9ea6d2da240fb9fd3fe79b479ecf8e734b2850","match_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd","node_hash":"293fc03ff4e39d7532467e4417ffcdb15f470f0bd3658dc1c021645ad53c9bef","own_hash":"c55f401232b19ff1133d77eac6c72868975064f177fc2d31ab8a181303c30599","text_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd"},"exq1ter":{"align_key":"282eea00b316a6950fb7179f6289a8b98359019e11cb23a3ff426cdecad90165","match_hash":"da88f35d24faf0099610c720953c4919647a5dfb9003889a15c464cd9c540fb1","node_hash":"062dd26bfb226ca9811d0dfb013df211cb201dfe7b4b25f9aba000f3212298d5","own_hash":"494e069be36a18ecaabf5a63be7c31617ade7a5460218ca83a2de3d8d8f964e0","text_hash":"da88f35d24faf0099610c720953c4919647a5dfb9003889a15c464cd9c540fb1"},"f463wj7":{"align_key":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61","match_hash":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61","node_hash":"f1cd6d5339c84d5c36f8e1ea484a8d837f3200f05d1b106f6119dfa30d846bfd","own_hash":"088eff622eb3086d68c4433cf5e1783dc9f2a6724c3536c31ec0bf80b5f89db5","text_hash":"b8736b999909049671d0ea075a42b308a5fbe2df1854899123fe09eb0ee9de61"},"j663d4h":{"align_key":"b8074a8bc013aa007023c4cce7334f1e1f22384d57a526ca22fd2f5589d5ab17","match_hash":"2e8313b033de5e17878ae778d6293bbd6e197a072f36acf7fe0840a690a7f31e","node_hash":"443539440dca73149ca75276492334ce9a5a6c50070137e8dc7bf414b75a27ab","own_hash":"acee4b6b1399bfde1dc67dea8d8c86b0db961d2af516ae4aaee26f294261456f","text_hash":"2e8313b033de5e17878ae778d6293bbd6e197a072f36acf7fe0840a690a7f31e"},"jvnys4s":{"align_key":"8f21603c39df123eb8680362184274aac9bdf1dbd485b3bba230960179071c95","match_hash":"3bcf9fa8412dca28be91d6423460233ddedd7add6c324d8a0f2b26038567d3d3","node_hash":"f5761f338b2878dff39a6ee724a75b13626dd625a0df96cf1cc6f9e8258cb7cc","own_hash":"7f0b9190d076cf01b51a8be863a51e9a8eeaab5d3ec8656202ff4d41578d4d82","text_hash":"3bcf9fa8412dca28be91d6423460233ddedd7add6c324d8a0f2b26038567d3d3"},"mzwwhxj":{"align_key":"8e7fe271cd1bab88d86c8900fe3ffae0ce95c55af6659df1fca278d359296e2d","match_hash":"9894a48790a8561966e68672fa13da20541d25a5178653720648f62cbc9ad36b","node_hash":"8f3e5338ba3c97efca4cef5c60d849757aed6334ee27ab807fa71d969089e46e","own_hash":"02ed6d50b8ac607b59e3f7b5116ee6143ed7e69af25b19c4037aa799bc330a88","text_hash":"5b1de19c9eb1399e54bde7e96f051cac8b337be1f307fac320292b8db7e39da7"},"r3e21zr":{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"71b1af6dd6a56f24cb3b338e0fe99df06b9154ecdf4b391b00b7a7e61e23a8ed","own_hash":"f7c9bbc8c7456e6dc435feeaaa02cbfdbe487523b020105793fb11b91d0894f5","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"},"r3e7hfs":{"align_key":"b902cc4550838229a710bfec4c38cbc7eb11082367a409df9135e7f007a96bda","match_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738","node_hash":"97a1f5b27bbfe8c9d2cec7212b519b4993506fc8080be3eff02f3aa31515a19d","own_hash":"6c7c11bf1b6260162c4141f1468aa6869f5c33359900497e2cdb1a051488bd62","text_hash":"24cb52bc2dd0b9ddd4650c011d76b585e76c4d4ecae8cf220c7600efb808d738"},"rdrqkv7":{"align_key":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b","match_hash":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b","node_hash":"87da3bd568286f3d0578d8a813496d15faafb62f4db44b07aa1c29e52cda34df","own_hash":"fb474321001657b59bccd3cae0987c544a4ef9a27214626d77695f4db237aa93","text_hash":"6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b"},"rr9rtf5":{"align_key":"769a4e6d0003189c7e96c5d9b7e810a0d11c3a12832527ec94b0f86d277f51ca","match_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19","node_hash":"8bd1fa377ec390fdc36e17baa8c3a39d2df7ef2a53cd0c0dc745ae40e98f99f8","own_hash":"6c7c11bf1b6260162c4141f1468aa6869f5c33359900497e2cdb1a051488bd62","text_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19"},"rsw24qc":{"align_key":"437c101749b0481e615fc7eed8dcacb536b65400c7e0695ad4cbb2bf2dae488c","match_hash":"2888dbfa0c08fd65c58c923b920860c76e13913d9fa93b03eccbbf7bdf53b882","node_hash":"caeb3ae57d0821b8ac961c9507debbcda2b118b16cdc010b83a143a782058d57","own_hash":"705f77e8520c8c5ce3a7a495a20d8a9c55baa9801e49a55c341f5de79eeede75","text_hash":"2888dbfa0c08fd65c58c923b920860c76e13913d9fa93b03eccbbf7bdf53b882"},"s0fn2fj":{"align_key":"b3108c22ec836d883e5116f7ed8546fa75390eb4aa960d78f2936115f69b6826","match_hash":"b6a4e12f700a6214bf02c9a437559e82fccdbecea7b575fa438a7d2e8c33071a","node_hash":"a05328e6d13edba652599aceb58c018dbccf54abd1f878c9f61623017960e1fe","own_hash":"34444fe0e6bdbf3927a579fe3286859d204ea0882a381b7769ffd4905551d24f","text_hash":"bd2abc746d51cf52d57a18632383245d585e29e265426967894eed99bc937561"},"t2ps2mp":{"align_key":"769a4e6d0003189c7e96c5d9b7e810a0d11c3a12832527ec94b0f86d277f51ca","match_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19","node_hash":"6d8056adfb50dc0b65a2a65b47eccc5f47d5482a8dfbefa0d8a06f6641a4be3a","own_hash":"c5749bee3a4b981fb79cf6db125ebb416dce5a83613a00feecd9d4cf75df9bb4","text_hash":"887fcea6a80333c6c02ae7e79735f0edad8d811f0b61431495f796f4bf6a7c19"},"tb49qpc":{"align_key":"cb0107aa0d562e8bd1188c33dc1ef5a1d07808e51edeabaa87465556d11b6ec8","match_hash":"1951e0b61441d7b817b4967b36eee007f8e7525ac75fd359437eddb871e59f72","node_hash":"154c910857f33c152fdc78e42582510327055e62da9a25ab10c3891eeaf69627","own_hash":"fb4cc66080def0b42a06f7aa566a24957c7345c77b8d873ca5c790704a890f47","text_hash":"1951e0b61441d7b817b4967b36eee007f8e7525ac75fd359437eddb871e59f72"},"vet7k0e":{"align_key":"c099142bc3186ded72786ba27e9ea6d2da240fb9fd3fe79b479ecf8e734b2850","match_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd","node_hash":"a3c82ab1e44bf4cffa1a59507f8bddc39a1ad191321ee803d0d60d96609a44c7","own_hash":"e813ff86a8920cef741e548eb5df545774b520d2fadbdda6e2b872a6729b9d26","text_hash":"219c4a6c86a716e99d5dc9611b61e584deb11b57e972d87d4b773387ebc3cadd"},"yd513mw":{"align_key":"b35d81b0ff35d6d05ad2c187a289d3017896427dfd8bb98160bc806ac0ec32f7","match_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e","node_hash":"8b8a9000e984c3efeb27fc40c8fbadd7349dd64860aa276dcd2a25a57ce52e87","own_hash":"2a5302e2041d0e59458fea212dd51b89299901318bfceeeff227c1a07d3f5d1a","text_hash":"9a2d5d28ee47736a00d074e066bb944154f95a192360c8c04226c3d12a4f7d1e"},"ytp1e5n":{"align_key":"610c235321e38145abff21e31eaab303d85a6ce0d1b9340a599dc821a04922f6","match_hash":"bcb4902ec69f2a7a724384f561ee9aa0e214c824c5882c1ad92777bee680aad4","node_hash":"7667a3a4bc2e8fec1dc7af53e19c414952f9ce926b1e37e8d9adca66dd8f8d1d","own_hash":"494e069be36a18ecaabf5a63be7c31617ade7a5460218ca83a2de3d8d8f964e0","text_hash":"f589ee914c46c4aea7ce2de49d925c433c914121f576b81a3eab692f17f179a3"}}} +{"doc":"vectors","id":"math-form","nodes":[{"id":"aaaaaaa","kind":"document","ord":0,"parent":null},{"attrs":{"level":1},"id":"bbbbbbb","kind":"section","ord":0,"parent":"aaaaaaa"},{"id":"ccccccc","kind":"paragraph","ord":0,"parent":"bbbbbbb"},{"content":[{"kind":"text","text":"The energy is "},{"kind":"equation","node":"eeeeeee"},{"kind":"text","text":" where "},{"form":"scienting-math/1","kind":"math","notation":"tex","source":"K_{r}"},{"kind":"text","text":" is the force constant."}],"id":"ddddddd","kind":"sentence","ord":0,"parent":"ccccccc"},{"content":{"form":"scienting-math/1","notation":"tex","source":"E=\\frac{1}{2}K_{r}x^{2}"},"id":"eeeeeee","kind":"equation","ord":0,"parent":"ddddddd"}],"own_content":{"aaaaaaa":{"kind":"document"},"bbbbbbb":{"heading":null,"kind":"section","label":null,"level":1},"ccccccc":{"kind":"paragraph"},"ddddddd":{"kind":"sentence","runs":[{"kind":"text","text":"The energy is "},{"kind":"equation"},{"kind":"text","text":" where "},{"kind":"math","notation":"tex","source":"K_{r}"},{"kind":"text","text":" is the force constant."}]},"eeeeeee":{"kind":"equation","label":null,"notation":"tex","source":"E=\\frac{1}{2}K_{r}x^{2}"}},"values":{"aaaaaaa":{"align_key":"0a0b7b0fad218642459f4be88e8a02e165b79eb087d7be0b33845f15d0dfc0b9","match_hash":"77244e317f4ace842f4027deb48c9063162f87b10a6d59cc18412945391f8b23","node_hash":"3d5c422271f7428e60428e337dc0bcb09d7f0a8ff77c309bdee3dfc277f8e83b","own_hash":"34444fe0e6bdbf3927a579fe3286859d204ea0882a381b7769ffd4905551d24f","text_hash":"06e060a25e16536e6d5f673d0d439ca35d5792f7b02ee57254a8932e5e691305"},"bbbbbbb":{"align_key":"0a0b7b0fad218642459f4be88e8a02e165b79eb087d7be0b33845f15d0dfc0b9","match_hash":"77244e317f4ace842f4027deb48c9063162f87b10a6d59cc18412945391f8b23","node_hash":"0dba0d12b31e1de737b37ededa816aa9ff8c864a04005c3f3ccaad1864a0fc5d","own_hash":"2e173cf74bdb3f542b050ad3556181eeebb58464a7a98765085b98ee7423227b","text_hash":"06e060a25e16536e6d5f673d0d439ca35d5792f7b02ee57254a8932e5e691305"},"ccccccc":{"align_key":"0a0b7b0fad218642459f4be88e8a02e165b79eb087d7be0b33845f15d0dfc0b9","match_hash":"77244e317f4ace842f4027deb48c9063162f87b10a6d59cc18412945391f8b23","node_hash":"87b388726c1ccd7b6f2f26db6912d261bc96ea8cd2d6a87e80e5bb7a3ca2a7f4","own_hash":"494e069be36a18ecaabf5a63be7c31617ade7a5460218ca83a2de3d8d8f964e0","text_hash":"06e060a25e16536e6d5f673d0d439ca35d5792f7b02ee57254a8932e5e691305"},"ddddddd":{"align_key":"0a0b7b0fad218642459f4be88e8a02e165b79eb087d7be0b33845f15d0dfc0b9","match_hash":"77244e317f4ace842f4027deb48c9063162f87b10a6d59cc18412945391f8b23","node_hash":"1282798a0fca7a4a61243a77a371bd9f2ca25868e85a4dc976db117082ce5139","own_hash":"bdf85965019bac6d61b8032a5caab32fdaf10fe912aa058e89ed93057a916788","text_hash":"06e060a25e16536e6d5f673d0d439ca35d5792f7b02ee57254a8932e5e691305"},"eeeeeee":{"align_key":"cd2822e88bcde089a4b8d4d60820a4c0c387d75c7fb1737b5e5ac93129194ff7","match_hash":"cd2822e88bcde089a4b8d4d60820a4c0c387d75c7fb1737b5e5ac93129194ff7","node_hash":"a0afe2c822540a5bdba87aab9459601f89624ee4a1192fcc7ea610646cea7804","own_hash":"453b9ae322705372c37ad8e1a30d712d037a1324c45d700f339229ae8b3013eb","text_hash":"4b8672dac603564cd19e17e0a59999e1eb9661d2179795992c3edeee9b094266"}}}