infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: rewrite docs
check / check (default, macos-latest) (push) Canceled after 0s
Details
check / check (default, ubuntu-latest) (push) Canceled after 0s
Details
check / check (py310, macos-latest) (push) Canceled after 0s
Details
check / check (py310, ubuntu-latest) (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
alexm committed 2026-10-12 07:55:22 -04:00
1 parent 8b2df58266
commit 002e2c3552
140 files changed
+4555 -701
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

fixtures/README.md
+53 -15
View File
@@ -1,31 +1,66 @@
# Fixtures
Example data for learning the formats and for testing implementations.
All text here is synthetic, written for these files and dedicated CC0; license fields inside the fixtures describe fictional works.
A reader, writer, converter or checker of stored papers needs files to test against, and a newcomer needs examples of what a stored paper looks like.
This folder holds that example data of manuspec (<https://git.scient.ing/infra/manuspec>), version 0.1.0.
All text is synthetic, written for these files and dedicated CC0; license fields inside the fixtures describe fictional works.
Three readers use these files.
A newcomer learns what a stored paper looks like and what can go wrong with one.
An implementer, who writes a reader, writer, converter or checker, tests it against the files.
An expert reviewing the work finds which cases the tests pin and what the published keys do and do not prove.
Each format here has the status `candidate` or `provisional` in `registry/formats.json`.
A hash is a fixed-length fingerprint of data: any change to the data changes it.
The hash used here is SHA-256, written in hexadecimal.
A package is one document stored as a folder of files, with a manifest listing every file and its hash.
A node is an addressable part of a document, such as a section, a sentence or a table cell.
A witness is a source file a reading came from, such as a PDF.
A receipt is a signed statement that a named person reviewed listed nodes.
## Packages
| Package | What it shows |
Each package under `packages/` is a valid document.
A conforming reader accepts each one and recomputes every hash it contains.
| Package | What it is for |
| --- | --- |
| `packages/hale2026harmonic/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper read once with mistakes and fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster that signed them, a dataset taken from the table, and derived MathML. |
| `packages/hale2026harmonic.checks-only/` | The same document prepared for another of the team's machines: text withheld, every hash an HMAC under the team key in `keys/`. |
| `packages/okafor2025open/` | An open-access paper established from a JATS file and a PDF that agree, so every sentence is corroborated. |
| `packages/smithlab2027charges/` | A short draft written in Typst, whose sentences cite sentences of the main example by address. |
| `packages/tiny2026example/` | A small document, the starting point of the correction vectors and the broken packages. |
| `packages/hale2026harmonic/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, and derived MathML. |
| `packages/hale2026harmonic.checks-only/` | The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` ([spec/05](../spec/05-hashes.md), [spec/12 section 12.5](../spec/12-package.md#125-text-policies)). It shows how a team can share its checks on a paywalled paper without sharing the paper. |
| `packages/okafor2025open/` | An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness. |
| `packages/smithlab2027charges/` | A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#r1gacps`, a sentence of the main example. |
| `packages/tiny2026example/` | A small document, the starting point of the correction vectors (`vectors/corrections.jsonl`) and of every broken package below. |
## Records
`records/` holds a valid example of every other format: anchors (a span, a leaf, a cell, a paragraph, and one keyed for a public layer), rights records, a team policy, backend declarations, team profiles, a layer with its annotations and the receipt that confirms them, submissions and grades, a project, a digitization with its confirmation and the dataset it yields, a view, a QA report, a draft snapshot and a draft's sentence IDs.
`records/` holds a valid example of most of the other formats.
[CHECK: spec/22 section 22.1 says "a valid record of every other format", but `scienting-stages/1`, `scienting-stamps/1`, `scienting-suspect/1` and `scienting-bundle/1` have no standalone file here; confirm the wording the author wants]
Each record is a single file, or a folder where a format needs several files.
| Files | What they are for |
| --- | --- |
| `anchor-span.json`, `anchor-leaf.json`, `anchor-cell.json`, `anchor-paragraph.json` | An anchor is an address pinned to a hash, with a quotation and context, so it can be found again after the text changes. These four anchors point at a span of a sentence, a sentence, a table cell and a paragraph of the main example. |
| `anchor-keyed.json` | An anchor that leaves the team, as in a public layer: its table context is stored as keyed hashes, so the labels cannot be read from it ([spec/06 section 6.3](../spec/06-anchors.md#63-keyed-anchors)). |
| `rights-open.json`, `rights-subscription.json`, `rights-collective.json` | A witness's license and what uses it allows, under an open license, a subscription and a collective license. |
| `policy.json` | A signed team decision that settles a use the rights record leaves unknown. |
| `backend-docling.json`, `backend-examplevendor.json` | The code and weight licenses of a conversion tool and the conditions attached to them. |
| `profile-acme.json`, `profile-smithlab.json` | What those conditions ask about a team: its purpose, its funding and whom it competes with. |
| `roster.json` | The keys that may sign, and for what. |
| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). |
| `submissions/`, `grades/` | A student's answers to exercises in the layer, and the grader's result for each. |
| `qa-report.json` | Reviewer error bounds for a delivery that covers two of the main example's receipts. |
| `project.json`, `digitization.json`, `digitization-confirmation.json`, `datasets/`, `view.vl.json`, `snapshot.json`, `draft-ids.jsonl` | A team project; values read off the main example's figure by hand, the receipt that confirms them, and the dataset they yield; a Vega-Lite chart pinned to that dataset; what the draft cites, pinned; and the sentence IDs of the draft's Typst project. These formats are `provisional`. |
## Broken packages
Each folder in `invalid/` holds a package that breaks exactly one rule, and `expect.json` names it.
Each folder in `invalid/` holds a `package/` that breaks exactly one rule, built from `packages/tiny2026example/` by one edit, and an `expect.json` that names the rule.
A conforming implementation refuses every one of them and should report that rule.
`registry/rules.json` lists every rule ID with its meaning, and a validator reports each failure as `path:line: rule: message` ([spec/22 section 22.2](../spec/22-conformance.md#222-validating-a-package)).
| Case | What is wrong | Rule |
| --- | --- | --- |
| `alias-dangling/` | An alias whose node does not exist. | `doc.id.successor` |
| `bad-id/` | IDs use Crockford base32, which has no i, l, o or u. | `package.schema` |
| `bad-id/` | An ID that uses a letter Crockford base32 lacks (i, l, o or u). | `package.schema` |
| `checks-only-plain/` | A checks-only package that still carries text and plain hashes. | `doc.policy.withheld` |
| `cite-not-reference/` | A cite run names a node that is not a reference. | `doc.run.cite` |
| `empty-optional-file/` | An optional JSONL file with no records. | `package.file.empty` |
@@ -33,8 +68,8 @@ A conforming implementation refuses every one of them and should report that rul
| `heading-not-first/` | A heading must be a section's first child. | `doc.tree.child` |
| `location-unknown-page/` | A location on a page no witness has. | `doc.location.page` |
| `manifest-hash/` | A file changed after the manifest was written. | `package.manifest.hash` |
| `not-canonical/` | A line that is valid JSON but not canonical JSON. | `package.file.canonical` |
| `receipt-root/` | A receipt whose Merkle root does not recompute. | `doc.receipt.root` |
| `not-canonical/` | A line that is valid JSON but not canonical JSON (the fixed form of RFC 8785, which gives equal data identical bytes). | `package.file.canonical` |
| `receipt-root/` | A receipt whose Merkle root (one hash that summarizes a list of hashes) does not recompute. | `doc.receipt.root` |
| `receipt-signature/` | A receipt changed after it was signed. | `doc.receipt.signature` |
| `root-mismatch/` | document.json's root is not the root node's node_hash. | `doc.hash.root` |
| `style-sup-sub/` | A run cannot be both a subscript and a superscript. | `doc.run.style` |
@@ -44,8 +79,11 @@ A conforming implementation refuses every one of them and should report that rul
## Keys
`keys/` holds the Ed25519 secret keys that signed the fixtures' receipts, and the team secret of the checks-only package.
They are published so that anyone can reproduce the signatures; never use them for anything else.
`keys/` holds the Ed25519 secret keys (`ana`, `ben`, `cara`, and `ben-spot`, whose signature picks the nodes of a spot-check) that signed the fixtures' receipts, and the team secret of the checks-only package.
Ed25519 is a signature scheme: the secret key makes a signature, and the public key in a roster verifies it.
These are published test keys.
Anyone can reproduce the signatures, so a signature made with these keys proves nothing about who made it.
Use them for nothing else.
## Sources