0 Fork 0

Files

T

feat: zipped packages, target set, checklists, audit fixes and a stress test

Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-10-14 20:07:22 -04:00

10 KiB

Raw Blame History

Fixtures

A reader, writer, converter or checker of stored papers needs files to test against, and a newcomer needs examples of what a stored paper looks like. This folder holds that example data of manuspec (https://git.scient.ing/infra/manuspec), version 0.1.0. All text is synthetic, written for these files and dedicated CC0; license fields inside the fixtures describe fictional works.

Three readers use these files. A newcomer learns what a stored paper looks like and what can go wrong with one. An implementer, who writes a reader, writer, converter or checker, tests it against the files. An expert reviewing the work finds which cases the tests pin and what the published keys do and do not prove.

Each format here has the status candidate or provisional in registry/formats.json.

A hash is a fixed-length fingerprint of data: any change to the data changes it. The hash used here is SHA-256, written in hexadecimal. A package is one document stored as a folder of files, with a manifest listing every file and its hash. A node is an addressable part of a document, such as a section, a sentence or a table cell. A witness is a source file a reading came from, such as a PDF. A receipt is a signed statement that a named person reviewed listed nodes.

Packages

Each package under packages/ is a valid document. A conforming reader accepts each one and recomputes every hash it contains.

Package	What it is for
`packages/hale2026harmonic/`	The main example, followed step by step in the walkthrough: a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, and derived MathML.
`packages/hale2026harmonic.checks-only/`	The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` (spec/05, spec/12 section 12.5). It shows how a team can share its checks on a paywalled paper without sharing the paper.
`packages/okafor2025open/`	An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness.
`packages/smithlab2027charges/`	A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#r1gacps`, a sentence of the main example.
`packages/tiny2026example/`	A small document, the starting point of the correction vectors (`vectors/corrections.jsonl`) and of every broken package below.
`packages/tiny2026example.sct.zip`	The same document as a zipped package, written the way a producer makes a byte-identical archive: entries under one folder named by the key, sorted by path, stored, dated 1980-01-01, with no extra fields (spec/12 section 12.2). A reader unpacks it to exactly the folder beside it.

Records

records/ holds a valid example of every other format that is a JSON file of its own. Three formats have their example elsewhere: the stage requirements (scienting-stages/1) and the stamp rules (scienting-stamps/1) are in registry/, and a bundle (scienting-bundle/1) is a SQLite file that tools/load.py --bundle builds from the packages. Each record is a single file, or a folder where a format needs several files.

Files	What they are for
`anchor-span.json`, `anchor-leaf.json`, `anchor-cell.json`, `anchor-paragraph.json`	An anchor is an address pinned to a hash, with a quotation and context, so it can be found again after the text changes. These four anchors point at a span of a sentence, a sentence, a table cell and a paragraph of the main example.
`anchor-keyed.json`	An anchor that leaves the team, as in a public layer: its table context is stored as keyed hashes, so the labels cannot be read from it (spec/06 section 6.3).
`rights-open.json`, `rights-subscription.json`, `rights-collective.json`	A witness's license and what uses it allows, under an open license, a subscription and a collective license.
`policy.json`	A signed team decision that settles a use the rights record leaves unknown.
`backend-docling.json`, `backend-examplevendor.json`	The code and weight licenses of a conversion tool and the conditions attached to them.
`profile-acme.json`, `profile-smithlab.json`	What those conditions ask about a team: its purpose, its funding and whom it competes with.
`roster.json`	The keys that may sign, the role of each key's holder in the team, and what each key may sign. Ana is the owner, so she may revoke anyone's receipt.
`layer/`	A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`).
`submissions/`, `grades/`	A student's answers to exercises in the layer, and the grader's result for each.
`suspect.json`	A doubtful reading raised for review: the converter's reading of an exponent in the main example, before the first correction answered it.
`qa-report.json`	Reviewer error bounds for a delivery that covers two of the main example's receipts, signed by the team's owner key.
`checklist/`	A conformance checklist (`checklist.json`) for an accessible HTML export of the main example, checked against WCAG 2.1 AA criterion by criterion, and the receipt that covers it at its hash (`receipt.json`). The format is `provisional`.
`project.json`, `digitization.json`, `digitization-confirmation.json`, `datasets/`, `view.vl.json`, `snapshot.json`, `draft-ids.jsonl`	A team project; values read off the main example's figure by hand, the receipt that confirms them, and the dataset they yield; a Vega-Lite chart pinned to that dataset; what the draft cites, pinned; and the sentence IDs of the draft's Typst project. These formats are `provisional`.

Broken packages

Each folder in invalid/ holds a package/ folder, or a package.sct.zip archive, that breaks exactly one rule, built from packages/tiny2026example/ by one edit, and an expect.json that names the rule. A conforming implementation refuses every one of them and should report that rule. registry/rules.json lists every rule ID with its meaning, and a validator reports each failure as path:line: rule: message (spec/22 section 22.2).

Case	What is wrong	Rule
`alias-dangling/`	An alias whose node does not exist.	`doc.id.successor`
`bad-id/`	An ID that uses a letter Crockford base32 lacks (i, l, o or u).	`package.schema`
`checks-only-plain/`	A checks-only package that still carries text and plain hashes.	`doc.policy.withheld`
`cite-not-reference/`	A cite run names a node that is not a reference.	`doc.run.cite`
`counts-wrong/`	document.json's counts are not the number of nodes and leaves.	`doc.counts`
`crlf/`	A file whose lines end in CR LF, which is not the canonical form.	`package.file.canonical`
`empty-optional-file/`	An optional JSONL file with no records.	`package.file.empty`
`hash-mismatch/`	A node_hash that does not recompute.	`doc.hash.node-hash`
`heading-not-first/`	A heading must be a section's first child.	`doc.tree.child`
`location-unknown-page/`	A location on a page no witness has.	`doc.location.page`
`manifest-hash/`	A file changed after the manifest was written.	`package.manifest.hash`
`manifest-key/`	The manifest names another key than document.json.	`package.manifest.key`
`missing-required-file/`	A required file is missing.	`package.file.required`
`not-canonical/`	A line that is valid JSON but not canonical JSON (the fixed form of RFC 8785, which gives equal data identical bytes).	`package.file.canonical`
`receipt-root/`	A receipt whose Merkle root (one hash that summarizes a list of hashes) does not recompute.	`doc.receipt.root`
`receipt-signature/`	A receipt changed after it was signed.	`doc.receipt.signature`
`receipt-unsorted-nodes/`	A receipt whose nodes are not sorted by ID.	`doc.receipt.nodes`
`retired-live/`	An ID that is both live and retired.	`doc.id.retired-live`
`rev-without-corrections/`	document.json's rev is not the number of corrections.	`doc.correction.rev`
`root-mismatch/`	document.json's root is not the root node's node_hash.	`doc.hash.root`
`signer-mismatch/`	A receipt whose signature was made with another key than the signer it names.	`doc.receipt.signature`
`style-sup-sub/`	A run cannot be both a subscript and a superscript.	`doc.run.style`
`text-edited/`	Text changed without its hashes.	`doc.hash.text-hash`
`unknown-kind/`	Kinds outside the list are refused.	`package.schema`
`unlisted-file/`	A file the manifest does not list.	`package.manifest.unlisted`
`unsafe-path/`	The manifest lists a path that leaves the package.	`package.path.unsafe`
`zip-symlink/`	A zipped package with an entry that is a symbolic link.	`package.zip.entry`
`zip-traversal/`	A zipped package with an entry whose path leaves the package folder.	`package.zip.entry`

Keys

keys/ holds the Ed25519 secret keys (ana, ben, cara, and ben-spot, whose signature picks the nodes of a spot-check) that signed the fixtures' receipts, and the team secret of the checks-only package. Ed25519 is a signature scheme: the secret key makes a signature, and the public key in a roster verifies it. These are published test keys. Anyone can reproduce the signatures, so a signature made with these keys proves nothing about who made it. Use them for nothing else.

Sources

src/ holds the hand-written descriptions of the three main documents, as first read; see src/README.md. Everything else here is generated by python3 tools/build.py, so change the sources or the builder, never the generated files.

10 KiB Raw Blame History