0 Fork 0

Files

T

feat: zipped packages, target set, checklists, audit fixes and a stress test

Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-10-14 20:07:22 -04:00

4.2 KiB

Raw Blame History

22. Conformance

22.1 The suite

Path	Holds	A conforming implementation
`vectors/*.jsonl`	Inputs and expected outputs, one case per line.	Computes every expected output from its input.
`fixtures/packages/`	Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document, as a folder and as the zip of 12.2.	Accepts each, and recomputes every value it contains.
`fixtures/records/`	A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `tools/load.py --bundle`.	Accepts each.
`fixtures/invalid/`	Invalid packages, each with `expect.json` naming the rule it breaks.	Refuses each, and SHOULD name that rule.
`fixtures/keys/`	The test keys that signed the fixtures and the team secret of the checks-only export.	Never uses them for anything else.

All fixture text is synthetic and dedicated to the public domain; license fields inside fixtures describe the fictional works. Each part's CI runs this suite against its own implementation, so the parts cannot drift apart silently.

22.2 Validating a package

A validator checks, in order, and reports each failure as path:line: rule: message:

The manifest: present, valid, sorted, every listed file present with its hash and size, nothing unlisted, paths safe and no symbolic links, required files present.
Every file in canonical form, byte for byte, and every record against its schema.
The tree (3), runs and references between nodes, and the counts.
The policy and hashing scheme (12.5).
The five values of every node, the root and meta_hash, when the text is present and the key, if any, is known.
The ID table, the corrections, the witnesses, pages, locations and references, and every file's order.
Receipts: IDs, roots, documents, node order, signatures against the package's roster, spot-check keys, seeds and samples, and the fidelity distribution.

22.3 Schemas

Schemas use draft 2020-12, restricted to the keywords $schema, $id, $ref, $defs, $comment, title, description, examples, default, deprecated, type, enum, const, properties, required, additionalProperties, patternProperties, propertyNames, minProperties, maxProperties, dependentRequired, items, prefixItems, minItems, maxItems, uniqueItems, contains, minLength, maxLength, pattern, format, minimum, maximum, exclusiveMinimum, exclusiveMaximum, oneOf, anyOf, allOf, not, if, then and else. Patterns avoid \d, \w, \s, \b and lookbehind, whose meaning differs between regular-expression dialects. A schema checks shape; the chapters' rules, such as the five values, are checked by validators.

22.4 Rules

registry/rules.json lists every rule ID with its meaning. Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is never reused.

Prefix	Covers
`package.`	The manifest, paths, files and schemas.
`doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.`	The tree, node content and runs.
`doc.hash.`	The five values, the root and the metadata hash.
`doc.id.`, `doc.order.`	The ID table and file order.
`doc.policy.`	Text policies and keyed hashing.
`doc.correction.`, `correction.`	A package's corrections, and the gate.
`doc.receipt.`, `doc.qa.`, `doc.fidelity.`	Receipts, spot-checks and levels.
`doc.witness.`, `doc.page.`, `doc.location.`, `doc.reference.`	Witnesses and geometry.
`grade.`	Grading.

22.5 Stability

Within a major version of a format, a valid file stays valid, and every value the chapters define is computed the same way. Rule IDs, enumeration values and format identifiers are never reused with another meaning. A candidate format changes only with a recorded reason and a new or changed vector; a provisional one may change more freely, and says so in registry/formats.json.

4.2 KiB Raw Blame History