alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Tools
This page is for anyone who wants to run the repository's checks, rebuild its examples or load a document into a database. Newcomers get each term defined where it first appears. Implementers get the commands and the traps.
These programs are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
They need Python 3.14 and two packages, jsonschema and cryptography, and are licensed Apache-2.0.
A package is the folder of files that holds one document, such as fixtures/packages/hale2026harmonic.
A vector is a test case: an input and the exact output every implementation must compute (see the vectors).
A fixture is an example file or package under fixtures/.
| Program | Does |
|---|---|
check.py |
Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. --quick skips the rebuild. |
build.py |
Builds fixtures/ and vectors/ from the hand-written sources in fixtures/src/ and the cases it defines. --out DIR builds into another folder. |
validate.py |
Validates packages, as folders or as zipped packages (.sct.zip, unpacked under the rules of spec/12 §12.2): python3 tools/validate.py PACKAGE [PACKAGE ...]. With --correction PROPOSALS it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything (spec/11 §11.4). |
load.py |
Validates packages, folders or zipped, then loads them into a SQLite database with the tables of sql/: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]. With --bundle --built-from REV it builds a new bundle instead, with the full-text search index (spec/13 §13.5). |
stress.py |
Stresses the oracle with random inputs from a seeded generator: python3 tools/stress.py [--seed N] [--rounds N]. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
oracle/ |
A compact implementation of every rule. build.py uses it to compute expected values, and check.py uses it to recompute them. |
When to run what
Each program is also a pixi task.
Pixi sets up a pinned Python and the other packages the repository uses.
The commands pixi run check, pixi run quick, pixi run build and pixi run load then work from anywhere in the repository.
pixi run check lints the Markdown first.
Run the programs directly from the repository root if you do not use pixi:
python3 tools/check.py # everything
python3 tools/check.py --quick # everything except the rebuild
python3 tools/build.py --out /tmp/rebuilt
python3 tools/validate.py fixtures/packages/hale2026harmonic
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
A passing check ends with All checks pass.
The load command prints hale2026harmonic: 75 nodes, one line per package, and writes lab.db with a folder lab.db.assets/ beside it for files that have no table of their own.
It validates every package first and loads nothing if one fails, printing each failure as package: where: rule: message.
The full and checks-only forms of one document share its key, so they go into separate databases.
Each program prints its options with --help and refuses options it does not know.
build.py rebuilds fixtures/ and vectors/ in place unless you pass --out.
Run pixi run check before every commit.
Run pixi run build after you change a fixture source, a rule in oracle/ or a case in build.py, and commit the regenerated files with the change.
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
About the oracle
An oracle is a reference computation: a slow, plain program whose answers count as the expected ones. It is not a production reader or writer. The oracle exists so that no expected value in this repository is typed by hand.
It favors clarity, and leaves the cryptography and schema validation to established packages.
Ed25519 (a digital signature scheme) and HKDF come from cryptography, and JSON Schema (a format that describes the shape of JSON data) validation from jsonschema.
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
Hashes are kept incrementally.
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
check.py compares the result after every correction with a slow computation that projects each node from its whole subtree.
Do not build on it as a library. Implementations follow the specification and test themselves against the vectors.