feat: validator, bundle builder and release checks for 0.1.0

- Add tools/validate.py: validates packages, and with --correction runs
  proposed corrections through the gate against a package without
  writing anything (spec/11 §11.4). The gate now refuses a proposal that
  fails the correction schema, rule correction.schema.
- tools/load.py validates every package before loading, and with
  --bundle --built-from builds a scienting-bundle/1 with the FTS5 search
  index and bundle_meta (spec/13 §13.5). Search rows are defined per
  unit (sentence, cell, equation outside a sentence, figure-text) with
  the parent kind as context.
- Roster keys carry a role, and a key with the role owner may revoke any
  receipt (spec/08 §8.7, spec/20); roster schema, store table and the
  fixture roster updated. The checker signs a trial revocation to prove
  the rule both ways.
- A qa: receipt raises the nodes it sampled except those in its errors
  (spec/08 §8.2); the example paper's fidelity counts become 27 F3,
  13 F2, 4 F1 and 1 F0.
- The destination chooses the keyed scheme, not the text policy:
  hmac-team between the team's machines, hkdf-pages outside the team
  (spec/05 §5.8, spec/12 §12.5).
- A TextQuoteSelector must repeat the anchor's plain quote and is absent
  for hashed or keyed anchors (spec/18 §18.3).
- Add fixtures/records/suspect.json, a scienting-suspect/1 record.
- Oracle: hashes are kept incrementally and checked after every
  correction against a full recomputation; load and write refuse unsafe
  manifest paths; Ed25519, HKDF and schema validation come from the
  cryptography and jsonschema packages; canonical JSON uses the built-in
  encoder with an exact RFC 8785 fallback.
- The tools need Python 3.14 with cryptography and jsonschema; the py310
  CI environment is dropped and the workflow runs the default only.
- Resolve every [CHECK] note in the docs, READMEs and context; add
  CONTRIBUTING.md; every tool answers --help and refuses unknown options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 18:58:00 -04:00
1 parent b357af84d8
commit 8c8b51cc3f
89 files changed
+1141 -513

No files matched your search

+19 -8
View File
@@ -6,8 +6,8 @@ Implementers get the commands and the traps.
These programs are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, and one loads packages into SQLite, a database stored in a single file.
They use only the Python standard library (Python 3.10 or later [CHECK: stated in README.md; pixi.toml pins 3.14 and I could only run on 3.13]) and are licensed Apache-2.0.
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
@@ -17,14 +17,16 @@ A fixture is an example file or package under `fixtures/`.
| --- | --- |
| `check.py` | Checks everything: schemas, records, packages, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip and the prose. `--quick` skips the rebuild. |
| `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
| `load.py` | Loads packages into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. |
| `validate.py` | Validates packages: `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
| `load.py` | Validates packages, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
| `oracle/` | A compact implementation of every rule. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. |
## When to run what
Each program is also a pixi task.
Pixi sets up a pinned Python and the other packages the repository uses.
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository [CHECK: pixi is not installed here, so the pixi forms were not run]. `pixi run check` lints the Markdown first.
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
`pixi run check` lints the Markdown first.
Run the programs directly from the repository root if you do not use pixi:
@@ -32,12 +34,16 @@ Run the programs directly from the repository root if you do not use pixi:
python3 tools/check.py # everything
python3 tools/check.py --quick # everything except the rebuild
python3 tools/build.py --out /tmp/rebuilt
python3 tools/validate.py fixtures/packages/hale2026harmonic
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
```
A passing check ends with `All checks pass.`
The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
`build.py` has no help option and ignores flags it does not know, so it rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
The full and checks-only forms of one document share its key, so they go into separate databases.
Each program prints its options with `--help` and refuses options it does not know.
`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
Run `pixi run check` before every commit.
Run `pixi run build` after you change a fixture source, a rule in `oracle/` or a case in `build.py`, and commit the regenerated files with the change.
@@ -49,9 +55,14 @@ An oracle is a reference computation: a slow, plain program whose answers count
It is not a production reader or writer.
The oracle exists so that no expected value in this repository is typed by hand.
It favors clarity over speed.
Its Ed25519 (a digital signature scheme) is the slow reference code of RFC 8032.
Its validator for JSON Schema (a format that describes the shape of JSON data) supports exactly the keyword subset the schemas use and refuses any schema that uses more.
It favors clarity, and leaves the cryptography and schema validation to established packages.
Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
Hashes are kept incrementally.
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
Do not build on it as a library.
Implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).