feat: validator, bundle builder and release checks for 0.1.0
- Add tools/validate.py: validates packages, and with --correction runs proposed corrections through the gate against a package without writing anything (spec/11 §11.4). The gate now refuses a proposal that fails the correction schema, rule correction.schema. - tools/load.py validates every package before loading, and with --bundle --built-from builds a scienting-bundle/1 with the FTS5 search index and bundle_meta (spec/13 §13.5). Search rows are defined per unit (sentence, cell, equation outside a sentence, figure-text) with the parent kind as context. - Roster keys carry a role, and a key with the role owner may revoke any receipt (spec/08 §8.7, spec/20); roster schema, store table and the fixture roster updated. The checker signs a trial revocation to prove the rule both ways. - A qa: receipt raises the nodes it sampled except those in its errors (spec/08 §8.2); the example paper's fidelity counts become 27 F3, 13 F2, 4 F1 and 1 F0. - The destination chooses the keyed scheme, not the text policy: hmac-team between the team's machines, hkdf-pages outside the team (spec/05 §5.8, spec/12 §12.5). - A TextQuoteSelector must repeat the anchor's plain quote and is absent for hashed or keyed anchors (spec/18 §18.3). - Add fixtures/records/suspect.json, a scienting-suspect/1 record. - Oracle: hashes are kept incrementally and checked after every correction against a full recomputation; load and write refuse unsafe manifest paths; Ed25519, HKDF and schema validation come from the cryptography and jsonschema packages; canonical JSON uses the built-in encoder with an exact RFC 8785 fallback. - The tools need Python 3.14 with cryptography and jsonschema; the py310 CI environment is dropped and the workflow runs the default only. - Resolve every [CHECK] note in the docs, READMEs and context; add CONTRIBUTING.md; every tool answers --help and refuses unknown options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
89 files changed
+1141
-513
No files matched your search
+19
-8
@@ -6,8 +6,8 @@ Implementers get the commands and the traps.
|
||||
|
||||
These programs are the repository's reference implementation.
|
||||
They apply every rule of the specification, so the rest of the repository can be checked against them.
|
||||
Two build and check the repository, and one loads packages into SQLite, a database stored in a single file.
|
||||
They use only the Python standard library (Python 3.10 or later [CHECK: stated in README.md; pixi.toml pins 3.14 and I could only run on 3.13]) and are licensed Apache-2.0.
|
||||
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
|
||||
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
|
||||
|
||||
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
|
||||
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
|
||||
@@ -17,14 +17,16 @@ A fixture is an example file or package under `fixtures/`.
|
||||
| --- | --- |
|
||||
| `check.py` | Checks everything: schemas, records, packages, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip and the prose. `--quick` skips the rebuild. |
|
||||
| `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
||||
| `load.py` | Loads packages into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. |
|
||||
| `validate.py` | Validates packages: `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
||||
| `load.py` | Validates packages, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
||||
| `oracle/` | A compact implementation of every rule. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. |
|
||||
|
||||
## When to run what
|
||||
|
||||
Each program is also a pixi task.
|
||||
Pixi sets up a pinned Python and the other packages the repository uses.
|
||||
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository [CHECK: pixi is not installed here, so the pixi forms were not run]. `pixi run check` lints the Markdown first.
|
||||
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
|
||||
`pixi run check` lints the Markdown first.
|
||||
|
||||
Run the programs directly from the repository root if you do not use pixi:
|
||||
|
||||
@@ -32,12 +34,16 @@ Run the programs directly from the repository root if you do not use pixi:
|
||||
python3 tools/check.py # everything
|
||||
python3 tools/check.py --quick # everything except the rebuild
|
||||
python3 tools/build.py --out /tmp/rebuilt
|
||||
python3 tools/validate.py fixtures/packages/hale2026harmonic
|
||||
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
|
||||
```
|
||||
|
||||
A passing check ends with `All checks pass.`
|
||||
The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
|
||||
`build.py` has no help option and ignores flags it does not know, so it rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
||||
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
|
||||
The full and checks-only forms of one document share its key, so they go into separate databases.
|
||||
Each program prints its options with `--help` and refuses options it does not know.
|
||||
`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
||||
|
||||
Run `pixi run check` before every commit.
|
||||
Run `pixi run build` after you change a fixture source, a rule in `oracle/` or a case in `build.py`, and commit the regenerated files with the change.
|
||||
@@ -49,9 +55,14 @@ An oracle is a reference computation: a slow, plain program whose answers count
|
||||
It is not a production reader or writer.
|
||||
The oracle exists so that no expected value in this repository is typed by hand.
|
||||
|
||||
It favors clarity over speed.
|
||||
Its Ed25519 (a digital signature scheme) is the slow reference code of RFC 8032.
|
||||
Its validator for JSON Schema (a format that describes the shape of JSON data) supports exactly the keyword subset the schemas use and refuses any schema that uses more.
|
||||
It favors clarity, and leaves the cryptography and schema validation to established packages.
|
||||
Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
|
||||
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
|
||||
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
|
||||
|
||||
Hashes are kept incrementally.
|
||||
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
|
||||
`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
||||
|
||||
Do not build on it as a library.
|
||||
Implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
|
||||
Reference in new issue
Block a user