feat: validator, bundle builder and release checks for 0.1.0

- Add tools/validate.py: validates packages, and with --correction runs
  proposed corrections through the gate against a package without
  writing anything (spec/11 §11.4). The gate now refuses a proposal that
  fails the correction schema, rule correction.schema.
- tools/load.py validates every package before loading, and with
  --bundle --built-from builds a scienting-bundle/1 with the FTS5 search
  index and bundle_meta (spec/13 §13.5). Search rows are defined per
  unit (sentence, cell, equation outside a sentence, figure-text) with
  the parent kind as context.
- Roster keys carry a role, and a key with the role owner may revoke any
  receipt (spec/08 §8.7, spec/20); roster schema, store table and the
  fixture roster updated. The checker signs a trial revocation to prove
  the rule both ways.
- A qa: receipt raises the nodes it sampled except those in its errors
  (spec/08 §8.2); the example paper's fidelity counts become 27 F3,
  13 F2, 4 F1 and 1 F0.
- The destination chooses the keyed scheme, not the text policy:
  hmac-team between the team's machines, hkdf-pages outside the team
  (spec/05 §5.8, spec/12 §12.5).
- A TextQuoteSelector must repeat the anchor's plain quote and is absent
  for hashed or keyed anchors (spec/18 §18.3).
- Add fixtures/records/suspect.json, a scienting-suspect/1 record.
- Oracle: hashes are kept incrementally and checked after every
  correction against a full recomputation; load and write refuse unsafe
  manifest paths; Ed25519, HKDF and schema validation come from the
  cryptography and jsonschema packages; canonical JSON uses the built-in
  encoder with an exact RFC 8785 fallback.
- The tools need Python 3.14 with cryptography and jsonschema; the py310
  CI environment is dropped and the workflow runs the default only.
- Resolve every [CHECK] note in the docs, READMEs and context; add
  CONTRIBUTING.md; every tool answers --help and refuses unknown options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tools/README.md

@@ -6,8 +6,8 @@ Implementers get the commands and the traps.
 
 These programs are the repository's reference implementation.
 They apply every rule of the specification, so the rest of the repository can be checked against them.
-Two build and check the repository, and one loads packages into SQLite, a database stored in a single file.
-They use only the Python standard library (Python 3.10 or later [CHECK: stated in README.md; pixi.toml pins 3.14 and I could only run on 3.13]) and are licensed Apache-2.0.
+Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
+They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
 
 A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
 A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
@@ -17,14 +17,16 @@ A fixture is an example file or package under `fixtures/`.
 | --- | --- |
 | `check.py` | Checks everything: schemas, records, packages, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip and the prose. `--quick` skips the rebuild. |
 | `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
-| `load.py` | Loads packages into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. |
+| `validate.py` | Validates packages: `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
+| `load.py` | Validates packages, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
 | `oracle/` | A compact implementation of every rule. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. |
 
 ## When to run what
 
 Each program is also a pixi task.
 Pixi sets up a pinned Python and the other packages the repository uses.
-The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository [CHECK: pixi is not installed here, so the pixi forms were not run]. `pixi run check` lints the Markdown first.
+The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
+`pixi run check` lints the Markdown first.
 
 Run the programs directly from the repository root if you do not use pixi:
 
@@ -32,12 +34,16 @@ Run the programs directly from the repository root if you do not use pixi:
 python3 tools/check.py            # everything
 python3 tools/check.py --quick    # everything except the rebuild
 python3 tools/build.py --out /tmp/rebuilt
+python3 tools/validate.py fixtures/packages/hale2026harmonic
 python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
 ```
 
 A passing check ends with `All checks pass.`
 The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
-`build.py` has no help option and ignores flags it does not know, so it rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
+It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
+The full and checks-only forms of one document share its key, so they go into separate databases.
+Each program prints its options with `--help` and refuses options it does not know.
+`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
 
 Run `pixi run check` before every commit.
 Run `pixi run build` after you change a fixture source, a rule in `oracle/` or a case in `build.py`, and commit the regenerated files with the change.
@@ -49,9 +55,14 @@ An oracle is a reference computation: a slow, plain program whose answers count
 It is not a production reader or writer.
 The oracle exists so that no expected value in this repository is typed by hand.
 
-It favors clarity over speed.
-Its Ed25519 (a digital signature scheme) is the slow reference code of RFC 8032.
-Its validator for JSON Schema (a format that describes the shape of JSON data) supports exactly the keyword subset the schemas use and refuses any schema that uses more.
+It favors clarity, and leaves the cryptography and schema validation to established packages.
+Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
+The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
+Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
+
+Hashes are kept incrementally.
+A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
+`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
 
 Do not build on it as a library.
 Implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).

tools/build.py

@@ -43,10 +43,10 @@ def secret(name):
 
 
 KEYS = {
-    "ana": {"name": "Ana Example", "purposes": ["sign"]},
-    "ben": {"name": "Ben Example", "purposes": ["sign"]},
-    "ben-spot": {"name": "Ben Example", "purposes": ["spot-check"]},
-    "cara": {"name": "Cara Example", "purposes": ["sign"]},
+    "ana": {"name": "Ana Example", "role": "owner", "purposes": ["sign"]},
+    "ben": {"name": "Ben Example", "role": "editor", "purposes": ["sign"]},
+    "ben-spot": {"name": "Ben Example", "role": "editor", "purposes": ["spot-check"]},
+    "cara": {"name": "Cara Example", "role": "reader", "purposes": ["sign"]},
 }
 TEAM_SECRET = hashlib.sha256(b"manuspec fixture team secret: smithlab").digest()
 
@@ -54,7 +54,7 @@ TEAM_SECRET = hashlib.sha256(b"manuspec fixture team secret: smithlab").digest()
 def roster():
     keys = []
     for kid, k in KEYS.items():
-        keys.append({"id": kid, "person": {"name": k["name"]}, "scheme": "ed25519",
+        keys.append({"id": kid, "person": {"name": k["name"]}, "role": k["role"], "scheme": "ed25519",
                      "public": base64.b64encode(crypto.ed25519_public(secret(kid))).decode(),
                      "purposes": k["purposes"], "registered_at": "2026-09-01T00:00:00Z", "revoked_at": None})
     return {"format": "scienting-roster/1", "store": STORE, "keys": keys}
@@ -118,9 +118,9 @@ def _math_node(kind, e):
 
 
 def expand(s, page=None):
-    page = s.get("page", page)
     if isinstance(s, str):
         s = {"s": [s]}
+    page = s.get("page", page)
     if "s" in s:
         kids = []
         n = {"kind": "sentence", "content": runs_of(s["s"], kids), "children": kids}
@@ -565,7 +565,6 @@ def build_hale(out):
     anchors = []
     for rk, bond, pre in rows:
         for st, (kc, rc) in (("A", ("_ak", "_ar")), ("B", ("_bk", "_br"))):
-            kname = pre + kc.replace("_a", "_a").replace("_b", "_b")
             kid, rid = names.ids[pre + kc], names.ids[pre + rc]
             kv, rv = norm(hashing.plain(final_doc, kid)), norm(hashing.plain(final_doc, rid))
             row_key = rk + "-" + st.lower()
@@ -1007,6 +1006,8 @@ def corrections_vectors():
         ("insert-text", {"op": "insert", "target": nm["@d"], "run": 0, "offset": 8, "text": "only ", "evidence": crop}),
         ("insert-text-no-crop", {"op": "insert", "target": nm["@d"], "run": 0, "offset": 8, "text": "only ",
                                  "evidence": person}),
+        ("insert-text-bad-run", {"op": "insert", "target": nm["@d"], "run": 7, "offset": 0, "text": "x",
+                                 "evidence": crop}),
         ("insert-node", {"op": "insert", "parent": nm["@p2"], "place_after": nm["@c"],
                          "node": {"kind": "sentence", "content": [{"kind": "text", "text": "A dropped sentence."}]},
                          "evidence": crop}),
@@ -1050,7 +1051,6 @@ def resolve_vectors(hale):
         nid = names.ids[a["node"]]
         if a.get("container_of"):
             nid = d0.nodes[nid]["parent"]
-            cid = cid
         span = None
         if "quote" in a:
             t = hashing.plain(d0, nid)
@@ -1134,6 +1134,12 @@ def records(out, hale):
         return AN.make_anchor(doc, vals, nid, rev, span=span, store=STORE)
 
     wr(out, "roster.json", ROSTER)
+    # The doubtful reading behind the first correction, as the converter raised it before anyone answered it.
+    sus = {"format": "scienting-suspect/1", "kind": "number", "node": I["@s4"], "a": "10-3", "b": "10\u22123",
+           "crop": crop_sha(I["@s4"]),
+           "note": "The text layer reads a hyphen on the line; the page image shows a raised minus three."}
+    sus = dict(sus, id=record_id(sha256_hex(jcs.dumpb(sus))))
+    wr(out, "suspect.json", sus)
     wr(out, "anchor-span.json", anchor("@s3", "first test case"))
     wr(out, "anchor-cell.json", anchor("@cc_ak"))
     wr(out, "anchor-paragraph.json", AN.make_anchor(doc, vals, doc.nodes[I["@s3"]]["parent"], rev, store=STORE))
@@ -1393,9 +1399,8 @@ def records(out, hale):
 # --- invalid packages ---------------------------------------------------------------
 
 
-def invalid(out):
+def invalid(out, tiny):
     base = os.path.join(out, "fixtures", "invalid")
-    tiny = tiny_package(None)
     files = tiny["files"]
     nodes = [json.loads(x) for x in files["nodes.jsonl"].decode().splitlines()]
     nm = tiny["names"].ids
@@ -1418,7 +1423,6 @@ def invalid(out):
         with open(os.path.join(base, name, "expect.json"), "wb") as fh:
             fh.write((jcs.dumps({"rule": rule, "note": note}) + "\n").encode())
 
-    write(os.path.join(out, "fixtures", "packages", "tiny2026example"), files)
     case("hash-mismatch", "doc.hash.node-hash", {"nodes.jsonl": edit_nodes(lambda ns: by(ns, "@a").update(node_hash="0" * 64))},
          "A node_hash that does not recompute.")
     case("text-edited", "doc.hash.text-hash", {"nodes.jsonl": edit_nodes(
@@ -1531,16 +1535,24 @@ def main(out):
     tiny = tiny_package(os.path.join(out, "fixtures", "packages", "tiny2026example"))
     records(out, hale)
     vectors(out, hale, tiny)
-    invalid(out)
+    invalid(out, tiny)
+
+
+def cli():
+    import argparse
+    ap = argparse.ArgumentParser(
+        description="Build every fixture and test vector from fixtures/src and the oracle. The build is "
+                    "deterministic, and tools/check.py rebuilds into a temporary folder to prove it.")
+    ap.add_argument("--out", metavar="DIR", default=ROOT,
+                    help="build into DIR instead of rewriting fixtures/ and vectors/ in this repository")
+    out = os.path.abspath(ap.parse_args().out)
+    if out != ROOT:
+        for sub in ("registry", "fixtures/src"):
+            dst = os.path.join(out, sub)
+            if not os.path.exists(dst):
+                shutil.copytree(os.path.join(ROOT, sub), dst)
+    main(out)
 
 
 if __name__ == "__main__":
-    out = ROOT
-    if "--out" in sys.argv:
-        out = sys.argv[sys.argv.index("--out") + 1]
-        if out != ROOT:
-            for sub in ("registry", "fixtures/src"):
-                dst = os.path.join(out, sub)
-                if not os.path.exists(dst):
-                    shutil.copytree(os.path.join(ROOT, sub), dst)
-    main(out)
+    cli()

tools/check.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """Check the whole repository. Exit status 0 means every check passed.
 
-Non-normative tooling; standard library only (it uses `jsonschema` as a second validator when installed).
+Non-normative tooling; it needs Python 3.14 and the packages pixi installs (`jsonschema`, `cryptography`).
 Usage: python3 tools/check.py [--quick]   (--quick skips the determinism rebuild)
 """
 
@@ -13,6 +13,7 @@ import re
 import sqlite3
 import subprocess
 import sys
+import shutil
 import tempfile
 from fractions import Fraction
 
@@ -27,6 +28,8 @@ from tools.oracle import package as PK  # noqa: E402
 from tools.oracle.model import Doc  # noqa: E402
 from tools.oracle.notation import fold_math  # noqa: E402
 from tools.oracle.schema import Store, validate as sv  # noqa: E402
+from jsonschema import Draft202012Validator  # noqa: E402
+from jsonschema.exceptions import SchemaError  # noqa: E402
 from tools.oracle.text import cp_to_utf16, letter_first, mint_node_id, norm, record_id, sha256_hex  # noqa: E402
 from tools.oracle.tree import Refusal  # noqa: E402
 
@@ -69,35 +72,20 @@ def check_schemas():
     for f in SCHEMA_FILES:
         s = STORE.load(f)
         ok("schemas", s.get("$id", "").endswith("/" + f), "%s: $id must end with its file name" % f)
+        try:
+            Draft202012Validator.check_schema(s)
+            meta = True
+        except SchemaError:
+            meta = False
+        ok("schemas", meta, "%s is not a valid draft 2020-12 schema" % f)
     for f in js(p("registry", "formats.json"))["formats"]:
         ok("schemas", os.path.exists(p("schemas", f["schema"])) or os.path.exists(p(f["schema"])),
            "%s names a missing schema %s" % (f["format"], f["schema"]))
-    try:
-        import jsonschema  # noqa: F401
-        from referencing import Registry, Resource
-        reg = Registry().with_resources(
-            [(STORE.load(f)["$id"], Resource.from_contents(STORE.load(f))) for f in SCHEMA_FILES])
-        COUNTS["second-validator"] = 0
-        globals()["SECOND"] = (jsonschema, reg)
-    except ImportError:
-        globals()["SECOND"] = None
-
-
-def second(name, rec):
-    if not SECOND:
-        return
-    jsonschema, reg = SECOND
-    v = jsonschema.Draft202012Validator(STORE.load(name), registry=reg)
-    errs = list(v.iter_errors(rec))
-    COUNTS["second-validator"] += 1
-    if errs:
-        FAILS.append("second-validator: %s: %s" % (name, errs[0].message))
 
 
 def valid(section, name, rec, where):
     e = schema_errors(name, rec)
     ok(section, not e, "%s: %s" % (where, e[:3]))
-    second(name, rec)
 
 
 def check_registry():
@@ -106,6 +94,13 @@ def check_registry():
     for rule in js(p("registry", "stamps", "scienting-stamps-1.json"))["rules"]:
         re.compile(rule["pattern"])
     rules = {r["id"] for r in js(p("registry", "rules.json"))["rules"]}
+    for f in sorted(os.listdir(p("tools", "oracle"))):
+        if f.endswith(".py"):
+            for rule in re.findall(r'(?:Refusal|rep\.add)\(\s*"([a-z][a-z0-9.-]+)"', open(p("tools", "oracle", f)).read()):
+                if not rule.endswith("."):
+                    ok("registry", rule in rules, "tools/oracle/%s raises %s, which is not in rules.json" % (f, rule))
+    for v in ("text-hash", "match-hash", "align-key", "own-hash", "node-hash"):
+        ok("registry", "doc.hash." + v in rules, "doc.hash.%s is not in rules.json" % v)
     for d in sorted(os.listdir(p("fixtures", "invalid"))):
         ok("registry", js(p("fixtures", "invalid", d, "expect.json"))["rule"] in rules, "%s: rule is not in rules.json" % d)
 
@@ -133,13 +128,28 @@ def check_records():
             else:
                 sch = FORMAT_SCHEMA[rec["format"]]
             valid("records", sch, rec, rel)
+            if rec.get("format") == "scienting-suspect/1":
+                ok("records", rec["id"] == record_id(sha256_hex(jcs.dumpb({k: v for k, v in rec.items() if k != "id"}))),
+                   "%s: id does not derive" % rel)
             if rec.get("signature"):
                 ok("records", RC.verify_signature(rec, roster), "%s: signature does not verify" % rel)
                 ok("records", rec["id"] == record_id(RC.record_hash(rec)), "%s: id does not derive" % rel)
+    # A project's settings.stages holds the id of a stage file.
+    stage_ids = {js(p("registry", "stages", f))["id"] for f in os.listdir(p("registry", "stages")) if f.endswith(".json")}
+    stages = js(os.path.join(R, "project.json")).get("settings", {}).get("stages")
+    ok("records", stages is None or stages in stage_ids, "project settings.stages names no stage file: %s" % stages)
     # Confirmed annotations are covered by the layer receipt at their content hash.
     lr = js(os.path.join(R, "layer", "receipt.json"))
     items = {(k, i): h for k, i, h in lr["records"]}
     for a in jl(os.path.join(R, "layer", "annotations.jsonl")):
+        sel = a["target"].get("selector", [])
+        anchors = [x["anchor"] for x in sel if x.get("type") == "AnchorSelector"]
+        for x in sel:
+            if x.get("type") == "TextQuoteSelector":
+                q = anchors[0].get("quote", {}) if anchors else {}
+                same = all(x.get(k, "") == q.get(k, "") for k in ("exact", "prefix", "suffix"))
+                ok("records", "exact" in q and same and set(x) <= {"type", "exact", "prefix", "suffix"},
+                   "a TextQuoteSelector must repeat the anchor's plain quote: " + a["id"])
         body = {k: v for k, v in a.items() if k not in ("status", "receipt", "modified")}
         ok("records", a["id"].rsplit("/", 1)[1] == record_id(sha256_hex(jcs.dumpb(
             {k: v for k, v in body.items() if k != "id"}))), "annotation id does not derive: " + a["id"])
@@ -150,12 +160,7 @@ def check_records():
 
 
 def state_of(pkg):
-    nodes = jl(os.path.join(pkg, "nodes.jsonl"))
-    key = js(os.path.join(pkg, "document.json"))["key"]
-    ids = jl(os.path.join(pkg, "ids.jsonl")) if os.path.exists(os.path.join(pkg, "ids.jsonl")) else []
-    cors = jl(os.path.join(pkg, "corrections.jsonl")) if os.path.exists(os.path.join(pkg, "corrections.jsonl")) else []
-    strip = ("text_hash", "match_hash", "align_key", "own_hash", "node_hash")
-    return CO.State(key, [{k: v for k, v in n.items() if k not in strip} for n in nodes], ids, cors)
+    return PK.read_state(pkg)
 
 
 def check_packages():
@@ -192,6 +197,142 @@ def check_packages():
         valid("packages", "mathml.schema.json", m, "derived/mathml.jsonl")
 
 
+def check_owner_revocation():
+    """A key the roster makes an owner may revoke anyone's receipt (spec/08 §8.7); any other key may not.
+
+    No fixture has such a revocation, so this signs one: Ben revokes a receipt of Ana's, once with the
+    roster role owner and once as the editor he is.
+    """
+    hp = p("fixtures", "packages", "hale2026harmonic")
+    revoker = "ben"
+    secret = bytes.fromhex(open(p("fixtures", "keys", revoker + ".ed25519.hex")).read().strip())
+    roster_rec = js(os.path.join(hp, "roster.json"))
+    as_owner = dict(roster_rec, keys=[dict(k, role="owner") if k["id"] == revoker else k for k in roster_rec["keys"]])
+    owner_roster = {k["id"]: k for k in as_owner["keys"]}
+    nodes = jl(os.path.join(hp, "nodes.jsonl"))
+    vals = {n["id"]: n for n in nodes}
+    document = js(os.path.join(hp, "document.json"))
+    recs, fidelity = jl(os.path.join(hp, "receipts.jsonl")), None
+    for target in recs:
+        if target["format"] != "scienting-receipt/1" or target["by"] == revoker:
+            continue
+        rev = RC.seal({"format": "scienting-revocation/1", "receipt": target["id"], "reason": "Owner check.",
+                       "by": revoker, "at": "2026-10-13T00:00:00Z"}, secret, revoker)
+        trial = sorted(recs + [rev], key=lambda r: (r["at"], r["id"]))
+        fidelity = RC.distribution(RC.effective_levels(nodes, vals, trial, owner_roster))
+        if fidelity != document["fidelity"]:
+            recs = trial
+            break
+    ok("packages", fidelity != document["fidelity"], "no receipt of the hale package matters to its levels")
+    document["fidelity"] = fidelity
+    manifest = js(os.path.join(hp, "MANIFEST.json"))
+    files, absent = {}, [f for f in manifest["files"] if f.get("present", True) is False]
+    for f in manifest["files"]:
+        if f.get("present", True):
+            with open(os.path.join(hp, f["path"]), "rb") as fh:
+                files[f["path"]] = fh.read()
+    files["receipts.jsonl"] = PK.jsonl(recs)
+    files["document.json"] = PK.line(document).encode("utf-8")
+    for roster, passes in ((as_owner, True), (roster_rec, False)):
+        files["roster.json"] = PK.line(roster).encode("utf-8")
+        with tempfile.TemporaryDirectory() as tmp:
+            PK.write(tmp, files, absent)
+            rules = {e["rule"] for e in PK.validate(tmp, STORE).errors}
+        role = next(k["role"] for k in roster["keys"] if k["id"] == revoker)
+        ok("packages", ("doc.fidelity.report" not in rules) == passes and rules <= {"doc.fidelity.report"},
+           "a revocation by a key with the role %s: %s" % (role, sorted(rules)))
+
+
+def check_unsafe_paths():
+    """Loading and writing refuse manifest paths that leave the package, even without a prior validation."""
+    with tempfile.TemporaryDirectory() as tmp:
+        pkg = os.path.join(tmp, "pkg")
+        shutil.copytree(p("fixtures", "packages", "tiny2026example"), pkg)
+        manifest = js(os.path.join(pkg, "MANIFEST.json"))
+        for bad in ("../outside", "/etc/outside", "assets/../../outside"):
+            m = dict(manifest, files=[{"path": bad, "sha256": "0" * 64, "size": 1}] + manifest["files"])
+            with open(os.path.join(pkg, "MANIFEST.json"), "w", encoding="utf-8") as f:
+                f.write(PK.line(m))
+            conn = sqlite3.connect(":memory:")
+            db.create(conn, p("sql"))
+            try:
+                db.load(conn, pkg, os.path.join(tmp, "assets"))
+                rule = None
+            except Refusal as e:
+                rule = e.rule
+            ok("packages", rule == "package.path.unsafe", "load accepted the manifest path %r" % bad)
+            try:
+                PK.write(os.path.join(tmp, "out"), {bad: b"x", "document.json": b"{}"})
+                rule = None
+            except Refusal as e:
+                rule = e.rule
+            ok("packages", rule == "package.path.unsafe", "write accepted the path %r" % bad)
+        ok("packages", not os.path.exists(os.path.join(tmp, "outside")), "a file was written outside the package")
+
+
+def reference_values(doc, hasher=hashing.PLAIN):
+    """The five values the slow way: every node projected from its whole subtree, as spec/05 §5.1 reads.
+
+    The oracle's hashing.Index joins its children's cached projections and rehashes only what a correction
+    changed; this independent computation must always agree with it.
+    """
+    values = {}
+    for nid in reversed(doc.preorder()):
+        st, _ = hashing.strict(doc, nid)
+        own = hasher(jcs.dumpb(hashing.own_content(doc, nid, lambda x: values[x]["text_hash"])))
+        values[nid] = {"text_hash": hasher(norm(hashing.plain(doc, nid))), "match_hash": hasher(st),
+                       "align_key": hasher(hashing.lossy_from_strict(st)), "own_hash": own,
+                       "node_hash": hasher(jcs.dumpb({"own": own, "children": [
+                           values[c]["node_hash"] for c in doc.kids(nid)]}))}
+    return values
+
+
+def check_incremental():
+    """After every correction of every fixture source, the incrementally kept values equal the slow ones."""
+    from tools.build import Names, build_document
+    team = hashing.Hasher(crypto.team_key(bytes.fromhex(
+        open(p("fixtures", "keys", "smithlab-team-secret.hex")).read().strip())))
+    Names.registry.clear()
+    # Sorted, the main example comes first, so the draft's references to its sentences resolve.
+    for f in sorted(os.listdir(p("fixtures", "src"))):
+        if not f.endswith(".json"):
+            continue
+        _, _, _, _, states, _ = build_document(js(p("fixtures", "src", f)))
+        keyed = None
+        for rev in sorted(states):
+            st = states[rev]
+            ok("packages", st.index().values == reference_values(st.doc()),
+               "%s rev %d: incremental values differ from the full computation" % (f, rev))
+            # The keyed values, carried forward from revision 0 by updates alone.
+            keyed = hashing.Index.build(st.doc(), team) if keyed is None else keyed.update(st.doc())
+            ok("packages", keyed.values == reference_values(st.doc(), team),
+               "%s rev %d: incremental keyed values differ from the full computation" % (f, rev))
+    Names.registry.clear()
+
+
+def check_gate():
+    """The gate runs on a proposal against a package's current state and leaves that state unchanged."""
+    st = state_of(p("fixtures", "packages", "hale2026harmonic"))
+    before = json.dumps(st.nodes, sort_keys=True)
+    target = next(n for n in st.nodes if n["kind"] == "sentence" and "Each bond" in hashing.plain(st.doc(), n["id"]))
+    base = {"format": "scienting-correction/1", "evidence": [{"kind": "person", "by": "ana"}], "author": {"key": "ana"},
+            "reason": "Gate check.", "at": "2026-10-14T09:00:00Z", "status": "proposed"}
+    st1, rec = CO.gate(st, dict(base, op="replace", target=target["id"], find="Each bond", replace="Every bond"), STORE)
+    ok("packages", rec["index"] == st.rev + 1 and st1.rev == st.rev + 1, "the gate's record has the next index")
+    ok("packages", json.dumps(st.nodes, sort_keys=True) == before, "the gate changed the state it was given")
+    valid("packages", "correction.schema.json", rec, "gate record")
+    for bad, rule in (({"op": "replace", "target": target["id"]}, "correction.schema"),
+                      (dict(base, op="replace", target=target["id"], find="0.01", replace="0.02"), "correction.digits"),
+                      (dict(base, op="insert", target=target["id"], run=9, offset=0, text="x",
+                            evidence=[{"kind": "crop", "sha256": "0" * 64}]), "correction.insert")):
+        try:
+            CO.gate(st, bad, STORE)
+            got = None
+        except Refusal as e:
+            got = e.rule
+        ok("packages", got == rule, "the gate gave %s for a proposal that breaks %s" % (got, rule))
+
+
 def check_invalid():
     base = p("fixtures", "invalid")
     for d in sorted(os.listdir(base)):
@@ -299,6 +440,8 @@ def check_vectors():
         try:
             st1, rec = CO.apply(st, c["correction"])
             v1 = hashing.compute(st1.doc())
+            ok("vectors", st1.index().values == reference_values(st1.doc()),
+               "correction %s: incremental values differ from the full computation" % c["id"])
             got = {"record": rec, "nodes": [dict(n, **v1[n["id"]]) for n in st1.nodes], "ids": st1.ids}
         except Refusal as e:
             got = {"refusal": e.rule}
@@ -330,7 +473,8 @@ def check_determinism():
     with tempfile.TemporaryDirectory() as tmp:
         subprocess.run([sys.executable, p("tools", "build.py"), "--out", tmp], check=True)
         for sub in ("fixtures", "vectors"):
-            cmp = filecmp.dircmp(p(sub), os.path.join(tmp, sub))
+            # Files the repository ignores, such as macOS's .DS_Store, are not part of the build.
+            cmp = filecmp.dircmp(p(sub), os.path.join(tmp, sub), ignore=filecmp.DEFAULT_IGNORES + [".DS_Store"])
             diffs = []
 
             def walk(c, prefix):
@@ -384,8 +528,7 @@ def check_sql():
     ok("sql", "FOREIGN KEY" not in ddl and "REFERENCES" not in ddl, "no foreign keys")
     with tempfile.TemporaryDirectory() as tmp:
         conn = sqlite3.connect(os.path.join(tmp, "bundle.db"))
-        db.create(conn, p("sql"))
-        conn.executescript(open(p("sql", "bundle-sqlite.sql"), encoding="utf-8").read())
+        db.create_bundle(conn, p("sql"))
         assets = os.path.join(tmp, "assets")
         pk = p("fixtures", "packages")
         for d in sorted(os.listdir(pk)):
@@ -393,29 +536,18 @@ def check_sql():
             if key is None:
                 # A second load of the same key goes to its own database.
                 c2 = sqlite3.connect(":memory:")
-                db.create(c2, p("sql"))
+                db.create_bundle(c2, p("sql"))
                 key = db.load(c2, os.path.join(pk, d), assets)
+                ok("sql", db.index(c2, key) == 0, "%s: withheld text was indexed" % d)
                 h = db.unload(c2, key, os.path.join(tmp, "out", d), assets)
             else:
+                db.index(conn, key)
                 h = db.unload(conn, key, os.path.join(tmp, "out", d), assets)
-            cmp = filecmp.dircmp(os.path.join(pk, d), os.path.join(tmp, "out", d))
             same = _same_tree(os.path.join(pk, d), os.path.join(tmp, "out", d))
             ok("sql", same, "%s: unload is not byte-identical" % d)
             with open(os.path.join(pk, d, "MANIFEST.json"), "rb") as f:
                 ok("sql", sha256_hex(f.read()) == h, "%s: package hash differs after unload" % d)
-        # The search index over sentences, captions, headings and cells.
-        for doc_key, nid, kind, parent in conn.execute(
-                "SELECT n.doc_key, n.id, n.kind, p.kind FROM nodes n JOIN nodes p ON p.doc_key = n.doc_key AND p.id = n.parent "
-                "WHERE n.kind IN ('sentence', 'cell', 'equation') AND n.withheld IS NULL").fetchall():
-            st = None
-        rows = conn.execute("SELECT doc_key, id, kind, content FROM nodes WHERE kind IN ('sentence','cell')").fetchall()
-        for doc_key, nid, kind, content in rows:
-            runs = json.loads(content)
-            text = "".join(r.get("text", "") if r["kind"] != "math" else "$" + r["source"] + "$" for r in runs
-                           if r["kind"] != "equation")
-            conn.execute("INSERT INTO search VALUES (?,?,?,?,?)", (doc_key, nid, kind, "", norm(text)))
-        hits = conn.execute("SELECT node FROM search WHERE search MATCH 'scanned'").fetchall()
-        ok("sql", len(hits) == 1, "full-text search finds the merged sentence")
+        check_bundle(conn)
         # Store records satisfy the store tables' CHECK constraints.
         R = p("fixtures", "records")
         lay = js(os.path.join(R, "layer", "layer.json"))
@@ -433,8 +565,8 @@ def check_sql():
         for m in prj["members"]:
             conn.execute("INSERT INTO members VALUES (?,?,?,?)", (prj["key"], m["person"], m["role"], jcs.dumps(m["keys"])))
         for k in js(os.path.join(R, "roster.json"))["keys"]:
-            conn.execute("INSERT INTO roster_keys VALUES (?,?,?,?,?,?,?,?)",
-                         ("smithlab", k["id"], jcs.dumps(k["person"]), k["scheme"], k["public"],
+            conn.execute("INSERT INTO roster_keys VALUES (?,?,?,?,?,?,?,?,?)",
+                         ("smithlab", k["id"], jcs.dumps(k["person"]), k["role"], k["scheme"], k["public"],
                           jcs.dumps(k["purposes"]), k["registered_at"], k["revoked_at"]))
         dg = js(os.path.join(R, "digitization.json"))
         conn.execute("INSERT INTO digitizations VALUES (?,?,?,?,?,?,?,?)",
@@ -444,6 +576,51 @@ def check_sql():
         ok("sql", True, "store records loaded")
 
 
+def check_bundle(conn):
+    """The bundle's search index and metadata (spec/13 §13.5), built by the oracle's db.index and db.write_meta."""
+    units = dict(conn.execute("SELECT unit, COUNT(*) FROM search GROUP BY unit").fetchall())
+    for u in ("sentence", "cell", "figure-text"):
+        ok("sql", units.get(u, 0) > 0, "the search index has no %s rows" % u)
+    # No fixture has an equation outside a sentence, so a small document shows both kinds of equation.
+    nodes = [{"id": "d", "kind": "document", "parent": None, "ord": 0},
+             {"id": "p", "kind": "paragraph", "parent": "d", "ord": 0},
+             {"id": "s", "kind": "sentence", "parent": "p", "ord": 0,
+              "content": [{"kind": "text", "text": "Then "}, {"kind": "equation", "node": "e1"}]},
+             {"id": "e1", "kind": "equation", "parent": "s", "ord": 0, "content": {"notation": "tex", "source": "a=b"}},
+             {"id": "e2", "kind": "equation", "parent": "p", "ord": 1, "content": {"notation": "tex", "source": "c=d"}}]
+    got = [(n, u, c) for _, n, u, c, _ in db.search_rows("k", nodes)]
+    ok("sql", got == [("s", "sentence", "paragraph"), ("e2", "equation", "paragraph")],
+       "search rows for a sentence with an inline equation and a display equation: %s" % got)
+    ok("sql", set(units) <= {"sentence", "cell", "equation", "figure-text"}, "unknown search units %s" % sorted(units))
+    expect = conn.execute(
+        "SELECT COUNT(*) FROM nodes n LEFT JOIN nodes q ON q.doc_key = n.doc_key AND q.id = n.parent "
+        "WHERE n.withheld IS NULL AND (n.kind IN ('sentence', 'cell') OR (n.kind = 'equation' AND q.kind <> 'sentence'))"
+    ).fetchone()[0]
+    ok("sql", sum(v for u, v in units.items() if u != "figure-text") == expect,
+       "the search index does not have one row per sentence, cell and equation outside a sentence")
+    inner = conn.execute(
+        "SELECT COUNT(*) FROM search s JOIN nodes n ON n.doc_key = s.doc_key AND n.id = s.node "
+        "JOIN nodes q ON q.doc_key = n.doc_key AND q.id = n.parent WHERE n.kind = 'equation' AND q.kind = 'sentence'"
+    ).fetchone()[0]
+    ok("sql", inner == 0, "an equation inside a sentence is indexed apart from its sentence")
+    bad = conn.execute(
+        "SELECT COUNT(*) FROM search s JOIN nodes n ON n.doc_key = s.doc_key AND n.id = s.node "
+        "LEFT JOIN nodes q ON q.doc_key = n.doc_key AND q.id = n.parent "
+        "WHERE s.unit <> 'figure-text' AND s.context IS NOT q.kind").fetchone()[0]
+    ok("sql", bad == 0, "a search row's context is not the kind of its node's parent")
+    hits = conn.execute("SELECT node FROM search WHERE search MATCH 'scanned'").fetchall()
+    ok("sql", len(hits) == 1, "full-text search finds the merged sentence")
+    hits = conn.execute("SELECT context FROM search WHERE unit = 'sentence' AND search MATCH 'Methods'").fetchall()
+    ok("sql", ("heading",) in hits, "a heading sentence is indexed with the context heading")
+    stores = [r[0] for r in conn.execute("SELECT DISTINCT store FROM documents WHERE store IS NOT NULL")]
+    db.write_meta(conn, stores[0] if len(stores) == 1 else "", "fixtures/packages")
+    meta = dict(conn.execute("SELECT name, value FROM bundle_meta").fetchall())
+    ok("sql", {"format", "store", "built_from", "spec"} <= set(meta), "bundle_meta lacks a required name")
+    ok("sql", meta["format"] in FORMAT_SCHEMA, "bundle_meta names an unregistered format")
+    top = re.search(r"^## (\S+)", open(p("CHANGELOG.md"), encoding="utf-8").read(), re.M).group(1)
+    ok("sql", meta["spec"] == top, "bundle_meta spec %s is not the changelog's latest version %s" % (meta["spec"], top))
+
+
 def _same_tree(a, b):
     fa = sorted(os.path.relpath(os.path.join(d, f), a) for d, _, fs in os.walk(a) for f in fs)
     fb = sorted(os.path.relpath(os.path.join(d, f), b) for d, _, fs in os.walk(b) for f in fs)
@@ -525,13 +702,24 @@ def check_links():
 
 
 def main():
+    import argparse
+    ap = argparse.ArgumentParser(
+        description="Check the whole repository: schemas, records, packages, broken packages, vectors, a "
+                    "byte-for-byte rebuild, the SQL round trip and the bundle, prose and links. "
+                    "Exit status 0 means every check passed.")
+    ap.add_argument("--quick", action="store_true", help="skip the byte-for-byte rebuild, the slowest check")
+    args = ap.parse_args()
     check_schemas()
     check_registry()
     check_records()
     check_packages()
+    check_owner_revocation()
+    check_unsafe_paths()
+    check_incremental()
+    check_gate()
     check_invalid()
     check_vectors()
-    if "--quick" not in sys.argv:
+    if not args.quick:
         check_determinism()
     check_sql()
     check_prose()

tools/load.py

@@ -1,8 +1,12 @@
 #!/usr/bin/env python3
 """Load packages into a SQLite database with the tables of sql/doc.sql and sql/store.sql.
 
-Non-normative tooling, standard library only.
+Non-normative tooling.
 Usage: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]
+       python3 tools/load.py --sqlite lab.db --bundle --built-from REV PACKAGE [PACKAGE ...]
+With --bundle the new file is a scienting-bundle/1: it also gets the tables of sql/bundle-sqlite.sql,
+the full-text search index and bundle_meta (spec/13 §13.5).
+Each package is validated first, and nothing is loaded if any package fails.
 Files without tables of their own (assets, datasets, derived files) are copied by SHA-256 into an
 asset store folder beside the database, named <database>.assets/.
 """
@@ -15,22 +19,46 @@ import sys
 ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 sys.path.insert(0, ROOT)
 
-from tools.oracle import db  # noqa: E402
+from tools.oracle import db, package  # noqa: E402
+from tools.oracle.schema import Store  # noqa: E402
 
 
 def main():
     ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
     ap.add_argument("--sqlite", required=True, help="the database file; created with the tables if new")
+    ap.add_argument("--bundle", action="store_true", help="build a new scienting-bundle/1 file with search")
+    ap.add_argument("--built-from", help="with --bundle: the store revision the packages come from")
+    ap.add_argument("--store", help="with --bundle: the store's name (default: the packages' common store)")
     ap.add_argument("packages", nargs="+", help="package folders")
     a = ap.parse_args()
+    if a.bundle and (os.path.exists(a.sqlite) or not a.built_from):
+        ap.error("--bundle needs --built-from and a database file that does not exist yet")
+    store, bad = Store(os.path.join(ROOT, "schemas")), False
+    for p in a.packages:
+        for e in package.validate(p, store).errors:
+            print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
+            bad = True
+    if bad:
+        sys.exit("Nothing was loaded.")
     new = not os.path.exists(a.sqlite)
     conn = sqlite3.connect(a.sqlite)
-    if new:
+    if a.bundle:
+        db.create_bundle(conn, os.path.join(ROOT, "sql"))
+    elif new:
         db.create(conn, os.path.join(ROOT, "sql"))
     for p in a.packages:
         key = db.load(conn, p, a.sqlite + ".assets")
         n = conn.execute("SELECT COUNT(*) FROM nodes WHERE doc_key = ?", (key,)).fetchone()[0]
-        print("%s: %d nodes" % (key, n))
+        if a.bundle:
+            print("%s: %d nodes, %d search rows" % (key, n, db.index(conn, key)))
+        else:
+            print("%s: %d nodes" % (key, n))
+    if a.bundle:
+        stores = {r[0] for r in conn.execute("SELECT DISTINCT store FROM documents WHERE store IS NOT NULL")}
+        store = a.store or (stores.pop() if len(stores) == 1 else None)
+        if store is None:
+            sys.exit("The packages name no single store; give one with --store.")
+        db.write_meta(conn, store, a.built_from)
     conn.commit()
 
 

tools/oracle/__init__.py

@@ -0,0 +1,13 @@
+"""The reference computations behind every expected value in this repository; see tools/README.md."""
+
+import importlib.util
+import sys
+
+if sys.version_info < (3, 14):
+    sys.exit("The manuspec tools need Python 3.14 or later; this is Python %d.%d." % sys.version_info[:2])
+
+_missing = [m for m in ("cryptography", "jsonschema") if importlib.util.find_spec(m) is None]
+if _missing:
+    sys.exit("The manuspec tools need %s %s; run them through pixi, or install %s with pip."
+             % ("the package" if len(_missing) == 1 else "the packages", " and ".join(_missing),
+                "it" if len(_missing) == 1 else "them"))

tools/oracle/corrections.py

@@ -9,6 +9,7 @@ from . import hashing
 from .model import LABELLED, LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc, CHILDREN
 from .tree import Refusal, canonical_runs, mint_ids, validate_structure, _flatten
 from . import jcs
+from .schema import validate as schema_validate
 from .text import digits, record_id, sha256_hex
 
 MERGEABLE = RUN_KINDS | {"paragraph", "list", "item", "quote", "caption", "figure-note", "table-note"}
@@ -27,11 +28,12 @@ EDITABLE_RUN_FIELD = {"text": "text", "link": "text", "cite": "text", "xref": "t
 class State:
     """The current state of a document: nodes, the ID table and the corrections applied."""
 
-    def __init__(self, key, nodes, ids=None, corrections=None):
+    def __init__(self, key, nodes, ids=None, corrections=None, indexes=None):
         self.key = key
         self.nodes = [dict(n) for n in nodes]
         self.ids = [dict(e) for e in (ids or [])]
         self.corrections = [dict(c) for c in (corrections or [])]
+        self._indexes = dict(indexes or {})
 
     @property
     def rev(self):
@@ -40,6 +42,12 @@ class State:
     def doc(self):
         return Doc(self.key, self.nodes)
 
+    def index(self, hasher=hashing.PLAIN):
+        """The node values under `hasher`, kept with the state so the next correction rehashes only what changes."""
+        if hasher.key not in self._indexes:
+            self._indexes[hasher.key] = hashing.Index.build(self.doc(), hasher)
+        return self._indexes[hasher.key]
+
     def used_ids(self):
         return {n["id"] for n in self.nodes} | {e["id"] for e in self.ids}
 
@@ -120,11 +128,28 @@ def _place(nodes, nid, parent, after):
     by_id[nid]["ord"] = pos
 
 
+def gate(state, proposal, store, hasher=hashing.PLAIN):
+    """Run the gate (spec/11 §11.4) on a proposed correction, leaving `state` unchanged.
+
+    Returns the state the correction would produce and the record it would append to corrections.jsonl, as apply
+    does. Raises Refusal naming the rule broken. `store` is the schema store that holds correction.schema.json.
+    """
+    errors = schema_validate(store, "correction.schema.json", proposal)
+    if errors:
+        raise Refusal("correction.schema", "; ".join(errors[:3]))
+    return apply(state, proposal, hasher)
+
+
+def _is_index(v):
+    return isinstance(v, int) and not isinstance(v, bool) and v >= 0
+
+
 def apply(state, c, hasher=hashing.PLAIN):
     """Apply correction `c` to `state` and return the recorded correction. Raises Refusal."""
     c = copy.deepcopy(c)
     doc = state.doc()
-    before_vals = hashing.compute(doc, hasher)
+    before_ix = state.index(hasher)
+    before_vals = before_ix.values
     nodes = [copy.deepcopy(n) for n in state.nodes]
     by_id = {n["id"]: n for n in nodes}
     op = c["op"]
@@ -194,11 +219,14 @@ def apply(state, c, hasher=hashing.PLAIN):
         node = by_id[target]
         if node["kind"] not in RUN_KINDS:
             raise Refusal("correction.insert", "text is inserted into runs", target)
-        r = node["content"][c["run"]]
-        f = EDITABLE_RUN_FIELD.get(r["kind"])
-        if not f or not (0 <= c["offset"] <= len(r[f])):
+        run, offset = c.get("run"), c.get("offset")
+        if not _is_index(run) or run >= len(node["content"]) or not _is_index(offset):
             raise Refusal("correction.insert", "bad run or offset", target)
-        node["content"][c["run"]] = dict(r, **{f: r[f][: c["offset"]] + c["text"] + r[f][c["offset"]:]})
+        r = node["content"][run]
+        f = EDITABLE_RUN_FIELD.get(r["kind"])
+        if not f or offset > len(r[f]) or not isinstance(c.get("text"), str):
+            raise Refusal("correction.insert", "bad run or offset", target)
+        node["content"][run] = dict(r, **{f: r[f][:offset] + c["text"] + r[f][offset:]})
         node["content"] = canonical_runs(node["content"])
         touched = [target]
 
@@ -212,10 +240,10 @@ def apply(state, c, hasher=hashing.PLAIN):
         node = by_id[target]
         if node["kind"] != "sentence":
             raise Refusal("correction.split", "only sentences are split", target)
-        k = c["offset"]
+        k = c.get("offset")
         first, second, pos = [], [], 0
         total = len(hashing.plain_runs(node["content"], doc))
-        if not 0 < k < total:
+        if not _is_index(k) or not 0 < k < total:
             raise Refusal("correction.split", "offset must fall inside the sentence", target)
         for r in node["content"]:
             ln = len(hashing.plain_runs([r], doc))
@@ -325,7 +353,8 @@ def apply(state, c, hasher=hashing.PLAIN):
 
     after_doc = Doc(state.key, nodes)
     validate_structure(after_doc)
-    after_vals = hashing.compute(after_doc, hasher)
+    after_ix = before_ix.update(after_doc)
+    after_vals = after_ix.values
 
     # Digits: a change to a digit needs crop evidence, and no leaf outside `touched` may change its digits.
     needs_crop = False
@@ -333,14 +362,14 @@ def apply(state, c, hasher=hashing.PLAIN):
         rep = c["replace"] if isinstance(c["replace"], str) else hashing.plain_runs(c["replace"], doc)
         needs_crop = digits(c["find"]) != digits(rep)
     elif op == "delete":
-        needs_crop = any(digits(hashing.plain(doc, x)) for x in touched)
+        needs_crop = any(digits(before_ix.plain(x)) for x in touched)
     elif op == "relabel":
         needs_crop = digits(record.get("from") or "") != digits(c.get("label") or "")
     if needs_crop and not _has_crop(c):
         raise Refusal("correction.digits", "a digit changed without crop evidence", target)
     for nid in doc.leaves(doc.root):
-        if nid in after_doc.nodes and nid not in touched:
-            if digits(hashing.plain(doc, nid)) != digits(hashing.plain(after_doc, nid)):
+        if nid in after_ix.entries and nid not in touched:
+            if digits(before_ix.plain(nid)) != digits(after_ix.plain(nid)):
                 raise Refusal("correction.digits", "an undeclared digit change", nid)
 
     for k in ("op", "target", "find", "replace", "run", "offset", "text", "parent", "place_after", "node", "with",
@@ -365,7 +394,7 @@ def apply(state, c, hasher=hashing.PLAIN):
         new_ids.append({"id": rid, "kind": "retired", "successor": succ, "node_kind": old_kinds[rid],
                         "rev": record["index"]})
     new_state = State(state.key, [after_doc.nodes[x] for x in after_doc.preorder()], new_ids,
-                      state.corrections + [record])
+                      state.corrections + [record], {hasher.key: after_ix})
     return new_state, record
 
 

tools/oracle/crypto.py

@@ -1,24 +1,22 @@
 """Keyed hashing, key derivation, Merkle roots and Ed25519.
 
 Non-normative; see spec/05-hashes.md and spec/08-fidelity.md.
-Ed25519 follows the reference code of RFC 8032, section 6; it is slow and is meant only for test vectors.
+HKDF and Ed25519 come from the `cryptography` package.
 """
 
 import hashlib
-import hmac
+
+from cryptography.exceptions import InvalidSignature
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
 
 from . import jcs
 
 
 def hkdf(ikm, salt, info, length=32):
     """HKDF-SHA-256, RFC 5869."""
-    prk = hmac.new(salt, ikm, hashlib.sha256).digest()
-    out, t, i = b"", b"", 1
-    while len(out) < length:
-        t = hmac.new(prk, t + info + bytes([i]), hashlib.sha256).digest()
-        out += t
-        i += 1
-    return out[:length]
+    return HKDF(algorithm=hashes.SHA256(), length=length, salt=salt, info=info).derive(ikm)
 
 
 def team_key(secret):
@@ -74,121 +72,22 @@ def receipt_items(receipt):
 
 # --- Ed25519 (RFC 8032) -------------------------------------------------------
 
-_p = 2**255 - 19
-_d = -121665 * pow(121666, _p - 2, _p) % _p
-_q = 2**252 + 27742317777372353535851937790883648493
-
-
-def _sha512(b):
-    return hashlib.sha512(b).digest()
-
-
-def _inv(x):
-    return pow(x, _p - 2, _p)
-
-
-def _add(P, Q):
-    A = (P[1] - P[0]) * (Q[1] - Q[0]) % _p
-    B = (P[1] + P[0]) * (Q[1] + Q[0]) % _p
-    C = 2 * P[3] * Q[3] * _d % _p
-    D = 2 * P[2] * Q[2] % _p
-    E, F, G, H = B - A, D - C, D + C, B + A
-    return (E * F % _p, G * H % _p, F * G % _p, E * H % _p)
-
-
-def _mul(s, P):
-    Q = (0, 1, 1, 0)
-    while s > 0:
-        if s & 1:
-            Q = _add(Q, P)
-        P = _add(P, P)
-        s >>= 1
-    return Q
-
-
-def _equal(P, Q):
-    if (P[0] * Q[2] - Q[0] * P[2]) % _p != 0:
-        return False
-    return (P[1] * Q[2] - Q[1] * P[2]) % _p == 0
-
-
-_modp_sqrt_m1 = pow(2, (_p - 1) // 4, _p)
-
-
-def _recover_x(y, sign):
-    if y >= _p:
-        return None
-    x2 = (y * y - 1) * _inv(_d * y * y + 1)
-    if x2 == 0:
-        return None if sign else 0
-    x = pow(x2, (_p + 3) // 8, _p)
-    if (x * x - x2) % _p != 0:
-        x = x * _modp_sqrt_m1 % _p
-    if (x * x - x2) % _p != 0:
-        return None
-    if (x & 1) != sign:
-        x = _p - x
-    return x
-
-
-_gy = 4 * _inv(5) % _p
-_gx = _recover_x(_gy, 0)
-_G = (_gx, _gy, 1, _gx * _gy % _p)
-
-
-def _compress(P):
-    zinv = _inv(P[2])
-    x = P[0] * zinv % _p
-    y = P[1] * zinv % _p
-    return int.to_bytes(y | ((x & 1) << 255), 32, "little")
-
-
-def _decompress(s):
-    if len(s) != 32:
-        return None
-    y = int.from_bytes(s, "little")
-    sign = y >> 255
-    y &= (1 << 255) - 1
-    x = _recover_x(y, sign)
-    if x is None:
-        return None
-    return (x, y, 1, x * y % _p)
-
-
-def _secret_expand(secret):
-    h = _sha512(secret)
-    a = int.from_bytes(h[:32], "little")
-    a &= (1 << 254) - 8
-    a |= 1 << 254
-    return a, h[32:]
-
 
 def ed25519_public(secret):
-    a, _ = _secret_expand(secret)
-    return _compress(_mul(a, _G))
+    """The 32-byte public key of a 32-byte Ed25519 secret."""
+    return Ed25519PrivateKey.from_private_bytes(secret).public_key().public_bytes_raw()
 
 
 def ed25519_sign(secret, msg):
-    a, prefix = _secret_expand(secret)
-    A = _compress(_mul(a, _G))
-    r = int.from_bytes(_sha512(prefix + msg), "little") % _q
-    R = _compress(_mul(r, _G))
-    h = int.from_bytes(_sha512(R + A + msg), "little") % _q
-    s = (r + h * a) % _q
-    return R + int.to_bytes(s, 32, "little")
+    return Ed25519PrivateKey.from_private_bytes(secret).sign(msg)
 
 
 def ed25519_verify(public, msg, signature):
+    """True when `signature` is a valid Ed25519 signature of `msg` under `public`; never raises."""
     if len(public) != 32 or len(signature) != 64:
         return False
-    A = _decompress(public)
-    if not A:
+    try:
+        Ed25519PublicKey.from_public_bytes(public).verify(signature, msg)
+    except (InvalidSignature, ValueError):
         return False
-    R = _decompress(signature[:32])
-    if not R:
-        return False
-    s = int.from_bytes(signature[32:], "little")
-    if s >= _q:
-        return False
-    h = int.from_bytes(_sha512(signature[:32] + public + msg), "little") % _q
-    return _equal(_mul(s, _G), _add(R, _mul(h, A)))
+    return True

tools/oracle/db.py

@@ -8,9 +8,14 @@ import json
 import os
 import shutil
 
-from . import jcs
-from .package import read_json, read_jsonl, write
-from .text import sha256_hex
+from . import hashing, jcs
+from .model import Doc
+from .package import SHA256_RE, read_json, read_jsonl, safe_path, write
+from .text import norm, sha256_hex
+from .tree import Refusal
+
+SPEC_VERSION = "0.1.0"
+BUNDLE_FORMAT = "scienting-bundle/1"
 
 STRUCTURED = {"MANIFEST.json", "document.json", "witnesses.json", "provenance.json", "roster.json", "nodes.jsonl",
               "pages.jsonl", "locations.jsonl", "references.jsonl", "corrections.jsonl", "receipts.jsonl",
@@ -44,10 +49,71 @@ def create(conn, sql_dir):
             conn.executescript(f.read())
 
 
+def create_bundle(conn, sql_dir):
+    """The tables of a bundle: those of create(), then sql/bundle-sqlite.sql (spec/13 §13.5)."""
+    create(conn, sql_dir)
+    with open(os.path.join(sql_dir, "bundle-sqlite.sql"), encoding="utf-8") as f:
+        conn.executescript(f.read())
+
+
+def search_rows(key, nodes):
+    """The bundle's search rows for one document, in document order (spec/13 §13.5).
+
+    One row per sentence, cell, equation outside a sentence and figure-text item, as
+    (doc_key, node, unit, context, text). For a node, `context` is the kind of its parent; a figure-text item
+    takes its panel as `node` and its `role` as `context`. Withheld nodes are skipped.
+    """
+    doc = Doc(key, nodes)
+    out = []
+    for nid in doc.order:
+        n = doc.nodes[nid]
+        kind = n["kind"]
+        parent = doc.nodes[n["parent"]]["kind"] if n.get("parent") else None
+        if n.get("withheld"):
+            continue
+        if kind in ("sentence", "cell") or (kind == "equation" and parent != "sentence"):
+            out.append((key, nid, kind, parent, norm(hashing.plain(doc, nid))))
+        elif kind == "panel":
+            for item in (n.get("attrs") or {}).get("figure_text", []):
+                out.append((key, nid, "figure-text", item["role"], norm(item["text"])))
+    return out
+
+
+def index(conn, key):
+    """Fill the bundle's `search` table for the loaded document `key`."""
+    nodes = []
+    for n in _q(conn, "SELECT id, kind, parent, ord, attrs, content, withheld FROM nodes WHERE doc_key = ? ORDER BY seq",
+                (key,)):
+        rec = {"id": n["id"], "kind": n["kind"], "parent": n["parent"], "ord": n["ord"]}
+        _opt(rec, "attrs", P(n["attrs"]))
+        _opt(rec, "content", P(n["content"]))
+        if n["withheld"]:
+            rec["withheld"] = True
+        nodes.append(rec)
+    rows = search_rows(key, nodes)
+    conn.executemany("INSERT INTO search VALUES (?, ?, ?, ?, ?)", rows)
+    return len(rows)
+
+
+def write_meta(conn, store, built_from, extra=None):
+    """Write `bundle_meta`: format, store, built_from and spec, then any `extra` pairs."""
+    pairs = {"format": BUNDLE_FORMAT, "store": store, "built_from": built_from, "spec": SPEC_VERSION}
+    pairs.update(extra or {})
+    conn.executemany("INSERT INTO bundle_meta VALUES (?, ?)", sorted(pairs.items()))
+
+
 def load(conn, root, asset_store):
-    """Load the package at `root`. Files without tables go to `asset_store`, by SHA-256."""
+    """Load the package at `root`. Files without tables go to `asset_store`, by SHA-256.
+
+    Validate the package first (package.validate); this only refuses paths and hashes that are unsafe to use.
+    """
     manifest = read_json(os.path.join(root, "MANIFEST.json"))
     listed = {f["path"]: f for f in manifest["files"]}
+    for path, f in listed.items():
+        if not safe_path(path):
+            raise Refusal("package.path.unsafe", "unsafe path %r" % path)
+        if not isinstance(f.get("sha256"), str) or not SHA256_RE.fullmatch(f["sha256"]):
+            raise Refusal("package.manifest.schema", "%s: sha256 is not 64 lowercase hex digits" % path)
 
     def jl(name):
         return read_jsonl(os.path.join(root, name)) if name in listed else []
@@ -241,6 +307,8 @@ def unload(conn, key, out, asset_store):
     absent = []
     for f in _q(conn, "SELECT * FROM package_files WHERE doc_key = ?", (key,)):
         if f["present"]:
+            if not SHA256_RE.fullmatch(f["sha256"]):
+                raise Refusal("package.manifest.schema", "%s: sha256 is not 64 lowercase hex digits" % f["path"])
             with open(os.path.join(asset_store, f["sha256"]), "rb") as fh:
                 files[f["path"]] = fh.read()
         else:

tools/oracle/hashing.py

@@ -214,26 +214,102 @@ def own_content(doc, nid, text_hash_of=None):
     return o
 
 
+class _Entry:
+    """One node's cached work: its five values, and the projections its parent joins."""
+
+    __slots__ = ("sig", "values", "plain", "joined_plain", "joined_strict", "complexes")
+
+    def __init__(self, sig, values, plain, joined_plain, joined_strict, complexes):
+        self.sig = sig
+        self.values = values
+        self.plain = plain
+        # What this node gives a container's projections: its text leaves joined, or None when it has none.
+        self.joined_plain = joined_plain
+        self.joined_strict = joined_strict
+        self.complexes = complexes
+
+
+def _sig(doc, nid):
+    """Everything a node's values depend on besides its children's values."""
+    n = doc.nodes[nid]
+    return (n["kind"], n.get("label"), n.get("attrs"), n.get("content"), tuple(doc.kids(nid)))
+
+
+class Index:
+    """The five values of every node of one document, kept so that a change rehashes only what it touched.
+
+    A node's values depend only on its own subtree, so after a change only the nodes whose own record or
+    children changed, and their ancestors, are rehashed. A container's projections are its children's
+    cached projections joined, so no container walks its subtree again.
+    """
+
+    def __init__(self, hasher=PLAIN):
+        self.hasher = hasher
+        self.entries = {}
+        self.values = {}
+
+    @classmethod
+    def build(cls, doc, hasher=PLAIN):
+        ix = cls(hasher)
+        ix._rehash(doc, set(doc.nodes))
+        return ix
+
+    def update(self, doc):
+        """A new Index for `doc`, a later state of this one's document; this Index is left unchanged."""
+        ix = Index(self.hasher)
+        ix.entries = {nid: e for nid, e in self.entries.items() if nid in doc.nodes}
+        ix.values = {nid: v for nid, v in self.values.items() if nid in doc.nodes}
+        dirty = {nid for nid in doc.nodes if nid not in ix.entries or ix.entries[nid].sig != _sig(doc, nid)}
+        for nid in list(dirty):
+            p = doc.nodes[nid].get("parent")
+            while p is not None and p not in dirty:
+                dirty.add(p)
+                p = doc.nodes[p].get("parent")
+        ix._rehash(doc, dirty)
+        return ix
+
+    def plain(self, nid):
+        """The node's plain projection."""
+        return self.entries[nid].plain
+
+    def _rehash(self, doc, dirty):
+        h = self.hasher
+        for nid in reversed(doc.preorder()):
+            if nid not in dirty:
+                continue
+            n = doc.nodes[nid]
+            k = n["kind"]
+            kids = doc.kids(nid)
+            if k in RUN_KINDS or k in MATH_KINDS or k in VERBATIM_KINDS:
+                p = plain(doc, nid)
+                st, cx = strict(doc, nid)
+                # An equation line is never a text leaf; a sentence's equations are part of its own runs.
+                joined = (None, None, []) if k == "equation-line" else (p, st, cx)
+            else:
+                ps, ss, cx = [], [], []
+                for c in kids:
+                    e = self.entries[c]
+                    if e.joined_plain is not None:
+                        ps.append(e.joined_plain)
+                        ss.append(e.joined_strict)
+                        cx.extend(e.complexes)
+                p = " ".join(ps)
+                st = _finish_strict(" ".join(ss))
+                joined = (p, " ".join(ss), cx) if ps else (None, None, [])
+            own = h(jcs.dumpb(own_content(doc, nid, lambda x: self.values[x]["text_hash"])))
+            self.values[nid] = {
+                "text_hash": h(norm(p)),
+                "match_hash": h(st),
+                "align_key": h(lossy_from_strict(st)),
+                "own_hash": own,
+                "node_hash": h(jcs.dumpb({"own": own, "children": [self.values[c]["node_hash"] for c in kids]})),
+            }
+            self.entries[nid] = _Entry(_sig(doc, nid), self.values[nid], p, *joined)
+
+
 def compute(doc, hasher=PLAIN):
-    """Compute the five content values of every node, bottom up. Returns {id: {...}}."""
-    values = {}
-
-    def text_hash(nid):
-        return values[nid]["text_hash"] if nid in values else hasher(norm(plain(doc, nid)))
-
-    for nid in reversed(doc.preorder()):
-        s, _ = strict(doc, nid)
-        th = hasher(norm(plain(doc, nid)))
-        own = hasher(jcs.dumpb(own_content(doc, nid, text_hash)))
-        node = hasher(jcs.dumpb({"own": own, "children": [values[c]["node_hash"] for c in doc.kids(nid)]}))
-        values[nid] = {
-            "text_hash": th,
-            "match_hash": hasher(s),
-            "align_key": hasher(lossy_from_strict(s)),
-            "own_hash": own,
-            "node_hash": node,
-        }
-    return values
+    """The five content values of every node, bottom up. Returns {id: {...}}."""
+    return Index.build(doc, hasher).values
 
 
 def meta_object(document):

tools/oracle/jcs.py

@@ -1,11 +1,18 @@
 """Canonical JSON, RFC 8785 (JSON Canonicalization Scheme).
 
 Non-normative.
-The normative rule is spec/01-conventions.md; this module exists so the vectors can be generated and checked.
+The normative rule is spec/01-conventions.md.
+Most values serialize through the standard library's C encoder, which escapes strings exactly as RFC 8785 does.
+Values it would write differently take the exact path below: floats, integers beyond 2^53, and member names
+outside the Basic Multilingual Plane, which RFC 8785 sorts by UTF-16 code units rather than by code points.
 """
 
+import json
 import math
 
+_C = json.JSONEncoder(ensure_ascii=False, separators=(",", ":"), sort_keys=True, allow_nan=False)
+_EXACT_INT = 2**53
+
 _ESC = {
     0x08: "\\b",
     0x09: "\\t",
@@ -82,8 +89,37 @@ def _key(s):
     return s.encode("utf-16-be")
 
 
+def _fast(v):
+    """True when the C encoder writes `v` exactly as RFC 8785 does."""
+    t = type(v)
+    if t is str or v is None or t is bool:
+        return True
+    if t is int:
+        return -_EXACT_INT <= v <= _EXACT_INT
+    if t is dict:
+        for k, x in v.items():
+            if type(k) is not str or (not k.isascii() and max(k) > "\uffff") or not _fast(x):
+                return False
+        return True
+    if t is list or t is tuple:
+        return all(_fast(x) for x in v)
+    return False
+
+
 def dumps(value):
     """Serialize `value` to canonical JSON text."""
+    if _fast(value):
+        text = _C.encode(value)
+        if not text.isascii():
+            try:
+                text.encode("utf-8")
+            except UnicodeEncodeError:
+                raise ValueError("lone surrogate in string") from None
+        return text
+    return _exact(value)
+
+
+def _exact(value):
     if value is None:
         return "null"
     if value is True:

tools/oracle/package.py

@@ -7,6 +7,7 @@ import base64
 import json
 import os
 import posixpath
+import re
 
 from . import crypto, hashing, jcs, receipts as R
 from .anchors import ADDRESS_RE
@@ -46,8 +47,11 @@ def witness_id(sha):
     return letter_first(bytes.fromhex(sha), 7)
 
 
+SHA256_RE = re.compile(r"[0-9a-f]{64}")
+
+
 def safe_path(p):
-    if not p or p.startswith("/") or "\\" in p or "\x00" in p:
+    if not isinstance(p, str) or not p or p.startswith("/") or "\\" in p or "\x00" in p:
         return False
     parts = p.split("/")
     return all(x not in ("", ".", "..") for x in parts) and posixpath.normpath(p) == p
@@ -58,6 +62,9 @@ def write(root, files, absent=()):
 
     Returns the package hash.
     """
+    for path in list(files) + [a["path"] for a in absent]:
+        if not safe_path(path) or path == "MANIFEST.json":
+            raise Refusal("package.path.unsafe", "unsafe path %r" % path)
     os.makedirs(root, exist_ok=True)
     entries = [dict(a) for a in absent]
     for path in sorted(files, key=lambda p: p.encode("utf-8")):
@@ -374,10 +381,23 @@ def _validate(root, store, key, rep):
             if qa["sample"] != R.sample(orig, seed, qa["rate"], qa["minimum"]):
                 rep.add("doc.qa.sample", r["id"], "the sample does not follow from the seed")
     if roster and not withheld:
-        lv = R.effective_levels(nodes, vals, recs, roster)
-        if R.distribution(lv) != document["fidelity"]:
+        if R.distribution(R.effective_levels(nodes, vals, recs, roster)) != document["fidelity"]:
             rep.add("doc.fidelity.report", "document.json", "the fidelity distribution does not recompute")
 
 
+def read_state(root):
+    """The current state of the package at `root`, for the gate (spec/11 §11.4): nodes without their values,
+    the ID table and the corrections applied so far. Validate the package first."""
+    from .corrections import State
+    present = {f["path"] for f in read_json(os.path.join(root, "MANIFEST.json"))["files"]}
+
+    def jl(name):
+        return read_jsonl(os.path.join(root, name)) if name in present else []
+
+    values = ("text_hash", "match_hash", "align_key", "own_hash", "node_hash")
+    nodes = [{k: v for k, v in n.items() if k not in values} for n in jl("nodes.jsonl")]
+    return State(read_json(os.path.join(root, "document.json"))["key"], nodes, jl("ids.jsonl"), jl("corrections.jsonl"))
+
+
 def n_root_hash(nodes, doc):
     return {n["id"]: n for n in nodes}[doc.root]["node_hash"]

tools/oracle/receipts.py

@@ -123,7 +123,9 @@ def revoked(receipt, revocations):
     return False
 
 
-def valid_revocations(revocations, receipts_by_id, roster, owners):
+def valid_revocations(revocations, receipts_by_id, roster):
+    """The revocations signed by the receipt's signer, the signer they name, or a key the roster makes an owner."""
+    owners = {k for k, v in roster.items() if v.get("role") == "owner"}
     out = []
     for r in revocations:
         if not verify_signature(r, roster):
@@ -139,7 +141,7 @@ def valid_revocations(revocations, receipts_by_id, roster, owners):
     return out
 
 
-def receipt_status(rec, records, roster, owners=()):
+def receipt_status(rec, records, roster):
     """`valid`, `revoked`, `suspended`, or `invalid` for one receipt."""
     receipts = [x for x in records if x["format"] == "scienting-receipt/1"]
     revs = [x for x in records if x["format"] == "scienting-revocation/1"]
@@ -151,26 +153,30 @@ def receipt_status(rec, records, roster, owners=()):
     key = roster.get(rec["by"])
     if not key or not key_valid_at(key, rec["at"]) or rec["signature"]["key"] != rec["by"]:
         return "invalid"
-    if revoked(rec, valid_revocations(revs, by_id, roster, owners)):
+    if revoked(rec, valid_revocations(revs, by_id, roster)):
         return "revoked"
     for q in receipts:
         qa = q.get("qa")
         if qa and qa.get("of") == rec["id"] and qa.get("errors"):
-            if receipt_status(q, records, roster, owners) == "valid":
+            if receipt_status(q, records, roster) == "valid":
                 return "suspended"
     return "valid"
 
 
-def effective_levels(nodes, values, records, roster, owners=()):
-    """Every leaf's level: its recorded level, raised by valid receipts that list it at its current hash."""
+def effective_levels(nodes, values, records, roster):
+    """Every leaf's level: its recorded level, raised by valid receipts that list it at its current hash.
+
+    A spot-check's `qa:` receipt does not raise the nodes it found in error (spec/08 §8.2).
+    """
     level = {n["id"]: n.get("fidelity") for n in nodes if "fidelity" in n}
     for rec in records:
         if rec["format"] != "scienting-receipt/1":
             continue
-        if receipt_status(rec, records, roster, owners) != "valid":
+        if receipt_status(rec, records, roster) != "valid":
             continue
+        wrong = {e["node"] for e in (rec.get("qa") or {}).get("errors", [])}
         for nid, h in rec.get("nodes", []):
-            if nid in level and values[nid]["node_hash"] == h:
+            if nid in level and nid not in wrong and values[nid]["node_hash"] == h:
                 if LEVELS.index(rec["level"]) > LEVELS.index(level[nid]):
                     level[nid] = rec["level"]
     return level

tools/oracle/schema.py

@@ -1,13 +1,15 @@
-"""A JSON Schema (draft 2020-12) validator for the subset of keywords the spec's schemas use.
+"""JSON Schema (draft 2020-12) validation with the `jsonschema` package, and the lint of the keyword subset.
 
 Non-normative.
-The spec's schemas are restricted to these keywords (spec/22-conformance.md) so that any 2020-12 validator, and
-this one, agree; CI also runs the `jsonschema` package when it is installed.
+The spec's schemas are restricted to the keywords below (spec/22-conformance.md) so that every 2020-12
+validator reads them alike; `lint` refuses any schema that leaves the subset.
 """
 
 import json
 import os
-import re
+
+from jsonschema import Draft202012Validator
+from referencing import Registry, Resource
 
 KEYWORDS = {
     "$schema", "$id", "$ref", "$defs", "$comment", "title", "description", "examples", "default", "deprecated",
@@ -24,9 +26,25 @@ class SchemaError(Exception):
 
 
 class Store:
+    """The schemas of one folder, linted on load, with a compiled validator per schema."""
+
     def __init__(self, root):
         self.root = root
         self.docs = {}
+        self._registry = None
+        self._validators = {}
+
+    def registry(self):
+        if self._registry is None:
+            names = sorted(f for f in os.listdir(self.root) if f.endswith(".schema.json"))
+            self._registry = Registry().with_resources(
+                [(self.load(f)["$id"], Resource.from_contents(self.load(f))) for f in names])
+        return self._registry
+
+    def validator(self, name):
+        if name not in self._validators:
+            self._validators[name] = Draft202012Validator(self.load(name), registry=self.registry())
+        return self._validators[name]
 
     def load(self, name):
         if name not in self.docs:
@@ -65,146 +83,7 @@ def lint(schema, where):
                     lint(s, "%s/%s/%d" % (where, k, i))
 
 
-def _type_ok(t, x):
-    if t == "null":
-        return x is None
-    if t == "boolean":
-        return isinstance(x, bool)
-    if t == "integer":
-        return isinstance(x, int) and not isinstance(x, bool)
-    if t == "number":
-        return isinstance(x, (int, float)) and not isinstance(x, bool)
-    if t == "string":
-        return isinstance(x, str)
-    if t == "array":
-        return isinstance(x, list)
-    if t == "object":
-        return isinstance(x, dict)
-    raise SchemaError("unknown type " + t)
-
-
-def _equal(a, b):
-    if isinstance(a, bool) or isinstance(b, bool):
-        return type(a) is type(b) and a == b
-    return a == b
-
-
 def validate(store, name, instance):
-    """Return a list of error strings; empty when the instance is valid."""
-    schema = store.load(name)
-    errors = []
-    _v(store, name, schema, instance, "", errors)
-    return errors
-
-
-def _v(store, base, s, x, path, errors):
-    if s is True:
-        return
-    if s is False:
-        errors.append(path + ": not allowed")
-        return
-    if "$ref" in s:
-        b2, target = store.deref(base, s["$ref"])
-        _v(store, b2, target, x, path, errors)
-    if "type" in s:
-        ts = s["type"] if isinstance(s["type"], list) else [s["type"]]
-        if not any(_type_ok(t, x) for t in ts):
-            errors.append("%s: expected %s" % (path or "/", "/".join(ts)))
-            return
-    if "const" in s and not _equal(s["const"], x):
-        errors.append("%s: expected %r" % (path or "/", s["const"]))
-    if "enum" in s and not any(_equal(e, x) for e in s["enum"]):
-        errors.append("%s: %r is not one of %r" % (path or "/", x, s["enum"]))
-    if isinstance(x, str):
-        if "minLength" in s and len(x) < s["minLength"]:
-            errors.append("%s: shorter than %d" % (path, s["minLength"]))
-        if "maxLength" in s and len(x) > s["maxLength"]:
-            errors.append("%s: longer than %d" % (path, s["maxLength"]))
-        if "pattern" in s and not re.search(s["pattern"], x):
-            errors.append("%s: %r does not match %s" % (path, x, s["pattern"]))
-    if isinstance(x, (int, float)) and not isinstance(x, bool):
-        if "minimum" in s and x < s["minimum"]:
-            errors.append("%s: below %s" % (path, s["minimum"]))
-        if "maximum" in s and x > s["maximum"]:
-            errors.append("%s: above %s" % (path, s["maximum"]))
-        if "exclusiveMinimum" in s and x <= s["exclusiveMinimum"]:
-            errors.append("%s: not above %s" % (path, s["exclusiveMinimum"]))
-        if "exclusiveMaximum" in s and x >= s["exclusiveMaximum"]:
-            errors.append("%s: not below %s" % (path, s["exclusiveMaximum"]))
-    if isinstance(x, list):
-        if "minItems" in s and len(x) < s["minItems"]:
-            errors.append("%s: fewer than %d items" % (path, s["minItems"]))
-        if "maxItems" in s and len(x) > s["maxItems"]:
-            errors.append("%s: more than %d items" % (path, s["maxItems"]))
-        if s.get("uniqueItems"):
-            seen = [json.dumps(i, sort_keys=True) for i in x]
-            if len(seen) != len(set(seen)):
-                errors.append("%s: items are not unique" % path)
-        pre = s.get("prefixItems", [])
-        for i, item in enumerate(x):
-            if i < len(pre):
-                _v(store, base, pre[i], item, "%s/%d" % (path, i), errors)
-            elif "items" in s:
-                _v(store, base, s["items"], item, "%s/%d" % (path, i), errors)
-        if "contains" in s:
-            if not any(not _sub(store, base, s["contains"], i) for i in x):
-                errors.append("%s: no item matches `contains`" % path)
-    if isinstance(x, dict):
-        if "minProperties" in s and len(x) < s["minProperties"]:
-            errors.append("%s: fewer than %d properties" % (path, s["minProperties"]))
-        if "maxProperties" in s and len(x) > s["maxProperties"]:
-            errors.append("%s: more than %d properties" % (path, s["maxProperties"]))
-        for r in s.get("required", []):
-            if r not in x:
-                errors.append("%s: missing %s" % (path or "/", r))
-        for k, deps in s.get("dependentRequired", {}).items():
-            if k in x:
-                for d in deps:
-                    if d not in x:
-                        errors.append("%s: %s requires %s" % (path or "/", k, d))
-        props = s.get("properties", {})
-        pprops = s.get("patternProperties", {})
-        for k, v in x.items():
-            matched = False
-            if k in props:
-                matched = True
-                _v(store, base, props[k], v, path + "/" + k, errors)
-            for pat, ps in pprops.items():
-                if re.search(pat, k):
-                    matched = True
-                    _v(store, base, ps, v, path + "/" + k, errors)
-            if not matched and "additionalProperties" in s:
-                ap = s["additionalProperties"]
-                if ap is False:
-                    errors.append("%s: unexpected property %s" % (path or "/", k))
-                else:
-                    _v(store, base, ap, v, path + "/" + k, errors)
-            if "propertyNames" in s:
-                _v(store, base, s["propertyNames"], k, path + "/" + k, errors)
-    for sub in s.get("allOf", []):
-        _v(store, base, sub, x, path, errors)
-    if "anyOf" in s:
-        if all(_sub(store, base, sub, x) for sub in s["anyOf"]):
-            errors.append("%s: matches none of anyOf" % (path or "/"))
-    if "oneOf" in s:
-        ok = [sub for sub in s["oneOf"] if not _sub(store, base, sub, x)]
-        if len(ok) != 1:
-            detail = ""
-            if not ok:
-                best = min((_sub(store, base, sub, x) for sub in s["oneOf"]), key=len)
-                detail = " (closest: %s)" % "; ".join(best[:3])
-            errors.append("%s: matches %d of oneOf%s" % (path or "/", len(ok), detail))
-    if "not" in s and not _sub(store, base, s["not"], x):
-        errors.append("%s: matches `not`" % (path or "/"))
-    if "if" in s:
-        if not _sub(store, base, s["if"], x):
-            if "then" in s:
-                _v(store, base, s["then"], x, path, errors)
-        elif "else" in s:
-            _v(store, base, s["else"], x, path, errors)
-
-
-def _sub(store, base, s, x):
-    e = []
-    _v(store, base, s, x, "", e)
-    return e
+    """Return a list of error strings, `path: message`, in path order; empty when the instance is valid."""
+    errors = sorted(store.validator(name).iter_errors(instance), key=lambda e: [str(x) for x in e.absolute_path])
+    return ["%s: %s" % ("/" + "/".join(str(x) for x in e.absolute_path), e.message) for e in errors]

tools/validate.py

@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+"""Validate packages, or run the correction gate on proposals against a package.
+
+Non-normative tooling.
+Usage: python3 tools/validate.py PACKAGE [PACKAGE ...]
+       python3 tools/validate.py --correction PROPOSALS PACKAGE
+A package is checked as spec/22 §22.2 describes, and each failure printed as `package: where: rule: message`.
+With --correction, PROPOSALS is a JSON file holding one proposed correction, or a JSON Lines file of several
+applied in order. Each runs through the gate (spec/11 §11.4) against the package's current state, and the record
+it would append to corrections.jsonl is printed, until one is refused. Nothing is written.
+Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.
+"""
+
+import argparse
+import json
+import os
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+
+from tools.oracle import corrections, jcs, package  # noqa: E402
+from tools.oracle.schema import Store  # noqa: E402
+from tools.oracle.tree import Refusal  # noqa: E402
+
+
+def read_proposals(path):
+    with open(path, encoding="utf-8") as f:
+        text = f.read()
+    try:
+        return [json.loads(text)]
+    except json.JSONDecodeError:
+        return [json.loads(ln) for ln in text.splitlines() if ln.strip()]
+
+
+def main():
+    lines = __doc__.splitlines()
+    ap = argparse.ArgumentParser(description=lines[0], epilog="\n".join(lines[5:]),
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--correction", metavar="PROPOSALS",
+                    help="a JSON or JSON Lines file of proposed corrections to run through the gate")
+    ap.add_argument("packages", nargs="+", metavar="PACKAGE", help="package folders")
+    a = ap.parse_args()
+    if a.correction and len(a.packages) != 1:
+        ap.error("--correction checks proposals against exactly one package")
+    store, bad = Store(os.path.join(ROOT, "schemas")), False
+    for p in a.packages:
+        for e in package.validate(p, store).errors:
+            print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
+            bad = True
+    if bad:
+        return 1
+    if not a.correction:
+        for p in a.packages:
+            print("%s: valid" % p)
+        return 0
+    pkg = a.packages[0]
+    state = package.read_state(pkg)
+    if any(n.get("withheld") for n in state.nodes):
+        print("%s: doc.policy.withheld: a package that withholds its text takes no corrections" % pkg,
+              file=sys.stderr)
+        return 1
+    for i, proposal in enumerate(read_proposals(a.correction), 1):
+        try:
+            state, record = corrections.gate(state, proposal, store)
+        except Refusal as e:
+            where = "%s:%d" % (a.correction, i) + (" " + e.node if e.node else "")
+            print("%s: %s: %s" % (where, e.rule, e.message), file=sys.stderr)
+            return 1
+        print(jcs.dumps(record))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())