feat: zipped packages, target set, checklists, audit fixes and a stress test

Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tools/build.py

@@ -14,6 +14,7 @@ import os
 import shutil
 import struct
 import sys
+import zipfile
 import zlib
 from fractions import Fraction
 
@@ -24,14 +25,17 @@ from tools.oracle import align as AL  # noqa: E402
 from tools.oracle import anchors as AN  # noqa: E402
 from tools.oracle import corrections as CO  # noqa: E402
 from tools.oracle import crypto, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI  # noqa: E402
+from tools.oracle import targets as TG  # noqa: E402
 from tools.oracle.model import LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc  # noqa: E402
 from tools.oracle.notation import fold_math  # noqa: E402
-from tools.oracle.package import jsonl, line, witness_id, write  # noqa: E402
+from tools.oracle.package import ZIP_EXTENSION, jsonl, line, witness_id, write, write_zip  # noqa: E402
+from tools.oracle.schema import Store  # noqa: E402
 from tools.oracle.text import letter_first, mint_node_id, norm, record_id, sha256_hex  # noqa: E402
 from tools.oracle.tree import Refusal, _flatten, mint_ids, validate_structure  # noqa: E402
 
 VERSION = "0.1.0"
 STORE = "smithlab"
+SCHEMAS = Store(os.path.join(ROOT, "schemas"))
 
 
 def H(s):
@@ -440,7 +444,7 @@ def build_document(src, used=None, extra_files=None):
         if before is not None:
             d = state.doc()
             c["offset"] = hashing.plain(d, c["target"]).index(before)
-        state, rec = CO.apply(state, c)
+        state, rec = CO.apply(state, c, store=SCHEMAS)
         if new_name:
             names.ids[new_name] = rec["created"][0]
         records.append(rec)
@@ -798,7 +802,11 @@ def vectors(out, hale, tiny):
              ("tex", "a \\cdot b"), ("typst", "a dot b"), ("tex", "a \\times b \\pm c"), ("tex", "\\left(x\\right)^2"),
              ("typst", "(a+b)^2"), ("tex", "\\frac{a}{b}"), ("typst", "a/b"), ("tex", "\\sqrt{2}"),
              ("tex", "\\mathbf{F}"), ("tex", "{ab}_2"), ("tex", "x_1_2"), ("tex", "\\Delta G^\\circ"),
-             ("tex", "E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2"), ("typst", "E = K_r (r - r_\"eq\")^2")]
+             ("tex", "E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2"), ("typst", "E = K_r (r - r_\"eq\")^2"),
+             # A script after an empty group applies to the atom before it, or to an empty base (the isotope idiom).
+             ("tex", "{}^{14}C"), ("tex", "x{}^2"), ("tex", "{}_2"), ("tex", "\\left)x\\right("),
+             # Letters are the Latin and Greek blocks: a fullwidth or modifier letter is complex, a Coptic letter too.
+             ("tex", "Ａ"), ("tex", "ʰ"), ("tex", "Ϣ"), ("tex", "ἀω")]
     wv(out, "notation.jsonl", [{"id": "%s:%s" % (n, s), "notation": n, "source": s, "canonical": fold_math(n, s)}
                                for n, s in notes])
     # Projections.
@@ -1018,12 +1026,21 @@ def corrections_vectors():
         ("merge", {"op": "merge", "target": nm["@c"], "with": nm["@d"], "joiner": " ", "evidence": person}),
         ("merge-not-adjacent", {"op": "merge", "target": nm["@a"], "with": nm["@c"], "evidence": person}),
         ("move", {"op": "move", "target": nm["@d"], "parent": nm["@p1"], "place_after": None, "evidence": person}),
-        ("reclassify", {"op": "reclassify", "target": nm["@p2"], "kind": "quote", "evidence": person}),
+        ("move-after-itself", {"op": "move", "target": nm["@d"], "parent": nm["@p2"], "place_after": nm["@d"],
+                               "evidence": person}),
+        ("reclassify", {"op": "reclassify", "target": nm["@p2"], "kind": "footnote", "evidence": person}),
+        ("reclassify-blocks", {"op": "reclassify", "target": nm["@p2"], "kind": "quote", "evidence": person}),
         ("reclassify-bad", {"op": "reclassify", "target": nm["@a"], "kind": "paragraph", "evidence": person}),
         ("relabel-section", {"op": "relabel", "target": nm["@sec"], "label": "2", "evidence": crop}),
+        ("relabel-remove", {"op": "relabel", "target": nm["@sec"], "label": None, "evidence": crop}),
         ("relabel-digit-no-crop", {"op": "relabel", "target": nm["@t"], "label": "Table 2", "evidence": person}),
         ("set-colspan", {"op": "set", "target": nm["@y1"], "attr": "colspan", "value": 2, "evidence": person}),
+        ("set-bad-value", {"op": "set", "target": nm["@y1"], "attr": "colspan", "value": "2", "evidence": person}),
         ("set-not-settable", {"op": "set", "target": nm["@y1"], "attr": "level", "value": 2, "evidence": person}),
+        ("insert-node-junk", {"op": "insert", "parent": nm["@p2"], "place_after": nm["@c"],
+                              "node": {"kind": "sentence", "content": [{"kind": "text", "text": "Junk."}], "foo": 1},
+                              "evidence": crop}),
+        ("delete-header-cell", {"op": "delete", "target": nm["@hx"], "evidence": person}),
     ]
     out = []
     vals0 = hashing.compute(st0.doc())
@@ -1032,7 +1049,7 @@ def corrections_vectors():
         c = dict(base, **c)
         case = {"id": cid, "doc": st0.key, "before": {"nodes": before, "ids": [], "rev": 0}, "correction": c}
         try:
-            st1, rec = CO.apply(st0, c)
+            st1, rec = CO.apply(st0, c, store=SCHEMAS)
             v1 = hashing.compute(st1.doc())
             case["expect"] = {"record": rec, "nodes": [dict(n, **v1[n["id"]]) for n in st1.nodes], "ids": st1.ids}
         except Refusal as e:
@@ -1242,6 +1259,10 @@ def records(out, hale):
             target(AN.make_anchor(doc, vals, I["@tab1"], rev, store=STORE)), "assessing", "confirmed"),
         ann({"type": "TextualBody", "kind": "exercise", "value": check_body["prompt"], "exercise": check_body},
             target(anchor("@s1")), "assessing", "confirmed"),
+        # A person's description of the figure, which an accessible export may use because it is confirmed.
+        ann({"type": "TextualBody", "kind": "description", "role": "alt", "language": "en",
+             "value": "Energy along the C–C bond stretch (panel a) and the fit residual (panel b)."},
+            target(AN.make_anchor(doc, vals, I["@fig1"], rev, store=STORE)), "describing", "confirmed"),
     ]
     anns.append(ann({"type": "TextualBody", "kind": "comment", "value": "Agreed; see the residual panel."},
                     {"type": "SpecificResource", "source": anns[2]["id"]}, "replying", created="2026-10-11T19:05:00Z"))
@@ -1367,12 +1388,48 @@ def records(out, hale):
     q2 = hale["by_name"]["q2"]
     n = len(q1["qa"]["sample"]) + len(q2["qa"]["sample"])
     k = len(q1["qa"]["errors"]) + len(q2["qa"]["errors"])
-    wr(out, "qa-report.json", {"format": "scienting-qa-report/1", "id": record_id(H("qa report 1")),
-                               "delivery": "smithlab evidence pilot (fictional)", "docs": ["smithlab/hale2026harmonic"],
-                               "receipts": [hale["by_name"]["r1"]["id"], hale["by_name"]["r4"]["id"]],
-                               "reviewers": [{"key": "ana", "sampled": n, "errors": k, "confidence": "0.95",
-                                              "method": "clopper-pearson", "bound": RC.bound_string(k, n)}],
-                               "at": "2026-10-12T10:00:00Z"})
+    wr(out, "qa-report.json", RC.seal({
+        "format": "scienting-qa-report/1", "delivery": "smithlab evidence pilot (fictional)",
+        "docs": ["smithlab/hale2026harmonic"], "receipts": [hale["by_name"]["r1"]["id"], hale["by_name"]["r4"]["id"]],
+        "reviewers": [{"key": "ana", "sampled": n, "errors": k, "confidence": "0.95", "method": "clopper-pearson",
+                       "bound": RC.bound_string(k, n)}],
+        "by": "ana", "at": "2026-10-12T10:00:00Z"}, secret("ana"), "ana"))
+    # A checklist for an accessible HTML export of the paper, and the receipt that covers it (spec/08 §8.11).
+    html_sha = H("accessible HTML export of hale2026harmonic (synthetic)")
+    checker = {"name": "example-checker", "version": "1.0"}
+    checklist = {"format": "scienting-checklist/1", "doc": "smithlab/hale2026harmonic", "rev": rev,
+                 "standard": {"name": "WCAG 2.1", "level": "AA", "url": "https://www.w3.org/TR/WCAG21/"},
+                 "export": {"kind": "html", "sha256": html_sha, "size": 48213},
+                 "items": [
+                     {"criterion": "1.1.1", "result": "pass", "method": "manual",
+                      "note": "Every figure has a confirmed description in the layer."},
+                     {"criterion": "1.2.1", "result": "not-applicable", "method": "manual", "note": "No audio or video."},
+                     {"criterion": "1.3.1", "result": "pass", "method": "manual",
+                      "note": "Headings, lists and table headers are marked up; each cell names its header path."},
+                     {"criterion": "1.3.2", "result": "pass", "method": "automated", "tool": checker},
+                     {"criterion": "1.4.3", "result": "pass", "method": "automated", "tool": checker},
+                     {"criterion": "1.4.5", "result": "pass", "method": "manual",
+                      "note": "Equations are MathML, never images of text."},
+                     {"criterion": "2.4.2", "result": "pass", "method": "automated", "tool": checker},
+                     {"criterion": "3.1.1", "result": "pass", "method": "automated", "tool": checker}],
+                 "by": "ana", "at": "2026-10-12T11:00:00Z"}
+    checklist_hash = H(jcs.dumpb(checklist))
+    checklist["id"] = record_id(checklist_hash)
+    wr(out, "checklist/checklist.json", checklist)
+    wr(out, "checklist/receipt.json", RC.seal({
+        "format": "scienting-receipt/1", "doc": "hale2026harmonic", "rev": rev, "level": "F3",
+        "scope": "targeted:checklist", "nodes": [], "records": [["checklist", checklist["id"], checklist_hash]],
+        "evidence": [{"kind": "source-file", "sha256": html_sha, "note": "The exported HTML."}],
+        "by": "ana", "at": "2026-10-12T11:30:00Z"}, secret("ana"), "ana"))
+    # The target set of targeted F3 (spec/08 §8.10) under each digit rule: the dataset's cells, the sentence a
+    # confirmed claim link anchors, and the panel a digitization reads from are the anchored nodes.
+    dataset_anchors = [json.loads(x) for x in
+                       hale["files"]["datasets/ds-hale-bond-params/anchors.jsonl"].decode().splitlines()]
+    anchored = sorted({a["anchor"]["node"] for a in dataset_anchors} | {I["@s2"], pa})
+    wv(out, "targets.jsonl", [{"id": "hale-" + rule, "package": "fixtures/packages/hale2026harmonic",
+                                "anchored": anchored, "digit_rule": rule,
+                                "expect": {"nodes": TG.target_set(doc, anchored, rule)}}
+                               for rule in TG.DIGIT_RULES])
     # Draft snapshot and sentence IDs.
     s5 = I["@s5"]
     snap_anchor = AN.make_anchor(doc, vals, s5, rev, store=STORE)
@@ -1462,6 +1519,72 @@ def invalid(out, tiny):
     case("style-sup-sub", "doc.run.style", {"nodes.jsonl": edit_nodes(lambda ns: by(ns, "@c").update(
         content=[{"kind": "text", "text": "A second paragraph.", "style": ["sub", "sup"]}]))},
          "A run cannot be both a subscript and a superscript.")
+    # Zipped packages that break spec/12 §12.2: the tiny package's files plus one bad entry.
+    tiny_dir = os.path.join(out, "fixtures", "packages", "tiny2026example")
+    entries = []
+    for rel in sorted(os.path.relpath(os.path.join(d, f), tiny_dir).replace(os.sep, "/")
+                      for d, _, fs in os.walk(tiny_dir) for f in fs):
+        with open(os.path.join(tiny_dir, rel), "rb") as fh:
+            entries.append(("tiny2026example/" + rel, fh.read(), 0o644))
+
+    def zip_case(name, rule, note, extra):
+        d = os.path.join(base, name)
+        os.makedirs(d)
+        with zipfile.ZipFile(os.path.join(d, "package" + ZIP_EXTENSION), "w") as zf:
+            for ename, data, mode in entries + extra:
+                zi = zipfile.ZipInfo(ename, date_time=(1980, 1, 1, 0, 0, 0))
+                zi.compress_type = zipfile.ZIP_STORED
+                zi.create_system = 3
+                zi.external_attr = mode << 16
+                zf.writestr(zi, data)
+        with open(os.path.join(d, "expect.json"), "wb") as fh:
+            fh.write((jcs.dumps({"rule": rule, "note": note}) + "\n").encode())
+
+    zip_case("zip-traversal", "package.zip.entry", "An archive entry whose path leaves the package folder.",
+             [("tiny2026example/../escape.txt", b"outside\n", 0o644)])
+    zip_case("zip-symlink", "package.zip.entry", "An archive entry that is a symbolic link.",
+             [("tiny2026example/assets/link", b"document.json", 0o120777)])
+    # Rules the audit found no broken package for.
+    case("missing-required-file", "package.file.required", {"provenance.json": None}, "A required file is missing.")
+    case("counts-wrong", "doc.counts", {"document.json": _doc_edit(files, counts={"nodes": 1, "leaves": 1})},
+         "document.json's counts are not the number of nodes and leaves.")
+    case("rev-without-corrections", "doc.correction.rev", {"document.json": _doc_edit(files, rev=3)},
+         "document.json's rev is not the number of corrections.")
+    case("receipt-unsorted-nodes", "doc.receipt.nodes", {"receipts.jsonl": _receipt_edit(
+        files, nodes=list(reversed([json.loads(x) for x in files["receipts.jsonl"].decode().splitlines()][0]["nodes"])))},
+        "A receipt whose nodes are not sorted by ID.")
+    case("retired-live", "doc.id.retired-live", {"ids.jsonl": jsonl([
+        {"id": nm["@a"], "kind": "retired", "successor": None, "node_kind": "sentence", "rev": 1}])},
+        "An ID that is both live and retired.")
+    case("manifest-key", "package.manifest.key", {}, "The manifest names another key than document.json.",
+         raw=lambda d: _manifest_edit(d, key="other2026example"))
+    case("unsafe-path", "package.path.unsafe", {}, "The manifest lists a path that leaves the package.",
+         raw=lambda d: _manifest_edit(d, files=[{"path": "../escape.txt", "sha256": "0" * 64, "size": 1}]
+                                       + json.load(open(os.path.join(d, "MANIFEST.json")))["files"]))
+    case("crlf", "package.file.canonical", {}, "A file whose lines end in CR LF, which is not the canonical form.",
+         raw=lambda d: _crlf(d))
+    signer = [json.loads(x) for x in files["receipts.jsonl"].decode().splitlines()][0]
+    case("signer-mismatch", "doc.receipt.signature", {"receipts.jsonl": jsonl([
+        RC.seal(dict(signer, by="cara"), secret("ana"), "ana")])},
+        "A receipt whose signature was made with another key than the signer it names.")
+
+
+def _manifest_edit(d, **kw):
+    p = os.path.join(d, "MANIFEST.json")
+    m = json.load(open(p, encoding="utf-8"))
+    m.update(kw)
+    open(p, "w", encoding="utf-8").write(jcs.dumps(m) + "\n")
+
+
+def _crlf(d):
+    p = os.path.join(d, "provenance.json")
+    data = open(p, "rb").read().replace(b"\n", b"\r\n")
+    open(p, "wb").write(data)
+    m = json.load(open(os.path.join(d, "MANIFEST.json"), encoding="utf-8"))
+    for f in m["files"]:
+        if f["path"] == "provenance.json":
+            f["sha256"], f["size"] = H(data), len(data)
+    open(os.path.join(d, "MANIFEST.json"), "w", encoding="utf-8").write(jcs.dumps(m) + "\n")
 
 
 def _swap_first(ns, sec):
@@ -1533,6 +1656,9 @@ def main(out):
     build_simple(out, "okafor2025open")
     build_simple(out, "smithlab2027charges")
     tiny = tiny_package(os.path.join(out, "fixtures", "packages", "tiny2026example"))
+    # The same package zipped as spec/12 §12.2 says a producer writes a byte-identical archive.
+    write_zip(os.path.join(out, "fixtures", "packages", "tiny2026example"),
+              os.path.join(out, "fixtures", "packages", "tiny2026example" + ZIP_EXTENSION))
     records(out, hale)
     vectors(out, hale, tiny)
     invalid(out, tiny)