feat: zipped packages, target set, checklists, audit fixes and a stress test

Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tools/oracle/corrections.py

@@ -6,8 +6,8 @@ Non-normative; see spec/11-corrections.md.
 import copy
 
 from . import hashing
-from .model import LABELLED, LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc, CHILDREN
-from .tree import Refusal, canonical_runs, mint_ids, validate_structure, _flatten
+from .model import ATTRS, LABELLED, LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc, CHILDREN
+from .tree import Refusal, canonical_runs, mint_ids, validate_references, validate_structure, _flatten
 from . import jcs
 from .schema import validate as schema_validate
 from .text import digits, record_id, sha256_hex
@@ -137,15 +137,33 @@ def gate(state, proposal, store, hasher=hashing.PLAIN):
     errors = schema_validate(store, "correction.schema.json", proposal)
     if errors:
         raise Refusal("correction.schema", "; ".join(errors[:3]))
-    return apply(state, proposal, hasher)
+    return apply(state, proposal, hasher, store)
 
 
 def _is_index(v):
     return isinstance(v, int) and not isinstance(v, bool) and v >= 0
 
 
-def apply(state, c, hasher=hashing.PLAIN):
-    """Apply correction `c` to `state` and return the recorded correction. Raises Refusal."""
+def id_order(e):
+    """The order of ids.jsonl (spec/04 §4.4): aliases first by ID, then retirements by revision and ID."""
+    return (0 if e["kind"] == "alias" else 1, e.get("rev", 0), e["id"])
+
+
+def _node_rule(error):
+    """The rule a node-schema error breaks, from the property the error names."""
+    path = error.split(":", 1)[0]
+    for prefix, rule in (("/attrs", "doc.node.attrs"), ("/label", "doc.node.label"), ("/content", "doc.node.content")):
+        if path.startswith(prefix):
+            return rule
+    return "doc.node.schema"
+
+
+def apply(state, c, hasher=hashing.PLAIN, store=None):
+    """Apply correction `c` to `state` and return the recorded correction. Raises Refusal.
+
+    With a schema `store`, every node the correction changed or created is also validated against
+    node.schema.json, which is how the gate enforces the attribute shapes of spec/03 §3.4 (rule 11 of §11.4).
+    """
     c = copy.deepcopy(c)
     doc = state.doc()
     before_ix = state.index(hasher)
@@ -308,7 +326,7 @@ def apply(state, c, hasher=hashing.PLAIN):
 
     elif op == "move":
         parent = c["parent"]
-        if parent not in by_id or parent in set(_subtree(doc, target)):
+        if parent not in by_id or parent in set(_subtree(doc, target)) or c.get("place_after") == target:
             raise Refusal("correction.move", "bad destination", target)
         _place(nodes, target, parent, c.get("place_after"))
         nodes = _rebuild(nodes)
@@ -326,6 +344,12 @@ def apply(state, c, hasher=hashing.PLAIN):
         node["kind"] = new_kind
         if new_kind not in LABELLED:
             node.pop("label", None)
+        # Each kind carries only its own attributes (spec/03 §3.4).
+        kept = {k: v for k, v in node.get("attrs", {}).items() if k in ATTRS.get(new_kind, ())}
+        if kept:
+            node["attrs"] = kept
+        else:
+            node.pop("attrs", None)
         record["from"] = old
         touched = [x for x in doc.leaves(target)]
 
@@ -334,7 +358,10 @@ def apply(state, c, hasher=hashing.PLAIN):
         if node["kind"] not in LABELLED:
             raise Refusal("correction.relabel", "%s takes no label" % node["kind"], target)
         record["from"] = node.get("label")
-        node["label"] = c["label"]
+        if c.get("label") is None:
+            node.pop("label", None)  # optional properties are omitted, never written as null (spec/03 §3.1)
+        else:
+            node["label"] = c["label"]
         touched = [target] if node["kind"] in LEAVES else []
 
     elif op == "set":
@@ -353,8 +380,15 @@ def apply(state, c, hasher=hashing.PLAIN):
 
     after_doc = Doc(state.key, nodes)
     validate_structure(after_doc)
+    validate_references(after_doc)
     after_ix = before_ix.update(after_doc)
     after_vals = after_ix.values
+    if store is not None:
+        changed = {x for x in set(touched) | set(created) | {target} if x in after_doc.nodes}
+        for nid in sorted(changed):
+            errors = schema_validate(store, "node.schema.json", dict(after_doc.nodes[nid], **after_vals[nid]))
+            if errors:
+                raise Refusal(_node_rule(errors[0]), errors[0], nid)
 
     # Digits: a change to a digit needs crop evidence, and no leaf outside `touched` may change its digits.
     needs_crop = False
@@ -393,24 +427,28 @@ def apply(state, c, hasher=hashing.PLAIN):
     for rid, succ in retired:
         new_ids.append({"id": rid, "kind": "retired", "successor": succ, "node_kind": old_kinds[rid],
                         "rev": record["index"]})
+    new_ids.sort(key=id_order)
     new_state = State(state.key, [after_doc.nodes[x] for x in after_doc.preorder()], new_ids,
                       state.corrections + [record], {hasher.key: after_ix})
     return new_state, record
 
 
 def _strip_edge(runs, end):
+    """Remove the whitespace at one edge of a split part, across as many text runs as it spans (spec/11 §11.1)."""
     from .text import is_ws
-    if not runs:
-        return
-    i = len(runs) - 1 if end else 0
-    r = runs[i]
-    if r["kind"] != "text":
-        return
-    t = r["text"]
-    if end:
-        while t and is_ws(t[-1]):
-            t = t[:-1]
-    else:
-        while t and is_ws(t[0]):
-            t = t[1:]
-    runs[i] = dict(r, text=t)
+    while runs:
+        i = len(runs) - 1 if end else 0
+        r = runs[i]
+        if r["kind"] != "text":
+            return
+        t = r["text"]
+        if end:
+            while t and is_ws(t[-1]):
+                t = t[:-1]
+        else:
+            while t and is_ws(t[0]):
+                t = t[1:]
+        if t:
+            runs[i] = dict(r, text=t)
+            return
+        del runs[i]