feat: zipped packages, target set, checklists, audit fixes and a stress test

Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tools/oracle/anchors.py

@@ -158,9 +158,12 @@ def _follow(state, nid):
 
 
 def _relabel_reverted_own(state, doc, nid, anchor):
-    """The own_hash the node had before the relabel corrections since the anchor, or None."""
-    rel = [c for c in state.corrections if c["index"] > anchor["rev"] and c.get("target") == nid]
-    if not rel or any(c["op"] != "relabel" for c in rel):
+    """The own_hash the node had before the relabel corrections since the anchor, or None when anything else
+    changed the node or what is below it since (spec/06 §6.5: its *only* changes were relabel corrections)."""
+    since = [c for c in state.corrections if c["index"] > anchor["rev"] and (
+        c.get("target") == nid or nid in (c.get("path") or []) or nid in (c.get("to_path") or []))]
+    rel = [c for c in since if c.get("target") == nid]
+    if not rel or any(c["op"] != "relabel" or c.get("target") != nid for c in since):
         return None
     n = dict(doc.nodes[nid], label=rel[0]["from"])
     tmp = Doc(doc.key, [n if x["id"] == nid else x for x in [doc.nodes[i] for i in doc.order]])
@@ -175,7 +178,11 @@ def resolve(state, anchor, threshold=Fraction(1, 4)):
     original = anchor["node"]
     nid, chain, via = _follow(state, original)
     live_kind = doc.nodes[nid]["kind"] if nid else None
-    kind = kind or live_kind or next((e["node_kind"] for e in state.ids if e["id"] == original), None)
+    table = {e["id"]: e for e in state.ids}
+    # The kind falls back to the retired entry the chain reached (spec/06 §6.5); an alias carries no kind.
+    retired_kind = next((table[x].get("node_kind") for x in reversed(chain)
+                         if table.get(x, {}).get("kind") == "retired"), None)
+    kind = kind or live_kind or retired_kind
     cls = anchor_class(kind) if kind else "leaf"
     is_span = "span" in anchor
     result = {"node": nid}
@@ -256,7 +263,7 @@ def resolve(state, anchor, threshold=Fraction(1, 4)):
         now = set(doc.leaves(nid))
         now_match = {values[x]["match_hash"] for x in now}
         held = sum(1 for i, m in orig if i in now or m in now_match)
-        if orig and 2 * held >= len(orig):
+        if 2 * held >= len(orig):
             return out("moved")
         return out("stale", reason="the successor holds fewer than half of the original leaves")
     if values[nid]["node_hash"] == anchor["node_hash"]:

tools/oracle/corrections.py

@@ -6,8 +6,8 @@ Non-normative; see spec/11-corrections.md.
 import copy
 
 from . import hashing
-from .model import LABELLED, LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc, CHILDREN
-from .tree import Refusal, canonical_runs, mint_ids, validate_structure, _flatten
+from .model import ATTRS, LABELLED, LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc, CHILDREN
+from .tree import Refusal, canonical_runs, mint_ids, validate_references, validate_structure, _flatten
 from . import jcs
 from .schema import validate as schema_validate
 from .text import digits, record_id, sha256_hex
@@ -137,15 +137,33 @@ def gate(state, proposal, store, hasher=hashing.PLAIN):
     errors = schema_validate(store, "correction.schema.json", proposal)
     if errors:
         raise Refusal("correction.schema", "; ".join(errors[:3]))
-    return apply(state, proposal, hasher)
+    return apply(state, proposal, hasher, store)
 
 
 def _is_index(v):
     return isinstance(v, int) and not isinstance(v, bool) and v >= 0
 
 
-def apply(state, c, hasher=hashing.PLAIN):
-    """Apply correction `c` to `state` and return the recorded correction. Raises Refusal."""
+def id_order(e):
+    """The order of ids.jsonl (spec/04 §4.4): aliases first by ID, then retirements by revision and ID."""
+    return (0 if e["kind"] == "alias" else 1, e.get("rev", 0), e["id"])
+
+
+def _node_rule(error):
+    """The rule a node-schema error breaks, from the property the error names."""
+    path = error.split(":", 1)[0]
+    for prefix, rule in (("/attrs", "doc.node.attrs"), ("/label", "doc.node.label"), ("/content", "doc.node.content")):
+        if path.startswith(prefix):
+            return rule
+    return "doc.node.schema"
+
+
+def apply(state, c, hasher=hashing.PLAIN, store=None):
+    """Apply correction `c` to `state` and return the recorded correction. Raises Refusal.
+
+    With a schema `store`, every node the correction changed or created is also validated against
+    node.schema.json, which is how the gate enforces the attribute shapes of spec/03 §3.4 (rule 11 of §11.4).
+    """
     c = copy.deepcopy(c)
     doc = state.doc()
     before_ix = state.index(hasher)
@@ -308,7 +326,7 @@ def apply(state, c, hasher=hashing.PLAIN):
 
     elif op == "move":
         parent = c["parent"]
-        if parent not in by_id or parent in set(_subtree(doc, target)):
+        if parent not in by_id or parent in set(_subtree(doc, target)) or c.get("place_after") == target:
             raise Refusal("correction.move", "bad destination", target)
         _place(nodes, target, parent, c.get("place_after"))
         nodes = _rebuild(nodes)
@@ -326,6 +344,12 @@ def apply(state, c, hasher=hashing.PLAIN):
         node["kind"] = new_kind
         if new_kind not in LABELLED:
             node.pop("label", None)
+        # Each kind carries only its own attributes (spec/03 §3.4).
+        kept = {k: v for k, v in node.get("attrs", {}).items() if k in ATTRS.get(new_kind, ())}
+        if kept:
+            node["attrs"] = kept
+        else:
+            node.pop("attrs", None)
         record["from"] = old
         touched = [x for x in doc.leaves(target)]
 
@@ -334,7 +358,10 @@ def apply(state, c, hasher=hashing.PLAIN):
         if node["kind"] not in LABELLED:
             raise Refusal("correction.relabel", "%s takes no label" % node["kind"], target)
         record["from"] = node.get("label")
-        node["label"] = c["label"]
+        if c.get("label") is None:
+            node.pop("label", None)  # optional properties are omitted, never written as null (spec/03 §3.1)
+        else:
+            node["label"] = c["label"]
         touched = [target] if node["kind"] in LEAVES else []
 
     elif op == "set":
@@ -353,8 +380,15 @@ def apply(state, c, hasher=hashing.PLAIN):
 
     after_doc = Doc(state.key, nodes)
     validate_structure(after_doc)
+    validate_references(after_doc)
     after_ix = before_ix.update(after_doc)
     after_vals = after_ix.values
+    if store is not None:
+        changed = {x for x in set(touched) | set(created) | {target} if x in after_doc.nodes}
+        for nid in sorted(changed):
+            errors = schema_validate(store, "node.schema.json", dict(after_doc.nodes[nid], **after_vals[nid]))
+            if errors:
+                raise Refusal(_node_rule(errors[0]), errors[0], nid)
 
     # Digits: a change to a digit needs crop evidence, and no leaf outside `touched` may change its digits.
     needs_crop = False
@@ -393,24 +427,28 @@ def apply(state, c, hasher=hashing.PLAIN):
     for rid, succ in retired:
         new_ids.append({"id": rid, "kind": "retired", "successor": succ, "node_kind": old_kinds[rid],
                         "rev": record["index"]})
+    new_ids.sort(key=id_order)
     new_state = State(state.key, [after_doc.nodes[x] for x in after_doc.preorder()], new_ids,
                       state.corrections + [record], {hasher.key: after_ix})
     return new_state, record
 
 
 def _strip_edge(runs, end):
+    """Remove the whitespace at one edge of a split part, across as many text runs as it spans (spec/11 §11.1)."""
     from .text import is_ws
-    if not runs:
-        return
-    i = len(runs) - 1 if end else 0
-    r = runs[i]
-    if r["kind"] != "text":
-        return
-    t = r["text"]
-    if end:
-        while t and is_ws(t[-1]):
-            t = t[:-1]
-    else:
-        while t and is_ws(t[0]):
-            t = t[1:]
-    runs[i] = dict(r, text=t)
+    while runs:
+        i = len(runs) - 1 if end else 0
+        r = runs[i]
+        if r["kind"] != "text":
+            return
+        t = r["text"]
+        if end:
+            while t and is_ws(t[-1]):
+                t = t[:-1]
+        else:
+            while t and is_ws(t[0]):
+                t = t[1:]
+        if t:
+            runs[i] = dict(r, text=t)
+            return
+        del runs[i]

tools/oracle/db.py

@@ -10,7 +10,7 @@ import shutil
 
 from . import hashing, jcs
 from .model import Doc
-from .package import SHA256_RE, read_json, read_jsonl, safe_path, write
+from .package import SHA256_RE, _all_files, read_json, read_jsonl, safe_path, write
 from .text import norm, sha256_hex
 from .tree import Refusal
 
@@ -107,6 +107,7 @@ def load(conn, root, asset_store):
 
     Validate the package first (package.validate); this only refuses paths and hashes that are unsafe to use.
     """
+    _all_files(root)  # refuses a symbolic link anywhere in the package before anything is copied
     manifest = read_json(os.path.join(root, "MANIFEST.json"))
     listed = {f["path"]: f for f in manifest["files"]}
     for path, f in listed.items():

tools/oracle/fingerprint.py

@@ -34,11 +34,16 @@ def is_stamp(line, size, rules):
     return None
 
 
-def fp_text(text):
-    s = unicodedata.normalize("NFKC", fold_plain_text(text))
+def _strip(s):
     return "".join(c for c in s if not is_ws(c) and unicodedata.category(c) not in ("Cc", "Cf"))
 
 
+def fp_text(text):
+    """Steps 3 and 4 of spec/07 §7.1: whitespace and control characters go first, so that "10 -20" and "10-20"
+    fold alike; then scripts and dashes fold as in the strict projection, then NFKC."""
+    return _strip(unicodedata.normalize("NFKC", fold_plain_text(_strip(text))))
+
+
 def fingerprint(page, rules):
     """`page`: {"size": [w, h], "lines": [{"text", "box"?}], "images": [sha256...]}.
 

tools/oracle/grade.py

@@ -3,18 +3,21 @@
 Non-normative; see spec/19-exercises.md.
 """
 
+from decimal import Decimal
 from fractions import Fraction
 
 from . import anchors as A
 from .tree import Refusal
 
 
-def dec(fr):
-    fr = Fraction(fr)
-    if fr.denominator == 1:
-        return str(fr.numerator)
-    s = "%.4f" % float(fr)
-    return s.rstrip("0").rstrip(".")
+def dec(d):
+    """A decimal string, exactly as the sum came out (spec/19 §19.4: points are decimal strings, summed exactly)."""
+    return format(Decimal(d), "f")
+
+
+def _zero_like(d):
+    """Zero with the scale of `d`, so a missed element worth "0.5" scores "0.0"."""
+    return Decimal(0).quantize(Decimal(d))
 
 
 def _resolved(state, anchor):
@@ -57,34 +60,34 @@ def grade(body, submission, state):
     items = []
 
     def item(name, pts, got, fb):
-        items.append({"element": name, "points": dec(pts if got else 0), "max": dec(pts), "feedback": fb})
+        items.append({"element": name, "points": dec(pts if got else _zero_like(pts)), "max": dec(pts), "feedback": fb})
 
     if kind in ("anchor", "check"):
         if kind == "check":
             v = key["verdict"]
             ok = ans.get("verdict") == v["value"]
-            item("verdict", Fraction(v["points"]), ok, v.get("feedback", {}).get("hit" if ok else "miss", ""))
+            item("verdict", v["points"], ok, v.get("feedback", {}).get("hit" if ok else "miss", ""))
         for i, el in enumerate(key["elements"]):
             ok = _any(state, rule, ans.get("anchors", []), el["anchors"])
-            item("element:%d" % i, Fraction(el["points"]), ok, el.get("feedback", {}).get("hit" if ok else "miss", ""))
+            item("element:%d" % i, el["points"], ok, el.get("feedback", {}).get("hit" if ok else "miss", ""))
     elif kind == "trace":
         chain = ans.get("chain", [])
         for i, link in enumerate(key["chain"]):
             ok = i < len(chain) and _any(state, rule, chain[i], link["anchors"])
-            item("link:%d" % i, Fraction(link["points"]), ok, link.get("feedback", {}).get("hit" if ok else "miss", ""))
+            item("link:%d" % i, link["points"], ok, link.get("feedback", {}).get("hit" if ok else "miss", ""))
     elif kind == "reproduce":
-        ok = abs(Fraction(ans["value"]) - Fraction(key["value"])) <= Fraction(key["tolerance"])
-        item("value", Fraction(key["points"]), ok, key.get("feedback", {}).get("hit" if ok else "miss", ""))
+        ok = abs(Decimal(ans["value"]) - Decimal(key["value"])) <= Decimal(key["tolerance"])
+        item("value", key["points"], ok, key.get("feedback", {}).get("hit" if ok else "miss", ""))
     elif kind == "verify":
         flags = ans.get("flags", [])
         for i, err in enumerate(key["errors"]):
             hit = [f for f in flags if _any(state, rule, [f["anchor"]], err["anchors"])]
             ok = any(f["verdict"] == err["verdict"] for f in hit)
             fb = err.get("feedback", {}).get("hit" if ok else "miss", "")
-            item("error:%d" % i, Fraction(err["points"]), ok, fb)
+            item("error:%d" % i, err["points"], ok, fb)
     else:
         raise Refusal("grade.kind", "unknown exercise kind " + kind)
-    score = sum(Fraction(x["points"]) for x in items)
-    total = sum(Fraction(x["max"]) for x in items)
+    score = sum((Decimal(x["points"]) for x in items), Decimal(0))
+    total = sum((Decimal(x["max"]) for x in items), Decimal(0))
     return {"format": "scienting-grade/1", "submission": submission["id"], "exercise": submission["exercise"],
             "score": dec(score), "max": dec(total), "items": items}

tools/oracle/model.py

@@ -69,6 +69,18 @@ LABELLED = frozenset(
      "code", "algorithm"]
 )
 
+# The attributes each kind may carry (spec/03 §3.4); kinds absent here carry none.
+ATTRS = {
+    "section": frozenset(["level"]),
+    "table": frozenset(["columns"]),
+    "cell": frozenset(["column", "rowspan", "colspan", "role"]),
+    "row": frozenset(["header_path"]),
+    "list": frozenset(["ordered"]),
+    "item": frozenset(["marker"]),
+    "theorem": frozenset(["variant"]),
+    "panel": frozenset(["image", "crop", "figure_text"]),
+}
+
 RUNS = frozenset(["text", "math", "cite", "xref", "fnmark", "link", "equation"])
 STYLES = frozenset(["em", "strong", "sup", "sub", "sc", "mono"])
 

tools/oracle/notation.py

@@ -79,14 +79,19 @@ class NotSimple(Exception):
     pass
 
 
+# The Latin and Greek letter blocks of spec/05 \u00a75.4: Basic Latin through Latin Extended-B, Greek and Coptic without
+# the Coptic letters, Latin Extended Additional, and Greek Extended.
+LETTER_RANGES = ((0x0041, 0x024F), (0x0370, 0x03E1), (0x03F0, 0x03FF), (0x1E00, 0x1EFF), (0x1F00, 0x1FFF))
+
+
 def is_letter(ch):
-    """A Latin or Greek letter, or the micro sign."""
+    """A letter of General Category L in the Latin or Greek blocks, or the micro sign."""
     if ch == "\u00b5":
         return True
     if not unicodedata.category(ch).startswith("L"):
         return False
-    name = unicodedata.name(ch, "")
-    return name.startswith("LATIN ") or name.startswith("GREEK ")
+    cp = ord(ch)
+    return any(lo <= cp <= hi for lo, hi in LETTER_RANGES)
 
 
 def fold_char(ch):
@@ -222,9 +227,9 @@ class _TexParser:
             elif kind == "ch" and val == "{":
                 inner = self.seq(True)
                 nxt = self.peek()
-                if nxt in (("ch", "_"), ("ch", "^")) and len(inner) != 1:
+                if nxt in (("ch", "_"), ("ch", "^")) and len(inner) > 1:
                     raise NotSimple("script on a multi-atom group")
-                atoms.extend(inner)
+                atoms.extend(inner)  # an empty group leaves a following script to the previous atom or an empty base
             else:
                 atoms.extend(self.command_or_char(kind, val))
 
@@ -245,7 +250,7 @@ class _TexParser:
             return self.arg()
         if val in ("left", "right"):
             tok = self.take()
-            if tok in (("ch", "("), ("ch", ")")):
+            if tok == ("ch", "(" if val == "left" else ")"):
                 return [Atom(tok[1])]
             raise NotSimple("\\left or \\right with another delimiter")
         raise NotSimple("command \\" + val)

tools/oracle/package.py

@@ -4,17 +4,26 @@ Non-normative; see spec/12-package.md and spec/22-conformance.md.
 """
 
 import base64
+import binascii
+import hashlib
 import json
 import os
 import posixpath
 import re
+import stat
+import tempfile
+import unicodedata
+import zipfile
 
 from . import crypto, hashing, jcs, receipts as R
 from .anchors import ADDRESS_RE
 from .model import LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc
 from .schema import validate as schema_validate
-from .text import letter_first, record_id, sha256_hex
-from .tree import Refusal, validate_structure
+from .text import KEY_RE, letter_first, record_id, sha256_hex
+from .tree import Refusal, validate_references, validate_structure
+
+# The reference tools' own limit on a package's total size (spec/12 §12.2 leaves the limit to the consumer).
+SIZE_LIMIT = 1 << 30
 
 JSONL = {
     "nodes.jsonl": "node.schema.json",
@@ -85,26 +94,46 @@ def write(root, files, absent=()):
     return sha256_hex(mb)
 
 
+def _refuse_constant(name):
+    raise ValueError("JSON has no canonical form for " + name)
+
+
+def _parse_int(text):
+    """Read a JSON integer as an ECMAScript engine would: exactly within 2^53, as a double beyond (spec/01 §1.1)."""
+    v = int(text)
+    return v if abs(v) <= 2**53 else float(v)
+
+
+def _canonical_bytes(data, where):
+    """Parse one canonical JSON value given as bytes; refuse anything else (spec/01 §1.1, §1.9).
+
+    Numbers are read as IEEE 754 doubles, as the canonical form writes them, and the comparison is on bytes, so a
+    carriage return, a byte order mark, a lone surrogate, NaN or invalid UTF-8 is refused, not silently translated.
+    """
+    try:
+        rec = json.loads(data.decode("utf-8"), parse_constant=_refuse_constant, parse_int=_parse_int)
+        canon = jcs.dumps(rec).encode("utf-8")
+    except (UnicodeDecodeError, ValueError, TypeError, OverflowError, RecursionError) as e:
+        raise Refusal("package.file.canonical", "%s: not canonical JSON (%s)" % (where, type(e).__name__))
+    if canon != data:
+        raise Refusal("package.file.canonical", "%s: not canonical JSON" % where)
+    return rec
+
+
 def read_jsonl(path):
-    out = []
-    with open(path, encoding="utf-8") as f:
-        for i, ln in enumerate(f, 1):
-            if not ln.endswith("\n"):
-                raise Refusal("package.file.newline", "%s:%d: every line ends with a newline" % (path, i))
-            rec = json.loads(ln)
-            if line(rec) != ln:
-                raise Refusal("package.file.canonical", "%s:%d: line is not canonical JSON" % (path, i))
-            out.append(rec)
-    return out
+    with open(path, "rb") as f:
+        data = f.read()
+    if data and not data.endswith(b"\n"):
+        raise Refusal("package.file.newline", "%s: every line ends with a newline" % path)
+    return [_canonical_bytes(ln, "%s:%d" % (path, i)) for i, ln in enumerate(data.split(b"\n")[:-1], 1)]
 
 
 def read_json(path):
-    with open(path, encoding="utf-8") as f:
-        text = f.read()
-    rec = json.loads(text)
-    if line(rec) != text:
-        raise Refusal("package.file.canonical", "%s: file is not canonical JSON" % path)
-    return rec
+    with open(path, "rb") as f:
+        data = f.read()
+    if not data.endswith(b"\n"):
+        raise Refusal("package.file.canonical", "%s: the file does not end with one newline" % path)
+    return _canonical_bytes(data[:-1], path)
 
 
 class Report:
@@ -119,55 +148,84 @@ class Report:
 
 
 def _all_files(root):
+    """Every file under `root` by its exact name, refusing any symbolic link on the way (spec/12 §12.2)."""
     out = []
-    for d, _, fs in os.walk(root):
+    for d, dirs, fs in os.walk(root):
+        for name in dirs + fs:
+            if os.path.islink(os.path.join(d, name)):
+                rel = os.path.relpath(os.path.join(d, name), root).replace(os.sep, "/")
+                raise Refusal("package.path.unsafe", "%s: symbolic links are refused" % rel)
         for f in fs:
             out.append(os.path.relpath(os.path.join(d, f), root).replace(os.sep, "/"))
     return sorted(out)
 
 
-def validate(root, store, key=None):
-    """Validate the package at `root`. `store` is a schema store; `key` a hashing key for keyed packages."""
+def _hash_file(full, budget):
+    """SHA-256 and size of a file read in chunks, refusing a read past `budget` bytes."""
+    h, n = hashlib.sha256(), 0
+    with open(full, "rb") as fh:
+        while True:
+            chunk = fh.read(1 << 20)
+            if not chunk:
+                break
+            n += len(chunk)
+            if n > budget:
+                raise Refusal("package.size", "the package's files exceed the limit of %d bytes" % SIZE_LIMIT)
+            h.update(chunk)
+    return h.hexdigest(), n
+
+
+def validate(root, store, key=None, limit=SIZE_LIMIT):
+    """Validate the package at `root`. `store` is a schema store; `key` a hashing key for keyed packages;
+    `limit` the most bytes the validator reads before refusing."""
     rep = Report()
     try:
-        _validate(root, store, key, rep)
+        _validate(root, store, key, rep, limit)
     except Refusal as e:
         rep.add(e.rule, e.node or "", e.message)
     return rep
 
 
-def _validate(root, store, key, rep):
+def _validate(root, store, key, rep, limit=SIZE_LIMIT):
     mpath = os.path.join(root, "MANIFEST.json")
-    if not os.path.exists(mpath):
+    if not os.path.lexists(mpath):
         raise Refusal("package.manifest.missing", "MANIFEST.json is missing")
+    actual = set(_all_files(root))
     manifest = read_json(mpath)
     for e in schema_validate(store, "manifest.schema.json", manifest):
         rep.add("package.manifest.schema", "MANIFEST.json", e)
+    if not rep:
+        return
     listed = {}
-    paths = [f["path"] for f in manifest.get("files", [])]
+    paths = [f["path"] for f in manifest["files"]]
     if paths != sorted(paths, key=lambda p: p.encode("utf-8")) or len(set(paths)) != len(paths):
         rep.add("package.manifest.order", "MANIFEST.json", "files must be sorted by path and unique")
-    for f in manifest.get("files", []):
+    folded = {}
+    budget = limit
+    for f in manifest["files"]:
         p = f["path"]
         listed[p] = f
-        if not safe_path(p):
+        if not safe_path(p) or p == "MANIFEST.json":
             rep.add("package.path.unsafe", p, "unsafe path")
             continue
-        full = os.path.join(root, p)
+        fold = unicodedata.normalize("NFC", p).casefold()
+        if fold in folded:
+            rep.add("package.manifest.order", p, "collides with %s under case folding" % folded[fold])
+        folded[fold] = p
         if f.get("present", True) is False:
             if not p.startswith("assets/"):
                 rep.add("package.manifest.absent", p, "only assets may be listed without their bytes")
-            if os.path.exists(full):
+            if p in actual:
                 rep.add("package.manifest.absent", p, "a file listed as absent is present")
             continue
-        if not os.path.exists(full):
+        if p not in actual:
             rep.add("package.manifest.missing-file", p, "listed file is missing")
             continue
-        with open(full, "rb") as fh:
-            data = fh.read()
-        if sha256_hex(data) != f["sha256"] or len(data) != f["size"]:
+        sha, size = _hash_file(os.path.join(root, p), budget)
+        budget -= size
+        if sha != f["sha256"] or size != f["size"]:
             rep.add("package.manifest.hash", p, "SHA-256 or size does not match the manifest")
-    for p in _all_files(root):
+    for p in sorted(actual):
         if p != "MANIFEST.json" and p not in listed:
             rep.add("package.manifest.unlisted", p, "file is not listed in the manifest")
     for p in REQUIRED:
@@ -223,28 +281,15 @@ def _validate(root, store, key, rep):
     if scheme == "hmac-team" and key is not None and document["hashing"]["key_id"] != crypto.key_id(key):
         rep.add("doc.hash.key", "document.json", "key_id does not match the key supplied")
 
-    # References inside runs.
+    # References between nodes, and the counts.
     kinds = {n["id"]: n["kind"] for n in nodes}
-    for n in nodes:
-        for r in n.get("content", []) if n["kind"] in RUN_KINDS else []:
-            if r["kind"] == "cite":
-                for ref in r["refs"]:
-                    if kinds.get(ref) != "reference":
-                        rep.add("doc.run.cite", n["id"], "cite names %s, which is not a reference" % ref)
-            if r["kind"] == "fnmark" and r.get("footnote") is not None and kinds.get(r["footnote"]) != "footnote":
-                rep.add("doc.run.fnmark", n["id"], "fnmark names a node that is not a footnote")
-            if r["kind"] == "xref" and r.get("target") and "#" not in r["target"]:
-                if r["target"] not in kinds:
-                    rep.add("doc.run.xref", n["id"], "xref names no node in this document")
-        if n["kind"] == "table":
-            for col in n.get("attrs", {}).get("columns", []):
-                for h in col["header_path"]:
-                    if kinds.get(h) != "cell":
-                        rep.add("doc.table.header", n["id"], "a header path names %s, not a cell" % h)
-        if n["kind"] == "row":
-            for h in n.get("attrs", {}).get("header_path", []):
-                if kinds.get(h) != "cell":
-                    rep.add("doc.table.header", n["id"], "a row header path names %s, not a cell" % h)
+    try:
+        validate_references(doc)
+    except Refusal as e:
+        rep.add(e.rule, e.node or "", e.message)
+    leaves_n = sum(1 for n in nodes if n["kind"] in LEAVES)
+    if document.get("counts") != {"nodes": len(nodes), "leaves": leaves_n}:
+        rep.add("doc.counts", "document.json", "counts do not recompute")
 
     # Hashes.
     hasher = hashing.PLAIN if scheme == "plain" else (hashing.Hasher(key) if key else None)
@@ -285,7 +330,7 @@ def _validate(root, store, key, rep):
     # Corrections.
     cors = data["corrections.jsonl"]
     for i, c in enumerate(cors, 1):
-        if c["index"] != i:
+        if c.get("index") != i:
             rep.add("doc.correction.index", "corrections.jsonl:%d" % i, "indexes count from 1 without gaps")
         if c["id"] != record_id(sha256_hex(jcs.dumpb({k: v for k, v in c.items() if k != "id"}))):
             rep.add("doc.correction.id", "corrections.jsonl:%d" % i, "id does not derive from the record")
@@ -361,16 +406,33 @@ def _validate(root, store, key, rep):
             ids_listed = [x[0] for x in r["nodes"]]
             if ids_listed != sorted(ids_listed) or len(set(ids_listed)) != len(ids_listed):
                 rep.add("doc.receipt.nodes", r["id"], "nodes sort by ID without repeats")
-        if roster and not R.verify_signature(r, roster):
-            rep.add("doc.receipt.signature", r["id"], "signature does not verify")
+        if roster:
+            signer = roster.get(r["by"])
+            if not R.verify_signature(r, roster):
+                rep.add("doc.receipt.signature", r["id"], "signature does not verify")
+            elif r["signature"]["key"] != r["by"] or not signer or not R.key_valid_at(signer, r["at"]):
+                rep.add("doc.receipt.signature", r["id"], "the signing key is not the signer's key, valid when signed")
         qa = r.get("qa")
         if qa:
             orig = by_id.get(qa["of"])
-            if not orig:
-                rep.add("doc.qa.of", r["id"], "the spot-checked receipt is not in this package")
+            if not orig or orig["format"] != "scienting-receipt/1":
+                rep.add("doc.qa.of", r["id"], "the spot-checked receipt is not a receipt in this package")
+                continue
+            if r["scope"] != "qa:" + qa["of"]:
+                rep.add("doc.qa.of", r["id"], "the scope names another receipt than qa.of")
+            if orig["at"] > r["at"]:
+                rep.add("doc.qa.of", r["id"], "a spot-check is signed after the receipt it checks")
+            orig_hashes = dict(orig["nodes"])
+            if [n for n, _ in r["nodes"]] != qa["sample"] or any(orig_hashes.get(n) != h for n, h in r["nodes"]):
+                rep.add("doc.qa.sample", r["id"], "the receipt's nodes are not the sampled nodes at the original's hashes")
+            if not {e["node"] for e in qa["errors"]} <= set(qa["sample"]):
+                rep.add("doc.qa.sample", r["id"], "an error names a node outside the sample")
+            try:
+                seed = base64.b64decode(qa["seed"], validate=True)
+            except (binascii.Error, ValueError):
+                rep.add("doc.qa.seed", r["id"], "the seed is not Base64")
                 continue
             ck = roster.get(qa["key"])
-            seed = base64.b64decode(qa["seed"])
             if roster:
                 if not ck or ck["scheme"] != "ed25519" or "spot-check" not in ck["purposes"]:
                     rep.add("doc.qa.key", r["id"], "the seed key is not a registered Ed25519 spot-check key")
@@ -380,7 +442,9 @@ def _validate(root, store, key, rep):
                     rep.add("doc.qa.seed", r["id"], "the seed is not the checker's signature over the receipt")
             if qa["sample"] != R.sample(orig, seed, qa["rate"], qa["minimum"]):
                 rep.add("doc.qa.sample", r["id"], "the sample does not follow from the seed")
-    if roster and not withheld:
+    if not withheld and (roster or not recs):
+        # Without a roster no receipt can be judged valid, so a package that carries receipts but no roster is
+        # checked against its store instead; one with neither reports the leaves' own levels.
         if R.distribution(R.effective_levels(nodes, vals, recs, roster)) != document["fidelity"]:
             rep.add("doc.fidelity.report", "document.json", "the fidelity distribution does not recompute")
 
@@ -401,3 +465,137 @@ def read_state(root):
 
 def n_root_hash(nodes, doc):
     return {n["id"]: n for n in nodes}[doc.root]["node_hash"]
+
+
+# --- zipped packages (spec/12 §12.2 and §12.8) -----------------------------------------
+
+ZIP_EXTENSION = ".sct.zip"
+# The reference tools' own limit on an archive's expanded size (spec/12 §12.2 leaves the limit to the consumer).
+ZIP_LIMIT = 1 << 30
+_ZIP_DATE = (1980, 1, 1, 0, 0, 0)
+_FLAG_ENCRYPTED = 0x1
+_FLAG_UTF8 = 0x800
+
+
+def is_zip(path):
+    return os.path.isfile(path) and path.lower().endswith(".zip")
+
+
+def write_zip(root, path):
+    """Zip the package folder at `root` the way spec/12 §12.2 says a producer writes a byte-identical archive.
+
+    Entries sit under one top-level folder named by the key, sorted by path, stored, dated 1980-01-01 00:00,
+    with no extra fields and no directory entries. Returns the SHA-256 of the archive.
+    """
+    key = read_json(os.path.join(root, "document.json"))["key"]
+    files = _all_files(root)
+    for p in files:
+        if p != "MANIFEST.json" and not safe_path(p):
+            raise Refusal("package.path.unsafe", "unsafe path %r" % p)
+    with zipfile.ZipFile(path, "w") as zf:
+        for p in sorted(files, key=lambda x: x.encode("utf-8")):
+            zi = zipfile.ZipInfo(key + "/" + p, date_time=_ZIP_DATE)
+            zi.compress_type = zipfile.ZIP_STORED
+            zi.create_system = 3
+            zi.external_attr = 0o644 << 16
+            with open(os.path.join(root, p), "rb") as f:
+                zf.writestr(zi, f.read())
+    with open(path, "rb") as f:
+        return sha256_hex(f.read())
+
+
+def _zip_entries(zf, limit):
+    """Check every entry of an open archive as spec/12 §12.2 requires; return the top-level folder."""
+    infos = zf.infolist()
+    if not infos:
+        raise Refusal("package.zip.entry", "the archive has no entries")
+    folder, seen, total = None, set(), 0
+    for zi in infos:
+        name = zi.filename
+        if not (zi.flag_bits & _FLAG_UTF8) and any(ord(ch) > 0x7F for ch in name):
+            raise Refusal("package.zip.entry", "%r: entry name is not marked as UTF-8" % name)
+        if zi.flag_bits & _FLAG_ENCRYPTED:
+            raise Refusal("package.zip.entry", "%s: encrypted entries are refused" % name)
+        top, sep, rest = name.partition("/")
+        if not sep or not KEY_RE.match(top):
+            raise Refusal("package.zip.entry", "%s: every entry sits under one folder named by the key" % name)
+        if folder is None:
+            folder = top
+        elif top != folder:
+            raise Refusal("package.zip.entry", "%s: entries sit under more than one top-level folder" % name)
+        if name in seen:
+            raise Refusal("package.zip.entry", "%s: duplicate entry" % name)
+        seen.add(name)
+        if zi.is_dir():
+            if rest != "" and not safe_path(rest.rstrip("/")):
+                raise Refusal("package.zip.entry", "%s: unsafe path" % name)
+            continue
+        if stat.S_ISLNK(zi.external_attr >> 16):
+            raise Refusal("package.zip.entry", "%s: symbolic links are refused" % name)
+        if zi.compress_type not in (zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED):
+            raise Refusal("package.zip.entry", "%s: entries are stored or deflated" % name)
+        if rest != "MANIFEST.json" and not safe_path(rest):
+            raise Refusal("package.zip.entry", "%s: unsafe path" % name)
+        total += zi.file_size
+        if total > limit:
+            raise Refusal("package.zip.size", "the archive expands past the limit of %d bytes" % limit)
+    return folder
+
+
+def read_zip(path, dest, limit=ZIP_LIMIT):
+    """Unpack a zipped package into the empty or new folder `dest`, refusing what spec/12 §12.2 refuses.
+
+    Returns the package folder, `dest/<key>`. Every byte written is counted against each entry's declared size
+    and against `limit`, so a header that lies about a size is refused too.
+    """
+    try:
+        zf = zipfile.ZipFile(path)
+    except (zipfile.BadZipFile, OSError) as e:
+        raise Refusal("package.zip.entry", "not a zip archive: %s" % e)
+    with zf:
+        folder = _zip_entries(zf, limit)
+        root = os.path.join(dest, folder)
+        os.makedirs(root)
+        written = 0
+        for zi in zf.infolist():
+            if zi.is_dir():
+                continue
+            rest = zi.filename.partition("/")[2]
+            full = os.path.join(root, rest)
+            os.makedirs(os.path.dirname(full), exist_ok=True)
+            try:
+                with zf.open(zi) as src, open(full, "wb") as dst:
+                    n = 0
+                    while True:
+                        chunk = src.read(1 << 16)
+                        if not chunk:
+                            break
+                        n += len(chunk)
+                        written += len(chunk)
+                        if n > zi.file_size or written > limit:
+                            raise Refusal("package.zip.size", "%s: holds more than its header declares" % zi.filename)
+                        dst.write(chunk)
+            except (zipfile.BadZipFile, RuntimeError, OSError) as e:
+                raise Refusal("package.zip.entry", "%s: %s" % (zi.filename, e))
+        return root
+
+
+def validate_zip(path, store, key=None, limit=ZIP_LIMIT):
+    """Validate a zipped package: unpack it under spec/12 §12.2's rules, then validate the folder."""
+    with tempfile.TemporaryDirectory() as tmp:
+        try:
+            root = read_zip(path, tmp, limit)
+        except Refusal as e:
+            rep = Report()
+            rep.add(e.rule, e.node or "", e.message)
+            return rep
+        rep = validate(root, store, key)
+        mpath = os.path.join(root, "MANIFEST.json")
+        if os.path.exists(mpath):
+            try:
+                mkey = read_json(mpath).get("key")
+            except (Refusal, ValueError):
+                mkey = None
+            if mkey != os.path.basename(root):
+                rep.add("package.zip.entry", os.path.basename(root), "the top-level folder is not named by the key")
+        return rep

tools/oracle/receipts.py

@@ -4,15 +4,17 @@ Non-normative; see spec/08-fidelity.md.
 """
 
 import base64
+import binascii
 import hashlib
 import math
 from fractions import Fraction
+from math import comb
 
 from . import crypto, jcs
 from .model import LEVELS
 from .text import record_id, sha256_hex
 
-SIGNED_FORMATS = ("scienting-receipt/1", "scienting-revocation/1", "scienting-policy/1")
+SIGNED_FORMATS = ("scienting-receipt/1", "scienting-revocation/1", "scienting-policy/1", "scienting-qa-report/1")
 
 
 def record_hash(rec):
@@ -37,13 +39,29 @@ def seal(rec, secret, key_id):
     return rec
 
 
+def _b64(s):
+    """Strict Base64 decoding; None for anything that is not Base64."""
+    try:
+        return base64.b64decode(s, validate=True)
+    except (binascii.Error, ValueError, TypeError):
+        return None
+
+
 def verify_signature(rec, roster):
+    """Whether the record's Ed25519 signature verifies under the roster key it names. Never raises."""
     sig = rec.get("signature") or {}
+    if not isinstance(sig, dict):
+        return False
     key = roster.get(sig.get("key"))
-    if not key or sig.get("scheme") != "ed25519" or key["scheme"] != "ed25519":
+    if not key or sig.get("scheme") != "ed25519" or key.get("scheme") != "ed25519":
+        return False
+    pub, value = _b64(key.get("public", "")), _b64(sig.get("value", ""))
+    if pub is None or value is None or len(pub) != 32 or len(value) != 64:
+        return False
+    try:
+        return crypto.ed25519_verify(pub, signing_payload(rec), value)
+    except (ValueError, TypeError):
         return False
-    pub = base64.b64decode(key["public"])
-    return crypto.ed25519_verify(pub, signing_payload(rec), base64.b64decode(sig["value"]))
 
 
 def key_valid_at(key, at):
@@ -103,10 +121,31 @@ def upper_bound(k, n, confidence="0.95"):
 
 
 def bound_string(k, n, confidence="0.95"):
-    """The bound as the spec reports it: six decimal places, rounded up."""
-    x = upper_bound(k, n, confidence)
-    v = math.ceil(x * 10**6 - 1e-9)
-    return "%d.%06d" % (v // 10**6, v % 10**6)
+    """The bound as the spec reports it: six decimal places, rounded up, computed exactly.
+
+    The reported value is the least multiple of 10^-6 at which the probability of at most k errors in n is
+    at most 1 - c, found by bisection with integer arithmetic, so no floating point enters the result.
+    """
+    if n <= 0:
+        raise ValueError("n must be positive")
+    if k >= n:
+        return "1.000000"
+    alpha = 1 - Fraction(confidence)
+    scale = 10**6
+
+    def tail_at_most(m):
+        # P[X <= k] at p = m / scale, compared with alpha, both scaled by scale**n and alpha's denominator.
+        total = sum(comb(n, i) * m**i * (scale - m)**(n - i) for i in range(k + 1))
+        return total * alpha.denominator <= alpha.numerator * scale**n
+
+    lo, hi = 0, scale  # the tail at 0 is 1 (above alpha); at 1 it is 0 (k < n)
+    while hi - lo > 1:
+        mid = (lo + hi) // 2
+        if tail_at_most(mid):
+            hi = mid
+        else:
+            lo = mid
+    return "%d.%06d" % (hi // scale, hi % scale)
 
 
 # --- levels -------------------------------------------------------------------
@@ -141,26 +180,57 @@ def valid_revocations(revocations, receipts_by_id, roster):
     return out
 
 
-def receipt_status(rec, records, roster):
-    """`valid`, `revoked`, `suspended`, or `invalid` for one receipt."""
-    receipts = [x for x in records if x["format"] == "scienting-receipt/1"]
-    revs = [x for x in records if x["format"] == "scienting-revocation/1"]
+def _base_status(rec, roster, valid_revs):
+    """`invalid`, `revoked` or `valid` for one receipt, before spot-checks are considered (spec/08 §8.6)."""
+    try:
+        if rec.get("root") != crypto.merkle_root(crypto.receipt_items(rec)):
+            return "invalid"
+        if rec["id"] != record_id(record_hash(rec)) or not verify_signature(rec, roster):
+            return "invalid"
+        key = roster.get(rec["by"])
+        if not key or not key_valid_at(key, rec["at"]) or rec["signature"]["key"] != rec["by"]:
+            return "invalid"
+    except (KeyError, TypeError, ValueError):
+        return "invalid"
+    return "revoked" if revoked(rec, valid_revs) else "valid"
+
+
+def statuses(records, roster):
+    """The status of every receipt in `records` (spec/08 §8.6), computed once.
+
+    A receipt is suspended when a valid spot-check of it found an error, and a spot-check's own validity may
+    depend on its own spot-check, so statuses are settled from the end of each chain of spot-checks; a chain
+    that loops back on itself settles nothing, and its receipts are invalid.
+    """
+    receipts = [x for x in records if x.get("format") == "scienting-receipt/1"]
+    revs = [x for x in records if x.get("format") == "scienting-revocation/1"]
     by_id = {x["id"]: x for x in receipts}
-    if rec.get("root") != crypto.merkle_root(crypto.receipt_items(rec)):
-        return "invalid"
-    if rec["id"] != record_id(record_hash(rec)) or not verify_signature(rec, roster):
-        return "invalid"
-    key = roster.get(rec["by"])
-    if not key or not key_valid_at(key, rec["at"]) or rec["signature"]["key"] != rec["by"]:
-        return "invalid"
-    if revoked(rec, valid_revocations(revs, by_id, roster)):
-        return "revoked"
+    valid_revs = valid_revocations(revs, by_id, roster)
+    base = {rec["id"]: _base_status(rec, roster, valid_revs) for rec in receipts}
+    checkers = {}
     for q in receipts:
         qa = q.get("qa")
-        if qa and qa.get("of") == rec["id"] and qa.get("errors"):
-            if receipt_status(q, records, roster) == "valid":
-                return "suspended"
-    return "valid"
+        if isinstance(qa, dict) and qa.get("errors") and base.get(q["id"]) == "valid" and qa.get("of") in by_id:
+            checkers.setdefault(qa["of"], []).append(q["id"])
+    final = {rid: s for rid, s in base.items() if s != "valid" or rid not in checkers}
+    pending = set(base) - set(final)
+    progress = True
+    while pending and progress:
+        progress = False
+        for rid in sorted(pending):
+            qs = checkers[rid]
+            if all(q in final for q in qs):
+                final[rid] = "suspended" if any(final[q] == "valid" for q in qs) else "valid"
+                pending.discard(rid)
+                progress = True
+    for rid in pending:
+        final[rid] = "invalid"
+    return final
+
+
+def receipt_status(rec, records, roster):
+    """`valid`, `revoked`, `suspended`, or `invalid` for one receipt among `records`."""
+    return statuses(records, roster).get(rec.get("id"), "invalid")
 
 
 def effective_levels(nodes, values, records, roster):
@@ -169,10 +239,11 @@ def effective_levels(nodes, values, records, roster):
     A spot-check's `qa:` receipt does not raise the nodes it found in error (spec/08 §8.2).
     """
     level = {n["id"]: n.get("fidelity") for n in nodes if "fidelity" in n}
+    status = statuses(records, roster)
     for rec in records:
         if rec["format"] != "scienting-receipt/1":
             continue
-        if receipt_status(rec, records, roster) != "valid":
+        if status.get(rec["id"]) != "valid":
             continue
         wrong = {e["node"] for e in (rec.get("qa") or {}).get("errors", [])}
         for nid, h in rec.get("nodes", []):

tools/oracle/targets.py

@@ -0,0 +1,69 @@
+"""The target set of targeted F3: the nodes a team's claims, data and quotations rest on.
+
+Non-normative; see spec/08-fidelity.md §8.10.
+"""
+
+from .model import LEAVES, MATH_KINDS
+from .text import is_digit
+
+DIGIT_RULES = ("all", "anchored-sections", "none")
+# The kinds that count as the section a node sits in: a section, or the front or back matter outside any.
+SECTION_KINDS = frozenset(["section", "front", "back"])
+DIGIT_KINDS = frozenset(["sentence", "cell"])
+DIGIT_RUNS = frozenset(["text", "math"])
+
+
+def has_digit_run(node):
+    """Whether a `text` or `math` run of the node holds a digit (General Category Nd)."""
+    for r in node.get("content") or []:
+        if not isinstance(r, dict) or r.get("kind") not in DIGIT_RUNS:
+            continue
+        s = r.get("text") if r["kind"] == "text" else r.get("source")
+        if any(is_digit(ch) for ch in s or ""):
+            return True
+    return False
+
+
+def enclosing_section(doc, nid):
+    """The nearest ancestor-or-self that is a section, or the front or back matter, else the root."""
+    if doc.nodes[nid]["kind"] in SECTION_KINDS:
+        return nid
+    for a in reversed(doc.ancestors(nid)):
+        if doc.nodes[a]["kind"] in SECTION_KINDS:
+            return a
+    return doc.root
+
+
+def target_set(doc, anchored, digit_rule="all"):
+    """The target set as `[[node, reason], …]` in document order, with `reason` the first that applies of
+    `anchored`, `equation` and `digits`.
+
+    `anchored` lists the live nodes that extracts, dataset values, confirmed claim links, draft quotations or
+    pinned citations anchor; a container among them contributes its leaves. Every equation and equation line
+    is a target. Under `all`, so is every sentence or cell with a digit in a text or math run; under
+    `anchored-sections`, only those in a section that holds an anchored node; under `none`, none.
+    """
+    if digit_rule not in DIGIT_RULES:
+        raise ValueError("digit rule must be one of %s" % ", ".join(DIGIT_RULES))
+    anchored = set(anchored)
+    unknown = sorted(a for a in anchored if a not in doc.nodes)
+    if unknown:
+        raise ValueError("anchored nodes are not live: %s" % ", ".join(unknown))
+    reasons = {}
+    for nid in sorted(anchored):
+        if doc.nodes[nid]["kind"] in LEAVES:
+            reasons[nid] = "anchored"
+        else:
+            for leaf in doc.leaves(nid):
+                reasons.setdefault(leaf, "anchored")
+    for nid in doc.preorder():
+        if doc.nodes[nid]["kind"] in MATH_KINDS:
+            reasons.setdefault(nid, "equation")
+    if digit_rule != "none":
+        sections = {enclosing_section(doc, a) for a in anchored}
+        for nid in doc.preorder():
+            n = doc.nodes[nid]
+            if n["kind"] in DIGIT_KINDS and has_digit_run(n):
+                if digit_rule == "all" or enclosing_section(doc, nid) in sections:
+                    reasons.setdefault(nid, "digits")
+    return [[nid, reasons[nid]] for nid in doc.preorder() if nid in reasons]

tools/oracle/tree.py

@@ -212,5 +212,38 @@ def _check_content(doc, nid):
         raise Refusal("doc.node.attrs", "a section needs a level", nid)
 
 
+def validate_references(doc):
+    """Check the references between nodes: cite, fnmark and xref runs, and the header paths of tables and rows
+    (spec/03 §3.4–3.5, spec/09 §9.2). Raises Refusal."""
+    kinds = {nid: doc.nodes[nid]["kind"] for nid in doc.order}
+    for nid in doc.order:
+        n = doc.nodes[nid]
+        if n["kind"] in RUN_KINDS and not n.get("withheld"):
+            for r in n.get("content") or []:
+                if r["kind"] == "cite":
+                    for ref in r["refs"]:
+                        if kinds.get(ref) != "reference":
+                            raise Refusal("doc.run.cite", "cite names %s, which is not a reference" % ref, nid)
+                if r["kind"] == "fnmark" and r.get("footnote") is not None and kinds.get(r["footnote"]) != "footnote":
+                    raise Refusal("doc.run.fnmark", "fnmark names a node that is not a footnote", nid)
+                if r["kind"] == "xref" and r.get("target") and "#" not in r["target"] and r["target"] not in kinds:
+                    raise Refusal("doc.run.xref", "xref names no node in this document", nid)
+        if n["kind"] == "table":
+            for col in n.get("attrs", {}).get("columns", []):
+                for h in col["header_path"]:
+                    if kinds.get(h) != "cell":
+                        raise Refusal("doc.table.header", "a header path names %s, not a cell" % h, nid)
+        if n["kind"] == "row":
+            for h in n.get("attrs", {}).get("header_path", []):
+                if kinds.get(h) != "cell":
+                    raise Refusal("doc.table.header", "a row header path names %s, not a cell" % h, nid)
+        if n["kind"] == "cell":
+            col = n.get("attrs", {}).get("column")
+            table = next((a for a in reversed(doc.ancestors(nid)) if kinds[a] == "table"), None)
+            columns = {c["id"] for c in doc.nodes[table].get("attrs", {}).get("columns", [])} if table else set()
+            if col is not None and columns and col not in columns:
+                raise Refusal("doc.table.header", "a cell names a column its table does not define", nid)
+
+
 def leaf_kinds():
     return LEAVES