feat: zipped packages, target set, checklists, audit fixes and a stress test

Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tools/load.py

@@ -4,6 +4,7 @@
 Non-normative tooling.
 Usage: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]
        python3 tools/load.py --sqlite lab.db --bundle --built-from REV PACKAGE [PACKAGE ...]
+A package is a folder, or a zipped package (`.sct.zip`), unpacked under the rules of spec/12 §12.2 first.
 With --bundle the new file is a scienting-bundle/1: it also gets the tables of sql/bundle-sqlite.sql,
 the full-text search index and bundle_meta (spec/13 §13.5).
 Each package is validated first, and nothing is loaded if any package fails.
@@ -15,6 +16,7 @@ import argparse
 import os
 import sqlite3
 import sys
+import tempfile
 
 ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 sys.path.insert(0, ROOT)
@@ -29,37 +31,50 @@ def main():
     ap.add_argument("--bundle", action="store_true", help="build a new scienting-bundle/1 file with search")
     ap.add_argument("--built-from", help="with --bundle: the store revision the packages come from")
     ap.add_argument("--store", help="with --bundle: the store's name (default: the packages' common store)")
-    ap.add_argument("packages", nargs="+", help="package folders")
+    ap.add_argument("--max-size", metavar="BYTES", type=int, default=package.ZIP_LIMIT,
+                    help="refuse a zipped package that expands past this many bytes (default %d)" % package.ZIP_LIMIT)
+    ap.add_argument("packages", nargs="+", help="package folders or zipped packages")
     a = ap.parse_args()
     if a.bundle and (os.path.exists(a.sqlite) or not a.built_from):
         ap.error("--bundle needs --built-from and a database file that does not exist yet")
     store, bad = Store(os.path.join(ROOT, "schemas")), False
-    for p in a.packages:
-        for e in package.validate(p, store).errors:
-            print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
-            bad = True
-    if bad:
-        sys.exit("Nothing was loaded.")
-    new = not os.path.exists(a.sqlite)
-    conn = sqlite3.connect(a.sqlite)
-    if a.bundle:
-        db.create_bundle(conn, os.path.join(ROOT, "sql"))
-    elif new:
-        db.create(conn, os.path.join(ROOT, "sql"))
-    for p in a.packages:
-        key = db.load(conn, p, a.sqlite + ".assets")
-        n = conn.execute("SELECT COUNT(*) FROM nodes WHERE doc_key = ?", (key,)).fetchone()[0]
+    with tempfile.TemporaryDirectory() as tmp:
+        folders = []
+        for i, p in enumerate(a.packages):
+            if package.is_zip(p):
+                try:
+                    folders.append(package.read_zip(p, os.path.join(tmp, str(i)), a.max_size))
+                except package.Refusal as e:
+                    print("%s: %s: %s" % (p, e.rule, e.message), file=sys.stderr)
+                    bad = True
+                    continue
+            else:
+                folders.append(p)
+            for e in package.validate(folders[-1], store).errors:
+                print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
+                bad = True
+        if bad:
+            sys.exit("Nothing was loaded.")
+        new = not os.path.exists(a.sqlite)
+        conn = sqlite3.connect(a.sqlite)
         if a.bundle:
-            print("%s: %d nodes, %d search rows" % (key, n, db.index(conn, key)))
-        else:
-            print("%s: %d nodes" % (key, n))
-    if a.bundle:
-        stores = {r[0] for r in conn.execute("SELECT DISTINCT store FROM documents WHERE store IS NOT NULL")}
-        store = a.store or (stores.pop() if len(stores) == 1 else None)
-        if store is None:
-            sys.exit("The packages name no single store; give one with --store.")
-        db.write_meta(conn, store, a.built_from)
-    conn.commit()
+            db.create_bundle(conn, os.path.join(ROOT, "sql"))
+        elif new:
+            db.create(conn, os.path.join(ROOT, "sql"))
+        for p in folders:
+            key = db.load(conn, p, a.sqlite + ".assets")
+            n = conn.execute("SELECT COUNT(*) FROM nodes WHERE doc_key = ?", (key,)).fetchone()[0]
+            if a.bundle:
+                print("%s: %d nodes, %d search rows" % (key, n, db.index(conn, key)))
+            else:
+                print("%s: %d nodes" % (key, n))
+        if a.bundle:
+            stores = {r[0] for r in conn.execute("SELECT DISTINCT store FROM documents WHERE store IS NOT NULL")}
+            store_name = a.store or (stores.pop() if len(stores) == 1 else None)
+            if store_name is None:
+                sys.exit("The packages name no single store; give one with --store.")
+            db.write_meta(conn, store_name, a.built_from)
+        conn.commit()
 
 
 if __name__ == "__main__":