infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: zipped packages, target set, checklists, audit fixes and a stress test

Browse Source 
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 20:07:22 -04:00
1 parent 8c8b51cc3f
commit ea0a7efa78
142 files changed
+2312 -268
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tools/oracle/package.py
+260 -62
View File
@@ -4,17 +4,26 @@ Non-normative; see spec/12-package.md and spec/22-conformance.md.
"""
import base64
import binascii
import hashlib
import json
import os
import posixpath
import re
import stat
import tempfile
import unicodedata
import zipfile
from . import crypto, hashing, jcs, receipts as R
from .anchors import ADDRESS_RE
from .model import LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc
from .schema import validate as schema_validate
from .text import letter_first, record_id, sha256_hex
from .tree import Refusal, validate_structure
from .text import KEY_RE, letter_first, record_id, sha256_hex
from .tree import Refusal, validate_references, validate_structure
# The reference tools' own limit on a package's total size (spec/12 §12.2 leaves the limit to the consumer).
SIZE_LIMIT = 1 << 30
JSONL = {
"nodes.jsonl": "node.schema.json",
@@ -85,26 +94,46 @@ def write(root, files, absent=()):
return sha256_hex(mb)
def _refuse_constant(name):
raise ValueError("JSON has no canonical form for " + name)
def _parse_int(text):
"""Read a JSON integer as an ECMAScript engine would: exactly within 2^53, as a double beyond (spec/01 §1.1)."""
v = int(text)
return v if abs(v) <= 2**53 else float(v)
def _canonical_bytes(data, where):
"""Parse one canonical JSON value given as bytes; refuse anything else (spec/01 §1.1, §1.9).
Numbers are read as IEEE 754 doubles, as the canonical form writes them, and the comparison is on bytes, so a
carriage return, a byte order mark, a lone surrogate, NaN or invalid UTF-8 is refused, not silently translated.
"""
try:
rec = json.loads(data.decode("utf-8"), parse_constant=_refuse_constant, parse_int=_parse_int)
canon = jcs.dumps(rec).encode("utf-8")
except (UnicodeDecodeError, ValueError, TypeError, OverflowError, RecursionError) as e:
raise Refusal("package.file.canonical", "%s: not canonical JSON (%s)" % (where, type(e).__name__))
if canon != data:
raise Refusal("package.file.canonical", "%s: not canonical JSON" % where)
return rec
def read_jsonl(path):
out = []
with open(path, encoding="utf-8") as f:
for i, ln in enumerate(f, 1):
if not ln.endswith("\n"):
raise Refusal("package.file.newline", "%s:%d: every line ends with a newline" % (path, i))
rec = json.loads(ln)
if line(rec) != ln:
raise Refusal("package.file.canonical", "%s:%d: line is not canonical JSON" % (path, i))
out.append(rec)
return out
with open(path, "rb") as f:
data = f.read()
if data and not data.endswith(b"\n"):
raise Refusal("package.file.newline", "%s: every line ends with a newline" % path)
return [_canonical_bytes(ln, "%s:%d" % (path, i)) for i, ln in enumerate(data.split(b"\n")[:-1], 1)]
def read_json(path):
with open(path, encoding="utf-8") as f:
text = f.read()
rec = json.loads(text)
if line(rec) != text:
raise Refusal("package.file.canonical", "%s: file is not canonical JSON" % path)
return rec
with open(path, "rb") as f:
data = f.read()
if not data.endswith(b"\n"):
raise Refusal("package.file.canonical", "%s: the file does not end with one newline" % path)
return _canonical_bytes(data[:-1], path)
class Report:
@@ -119,55 +148,84 @@ class Report:
def _all_files(root):
"""Every file under `root` by its exact name, refusing any symbolic link on the way (spec/12 §12.2)."""
out = []
for d, _, fs in os.walk(root):
for d, dirs, fs in os.walk(root):
for name in dirs + fs:
if os.path.islink(os.path.join(d, name)):
rel = os.path.relpath(os.path.join(d, name), root).replace(os.sep, "/")
raise Refusal("package.path.unsafe", "%s: symbolic links are refused" % rel)
for f in fs:
out.append(os.path.relpath(os.path.join(d, f), root).replace(os.sep, "/"))
return sorted(out)
def validate(root, store, key=None):
"""Validate the package at `root`. `store` is a schema store; `key` a hashing key for keyed packages."""
def _hash_file(full, budget):
"""SHA-256 and size of a file read in chunks, refusing a read past `budget` bytes."""
h, n = hashlib.sha256(), 0
with open(full, "rb") as fh:
while True:
chunk = fh.read(1 << 20)
if not chunk:
break
n += len(chunk)
if n > budget:
raise Refusal("package.size", "the package's files exceed the limit of %d bytes" % SIZE_LIMIT)
h.update(chunk)
return h.hexdigest(), n
def validate(root, store, key=None, limit=SIZE_LIMIT):
"""Validate the package at `root`. `store` is a schema store; `key` a hashing key for keyed packages;
`limit` the most bytes the validator reads before refusing."""
rep = Report()
try:
_validate(root, store, key, rep)
_validate(root, store, key, rep, limit)
except Refusal as e:
rep.add(e.rule, e.node or "", e.message)
return rep
def _validate(root, store, key, rep):
def _validate(root, store, key, rep, limit=SIZE_LIMIT):
mpath = os.path.join(root, "MANIFEST.json")
if not os.path.exists(mpath):
if not os.path.lexists(mpath):
raise Refusal("package.manifest.missing", "MANIFEST.json is missing")
actual = set(_all_files(root))
manifest = read_json(mpath)
for e in schema_validate(store, "manifest.schema.json", manifest):
rep.add("package.manifest.schema", "MANIFEST.json", e)
if not rep:
return
listed = {}
paths = [f["path"] for f in manifest.get("files", [])]
paths = [f["path"] for f in manifest["files"]]
if paths != sorted(paths, key=lambda p: p.encode("utf-8")) or len(set(paths)) != len(paths):
rep.add("package.manifest.order", "MANIFEST.json", "files must be sorted by path and unique")
for f in manifest.get("files", []):
folded = {}
budget = limit
for f in manifest["files"]:
p = f["path"]
listed[p] = f
if not safe_path(p):
if not safe_path(p) or p == "MANIFEST.json":
rep.add("package.path.unsafe", p, "unsafe path")
continue
full = os.path.join(root, p)
fold = unicodedata.normalize("NFC", p).casefold()
if fold in folded:
rep.add("package.manifest.order", p, "collides with %s under case folding" % folded[fold])
folded[fold] = p
if f.get("present", True) is False:
if not p.startswith("assets/"):
rep.add("package.manifest.absent", p, "only assets may be listed without their bytes")
if os.path.exists(full):
if p in actual:
rep.add("package.manifest.absent", p, "a file listed as absent is present")
continue
if not os.path.exists(full):
if p not in actual:
rep.add("package.manifest.missing-file", p, "listed file is missing")
continue
with open(full, "rb") as fh:
data = fh.read()
if sha256_hex(data) != f["sha256"] or len(data) != f["size"]:
sha, size = _hash_file(os.path.join(root, p), budget)
budget -= size
if sha != f["sha256"] or size != f["size"]:
rep.add("package.manifest.hash", p, "SHA-256 or size does not match the manifest")
for p in _all_files(root):
for p in sorted(actual):
if p != "MANIFEST.json" and p not in listed:
rep.add("package.manifest.unlisted", p, "file is not listed in the manifest")
for p in REQUIRED:
@@ -223,28 +281,15 @@ def _validate(root, store, key, rep):
if scheme == "hmac-team" and key is not None and document["hashing"]["key_id"] != crypto.key_id(key):
rep.add("doc.hash.key", "document.json", "key_id does not match the key supplied")
# References inside runs.
# References between nodes, and the counts.
kinds = {n["id"]: n["kind"] for n in nodes}
for n in nodes:
for r in n.get("content", []) if n["kind"] in RUN_KINDS else []:
if r["kind"] == "cite":
for ref in r["refs"]:
if kinds.get(ref) != "reference":
rep.add("doc.run.cite", n["id"], "cite names %s, which is not a reference" % ref)
if r["kind"] == "fnmark" and r.get("footnote") is not None and kinds.get(r["footnote"]) != "footnote":
rep.add("doc.run.fnmark", n["id"], "fnmark names a node that is not a footnote")
if r["kind"] == "xref" and r.get("target") and "#" not in r["target"]:
if r["target"] not in kinds:
rep.add("doc.run.xref", n["id"], "xref names no node in this document")
if n["kind"] == "table":
for col in n.get("attrs", {}).get("columns", []):
for h in col["header_path"]:
if kinds.get(h) != "cell":
rep.add("doc.table.header", n["id"], "a header path names %s, not a cell" % h)
if n["kind"] == "row":
for h in n.get("attrs", {}).get("header_path", []):
if kinds.get(h) != "cell":
rep.add("doc.table.header", n["id"], "a row header path names %s, not a cell" % h)
try:
validate_references(doc)
except Refusal as e:
rep.add(e.rule, e.node or "", e.message)
leaves_n = sum(1 for n in nodes if n["kind"] in LEAVES)
if document.get("counts") != {"nodes": len(nodes), "leaves": leaves_n}:
rep.add("doc.counts", "document.json", "counts do not recompute")
# Hashes.
hasher = hashing.PLAIN if scheme == "plain" else (hashing.Hasher(key) if key else None)
@@ -285,7 +330,7 @@ def _validate(root, store, key, rep):
# Corrections.
cors = data["corrections.jsonl"]
for i, c in enumerate(cors, 1):
if c["index"] != i:
if c.get("index") != i:
rep.add("doc.correction.index", "corrections.jsonl:%d" % i, "indexes count from 1 without gaps")
if c["id"] != record_id(sha256_hex(jcs.dumpb({k: v for k, v in c.items() if k != "id"}))):
rep.add("doc.correction.id", "corrections.jsonl:%d" % i, "id does not derive from the record")
@@ -361,16 +406,33 @@ def _validate(root, store, key, rep):
ids_listed = [x[0] for x in r["nodes"]]
if ids_listed != sorted(ids_listed) or len(set(ids_listed)) != len(ids_listed):
rep.add("doc.receipt.nodes", r["id"], "nodes sort by ID without repeats")
if roster and not R.verify_signature(r, roster):
rep.add("doc.receipt.signature", r["id"], "signature does not verify")
if roster:
signer = roster.get(r["by"])
if not R.verify_signature(r, roster):
rep.add("doc.receipt.signature", r["id"], "signature does not verify")
elif r["signature"]["key"] != r["by"] or not signer or not R.key_valid_at(signer, r["at"]):
rep.add("doc.receipt.signature", r["id"], "the signing key is not the signer's key, valid when signed")
qa = r.get("qa")
if qa:
orig = by_id.get(qa["of"])
if not orig:
rep.add("doc.qa.of", r["id"], "the spot-checked receipt is not in this package")
if not orig or orig["format"] != "scienting-receipt/1":
rep.add("doc.qa.of", r["id"], "the spot-checked receipt is not a receipt in this package")
continue
if r["scope"] != "qa:" + qa["of"]:
rep.add("doc.qa.of", r["id"], "the scope names another receipt than qa.of")
if orig["at"] > r["at"]:
rep.add("doc.qa.of", r["id"], "a spot-check is signed after the receipt it checks")
orig_hashes = dict(orig["nodes"])
if [n for n, _ in r["nodes"]] != qa["sample"] or any(orig_hashes.get(n) != h for n, h in r["nodes"]):
rep.add("doc.qa.sample", r["id"], "the receipt's nodes are not the sampled nodes at the original's hashes")
if not {e["node"] for e in qa["errors"]} <= set(qa["sample"]):
rep.add("doc.qa.sample", r["id"], "an error names a node outside the sample")
try:
seed = base64.b64decode(qa["seed"], validate=True)
except (binascii.Error, ValueError):
rep.add("doc.qa.seed", r["id"], "the seed is not Base64")
continue
ck = roster.get(qa["key"])
seed = base64.b64decode(qa["seed"])
if roster:
if not ck or ck["scheme"] != "ed25519" or "spot-check" not in ck["purposes"]:
rep.add("doc.qa.key", r["id"], "the seed key is not a registered Ed25519 spot-check key")
@@ -380,7 +442,9 @@ def _validate(root, store, key, rep):
rep.add("doc.qa.seed", r["id"], "the seed is not the checker's signature over the receipt")
if qa["sample"] != R.sample(orig, seed, qa["rate"], qa["minimum"]):
rep.add("doc.qa.sample", r["id"], "the sample does not follow from the seed")
if roster and not withheld:
if not withheld and (roster or not recs):
# Without a roster no receipt can be judged valid, so a package that carries receipts but no roster is
# checked against its store instead; one with neither reports the leaves' own levels.
if R.distribution(R.effective_levels(nodes, vals, recs, roster)) != document["fidelity"]:
rep.add("doc.fidelity.report", "document.json", "the fidelity distribution does not recompute")
@@ -401,3 +465,137 @@ def read_state(root):
def n_root_hash(nodes, doc):
return {n["id"]: n for n in nodes}[doc.root]["node_hash"]
# --- zipped packages (spec/12 §12.2 and §12.8) -----------------------------------------
ZIP_EXTENSION = ".sct.zip"
# The reference tools' own limit on an archive's expanded size (spec/12 §12.2 leaves the limit to the consumer).
ZIP_LIMIT = 1 << 30
_ZIP_DATE = (1980, 1, 1, 0, 0, 0)
_FLAG_ENCRYPTED = 0x1
_FLAG_UTF8 = 0x800
def is_zip(path):
return os.path.isfile(path) and path.lower().endswith(".zip")
def write_zip(root, path):
"""Zip the package folder at `root` the way spec/12 §12.2 says a producer writes a byte-identical archive.
Entries sit under one top-level folder named by the key, sorted by path, stored, dated 1980-01-01 00:00,
with no extra fields and no directory entries. Returns the SHA-256 of the archive.
"""
key = read_json(os.path.join(root, "document.json"))["key"]
files = _all_files(root)
for p in files:
if p != "MANIFEST.json" and not safe_path(p):
raise Refusal("package.path.unsafe", "unsafe path %r" % p)
with zipfile.ZipFile(path, "w") as zf:
for p in sorted(files, key=lambda x: x.encode("utf-8")):
zi = zipfile.ZipInfo(key + "/" + p, date_time=_ZIP_DATE)
zi.compress_type = zipfile.ZIP_STORED
zi.create_system = 3
zi.external_attr = 0o644 << 16
with open(os.path.join(root, p), "rb") as f:
zf.writestr(zi, f.read())
with open(path, "rb") as f:
return sha256_hex(f.read())
def _zip_entries(zf, limit):
"""Check every entry of an open archive as spec/12 §12.2 requires; return the top-level folder."""
infos = zf.infolist()
if not infos:
raise Refusal("package.zip.entry", "the archive has no entries")
folder, seen, total = None, set(), 0
for zi in infos:
name = zi.filename
if not (zi.flag_bits & _FLAG_UTF8) and any(ord(ch) > 0x7F for ch in name):
raise Refusal("package.zip.entry", "%r: entry name is not marked as UTF-8" % name)
if zi.flag_bits & _FLAG_ENCRYPTED:
raise Refusal("package.zip.entry", "%s: encrypted entries are refused" % name)
top, sep, rest = name.partition("/")
if not sep or not KEY_RE.match(top):
raise Refusal("package.zip.entry", "%s: every entry sits under one folder named by the key" % name)
if folder is None:
folder = top
elif top != folder:
raise Refusal("package.zip.entry", "%s: entries sit under more than one top-level folder" % name)
if name in seen:
raise Refusal("package.zip.entry", "%s: duplicate entry" % name)
seen.add(name)
if zi.is_dir():
if rest != "" and not safe_path(rest.rstrip("/")):
raise Refusal("package.zip.entry", "%s: unsafe path" % name)
continue
if stat.S_ISLNK(zi.external_attr >> 16):
raise Refusal("package.zip.entry", "%s: symbolic links are refused" % name)
if zi.compress_type not in (zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED):
raise Refusal("package.zip.entry", "%s: entries are stored or deflated" % name)
if rest != "MANIFEST.json" and not safe_path(rest):
raise Refusal("package.zip.entry", "%s: unsafe path" % name)
total += zi.file_size
if total > limit:
raise Refusal("package.zip.size", "the archive expands past the limit of %d bytes" % limit)
return folder
def read_zip(path, dest, limit=ZIP_LIMIT):
"""Unpack a zipped package into the empty or new folder `dest`, refusing what spec/12 §12.2 refuses.
Returns the package folder, `dest/<key>`. Every byte written is counted against each entry's declared size
and against `limit`, so a header that lies about a size is refused too.
"""
try:
zf = zipfile.ZipFile(path)
except (zipfile.BadZipFile, OSError) as e:
raise Refusal("package.zip.entry", "not a zip archive: %s" % e)
with zf:
folder = _zip_entries(zf, limit)
root = os.path.join(dest, folder)
os.makedirs(root)
written = 0
for zi in zf.infolist():
if zi.is_dir():
continue
rest = zi.filename.partition("/")[2]
full = os.path.join(root, rest)
os.makedirs(os.path.dirname(full), exist_ok=True)
try:
with zf.open(zi) as src, open(full, "wb") as dst:
n = 0
while True:
chunk = src.read(1 << 16)
if not chunk:
break
n += len(chunk)
written += len(chunk)
if n > zi.file_size or written > limit:
raise Refusal("package.zip.size", "%s: holds more than its header declares" % zi.filename)
dst.write(chunk)
except (zipfile.BadZipFile, RuntimeError, OSError) as e:
raise Refusal("package.zip.entry", "%s: %s" % (zi.filename, e))
return root
def validate_zip(path, store, key=None, limit=ZIP_LIMIT):
"""Validate a zipped package: unpack it under spec/12 §12.2's rules, then validate the folder."""
with tempfile.TemporaryDirectory() as tmp:
try:
root = read_zip(path, tmp, limit)
except Refusal as e:
rep = Report()
rep.add(e.rule, e.node or "", e.message)
return rep
rep = validate(root, store, key)
mpath = os.path.join(root, "MANIFEST.json")
if os.path.exists(mpath):
try:
mkey = read_json(mpath).get("key")
except (Refusal, ValueError):
mkey = None
if mkey != os.path.basename(root):
rep.add("package.zip.entry", os.path.basename(root), "the top-level folder is not named by the key")
return rep