feat: zipped packages, target set, checklists, audit fixes and a stress test
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
142 files changed
+2312
-268
No files matched your search
+15
-2
@@ -30,6 +30,7 @@ A conforming reader accepts each one and recomputes every hash it contains.
|
||||
| `packages/okafor2025open/` | An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness. |
|
||||
| `packages/smithlab2027charges/` | A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#r1gacps`, a sentence of the main example. |
|
||||
| `packages/tiny2026example/` | A small document, the starting point of the correction vectors (`vectors/corrections.jsonl`) and of every broken package below. |
|
||||
| `packages/tiny2026example.sct.zip` | The same document as a zipped package, written the way a producer makes a byte-identical archive: entries under one folder named by the key, sorted by path, stored, dated 1980-01-01, with no extra fields ([spec/12 section 12.2](../spec/12-package.md#122-paths)). A reader unpacks it to exactly the folder beside it. |
|
||||
|
||||
## Records
|
||||
|
||||
@@ -49,12 +50,13 @@ Each record is a single file, or a folder where a format needs several files.
|
||||
| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). |
|
||||
| `submissions/`, `grades/` | A student's answers to exercises in the layer, and the grader's result for each. |
|
||||
| `suspect.json` | A doubtful reading raised for review: the converter's reading of an exponent in the main example, before the first correction answered it. |
|
||||
| `qa-report.json` | Reviewer error bounds for a delivery that covers two of the main example's receipts. |
|
||||
| `qa-report.json` | Reviewer error bounds for a delivery that covers two of the main example's receipts, signed by the team's owner key. |
|
||||
| `checklist/` | A conformance checklist (`checklist.json`) for an accessible HTML export of the main example, checked against WCAG 2.1 AA criterion by criterion, and the receipt that covers it at its hash (`receipt.json`). The format is `provisional`. |
|
||||
| `project.json`, `digitization.json`, `digitization-confirmation.json`, `datasets/`, `view.vl.json`, `snapshot.json`, `draft-ids.jsonl` | A team project; values read off the main example's figure by hand, the receipt that confirms them, and the dataset they yield; a Vega-Lite chart pinned to that dataset; what the draft cites, pinned; and the sentence IDs of the draft's Typst project. These formats are `provisional`. |
|
||||
|
||||
## Broken packages
|
||||
|
||||
Each folder in `invalid/` holds a `package/` that breaks exactly one rule, built from `packages/tiny2026example/` by one edit, and an `expect.json` that names the rule.
|
||||
Each folder in `invalid/` holds a `package/` folder, or a `package.sct.zip` archive, that breaks exactly one rule, built from `packages/tiny2026example/` by one edit, and an `expect.json` that names the rule.
|
||||
A conforming implementation refuses every one of them and should report that rule.
|
||||
`registry/rules.json` lists every rule ID with its meaning, and a validator reports each failure as `path:line: rule: message` ([spec/22 section 22.2](../spec/22-conformance.md#222-validating-a-package)).
|
||||
|
||||
@@ -64,19 +66,30 @@ A conforming implementation refuses every one of them and should report that rul
|
||||
| `bad-id/` | An ID that uses a letter Crockford base32 lacks (i, l, o or u). | `package.schema` |
|
||||
| `checks-only-plain/` | A checks-only package that still carries text and plain hashes. | `doc.policy.withheld` |
|
||||
| `cite-not-reference/` | A cite run names a node that is not a reference. | `doc.run.cite` |
|
||||
| `counts-wrong/` | document.json's counts are not the number of nodes and leaves. | `doc.counts` |
|
||||
| `crlf/` | A file whose lines end in CR LF, which is not the canonical form. | `package.file.canonical` |
|
||||
| `empty-optional-file/` | An optional JSONL file with no records. | `package.file.empty` |
|
||||
| `hash-mismatch/` | A node_hash that does not recompute. | `doc.hash.node-hash` |
|
||||
| `heading-not-first/` | A heading must be a section's first child. | `doc.tree.child` |
|
||||
| `location-unknown-page/` | A location on a page no witness has. | `doc.location.page` |
|
||||
| `manifest-hash/` | A file changed after the manifest was written. | `package.manifest.hash` |
|
||||
| `manifest-key/` | The manifest names another key than document.json. | `package.manifest.key` |
|
||||
| `missing-required-file/` | A required file is missing. | `package.file.required` |
|
||||
| `not-canonical/` | A line that is valid JSON but not canonical JSON (the fixed form of RFC 8785, which gives equal data identical bytes). | `package.file.canonical` |
|
||||
| `receipt-root/` | A receipt whose Merkle root (one hash that summarizes a list of hashes) does not recompute. | `doc.receipt.root` |
|
||||
| `receipt-signature/` | A receipt changed after it was signed. | `doc.receipt.signature` |
|
||||
| `receipt-unsorted-nodes/` | A receipt whose nodes are not sorted by ID. | `doc.receipt.nodes` |
|
||||
| `retired-live/` | An ID that is both live and retired. | `doc.id.retired-live` |
|
||||
| `rev-without-corrections/` | document.json's rev is not the number of corrections. | `doc.correction.rev` |
|
||||
| `root-mismatch/` | document.json's root is not the root node's node_hash. | `doc.hash.root` |
|
||||
| `signer-mismatch/` | A receipt whose signature was made with another key than the signer it names. | `doc.receipt.signature` |
|
||||
| `style-sup-sub/` | A run cannot be both a subscript and a superscript. | `doc.run.style` |
|
||||
| `text-edited/` | Text changed without its hashes. | `doc.hash.text-hash` |
|
||||
| `unknown-kind/` | Kinds outside the list are refused. | `package.schema` |
|
||||
| `unlisted-file/` | A file the manifest does not list. | `package.manifest.unlisted` |
|
||||
| `unsafe-path/` | The manifest lists a path that leaves the package. | `package.path.unsafe` |
|
||||
| `zip-symlink/` | A zipped package with an entry that is a symbolic link. | `package.zip.entry` |
|
||||
| `zip-traversal/` | A zipped package with an entry whose path leaves the package folder. | `package.zip.entry` |
|
||||
|
||||
## Keys
|
||||
|
||||
|
||||
Reference in new issue
Block a user