alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4.7 KiB
12. The package
A document travels as a package: a folder named by its key, or the same folder zipped.
12.1 Files
| Path | Required | Holds |
|---|---|---|
MANIFEST.json |
Yes | The manifest (12.3). |
document.json |
Yes | The document (2.2). |
nodes.jsonl |
Yes | Every node, in document order (3). |
witnesses.json |
Yes | Every witness, with its rights (2.3). |
provenance.json |
Yes | Tools and versions, backends, cached readings, stages and where each ran. |
pages.jsonl |
No | Every page of every witness (7.3). |
locations.jsonl |
No | Geometry per node and page (7.4). |
references.jsonl |
No | Parsed references and their resolution (10). |
corrections.jsonl |
No | Every correction, in order (11). |
receipts.jsonl |
No | Receipts and revocations, sorted by at and then id (8). |
ids.jsonl |
No | Retired IDs and legacy aliases (4.4). |
suspects.jsonl |
No | Unresolved suspects, sorted by id (5.7). |
roster.json |
No | The roster its receipts need (8.7). |
assets/ |
No | Images and crops, as <sha256>.<extension>, present or listed only. |
datasets/<id>/ |
No | Datasets (15). |
derived/ |
No | Rebuildable renderings, never authoritative (12.7). |
An optional JSON Lines file with no records is omitted. Every file is in the canonical form of 1.9.
12.2 Paths
Paths use /, are relative, have no empty, . or .. segment, no backslash and no NUL, and are already normalized.
No two paths are the same once case-folded, so a package reads alike on every file system.
A package folder carries no symbolic links, and a consumer refuses one that does, as it refuses a package whose files exceed its own size limit.
In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit.
A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields; fixtures/packages/tiny2026example.sct.zip is such an archive.
12.3 The manifest
MANIFEST.json, scienting-package/1, is {format, doc, key, files}: the package format, scienting-doc/1, the document key, and every other file as {path, sha256, size}, sorted by the UTF-8 bytes of the path.
An asset whose bytes stay in the asset store is listed with present: false; nothing else may be.
The SHA-256 of MANIFEST.json pins the whole package.
12.4 Provenance
provenance.json holds tools ({name, version}), and optionally backends, readings (each cached reading's ID, witness, backend and SHA-256), stages (each stage's name, tool, input hashes, output hash, the processor where it ran, and the uses it ran under) and a session.
Every stage that uses a model or an outside service records where it ran (14).
12.5 Text policies
| Policy | Means |
|---|---|
full |
Every leaf keeps its content. |
checks-only |
Every leaf has withheld: true and no content; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: hmac-team between the team's machines and hkdf-pages outside the team (5.8). |
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
Its text lives in the team's private asset store, and only key holders can check its values.
fixtures/packages/hale2026harmonic.checks-only is such an export of hale2026harmonic to another machine of the team, under the team key in fixtures/keys/.
12.6 Assets
Assets are named by SHA-256, so every read is rehashed and a regenerated crop is a different asset.
Bytes that live only in the asset store are listed in the manifest with present: false, with their size.
12.7 Derived files
derived/ holds what a pinned tool makes from the package, such as derived/mathml.jsonl, whose lines are {node, run, tool, mathml} with run null for an equation node.
Nothing derived is content, and a consumer may delete and rebuild it.
12.8 Media type and extension
A zipped package has the extension .sct.zip and the media type application/vnd.scienting.doc+zip.