alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
72 lines
4.7 KiB
Markdown
72 lines
4.7 KiB
Markdown
# 12. The package
|
|
|
|
A document travels as a package: a folder named by its key, or the same folder zipped.
|
|
|
|
## 12.1 Files
|
|
|
|
| Path | Required | Holds |
|
|
| --- | --- | --- |
|
|
| `MANIFEST.json` | Yes | The manifest ([12.3](#123-the-manifest)). |
|
|
| `document.json` | Yes | The document ([2.2](02-model.md#22-documentjson)). |
|
|
| `nodes.jsonl` | Yes | Every node, in document order ([3](03-nodes.md)). |
|
|
| `witnesses.json` | Yes | Every witness, with its rights ([2.3](02-model.md#23-witnessesjson)). |
|
|
| `provenance.json` | Yes | Tools and versions, backends, cached readings, stages and where each ran. |
|
|
| `pages.jsonl` | No | Every page of every witness ([7.3](07-geometry.md#73-pagesjsonl)). |
|
|
| `locations.jsonl` | No | Geometry per node and page ([7.4](07-geometry.md#74-locationsjsonl)). |
|
|
| `references.jsonl` | No | Parsed references and their resolution ([10](10-references.md)). |
|
|
| `corrections.jsonl` | No | Every correction, in order ([11](11-corrections.md)). |
|
|
| `receipts.jsonl` | No | Receipts and revocations, sorted by `at` and then `id` ([8](08-fidelity.md)). |
|
|
| `ids.jsonl` | No | Retired IDs and legacy aliases ([4.4](04-identity.md#44-idsjsonl)). |
|
|
| `suspects.jsonl` | No | Unresolved suspects, sorted by `id` ([5.7](05-hashes.md#57-suspects)). |
|
|
| `roster.json` | No | The roster its receipts need ([8.7](08-fidelity.md#87-revocations-and-the-roster)). |
|
|
| `assets/` | No | Images and crops, as `<sha256>.<extension>`, present or listed only. |
|
|
| `datasets/<id>/` | No | Datasets ([15](15-datasets.md)). |
|
|
| `derived/` | No | Rebuildable renderings, never authoritative ([12.7](#127-derived-files)). |
|
|
|
|
An optional JSON Lines file with no records is omitted.
|
|
Every file is in the canonical form of [1.9](01-conventions.md#19-files).
|
|
|
|
## 12.2 Paths
|
|
|
|
Paths use `/`, are relative, have no empty, `.` or `..` segment, no backslash and no NUL, and are already normalized.
|
|
No two paths are the same once case-folded, so a package reads alike on every file system.
|
|
A package folder carries no symbolic links, and a consumer refuses one that does, as it refuses a package whose files exceed its own size limit.
|
|
In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit.
|
|
A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields; `fixtures/packages/tiny2026example.sct.zip` is such an archive.
|
|
|
|
## 12.3 The manifest
|
|
|
|
`MANIFEST.json`, `scienting-package/1`, is `{format, doc, key, files}`: the package format, `scienting-doc/1`, the document key, and every other file as `{path, sha256, size}`, sorted by the UTF-8 bytes of the path.
|
|
An asset whose bytes stay in the asset store is listed with `present: false`; nothing else may be.
|
|
The SHA-256 of `MANIFEST.json` pins the whole package.
|
|
|
|
## 12.4 Provenance
|
|
|
|
`provenance.json` holds `tools` (`{name, version}`), and optionally `backends`, `readings` (each cached reading's ID, witness, backend and SHA-256), `stages` (each stage's name, tool, input hashes, output hash, the `processor` where it ran, and the uses it ran under) and a `session`.
|
|
Every stage that uses a model or an outside service records where it ran ([14](14-rights.md)).
|
|
|
|
## 12.5 Text policies
|
|
|
|
| Policy | Means |
|
|
| --- | --- |
|
|
| `full` | Every leaf keeps its content. |
|
|
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). |
|
|
|
|
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
|
|
Its text lives in the team's private asset store, and only key holders can check its values.
|
|
`fixtures/packages/hale2026harmonic.checks-only` is such an export of `hale2026harmonic` to another machine of the team, under the team key in `fixtures/keys/`.
|
|
|
|
## 12.6 Assets
|
|
|
|
Assets are named by SHA-256, so every read is rehashed and a regenerated crop is a different asset.
|
|
Bytes that live only in the asset store are listed in the manifest with `present: false`, with their size.
|
|
|
|
## 12.7 Derived files
|
|
|
|
`derived/` holds what a pinned tool makes from the package, such as `derived/mathml.jsonl`, whose lines are `{node, run, tool, mathml}` with `run` null for an equation node.
|
|
Nothing derived is content, and a consumer may delete and rebuild it.
|
|
|
|
## 12.8 Media type and extension
|
|
|
|
A zipped package has the extension `.sct.zip` and the media type `application/vnd.scienting.doc+zip`.
|