alexmandClaude
a09d158e16
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
openalex in same_as, else the folded title, year and family name. A
collision lengthens the later key by two characters; a key is minted
once and never recomputed. A document key adds .si, .v<seq> or .<type>
for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
every node and record ID was re-minted and the docs rewritten to match.
The validator checks the document key against the work key and version
(doc.key.work) and the hash part against the identifier (doc.key.suffix).
vectors/keys.jsonl pins the cases paperstack's tests pin.
Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
implemented in tools/oracle/vrf.py in plain Python and reproduced
against the RFC's Example 16; the sample ranks by the VRF output. The
earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
verify_inclusion, consistency_proof, verify_consistency) with
scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
checkpoints with key hashes as transparency logs share them, the rule
that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).
Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
the checker refuses another version.
Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
checkpoint records; a check that every ID and hash the prose quotes
exists in the fixtures or vectors; three more broken packages
(history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
section, media-type registration and OCI distribution notes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
72 lines
4.7 KiB
Markdown
72 lines
4.7 KiB
Markdown
# 12. The package
|
|
|
|
A document travels as a package: a folder named by its key, or the same folder zipped.
|
|
|
|
## 12.1 Files
|
|
|
|
| Path | Required | Holds |
|
|
| --- | --- | --- |
|
|
| `MANIFEST.json` | Yes | The manifest ([12.3](#123-the-manifest)). |
|
|
| `document.json` | Yes | The document ([2.2](02-model.md#22-documentjson)). |
|
|
| `nodes.jsonl` | Yes | Every node, in document order ([3](03-nodes.md)). |
|
|
| `witnesses.json` | Yes | Every witness, with its rights ([2.3](02-model.md#23-witnessesjson)). |
|
|
| `provenance.json` | Yes | Tools and versions, backends, cached readings, stages and where each ran. |
|
|
| `pages.jsonl` | No | Every page of every witness ([7.3](07-geometry.md#73-pagesjsonl)). |
|
|
| `locations.jsonl` | No | Geometry per node and page ([7.4](07-geometry.md#74-locationsjsonl)). |
|
|
| `references.jsonl` | No | Parsed references and their resolution ([10](10-references.md)). |
|
|
| `corrections.jsonl` | No | Every correction, in order ([11](11-corrections.md)). |
|
|
| `receipts.jsonl` | No | Receipts and revocations, sorted by `at` and then `id` ([8](08-fidelity.md)). |
|
|
| `ids.jsonl` | No | Retired IDs and legacy aliases ([4.4](04-identity.md#44-idsjsonl)). |
|
|
| `suspects.jsonl` | No | Unresolved suspects, sorted by `id` ([5.7](05-hashes.md#57-suspects)). |
|
|
| `roster.json` | No | The roster its receipts need ([8.7](08-fidelity.md#87-revocations-and-the-roster)). |
|
|
| `assets/` | No | Images and crops, as `<sha256>.<extension>`, present or listed only. |
|
|
| `datasets/<id>/` | No | Datasets ([15](15-datasets.md)). |
|
|
| `derived/` | No | Rebuildable renderings, never authoritative ([12.7](#127-derived-files)). |
|
|
|
|
An optional JSON Lines file with no records is omitted.
|
|
Every file is in the canonical form of [1.9](01-conventions.md#19-files).
|
|
|
|
## 12.2 Paths
|
|
|
|
Paths use `/`, are relative, have no empty, `.` or `..` segment, no backslash and no NUL, and are already normalized.
|
|
No two paths are the same once case-folded, so a package reads alike on every file system.
|
|
A package folder carries no symbolic links, and a consumer refuses one that does, as it refuses a package whose files exceed its own size limit.
|
|
In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit.
|
|
A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields; `fixtures/packages/tiny2026tiny-j4qm.sct.zip` is such an archive.
|
|
|
|
## 12.3 The manifest
|
|
|
|
`MANIFEST.json`, `scienting-package/1`, is `{format, doc, key, files}`: the package format, `scienting-doc/1`, the document key, and every other file as `{path, sha256, size}`, sorted by the UTF-8 bytes of the path.
|
|
An asset whose bytes stay in the asset store is listed with `present: false`; nothing else may be.
|
|
The SHA-256 of `MANIFEST.json` pins the whole package.
|
|
|
|
## 12.4 Provenance
|
|
|
|
`provenance.json` holds `tools` (`{name, version}`), and optionally `backends`, `readings` (each cached reading's ID, witness, backend and SHA-256), `stages` (each stage's name, tool, input hashes, output hash, the `processor` where it ran, and the uses it ran under) and a `session`.
|
|
Every stage that uses a model or an outside service records where it ran ([14](14-rights.md)).
|
|
|
|
## 12.5 Text policies
|
|
|
|
| Policy | Means |
|
|
| --- | --- |
|
|
| `full` | Every leaf keeps its content. |
|
|
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). |
|
|
|
|
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
|
|
Its text lives in the team's private asset store, and only key holders can check its values.
|
|
`fixtures/packages/hale2026harmonic-damb.checks-only` is such an export of `hale2026harmonic-damb` to another machine of the team, under the team key in `fixtures/keys/`.
|
|
|
|
## 12.6 Assets
|
|
|
|
Assets are named by SHA-256, so every read is rehashed and a regenerated crop is a different asset.
|
|
Bytes that live only in the asset store are listed in the manifest with `present: false`, with their size.
|
|
|
|
## 12.7 Derived files
|
|
|
|
`derived/` holds what a pinned tool makes from the package, such as `derived/mathml.jsonl`, whose lines are `{node, run, tool, mathml}` with `run` null for an equation node.
|
|
Nothing derived is content, and a consumer may delete and rebuild it.
|
|
|
|
## 12.8 Media type and extension
|
|
|
|
A zipped package has the extension `.sct.zip` and the media type `application/vnd.scienting.doc+zip`.
|