alexmandClaude
a09d158e16
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
openalex in same_as, else the folded title, year and family name. A
collision lengthens the later key by two characters; a key is minted
once and never recomputed. A document key adds .si, .v<seq> or .<type>
for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
every node and record ID was re-minted and the docs rewritten to match.
The validator checks the document key against the work key and version
(doc.key.work) and the hash part against the identifier (doc.key.suffix).
vectors/keys.jsonl pins the cases paperstack's tests pin.
Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
implemented in tools/oracle/vrf.py in plain Python and reproduced
against the RFC's Example 16; the sample ranks by the VRF output. The
earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
verify_inclusion, consistency_proof, verify_consistency) with
scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
checkpoints with key hashes as transparency logs share them, the rule
that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).
Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
the checker refuses another version.
Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
checkpoint records; a check that every ID and hash the prose quotes
exists in the fixtures or vectors; three more broken packages
(history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
section, media-type registration and OCI distribution notes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5.6 KiB
1. Shared conventions
These rules apply to every format.
1.1 Canonical JSON
Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with \b, \t, \n, \f, \r, \" and \\ for those characters and \u00xx with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
Strings are not normalized: canonical JSON keeps every code point as given.
A document containing a lone surrogate, NaN or an infinity has no canonical form and is invalid.
Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
The vectors in canonical-json.jsonl give JSON text to parse as IEEE 754 doubles and the canonical output.
A store may keep its own canonical form for its stored hashes; that form is outside this text.
1.2 Numbers
A decimal that carries meaning is a string, so that 310.0 and 1.5260 keep their printed digits.
The schemas call this a decimal: an optional minus sign, an integer part without leading zeros, and an optional fraction.
JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
Geometry is in PDF points. A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
1.3 Hashes
A hash is SHA-256, written as 64 lowercase hexadecimal digits. Six digits are for display and for pins in addresses; no record stores a shortened hash. Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as 5.8 defines.
1.4 Text normalization for hashing
To normalize a string for hashing:
- Apply Unicode Normalization Form C.
- Replace every maximal run of whitespace characters with one U+0020.
- Remove a leading and a trailing U+0020.
Whitespace means the 25 characters with the Unicode property White_Space: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
U+200B ZERO WIDTH SPACE is not whitespace.
Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
The vectors in normalize.jsonl cover the cases, including the angstrom sign, which NFC turns into Å.
Normalization forms, General Categories and the White_Space property are those of Unicode 16.0.
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference tools refuse to run on another, and moving the formats to a later version is a change to every format that hashes text (0.3).
1.5 Character offsets
Offsets count Unicode code points, never UTF-16 code units or bytes.
An offset pair [start, end] names the code points from start up to but not including end.
A browser implementation converts to UTF-16 at its edge; offsets.jsonl gives both forms for text with characters outside the Basic Multilingual Plane.
1.6 Times
A time is UTC, written YYYY-MM-DDTHH:MM:SSZ, with no fractions.
A date is YYYY, YYYY-MM or YYYY-MM-DD.
1.7 Determinism
Every value these specifications define is a pure function of its inputs. Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently. From the cached readings onward, the same inputs and tool versions MUST give byte-identical files. No file carries the time it was written, unless the time is part of the record, as a receipt's is.
1.8 IDs and keys
| Kind | Form | Pattern |
|---|---|---|
| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | ^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$ |
| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | ^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$ |
| Work key | A Scholar-style part, a hyphen and 4 to 51 characters of Crockford base32 (2.4). At most 128 characters. | ^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}$ |
| Document key | A work key, with a dotted suffix for a version other than the version of record. At most 128 characters. | ^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\.[a-z0-9][a-z0-9-]*)*$ |
| Key | A store, project or source key, or a legacy document key kept as an alias. At most 128 characters. | ^[a-z0-9][a-z0-9._-]*$ |
| Alias | A legacy ID; it always holds a ., _ or -, so it never looks like a node ID. |
^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$ |
| Dataset ID | ds- and a key. |
^ds-[a-z0-9][a-z0-9._-]*$ |
The Crockford alphabet is 0123456789abcdefghjkmnpqrstvwxyz; it has no i, l, o or u.
Its 22 letters are abcdefghjkmnpqrstvwxyz.
Because the first character is a letter, every ID is a valid HTML id, CSS selector and URL fragment without escaping.
A key qualified by its store, smithlab/cornell1995second, names a document anywhere.
1.9 Files
JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A. JSON files hold one value in canonical JSON followed by one U+000A. So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence. Each JSON Lines file has a defined order, given where the file is defined.
1.10 Prose
Prose in these formats, and in this repository, is US English, one sentence per line.