alexmandClaude
a09d158e16
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
openalex in same_as, else the folded title, year and family name. A
collision lengthens the later key by two characters; a key is minted
once and never recomputed. A document key adds .si, .v<seq> or .<type>
for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
every node and record ID was re-minted and the docs rewritten to match.
The validator checks the document key against the work key and version
(doc.key.work) and the hash part against the identifier (doc.key.suffix).
vectors/keys.jsonl pins the cases paperstack's tests pin.
Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
implemented in tools/oracle/vrf.py in plain Python and reproduced
against the RFC's Example 16; the sample ranks by the VRF output. The
earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
verify_inclusion, consistency_proof, verify_consistency) with
scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
checkpoints with key hashes as transparency logs share them, the rule
that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).
Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
the checker refuses another version.
Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
checkpoint records; a check that every ID and hash the prose quotes
exists in the fixtures or vectors; three more broken packages
(history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
section, media-type registration and OCI distribution notes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
94 lines
5.6 KiB
Markdown
94 lines
5.6 KiB
Markdown
# 1. Shared conventions
|
|
|
|
These rules apply to every format.
|
|
|
|
## 1.1 Canonical JSON
|
|
|
|
Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
|
|
In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with `\b`, `\t`, `\n`, `\f`, `\r`, `\"` and `\\` for those characters and `\u00xx` with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
|
|
Strings are not normalized: canonical JSON keeps every code point as given.
|
|
A document containing a lone surrogate, `NaN` or an infinity has no canonical form and is invalid.
|
|
|
|
Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
|
|
The vectors in `canonical-json.jsonl` give JSON text to parse as IEEE 754 doubles and the canonical output.
|
|
|
|
A store may keep its own canonical form for its stored hashes; that form is outside this text.
|
|
|
|
## 1.2 Numbers
|
|
|
|
A decimal that carries meaning is a string, so that `310.0` and `1.5260` keep their printed digits.
|
|
The schemas call this a `decimal`: an optional minus sign, an integer part without leading zeros, and an optional fraction.
|
|
JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
|
|
|
|
Geometry is in PDF points.
|
|
A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
|
|
|
|
## 1.3 Hashes
|
|
|
|
A hash is SHA-256, written as 64 lowercase hexadecimal digits.
|
|
Six digits are for display and for pins in addresses; no record stores a shortened hash.
|
|
Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as [5.8](05-hashes.md#58-keyed-hashes) defines.
|
|
|
|
## 1.4 Text normalization for hashing
|
|
|
|
To normalize a string for hashing:
|
|
|
|
1. Apply Unicode Normalization Form C.
|
|
2. Replace every maximal run of whitespace characters with one U+0020.
|
|
3. Remove a leading and a trailing U+0020.
|
|
|
|
Whitespace means the 25 characters with the Unicode property `White_Space`: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
|
|
U+200B ZERO WIDTH SPACE is not whitespace.
|
|
|
|
Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
|
|
The vectors in `normalize.jsonl` cover the cases, including the angstrom sign, which NFC turns into Å.
|
|
|
|
Normalization forms, General Categories and the `White_Space` property are those of Unicode 16.0.
|
|
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference tools refuse to run on another, and moving the formats to a later version is a change to every format that hashes text ([0.3](00-introduction.md#03-format-identifiers-and-versions)).
|
|
|
|
## 1.5 Character offsets
|
|
|
|
Offsets count Unicode code points, never UTF-16 code units or bytes.
|
|
An offset pair `[start, end]` names the code points from `start` up to but not including `end`.
|
|
A browser implementation converts to UTF-16 at its edge; `offsets.jsonl` gives both forms for text with characters outside the Basic Multilingual Plane.
|
|
|
|
## 1.6 Times
|
|
|
|
A time is UTC, written `YYYY-MM-DDTHH:MM:SSZ`, with no fractions.
|
|
A date is `YYYY`, `YYYY-MM` or `YYYY-MM-DD`.
|
|
|
|
## 1.7 Determinism
|
|
|
|
Every value these specifications define is a pure function of its inputs.
|
|
Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently.
|
|
From the cached readings onward, the same inputs and tool versions MUST give byte-identical files.
|
|
No file carries the time it was written, unless the time is part of the record, as a receipt's is.
|
|
|
|
## 1.8 IDs and keys
|
|
|
|
| Kind | Form | Pattern |
|
|
| --- | --- | --- |
|
|
| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$` |
|
|
| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$` |
|
|
| Work key | A Scholar-style part, a hyphen and 4 to 51 characters of Crockford base32 ([2.4](02-model.md#24-work-and-document-keys)). At most 128 characters. | `^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}$` |
|
|
| Document key | A work key, with a dotted suffix for a version other than the version of record. At most 128 characters. | `^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\.[a-z0-9][a-z0-9-]*)*$` |
|
|
| Key | A store, project or source key, or a legacy document key kept as an alias. At most 128 characters. | `^[a-z0-9][a-z0-9._-]*$` |
|
|
| Alias | A legacy ID; it always holds a `.`, `_` or `-`, so it never looks like a node ID. | `^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$` |
|
|
| Dataset ID | `ds-` and a key. | `^ds-[a-z0-9][a-z0-9._-]*$` |
|
|
|
|
The Crockford alphabet is `0123456789abcdefghjkmnpqrstvwxyz`; it has no `i`, `l`, `o` or `u`.
|
|
Its 22 letters are `abcdefghjkmnpqrstvwxyz`.
|
|
Because the first character is a letter, every ID is a valid HTML `id`, CSS selector and URL fragment without escaping.
|
|
A key qualified by its store, `smithlab/cornell1995second`, names a document anywhere.
|
|
|
|
## 1.9 Files
|
|
|
|
JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A.
|
|
JSON files hold one value in canonical JSON followed by one U+000A.
|
|
So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence.
|
|
Each JSON Lines file has a defined order, given where the file is defined.
|
|
|
|
## 1.10 Prose
|
|
|
|
Prose in these formats, and in this repository, is US English, one sentence per line.
|