4.8 KiB
1. Shared conventions
These rules apply to every format.
1.1 Canonical JSON
Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with \b, \t, \n, \f, \r, \" and \\ for those characters and \u00xx with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
Strings are not normalized: canonical JSON keeps every code point as given.
A document containing a lone surrogate, NaN or an infinity has no canonical form and is invalid.
Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
The vectors in canonical-json.jsonl give JSON text to parse as IEEE 754 doubles and the canonical output.
Lode keeps its own canonical form for its stored hashes; that form is Lode's and is outside this text.
1.2 Numbers
A decimal that carries meaning is a string, so that 310.0 and 1.5260 keep their printed digits.
The schemas call this a decimal: an optional minus sign, an integer part without leading zeros, and an optional fraction.
JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
Geometry is in PDF points. A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
1.3 Hashes
A hash is SHA-256, written as 64 lowercase hexadecimal digits. Six digits are for display and for pins in addresses; no record stores a shortened hash. Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as 5.8 defines.
1.4 Text normalization for hashing
To normalize a string for hashing:
- Apply Unicode Normalization Form C.
- Replace every maximal run of whitespace characters with one U+0020.
- Remove a leading and a trailing U+0020.
Whitespace means the 25 characters with the Unicode property White_Space: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
U+200B ZERO WIDTH SPACE is not whitespace.
Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
The vectors in normalize.jsonl cover the cases, including the angstrom sign, which NFC turns into Å.
1.5 Character offsets
Offsets count Unicode code points, never UTF-16 code units or bytes.
An offset pair [start, end] names the code points from start up to but not including end.
A browser implementation converts to UTF-16 at its edge; offsets.jsonl gives both forms for text with characters outside the Basic Multilingual Plane.
1.6 Times
A time is UTC, written YYYY-MM-DDTHH:MM:SSZ, with no fractions.
A date is YYYY, YYYY-MM or YYYY-MM-DD.
1.7 Determinism
Every value these specifications define is a pure function of its inputs. Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently. From the cached readings onward, the same inputs and tool versions MUST give byte-identical files. No file carries the time it was written, unless the time is part of the record, as a receipt's is.
1.8 IDs and keys
| Kind | Form | Pattern |
|---|---|---|
| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | ^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$ |
| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | ^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$ |
| Key | A document, work, store, project or source key, as Lode's source keys are. At most 128 characters. | ^[a-z0-9][a-z0-9._-]*$ |
| Alias | A legacy ID; it always holds a ., _ or -, so it never looks like a node ID. |
^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$ |
| Dataset ID | ds- and a key. |
^ds-[a-z0-9][a-z0-9._-]*$ |
The Crockford alphabet is 0123456789abcdefghjkmnpqrstvwxyz; it has no i, l, o or u.
Its 22 letters are abcdefghjkmnpqrstvwxyz.
Because the first character is a letter, every ID is a valid HTML id, CSS selector and URL fragment without escaping.
A key qualified by its store, smithlab/cornell1995second, names a document anywhere.
1.9 Files
JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A. JSON files hold one value in canonical JSON followed by one U+000A. So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence. Each JSON Lines file has a defined order, given where the file is defined.
1.10 Prose
Prose in these formats, and in this repository, is US English, one sentence per line.