Initial commit

spec/01-conventions.md

@@ -0,0 +1,88 @@
+# 1. Shared conventions
+
+These rules apply to every format.
+
+## 1.1 Canonical JSON
+
+Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
+In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with `\b`, `\t`, `\n`, `\f`, `\r`, `\"` and `\\` for those characters and `\u00xx` with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
+Strings are not normalized: canonical JSON keeps every code point as given.
+A document containing a lone surrogate, `NaN` or an infinity has no canonical form and is invalid.
+
+Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
+The vectors in `canonical-json.jsonl` give JSON text to parse as IEEE 754 doubles and the canonical output.
+
+Lode keeps its own canonical form for its stored hashes; that form is Lode's and is outside this text.
+
+## 1.2 Numbers
+
+A decimal that carries meaning is a string, so that `310.0` and `1.5260` keep their printed digits.
+The schemas call this a `decimal`: an optional minus sign, an integer part without leading zeros, and an optional fraction.
+JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
+
+Geometry is in PDF points.
+A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
+
+## 1.3 Hashes
+
+A hash is SHA-256, written as 64 lowercase hexadecimal digits.
+Six digits are for display and for pins in addresses; no record stores a shortened hash.
+Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as [5.8](05-hashes.md#58-keyed-hashes) defines.
+
+## 1.4 Text normalization for hashing
+
+To normalize a string for hashing:
+
+1. Apply Unicode Normalization Form C.
+2. Replace every maximal run of whitespace characters with one U+0020.
+3. Remove a leading and a trailing U+0020.
+
+Whitespace means the 25 characters with the Unicode property `White_Space`: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
+U+200B ZERO WIDTH SPACE is not whitespace.
+
+Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
+The vectors in `normalize.jsonl` cover the cases, including the angstrom sign, which NFC turns into Å.
+
+## 1.5 Character offsets
+
+Offsets count Unicode code points, never UTF-16 code units or bytes.
+An offset pair `[start, end]` names the code points from `start` up to but not including `end`.
+A browser implementation converts to UTF-16 at its edge; `offsets.jsonl` gives both forms for text with characters outside the Basic Multilingual Plane.
+
+## 1.6 Times
+
+A time is UTC, written `YYYY-MM-DDTHH:MM:SSZ`, with no fractions.
+A date is `YYYY`, `YYYY-MM` or `YYYY-MM-DD`.
+
+## 1.7 Determinism
+
+Every value these specifications define is a pure function of its inputs.
+Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently.
+From the cached readings onward, the same inputs and tool versions MUST give byte-identical files.
+No file carries the time it was written, unless the time is part of the record, as a receipt's is.
+
+## 1.8 IDs and keys
+
+| Kind | Form | Pattern |
+| --- | --- | --- |
+| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$` |
+| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$` |
+| Key | A document, work, store, project or source key, as Lode's source keys are. At most 128 characters. | `^[a-z0-9][a-z0-9._-]*$` |
+| Alias | A legacy ID; it always holds a `.`, `_` or `-`, so it never looks like a node ID. | `^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$` |
+| Dataset ID | `ds-` and a key. | `^ds-[a-z0-9][a-z0-9._-]*$` |
+
+The Crockford alphabet is `0123456789abcdefghjkmnpqrstvwxyz`; it has no `i`, `l`, `o` or `u`.
+Its 22 letters are `abcdefghjkmnpqrstvwxyz`.
+Because the first character is a letter, every ID is a valid HTML `id`, CSS selector and URL fragment without escaping.
+A key qualified by its store, `smithlab/cornell1995second`, names a document anywhere.
+
+## 1.9 Files
+
+JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A.
+JSON files hold one value in canonical JSON followed by one U+000A.
+So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence.
+Each JSON Lines file has a defined order, given where the file is defined.
+
+## 1.10 Prose
+
+Prose in these formats, and in this repository, is US English, one sentence per line.