101 lines
6.6 KiB
Markdown
101 lines
6.6 KiB
Markdown
# 1. Shared conventions
|
|
|
|
These rules apply to every format.
|
|
|
|
## 1.1 Canonical JSON
|
|
|
|
Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
|
|
In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with `\b`, `\t`, `\n`, `\f`, `\r`, `\"` and `\\` for those characters and `\u00xx` with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
|
|
Strings are not normalized: canonical JSON keeps every code point as given.
|
|
A document containing a lone surrogate, `NaN` or an infinity has no canonical form and is invalid.
|
|
|
|
Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
|
|
The vectors in `canonical-json.jsonl` give JSON text to parse as IEEE 754 doubles and the canonical output.
|
|
|
|
A store may keep its own canonical form for its stored hashes; that form is outside this text.
|
|
|
|
## 1.2 Numbers
|
|
|
|
A decimal that carries meaning is a string, so that `310.0` and `1.5260` keep their printed digits.
|
|
The schemas call this a `decimal`: an optional minus sign, an integer part without leading zeros, and an optional fraction.
|
|
JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
|
|
|
|
Geometry is in PDF points.
|
|
A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
|
|
|
|
## 1.3 Hashes
|
|
|
|
A hash is SHA-256, written as 64 lowercase hexadecimal digits.
|
|
Six digits are for display and for pins in addresses; no record stores a shortened hash.
|
|
`document.json` names the function in `hashing.algorithm`, `sha-256` in this version, and every hash in the package, its records and its proofs uses the one function the document names; a consumer refuses an algorithm it does not implement (`doc.hash.algorithm`).
|
|
Property names such as `sha256` and `page_fp` belong to that function, so a later algorithm arrives as a new major version of the formats whose values it changes, with the name in the data letting two versions share a store.
|
|
Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as [5.8](05-hashes.md#58-keyed-hashes) defines.
|
|
|
|
## 1.4 Text normalization for hashing
|
|
|
|
To normalize a string for hashing:
|
|
|
|
1. Apply Unicode Normalization Form C.
|
|
2. Replace every maximal run of whitespace characters with one U+0020.
|
|
3. Remove a leading and a trailing U+0020.
|
|
|
|
Whitespace means the 25 characters with the Unicode property `White_Space`: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
|
|
U+200B ZERO WIDTH SPACE is not whitespace.
|
|
|
|
Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
|
|
The vectors in `normalize.jsonl` cover the cases, including the angstrom sign, which NFC turns into Å.
|
|
|
|
Normalization forms, General Categories and the `White_Space` property are those of Unicode 16.0.
|
|
<<<<<<< HEAD
|
|
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference implementation refuses to import on another, and moving the formats to a later version is a change to every format that hashes text ([0.3](00-introduction.md#03-format-identifiers-and-versions)).
|
|
||||||| parent of 4fa326d (First implementation (#1))
|
|
=======
|
|
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference tools refuse to run on another, and moving the formats to a later version is a change to every format that hashes text ([0.3](00-introduction.md#03-format-identifiers-and-versions)).
|
|
>>>>>>> 4fa326d (First implementation (#1))
|
|
|
|
## 1.5 Character offsets
|
|
|
|
Offsets count Unicode code points, never UTF-16 code units or bytes.
|
|
An offset pair `[start, end]` names the code points from `start` up to but not including `end`.
|
|
A browser implementation converts to UTF-16 at its edge; `offsets.jsonl` gives both forms for text with characters outside the Basic Multilingual Plane.
|
|
|
|
## 1.6 Times
|
|
|
|
A time is UTC, written `YYYY-MM-DDTHH:MM:SSZ`, with no fractions.
|
|
A date is `YYYY`, `YYYY-MM` or `YYYY-MM-DD`.
|
|
|
|
## 1.7 Determinism
|
|
|
|
Every value these specifications define is a pure function of its inputs.
|
|
Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently.
|
|
From the cached readings onward, the same inputs and tool versions MUST give byte-identical files.
|
|
No file carries the time it was written, unless the time is part of the record, as a receipt's is.
|
|
|
|
## 1.8 IDs and keys
|
|
|
|
| Kind | Form | Pattern |
|
|
| --- | --- | --- |
|
|
| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$` |
|
|
| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$` |
|
|
| Work key | A Scholar-style part, a hyphen and 4 to 51 characters of Crockford base32 ([2.4](02-model.md#24-work-and-document-keys)). At most 128 characters. | `^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}$` |
|
|
| Document key | A work key, with a dotted suffix for a version other than the version of record. At most 128 characters. | `^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\.[a-z0-9][a-z0-9-]*)*$` |
|
|
| Key | A store, project or source key, or a legacy document key kept as an alias. At most 128 characters. | `^[a-z0-9][a-z0-9._-]*$` |
|
|
| Alias | A legacy ID; it always holds a `.`, `_` or `-`, so it never looks like a node ID. | `^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$` |
|
|
| Dataset ID | `ds-` and a key. | `^ds-[a-z0-9][a-z0-9._-]*$` |
|
|
|
|
The Crockford alphabet is `0123456789abcdefghjkmnpqrstvwxyz`; it has no `i`, `l`, `o` or `u`.
|
|
Its 22 letters are `abcdefghjkmnpqrstvwxyz`.
|
|
Because the first character is a letter, every ID is a valid HTML `id`, CSS selector and URL fragment without escaping.
|
|
A key qualified by its store, `smithlab/cornell1995second`, names a document anywhere.
|
|
|
|
## 1.9 Files
|
|
|
|
JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A.
|
|
JSON files hold one value in canonical JSON followed by one U+000A.
|
|
So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence.
|
|
Each JSON Lines file has a defined order, given where the file is defined.
|
|
|
|
## 1.10 Prose
|
|
|
|
Prose in these formats, and in this repository, is US English, one sentence per line.
|