Initial commit
This commit is contained in:
373 files changed
+16704
No files matched your search
@@ -0,0 +1,68 @@
|
||||
# 0. Introduction
|
||||
|
||||
## 0.1 Scope
|
||||
|
||||
These specifications define the data the Scientific Computing Studio's parts write and read: documents and their packages, anchors, receipts, rights records, layers, exercises, datasets, digitizations, views, projects and bundles.
|
||||
They define data, not programs.
|
||||
A conforming file can be read, checked and loaded without any studio code, and every rule here is stated so that a second implementation, in any language, computes the same bytes.
|
||||
|
||||
The design document, *The Scientific Computing Studio: design of Lode, paperpack and Varve*, explains why each format exists.
|
||||
This text says exactly what each format is.
|
||||
Where the two differ, this text governs, and [DECISIONS.md](../DECISIONS.md) records why it differs.
|
||||
|
||||
## 0.2 Conformance language
|
||||
|
||||
The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY and OPTIONAL are to be interpreted as described in RFC 2119 and RFC 8174 when, and only when, they appear in capitals.
|
||||
|
||||
Three classes of implementation are distinguished.
|
||||
|
||||
| Class | Does | Must |
|
||||
| --- | --- | --- |
|
||||
| Producer | Writes files in a format. | Write only files that validate, with values computed as specified. |
|
||||
| Consumer | Reads files in a format. | Accept every valid file, and refuse every file the suite lists as invalid, naming the rule. |
|
||||
| Validator | Decides whether a file is valid. | Check every rule in [22](22-conformance.md) that the file's contents allow it to check. |
|
||||
|
||||
An implementation states, for each format, which versions it reads and writes.
|
||||
|
||||
## 0.3 Format identifiers and versions
|
||||
|
||||
Every format has an identifier of the form `scienting-<name>/<major>`, such as `scienting-doc/1`.
|
||||
A file or record that can travel on its own carries its identifier in a `format` property.
|
||||
Records inside a package file whose format the manifest fixes, such as the lines of `nodes.jsonl`, do not repeat it.
|
||||
|
||||
A change that makes a valid file invalid, or that changes a computed value, is a breaking change and needs a new major version, with a migration.
|
||||
Adding an optional property, a node kind, a run kind or an enumeration value is a minor change.
|
||||
A consumer of version *n* MUST refuse a major version it does not know, and SHOULD warn on, but accept, enumeration values added in a later minor version.
|
||||
|
||||
The specification as a whole is versioned with semantic versioning; this is version 0.1.0.
|
||||
[registry/formats.json](../registry/formats.json) lists every format with its media type, schema and status.
|
||||
|
||||
## 0.4 Status
|
||||
|
||||
Each format has one of three statuses.
|
||||
|
||||
| Status | Meaning |
|
||||
| --- | --- |
|
||||
| `candidate` | Complete, with vectors and fixtures, and ready to implement. Changes before 1.0 need a recorded decision. |
|
||||
| `provisional` | Complete, but its milestone has not come, and it may change with less ceremony. |
|
||||
| `reserved` | Named and shaped, but not yet specified in full. |
|
||||
|
||||
The design builds in levels, and level 1 implements a subset.
|
||||
Fields that level 1 does not use, such as rights beyond the license, receipts and spot-checks, are defined now so that formats never change when a team moves up a level.
|
||||
A consumer that does not use a field MUST still preserve it when it rewrites a file.
|
||||
|
||||
## 0.5 How the repository is arranged
|
||||
|
||||
| Path | Holds | Normative |
|
||||
| --- | --- | --- |
|
||||
| `spec/` | These chapters. | Yes |
|
||||
| `schemas/` | JSON Schemas, draft 2020-12. | Yes |
|
||||
| `sql/` | Table definitions for SQLite, Dolt and PostgreSQL, and the SQLite bundle. | Yes |
|
||||
| `context/` | The JSON-LD context for layers. | Yes |
|
||||
| `registry/` | Formats, rule IDs, stamp rules and the default stage requirements. | Yes |
|
||||
| `vectors/` | Test vectors, one case per line. | Yes |
|
||||
| `fixtures/` | Valid packages and records, invalid packages, and the keys that signed them. | Yes |
|
||||
| `tools/` | The oracle that computed the vectors, the builder and the checker. | No |
|
||||
|
||||
The oracle exists so that no vector is typed by hand.
|
||||
It is not a library: implementations conform to the chapters and the vectors, not to the oracle, and `scienting-core` is the reference implementation.
|
||||
@@ -0,0 +1,88 @@
|
||||
# 1. Shared conventions
|
||||
|
||||
These rules apply to every format.
|
||||
|
||||
## 1.1 Canonical JSON
|
||||
|
||||
Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
|
||||
In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with `\b`, `\t`, `\n`, `\f`, `\r`, `\"` and `\\` for those characters and `\u00xx` with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
|
||||
Strings are not normalized: canonical JSON keeps every code point as given.
|
||||
A document containing a lone surrogate, `NaN` or an infinity has no canonical form and is invalid.
|
||||
|
||||
Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
|
||||
The vectors in `canonical-json.jsonl` give JSON text to parse as IEEE 754 doubles and the canonical output.
|
||||
|
||||
Lode keeps its own canonical form for its stored hashes; that form is Lode's and is outside this text.
|
||||
|
||||
## 1.2 Numbers
|
||||
|
||||
A decimal that carries meaning is a string, so that `310.0` and `1.5260` keep their printed digits.
|
||||
The schemas call this a `decimal`: an optional minus sign, an integer part without leading zeros, and an optional fraction.
|
||||
JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
|
||||
|
||||
Geometry is in PDF points.
|
||||
A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
|
||||
|
||||
## 1.3 Hashes
|
||||
|
||||
A hash is SHA-256, written as 64 lowercase hexadecimal digits.
|
||||
Six digits are for display and for pins in addresses; no record stores a shortened hash.
|
||||
Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as [5.8](05-hashes.md#58-keyed-hashes) defines.
|
||||
|
||||
## 1.4 Text normalization for hashing
|
||||
|
||||
To normalize a string for hashing:
|
||||
|
||||
1. Apply Unicode Normalization Form C.
|
||||
2. Replace every maximal run of whitespace characters with one U+0020.
|
||||
3. Remove a leading and a trailing U+0020.
|
||||
|
||||
Whitespace means the 25 characters with the Unicode property `White_Space`: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
|
||||
U+200B ZERO WIDTH SPACE is not whitespace.
|
||||
|
||||
Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
|
||||
The vectors in `normalize.jsonl` cover the cases, including the angstrom sign, which NFC turns into Å.
|
||||
|
||||
## 1.5 Character offsets
|
||||
|
||||
Offsets count Unicode code points, never UTF-16 code units or bytes.
|
||||
An offset pair `[start, end]` names the code points from `start` up to but not including `end`.
|
||||
A browser implementation converts to UTF-16 at its edge; `offsets.jsonl` gives both forms for text with characters outside the Basic Multilingual Plane.
|
||||
|
||||
## 1.6 Times
|
||||
|
||||
A time is UTC, written `YYYY-MM-DDTHH:MM:SSZ`, with no fractions.
|
||||
A date is `YYYY`, `YYYY-MM` or `YYYY-MM-DD`.
|
||||
|
||||
## 1.7 Determinism
|
||||
|
||||
Every value these specifications define is a pure function of its inputs.
|
||||
Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently.
|
||||
From the cached readings onward, the same inputs and tool versions MUST give byte-identical files.
|
||||
No file carries the time it was written, unless the time is part of the record, as a receipt's is.
|
||||
|
||||
## 1.8 IDs and keys
|
||||
|
||||
| Kind | Form | Pattern |
|
||||
| --- | --- | --- |
|
||||
| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$` |
|
||||
| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$` |
|
||||
| Key | A document, work, store, project or source key, as Lode's source keys are. At most 128 characters. | `^[a-z0-9][a-z0-9._-]*$` |
|
||||
| Alias | A legacy ID; it always holds a `.`, `_` or `-`, so it never looks like a node ID. | `^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$` |
|
||||
| Dataset ID | `ds-` and a key. | `^ds-[a-z0-9][a-z0-9._-]*$` |
|
||||
|
||||
The Crockford alphabet is `0123456789abcdefghjkmnpqrstvwxyz`; it has no `i`, `l`, `o` or `u`.
|
||||
Its 22 letters are `abcdefghjkmnpqrstvwxyz`.
|
||||
Because the first character is a letter, every ID is a valid HTML `id`, CSS selector and URL fragment without escaping.
|
||||
A key qualified by its store, `smithlab/cornell1995second`, names a document anywhere.
|
||||
|
||||
## 1.9 Files
|
||||
|
||||
JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A.
|
||||
JSON files hold one value in canonical JSON followed by one U+000A.
|
||||
So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence.
|
||||
Each JSON Lines file has a defined order, given where the file is defined.
|
||||
|
||||
## 1.10 Prose
|
||||
|
||||
Prose in these formats, and in this repository, is US English, one sentence per line.
|
||||
@@ -0,0 +1,64 @@
|
||||
# 2. Works, versions, witnesses and documents
|
||||
|
||||
## 2.1 Four objects
|
||||
|
||||
| Object | Is | Identified by |
|
||||
| --- | --- | --- |
|
||||
| Work | The intellectual work, across all its versions. | A key, and `same_as` identifiers: `doi:`, `arxiv:`, `isbn:`, `pmid:`, `pmcid:`, `url:` or `lode:`. |
|
||||
| Version | One expression of the work: a preprint, an accepted manuscript, the version of record, an erratum, Supporting Information, a draft. | The work's key and a version type, with an optional sequence number and date. |
|
||||
| Witness | One file of a version: a PDF, JATS XML, a LaTeX source archive, an HTML page, an EPUB, a Typst project. | The SHA-256 of its bytes. |
|
||||
| Document | The text established for one version from one or more witnesses. | A key, and a root hash per revision. |
|
||||
|
||||
A document's key is its work's key, with a suffix for anything other than the version of record: `cornell1995second`, `cornell1995second.si`, `dong2026machine.v2`.
|
||||
Supporting Information is its own document, with its own pages, numbering and references, related to its article by `supplement-of`.
|
||||
A textbook is a work whose chapters are documents related by `part-of`.
|
||||
A draft is a document of kind `draft`.
|
||||
|
||||
Keys are immutable once a document is shared.
|
||||
A renamed document keeps its old keys in `aliases`.
|
||||
|
||||
## 2.2 `document.json`
|
||||
|
||||
| Property | Required | Holds |
|
||||
| --- | --- | --- |
|
||||
| `format` | Yes | `scienting-doc/1`. |
|
||||
| `key` | Yes | The document key. |
|
||||
| `store` | No | The store that holds it, when the package says so. |
|
||||
| `aliases` | No | Earlier keys. |
|
||||
| `kind` | Yes | `article`, `supplement`, `chapter`, `book`, `thesis`, `report`, `draft`, `course-material` or `other`. |
|
||||
| `work` | Yes | `{key, same_as}`. |
|
||||
| `version` | Yes | `{type, seq?, date?}`, where `type` is `preprint`, `submitted`, `accepted`, `version-of-record`, `erratum`, `retraction`, `supplement`, `draft` or `other`. |
|
||||
| `relations` | Yes | A list of `{type, target}`; `type` is `supplement-of`, `part-of`, `version-of`, `erratum-of`, `retraction-of` or `translation-of`, and `target` a key, optionally store-qualified. |
|
||||
| `meta` | Yes | The version's bibliographic record, as one CSL-JSON item. |
|
||||
| `language` | Yes | A BCP 47 tag. |
|
||||
| `license` | No | The document's license as stated: `{spdx?, text?, url?}`. Rights are per witness ([14](14-rights.md)). |
|
||||
| `policy` | Yes | The text policy: `full`, or `checks-only` ([12.5](12-package.md#125-text-policies)). |
|
||||
| `hashing` | Yes | How content values are computed: `{scheme: "plain"}`, a team HMAC or a page-derived key ([5.8](05-hashes.md#58-keyed-hashes)). |
|
||||
| `root` | Yes | The root node's `node_hash`. |
|
||||
| `meta_hash` | Yes | The metadata hash ([5.9](05-hashes.md#59-the-metadata-hash)). |
|
||||
| `rev` | Yes | The revision: the index of the last correction applied, or 0. |
|
||||
| `counts` | Yes | `{nodes, leaves}`. |
|
||||
| `fidelity` | Yes | `{lowest, counts}`: the lowest level among leaves, or null when there are none, and the number of leaves at each level, computed as [8.2](08-fidelity.md#82-levels) defines. |
|
||||
|
||||
The schema is `document.schema.json`.
|
||||
|
||||
## 2.3 `witnesses.json`
|
||||
|
||||
`witnesses.json` is an object with one property, `witnesses`, a non-empty list in which each witness is:
|
||||
|
||||
| Property | Required | Holds |
|
||||
| --- | --- | --- |
|
||||
| `id` | Yes | The witness ID: seven letter-first base32 characters encoding the SHA-256, as [4.6](04-identity.md#46-record-ids) defines. |
|
||||
| `sha256` | Yes | The SHA-256 of the file. |
|
||||
| `size` | Yes | Its size in bytes. |
|
||||
| `kind` | Yes | `pdf`, `jats`, `latex`, `html`, `epub`, `typst`, `xml`, `image` or `other`. |
|
||||
| `media_type` | Yes | Its media type. |
|
||||
| `pages` | No | The number of pages, for a witness that has pages. |
|
||||
| `text_layer` | No | Whether a PDF has a text layer; a scan has none. |
|
||||
| `version_label` | No | The version as the witness labels it. |
|
||||
| `origin` | Yes | `{source, url?, doi?, retrieved_at?, note?}`, where `source` is `publisher`, `repository`, `preprint-server`, `author`, `team`, `licensed-delivery`, `library` or `other`. |
|
||||
| `rights` | Yes | The rights record, `scienting-rights/1` ([14](14-rights.md)). |
|
||||
|
||||
The witness's bytes are not part of the package; they live in the team's asset store, by SHA-256.
|
||||
The SHA-256 is the exact identity of one file.
|
||||
Because publishers stamp each download, two copies of one paper rarely share it; page fingerprints identify their pages instead ([7](07-geometry.md)).
|
||||
@@ -0,0 +1,127 @@
|
||||
# 3. Nodes
|
||||
|
||||
A document is a tree of nodes.
|
||||
`nodes.jsonl` holds one node per line, in document order: the root first, then each node before its children, and siblings in order.
|
||||
|
||||
## 3.1 The node record
|
||||
|
||||
| Property | Required | Holds |
|
||||
| --- | --- | --- |
|
||||
| `id` | Yes | The node ID ([4](04-identity.md)). |
|
||||
| `kind` | Yes | One of the kinds in [3.2](#32-kinds-and-children). |
|
||||
| `parent` | Yes | The parent's ID, or null for the root. |
|
||||
| `ord` | Yes | The position among its siblings, from 0. |
|
||||
| `label` | No | The printed label (`2.3`, `Figure 2`, `(1)`, `a`), for kinds that take one. |
|
||||
| `attrs` | No | Structural attributes, by kind ([3.4](#34-attributes)). |
|
||||
| `content` | No | The node's own content, by kind ([3.3](#33-content)). |
|
||||
| `withheld` | No | `true` when the policy withholds the content ([12.5](12-package.md#125-text-policies)). |
|
||||
| `fidelity` | No | For leaves, the level the pipeline established: `F0`, `F1` or `F2` ([8.1](08-fidelity.md#81-levels-of-fidelity)). |
|
||||
| `prov` | No | `{witnesses, backend?, reading?, corroborated_by?}`: the witness IDs the text came from, the backend, and the witnesses that agree. |
|
||||
| `text_hash`, `match_hash`, `align_key`, `own_hash`, `node_hash` | Yes | The five content values ([5](05-hashes.md)). |
|
||||
|
||||
Optional properties are omitted, never written as null.
|
||||
The schema is `node.schema.json`.
|
||||
|
||||
## 3.2 Kinds and children
|
||||
|
||||
| Kind | May contain | Content | Label |
|
||||
| --- | --- | --- | --- |
|
||||
| `document` | `front`, `section`, `back` | | |
|
||||
| `front` | `title`, `authors`, `abstract`, `keywords`, `notes` | | |
|
||||
| `title` | `sentence` | | |
|
||||
| `authors` | | runs | |
|
||||
| `abstract` | blocks, `section` | | |
|
||||
| `keywords` | | runs | |
|
||||
| `notes` | `footnote`, `paragraph` | | |
|
||||
| `section` | `heading` first, then blocks and `section` | | Yes |
|
||||
| `heading` | `sentence` | | |
|
||||
| `paragraph` | `sentence` | | |
|
||||
| `sentence` | `equation` | runs | |
|
||||
| `equation` | `equation-line` | math | Yes |
|
||||
| `equation-line` | | math | Yes |
|
||||
| `figure` | `panel`, `caption`, `figure-note` | | Yes |
|
||||
| `panel` | | | Yes |
|
||||
| `caption` | `sentence` | | |
|
||||
| `figure-note` | `sentence` | | |
|
||||
| `table` | `caption`, `table-head`, `table-body`, `table-note` | | Yes |
|
||||
| `table-head`, `table-body` | `row` | | |
|
||||
| `table-note` | `sentence` | | |
|
||||
| `row` | `cell` | | |
|
||||
| `cell` | | runs | |
|
||||
| `list` | `item` | | |
|
||||
| `item`, `quote` | blocks | | |
|
||||
| `code`, `algorithm` | | verbatim | Yes |
|
||||
| `theorem` | blocks | | Yes |
|
||||
| `footnote` | `sentence` | | Yes |
|
||||
| `back` | `acknowledgments`, `references`, `appendix`, `notes` | | |
|
||||
| `acknowledgments` | blocks | | |
|
||||
| `references` | `reference` | | |
|
||||
| `reference` | | runs | Yes |
|
||||
| `appendix` | `section` | | |
|
||||
|
||||
Blocks are `paragraph`, `equation`, `figure`, `table`, `list`, `quote`, `code`, `algorithm`, `theorem` and `footnote`.
|
||||
|
||||
These rules also hold:
|
||||
|
||||
1. Exactly one node is a root, and it is a `document`.
|
||||
2. A `heading` is a section's first child, and a section has at most one.
|
||||
3. `front`, `back`, `title`, `authors`, `abstract`, `keywords`, `caption`, `table-head`, `table-body`, `acknowledgments` and `references` occur at most once among their siblings.
|
||||
4. A `table` has exactly one `table-body`.
|
||||
5. `title`, `heading`, `paragraph`, `caption`, `figure-note`, `table-note`, `table-head`, `table-body`, `row`, `list`, `item`, `quote`, `theorem`, `footnote` and `appendix` have at least one child.
|
||||
6. Only the kinds marked in the table take a `label`.
|
||||
7. A body without headings, as in a letter, is a section with no heading.
|
||||
|
||||
Kinds outside this list are refused; a new kind is a minor version.
|
||||
Page breaks, running heads and columns are not nodes: they are geometry ([7](07-geometry.md)).
|
||||
|
||||
## 3.3 Content
|
||||
|
||||
| Content | Kinds | Shape |
|
||||
| --- | --- | --- |
|
||||
| Runs | `sentence`, `cell`, `reference`, `authors`, `keywords` | A non-empty list of inline runs ([3.5](#35-inline-runs)). |
|
||||
| Math | `equation`, `equation-line` | `{notation, source}`: `tex` or `typst`, and the source as written. |
|
||||
| Verbatim | `code`, `algorithm` | `{language?, text}`. |
|
||||
|
||||
Other kinds have no `content`.
|
||||
Derived renderings, such as MathML, are not content; they live in `derived/` ([12.7](12-package.md#127-derived-files)).
|
||||
|
||||
## 3.4 Attributes
|
||||
|
||||
| Kind | Attribute | Holds |
|
||||
| --- | --- | --- |
|
||||
| `section` | `level` (required) | 1 for a section, 2 for a subsection, and so on. |
|
||||
| `table` | `columns` | The column definitions: `[{id, header_path}]`, where `header_path` lists header cell IDs, outermost first ([9.2](09-figures-tables-equations.md#92-tables)). |
|
||||
| `cell` | `column`, `rowspan`, `colspan`, `role` | The column's ID, spans (default 1), and `data` (default), `column-header`, `row-header` or `row-group-header`. |
|
||||
| `row` | `header_path` | The cell IDs that label the row, outermost first. |
|
||||
| `list` | `ordered` | Whether the list is numbered. |
|
||||
| `item` | `marker` | The printed marker. |
|
||||
| `theorem` | `variant` | The printed kind: Theorem, Lemma, Definition, Proof, Remark. |
|
||||
| `panel` | `image`, `crop`, `figure_text` | The image asset, a crop, and the text found inside the panel ([9.1](09-figures-tables-equations.md#91-figures-and-panels)). |
|
||||
|
||||
No other kind carries `attrs`, and each kind carries only its own attributes.
|
||||
|
||||
## 3.5 Inline runs
|
||||
|
||||
| Run | Properties | Plain projection |
|
||||
| --- | --- | --- |
|
||||
| `text` | `text`, and optional `style`: a sorted list of distinct values from `em`, `strong`, `sup`, `sub`, `sc`, `mono`, never both `sup` and `sub`. | The text. |
|
||||
| `math` | `notation` (`tex` or `typst`), `source`. | `$` + source + `$` |
|
||||
| `cite` | `refs`: reference node IDs; `text`: the printed form, such as `23`, `[1,2]` or `(Smith, 2019)`. | The printed form. |
|
||||
| `xref` | `target`: a node ID in this document, an address in another, or null when not yet resolved; `text`. | The printed form. |
|
||||
| `fnmark` | `footnote`: a footnote's ID, or null; `text`: the mark. | The mark. |
|
||||
| `link` | `url`, `text`. | The text. |
|
||||
| `equation` | `node`: the ID of an equation child of the sentence. | `$$` + that equation's source + `$$` |
|
||||
|
||||
Runs are canonical: no `text` run is empty, and no two adjacent `text` runs have the same style list.
|
||||
A sentence's `equation` runs name its `equation` children, each once, in the children's order, and a sentence has no other children.
|
||||
A display equation the sentence runs through is thus part of the sentence, while keeping its own ID and hashes.
|
||||
|
||||
## 3.6 The plain projection
|
||||
|
||||
The plain projection of a node with runs is the concatenation of its runs' projections.
|
||||
That of an equation or an equation line is `$$` + source + `$$`, and that of a verbatim node is its text.
|
||||
That of any other node is the plain projections of its text leaves joined with one U+0020.
|
||||
|
||||
A node's text leaves are found by walking its subtree in document order and stopping at `sentence`, `cell`, `reference`, `authors`, `keywords`, `code` and `algorithm` nodes, and at `equation` nodes whose parent is not a sentence; an `equation-line` is never a text leaf.
|
||||
|
||||
Offsets, quotations and the text hash are all defined on the plain projection as stored, before normalization.
|
||||
@@ -0,0 +1,73 @@
|
||||
# 4. Identity
|
||||
|
||||
A node's ID is a name, not a position.
|
||||
It carries no meaning, and once a document is established it is never recomputed, reassigned or reused.
|
||||
|
||||
## 4.1 Letter-first base32
|
||||
|
||||
To encode a SHA-256 digest *d* (32 bytes) as a letter-first ID of *n* characters:
|
||||
|
||||
1. The first character is the letter at index `d[0] mod 22` of `abcdefghjkmnpqrstvwxyz`.
|
||||
2. Read the bytes `d[1]` to `d[31]` as one big-endian bit string.
|
||||
3. Each further character is the Crockford character for the next five bits, from the most significant end.
|
||||
|
||||
Node IDs have seven characters, about 34.5 bits; derived record IDs have 13.
|
||||
`ids.jsonl` in the vectors gives digests and their encodings.
|
||||
|
||||
## 4.2 Minting node IDs
|
||||
|
||||
A node's ID is minted from SHA-256 of the canonical JSON of the array `["scienting-id/1", doc_key, kind, text_hash, occurrence]`, encoded with seven characters.
|
||||
`text_hash` is the plain, never keyed, text hash of the node ([5.1](05-hashes.md#51-the-five-values)).
|
||||
|
||||
When a document is established, nodes are minted in document order.
|
||||
A node's occurrence starts at the number of earlier nodes with the same kind and text hash; while the resulting ID is already taken, the occurrence increases by one.
|
||||
So a fresh conversion of the same input mints the same IDs.
|
||||
|
||||
A node added later, by an `insert` or a `split`, starts at occurrence 0 and increases while the ID is taken by any live, retired or alias ID of the document.
|
||||
|
||||
Columns and figure-text items are not nodes, but they take IDs from the same rule and the same document-wide set, after the nodes:
|
||||
a column with kind `column` and the text hash of its header cells' plain projections joined with U+0020;
|
||||
a figure-text item with kind `figure-text` and the text hash of its text.
|
||||
|
||||
## 4.3 Permanence, retirement and aliases
|
||||
|
||||
| Rule | Detail |
|
||||
| --- | --- |
|
||||
| Permanence | Once established, an ID never changes, and no other node ever takes it. |
|
||||
| Retirement | A `merge` or a `delete` retires IDs ([11](11-corrections.md)). A merge records the surviving node as the successor; a delete records none. |
|
||||
| Aliases | Legacy IDs, such as the `b-p03-text-004` block IDs Lode's extracts use today, are recorded as aliases of the node that now holds that text. |
|
||||
| Labels | What people say, such as "Eq. 3" or "Table S1", is a `label`, never an ID. |
|
||||
| Display paths | Tools may show a path such as `§2.3 ¶4 s2`; it is never stored or cited. |
|
||||
|
||||
## 4.4 `ids.jsonl`
|
||||
|
||||
Each line is one entry.
|
||||
|
||||
| Property | Holds |
|
||||
| --- | --- |
|
||||
| `id` | The retired node ID, or the alias. |
|
||||
| `kind` | `retired` or `alias`. |
|
||||
| `successor` | For a retirement, the successor's ID, or null for a deletion; for an alias, the live node it names. |
|
||||
| `node_kind` | For a retirement, the kind the node had. |
|
||||
| `rev` | For a retirement, the index of the correction that retired it. |
|
||||
|
||||
Aliases come first, sorted by ID; retirements follow, sorted by `rev` and then ID.
|
||||
No ID is both live and in the table, every successor is live or itself in the table, and every alias names a live node.
|
||||
|
||||
## 4.5 Store-qualified names
|
||||
|
||||
A document key qualified by its store, `smithlab/cornell1995second`, names a document anywhere, and an address with a store names a node anywhere ([6.1](06-anchors.md#61-addresses)).
|
||||
|
||||
## 4.6 Record IDs
|
||||
|
||||
A record that is never edited takes its ID from its own content, as 13 letter-first characters of a hash:
|
||||
|
||||
| Record | Hash |
|
||||
| --- | --- |
|
||||
| Receipt, revocation, policy | The record hash: SHA-256 of the record without `id` and `signature` ([8.4](08-fidelity.md#84-record-hashes-and-signatures)). |
|
||||
| Correction | SHA-256 of the record without `id`. |
|
||||
| Annotation | SHA-256 of the annotation without `id`, `status`, `receipt` and `modified`, as created. |
|
||||
| Submission, digitization | SHA-256 of the record without `id`. |
|
||||
|
||||
A witness ID is the seven-character letter-first encoding of the witness's SHA-256 itself.
|
||||
Other records, such as layers and qa reports, MAY use any record ID unique in their store.
|
||||
@@ -0,0 +1,193 @@
|
||||
# 5. Content values
|
||||
|
||||
Each node carries five content values, each for one job.
|
||||
No value covers the node's own ID, its geometry, its provenance or its fidelity.
|
||||
So re-witnessing a page or proofreading a sentence changes none of them, while changing one character changes the `text_hash` and `node_hash` of that sentence.
|
||||
|
||||
In this chapter *H* is SHA-256 written as hexadecimal; under a keyed scheme, *H* is HMAC-SHA-256 under the key ([5.8](#58-keyed-hashes)).
|
||||
|
||||
## 5.1 The five values
|
||||
|
||||
| Value | Definition | Used for |
|
||||
| --- | --- | --- |
|
||||
| `text_hash` | *H*(normalize(plain projection)) | Quotations and Lode's verbatim check, where every character counts. |
|
||||
| `match_hash` | *H*(strict projection) | Certifying that two witnesses agree, and recognizing the same text elsewhere. |
|
||||
| `align_key` | *H*(lossy projection) | Proposing alignments. It never certifies anything. |
|
||||
| `own_hash` | *H*(canonical JSON of the node's own content, [5.2](#52-own-content)) | Anchors on containers that have content of their own. |
|
||||
| `node_hash` | *H*(canonical JSON of `{"own": own_hash, "children": [each child's node_hash, in order]}`) | Pins on leaves; the root hash; knowing that something below has changed. |
|
||||
|
||||
Normalization is that of [1.4](01-conventions.md#14-text-normalization-for-hashing).
|
||||
Because each `node_hash` covers its children's, the root's `node_hash` is a Merkle root: a correction to one sentence changes that sentence's values and the `node_hash` of each ancestor, and nothing else.
|
||||
`hashes.jsonl` gives every value for every node of a small document, with each node's own content.
|
||||
|
||||
## 5.2 Own content
|
||||
|
||||
A node's own content is an object with its `kind` and these members, and nothing else:
|
||||
|
||||
| Kinds | Members |
|
||||
| --- | --- |
|
||||
| Kinds that take a label | `label`: the label, or null. |
|
||||
| `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content. |
|
||||
| `equation`, `equation-line` | `notation`, `source`. |
|
||||
| `code`, `algorithm` | `language` (or null), `text`. |
|
||||
| `section` | `level`, and `heading`: the `text_hash` of its heading, or null. |
|
||||
| `table` | `columns`, as stored. |
|
||||
| `cell` | `column`, `rowspan`, `colspan` and `role`, with defaults 1, 1 and `data` when absent. |
|
||||
| `theorem` | `variant`, or null. |
|
||||
|
||||
So paragraphs, rows, lists, items and the other containers without content have the same `own_hash` as every node of their kind, and are never judged by it.
|
||||
Own content covers stored values exactly; it is not normalized.
|
||||
A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table.
|
||||
|
||||
## 5.3 The strict projection
|
||||
|
||||
The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement.
|
||||
It is computed over a node's runs, or its math, as follows; for other nodes it is the strict projections of the text leaves joined with U+0020, then step 3.
|
||||
|
||||
1. Apply NFC to every string, then project each run:
|
||||
|
||||
| Run | Strict projection |
|
||||
| --- | --- |
|
||||
| `text` with `sup` | `^{` + inner + `}`, or only primes when the content is only primes ([5.4](#54-simple-notation)) |
|
||||
| `text` with `sub` | `_{` + inner + `}` |
|
||||
| other `text`, `link` | The text, with Unicode script characters folded and dashes before digits folded ([5.4](#54-simple-notation)) |
|
||||
| `math`, `equation` | The canonical linear form when the source is simple; U+FFFC when it is complex |
|
||||
| `cite`, `xref`, `fnmark` | U+FFF9 + the printed form with whitespace removed + U+FFFB |
|
||||
|
||||
Concatenate the results.
|
||||
2. Apply NFKC, which folds ligatures, the micro sign and Greek variant letters.
|
||||
3. Collapse whitespace to one U+0020 and trim, as [1.4](01-conventions.md#14-text-normalization-for-hashing) does.
|
||||
|
||||
Folding scripts before NFKC matters: NFKC alone would turn a superscript ² into a plain 2 and erase the structure the projection exists to keep.
|
||||
Marking citations matters for the same reason: "10" followed by citation 3 must not agree with the text "103".
|
||||
|
||||
Each complex run also yields a complex key, `notation:source`, with the source in NFC and every whitespace character removed.
|
||||
A node's complex keys are listed in order.
|
||||
Two readings agree only when their `match_hash` values are equal and their complex keys are equal, run by run; this is stage 1 of canonical math.
|
||||
Stage 2, a later minor version, will replace U+FFFC with canonical MathML; it changes `match_hash` and `align_key` values but no `text_hash`, `own_hash` or `node_hash`, so no anchor goes stale.
|
||||
|
||||
## 5.4 Simple notation
|
||||
|
||||
A character is simple, after folding, when it is:
|
||||
|
||||
1. a letter: General Category L and Script Latin or Greek, or U+00B5 MICRO SIGN;
|
||||
2. a digit: General Category Nd; or
|
||||
3. one of `+ − = × ⋅ / < > ≤ ≥ ± ( ) ′ . ,`.
|
||||
|
||||
Inside notation, characters fold: U+002D and U+2013 to U+2212 MINUS SIGN; U+00B7, U+2219 and U+22C5 to U+22C5 DOT OPERATOR; U+0027 and U+2032 to U+2032 PRIME.
|
||||
|
||||
**The canonical linear form.**
|
||||
Notation is a sequence of atoms; an atom has a base (possibly empty), a count of primes, an optional subscript and an optional superscript, each itself a sequence of atoms.
|
||||
An atom is written as its base, then its primes as U+2032, then `_{`, the subscript and `}`, then `^{`, the superscript and `}`.
|
||||
A superscript that holds only bare primes becomes primes on its atom.
|
||||
There are no spaces in the form, and a script with no base attaches to the character before it, so all of `$H_2O$`, `H$_2$O`, `$\mathrm{H}_{2}\mathrm{O}$`, `H<sub>2</sub>O` and `H₂O` project to `H_{2}O`.
|
||||
|
||||
**TeX sources** are read as tokens: control words (`\` and ASCII letters, with following whitespace skipped), control symbols (`\` and one other character), and single characters, with whitespace and `~` ignored.
|
||||
|
||||
| Token | Meaning |
|
||||
| --- | --- |
|
||||
| `{` … `}` | A group; its atoms are spliced in. A script on a group of more than one atom makes the source complex. |
|
||||
| `_`, `^` | A subscript or superscript of the previous atom (or of an empty base), taking one argument: a group, a command or a character. A second script of the same kind is complex. |
|
||||
| `'` | A prime on the previous atom. |
|
||||
| `\mathrm`, `\text`, `\textrm`, `\mathup` | A wrapper: its argument's atoms are spliced in. |
|
||||
| `\rm` | Ignored. |
|
||||
| `\,` `\;` `\:` `\!`, a backslash before a space, `\quad` `\qquad` `\enspace` `\thinspace` `\medspace` `\thickspace` | Spacing, ignored. |
|
||||
| `\alpha` … `\omega`, `\varepsilon`, `\vartheta`, `\varkappa`, `\varpi`, `\varrho`, `\varsigma`, `\varphi`, `\Gamma` … `\Omega` | The Greek letter TeX typesets; `\epsilon` is ϵ and `\phi` is ϕ. |
|
||||
| `\times` `\cdot` `\pm` `\le` `\leq` `\ge` `\geq` `\lt` `\gt` `\prime` `\AA` | × ⋅ ± ≤ ≤ ≥ ≥ < > ′ Å |
|
||||
| `\left(`, `\right)` | Parentheses; any other delimiter is complex. |
|
||||
| A simple character | An atom. |
|
||||
| Anything else | The source is complex. |
|
||||
|
||||
**Typst sources** are read as tokens: strings in double quotes (complex if they contain a backslash), identifiers (runs of letters, with `.` joining modifiers such as `plus.minus`), numbers (digits with an optional fraction), the two-character operators `<=` and `>=`, and single characters, with whitespace ignored.
|
||||
|
||||
| Token | Meaning |
|
||||
| --- | --- |
|
||||
| `_`, `^` | A script of the previous atom; its argument is a parenthesized group, whose parentheses are dropped, or one token. |
|
||||
| `(` … `)` elsewhere | Literal parentheses around a group. |
|
||||
| `'` | A prime. |
|
||||
| A string | Its characters, with whitespace removed. |
|
||||
| A one-letter identifier, a number | Its characters. |
|
||||
| `upright(…)` | A wrapper. |
|
||||
| `alpha` … `omega`, `Alpha` … `Omega`, and `epsilon.alt`, `theta.alt`, `phi.alt`, `pi.alt`, `rho.alt`, `sigma.alt`, `kappa.alt`, `beta.alt` | The Greek letter Typst renders; `epsilon` is ε and `phi` is φ. |
|
||||
| `plus` `minus` `times` `dot` `dot.op` `dot.c` `plus.minus` `lt` `gt` `lt.eq` `gt.eq` `eq` `prime` `slash`, and `<=` `>=` | + − × ⋅ ⋅ ⋅ ± < > ≤ ≥ = ′ / ≤ ≥ |
|
||||
| `thin` `med` `thick` `quad` `wide` `space` | Spacing, ignored. |
|
||||
| `/`, `*`, `#`, `&`, brackets, braces, any other identifier, and the operators `->`, `=>`, `!=`, `<-`, `==`, `:=` | Complex. In Typst `/` is a fraction. |
|
||||
|
||||
NFKC later folds the variant letters (ϵ to ε, ϕ to φ, ϑ to θ), so TeX's and Typst's defaults agree in the projection.
|
||||
|
||||
**Styled scripts.**
|
||||
The inner form of a `sup` or `sub` text run is its text with whitespace removed and characters folded, when every character is simple or a Unicode script character; otherwise it is the text with whitespace collapsed and trimmed.
|
||||
|
||||
**Unicode script characters** are those whose decomposition mapping is tagged `<super>` or `<sub>`, such as ², ⁻ and ₂.
|
||||
In text, each maximal run of script characters of one kind folds to `^{` or `_{`, the folded bases, and `}`, when every base is simple; otherwise the run is left for NFKC to flatten.
|
||||
|
||||
**Dashes in text.**
|
||||
Outside notation, U+002D and U+2013 immediately before a digit become U+2212, so "10-20" and "10–20" agree; "CT–CT" keeps its dash.
|
||||
|
||||
`notation.jsonl` covers every rule, and `projections.jsonl` with `projection-groups.jsonl` covers H₂O, cm⁻¹, 10⁻³ and K_r in every spelling, ligatures, the micro sign, citation marks and "103" against 10³.
|
||||
|
||||
## 5.5 The lossy projection
|
||||
|
||||
The lossy projection starts from the strict projection and:
|
||||
|
||||
1. removes each `_{` and `^{` and the `}` that closes it, leaving the script's content in place, while other braces stay;
|
||||
2. removes each mark, from U+FFF9 to the next U+FFFB;
|
||||
3. applies Unicode full case folding, then NFKC;
|
||||
4. deletes every whitespace character, every character of General Category Pd, and U+00AD SOFT HYPHEN.
|
||||
|
||||
So "103" and 10³ share an `align_key` while their `match_hash` values differ, and "bond- length", "bond-length" and "bond length" share one too.
|
||||
A node's prose length is the number of code points in its lossy projection other than U+FFFC.
|
||||
|
||||
## 5.6 Alignment
|
||||
|
||||
Two readings of a document are aligned as whole sentence sequences, never paragraph by paragraph, because backends disagree about paragraphs at page and column breaks.
|
||||
Each reading is a list of sentences in document order, each with its strict projection, lossy projection, complex keys, container and context (the kind of its container: paragraph, heading, caption and so on).
|
||||
|
||||
1. **Anchors.** A sentence is a candidate when its lossy projection occurs exactly once in each reading and its prose length is at least 20.
|
||||
Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
|
||||
2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections.
|
||||
Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
|
||||
3. **Compare** each pair.
|
||||
They agree when their strict projections and complex keys are equal.
|
||||
Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`.
|
||||
The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`.
|
||||
4. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
|
||||
5. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
|
||||
|
||||
Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted.
|
||||
`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order.
|
||||
|
||||
## 5.7 Suspects
|
||||
|
||||
A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note.
|
||||
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved.
|
||||
|
||||
## 5.8 Keyed hashes
|
||||
|
||||
A hash of a short text is not a secret: a withheld cell reading "310.0" is recovered by hashing guesses.
|
||||
So every content value that leaves a team's store is keyed.
|
||||
|
||||
| Where the value goes | `hashing` | Key |
|
||||
| --- | --- | --- |
|
||||
| The team's own store and bundles | `{"scheme": "plain"}` | None. |
|
||||
| An export to another machine of the team | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
|
||||
| A public layer, or any export on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
|
||||
|
||||
A keyed scheme replaces SHA-256 with HMAC-SHA-256 under its key in every content value, including the Merkle construction of `node_hash`, and in `meta_hash`.
|
||||
|
||||
**The team key** is HKDF-SHA-256 (RFC 5869) with the team secret, at least 32 random bytes, as input keying material, the salt `scienting-team-key/1`, the info `scienting-hash/1` and length 32.
|
||||
Its `key_id` is the first 16 hexadecimal digits of SHA-256 of the bytes `scienting-key-id/1` followed by the key.
|
||||
|
||||
**The page key** is HKDF-SHA-256 with the concatenated 32-byte page fingerprints of the witness, in page order, as input keying material, the salt `scienting-page-key/1`, the UTF-8 of the context as info, and length 32.
|
||||
The context names the export, such as `layer:smithlab/kp4m2x7q`, so two exports never share a key.
|
||||
The export lists each page as its page hash, SHA-256 of `scienting-page-hash/1` followed by the fingerprint's 32 bytes, never the fingerprint itself.
|
||||
A reader who holds the paper computes the fingerprints from their own copy, checks them against the page hashes and derives the key; someone who does not hold it sees only hashes of hashes.
|
||||
|
||||
In keyed anchors, quotations become MACs over the normalized quotation and context texts become MACs over their normalized text ([6.3](06-anchors.md#63-keyed-anchors)).
|
||||
Open-access documents under a license that allows redistribution need none of this.
|
||||
`keyed.jsonl` gives the derivations and a case a non-holder cannot verify.
|
||||
|
||||
## 5.9 The metadata hash
|
||||
|
||||
`meta_hash` is *H* of the canonical JSON of `{work, version, relations, meta, language, license}` from `document.json`, with `license` null when absent.
|
||||
Fixing an affiliation changes `meta_hash` and not the root.
|
||||
@@ -0,0 +1,120 @@
|
||||
# 6. Addresses and anchors
|
||||
|
||||
## 6.1 Addresses
|
||||
|
||||
An address names a node, optionally pinned to a hash and narrowed to a span:
|
||||
|
||||
```abnf
|
||||
address = [store "/"] key "#" node ["@" pin] ["[" start ":" end "]"]
|
||||
store = key
|
||||
key = lowalnum *(lowalnum / "." / "_" / "-")
|
||||
node = key ; a node ID or an alias
|
||||
pin = 6HEXLOWER ; the first six digits of a node_hash
|
||||
start = "0" / (%x31-39 *DIGIT)
|
||||
end = "0" / (%x31-39 *DIGIT)
|
||||
lowalnum = %x61-7A / DIGIT
|
||||
```
|
||||
|
||||
`start` is not greater than `end`, and both are code-point offsets into the plain projection of the node in the pinned state.
|
||||
|
||||
| Example | Names |
|
||||
| --- | --- |
|
||||
| `cornell1995second#k3m9q2a` | A sentence, whatever its current text. |
|
||||
| `cornell1995second#k3m9q2a@4f1a0c` | That sentence as it read when its `node_hash` began `4f1a0c`. |
|
||||
| `cornell1995second#k3m9q2a@4f1a0c[12:57]` | Code points 12 to 57 of its plain projection in that state. |
|
||||
| `smithlab/cornell1995second#k3m9q2a` | The same sentence in the `smithlab` store. |
|
||||
|
||||
On the web the node ID is the fragment: `…/smithlab/cornell1995second#k3m9q2a` lands on the sentence.
|
||||
`addresses.jsonl` gives valid and invalid addresses and how each parses.
|
||||
|
||||
## 6.2 The anchor
|
||||
|
||||
An anchor, `scienting-anchor/1`, is the stored form of a pinned address, with enough context to survive change.
|
||||
It is what Lode extracts, annotations, dataset values and draft citations use.
|
||||
|
||||
| Property | Required | Holds |
|
||||
| --- | --- | --- |
|
||||
| `format` | Yes | `scienting-anchor/1`. |
|
||||
| `store` | No | The store. |
|
||||
| `doc` | Yes | The document key. |
|
||||
| `node` | Yes | The node ID, or a legacy alias. |
|
||||
| `kind` | No, RECOMMENDED | The node's kind when the anchor was made. |
|
||||
| `own_hash`, `node_hash` | Yes | The node's values when the anchor was made. |
|
||||
| `span` | No | `[start, end]` in code points; present for span anchors. |
|
||||
| `quote` | With a span, or on a leaf | `{exact, prefix?, suffix?}`, or `{exact_hash, length, prefix?, suffix?}` for quotations longer than 200 code points. |
|
||||
| `context` | On a cell | `{table_label, row_path, column_path}`: the table's label and the normalized plain projections of the row's and column's header cells, outermost first. |
|
||||
| `leaves` | On a container without content | `[[id, match_hash], …]`: every leaf below the node, in document order. |
|
||||
| `rev` | Yes | The document's revision when the anchor was made. |
|
||||
| `keyed` | No | The hashing scheme, when the anchor leaves the team ([6.3](#63-keyed-anchors)). |
|
||||
|
||||
`exact` is the substring of the stored plain projection at the span, or the whole projection for a leaf anchor.
|
||||
`prefix` and `suffix` are up to 32 code points before and after it, omitted when empty.
|
||||
`exact_hash` is SHA-256 of the normalized quotation, and `length` its length in code points.
|
||||
|
||||
Revisions are counted, not timed, because anchors are made on different machines and clocks disagree.
|
||||
|
||||
## 6.3 Keyed anchors
|
||||
|
||||
When an anchor leaves the team, in a public layer or on a work whose text is withheld, it carries `keyed`, and:
|
||||
|
||||
1. `own_hash` and `node_hash` are computed under the key;
|
||||
2. `quote` becomes `{mac, length, prefix_mac?, suffix_mac?}`, each MAC over the normalized text; and
|
||||
3. `context` becomes `{table_label_mac, row_path_mac, column_path_mac}` in the same way.
|
||||
|
||||
Such an anchor can be verified, and its span shown, only over a copy of the paper the reader holds.
|
||||
|
||||
## 6.4 Four kinds of anchor
|
||||
|
||||
| Kind | Points at | Judged by |
|
||||
| --- | --- | --- |
|
||||
| Span | Part of a leaf's text: the anchor has a `span`. | The quote, then `node_hash`. |
|
||||
| Leaf | A whole `sentence`, `cell`, `reference`, `authors`, `keywords`, `equation`, `equation-line`, `code` or `algorithm`. | `node_hash`. |
|
||||
| Container with content | A `section`, `figure`, `panel`, `table`, `theorem` or `footnote`. | `own_hash`, with `node_hash` reported. |
|
||||
| Container without content | Any other node. | Its ID and its leaves. |
|
||||
|
||||
## 6.5 Resolution
|
||||
|
||||
To resolve an anchor against a document's current state (its nodes, its ID table and its corrections):
|
||||
|
||||
**Follow the ID.**
|
||||
If the anchor's node is live, use it.
|
||||
Otherwise follow the ID table: an alias leads to its node, a retirement to its successor, repeatedly, until a live node is reached, a deletion ends the chain, or an ID is unknown or repeats.
|
||||
The anchor is *retired* when the chain passed a retirement; an alias alone does not retire it.
|
||||
The kind is the anchor's `kind`, else the live node's, else the retired entry's `node_kind`.
|
||||
|
||||
**Find the quote.**
|
||||
The quote's hits in a text are the positions where `exact` occurs, or, for a hashed quote, where a window of `length` code points has that hash.
|
||||
When there is more than one hit and the anchor has a prefix or suffix, keep the hits preceded by the prefix and followed by the suffix.
|
||||
|
||||
**Judge**, by kind:
|
||||
|
||||
| Kind | Current | Changed below | Moved | Reattached | Otherwise |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Span or leaf | Not retired, and `node_hash` equal. | Never. | Retired, and the successor's text has exactly one hit (a span) or at least one (a leaf); a span takes the new position. | Not retired, changed, and exactly one hit, with the new position; or a changed node whose only changes since `rev` were `relabel` corrections, and whose own content with the earliest of those corrections' `from` label has the anchored `own_hash`. | Stale. |
|
||||
| Container with content | Not retired, and `own_hash` and `node_hash` equal. | Not retired, `own_hash` equal, `node_hash` not. | Retired, and the successor's `own_hash` equals the anchored one. | Not retired, and only `relabel` corrections changed it, judged as for leaves. | Stale. |
|
||||
| Container without content | Live and `node_hash` equal. | Live and `node_hash` not equal. | Retired, and the successor's leaves hold at least half of the anchored `leaves`, each matched by ID or else by `match_hash`. | Never. | Stale. |
|
||||
|
||||
**The cell rule.**
|
||||
A cell anchor whose ID led to no live node (a deletion, or an unknown ID), or whose successor fails the leaf rule, is reattached when exactly one cell in the document has the anchored `context` and exactly one hit of the quote.
|
||||
This finds a value again even in a table with multi-level headers or row labels repeated across parameter sets.
|
||||
|
||||
The result is `{result, node, node_hash?, span?, drift?, via?, reason?}`: the outcome, the live node and its `node_hash`, the new span, the drift of a `changed-below` result, the IDs followed, and a reason for `stale` and relabel results.
|
||||
A `moved` or `reattached` anchor is re-pinned to the live node, and the move is recorded.
|
||||
|
||||
Checks are stricter than readers: a Lode extract is a span or leaf anchor and never accepts `changed-below`, because a check is about exact words.
|
||||
`resolve.jsonl` resolves seventeen anchors made at revision 0 against `fixtures/packages/hale2026harmonic` at revision 12, with every outcome.
|
||||
|
||||
## 6.6 Drift
|
||||
|
||||
`changed-below` must not last forever.
|
||||
For a container *c* and an anchor made at revision *r*:
|
||||
|
||||
1. The corrections that count are those with `index` greater than *r* whose `path` or `to_path` contains *c* and whose `touched` list is not empty.
|
||||
2. `count` is their number, and *T* the union of their `touched` lists.
|
||||
3. *L* is the set of leaves below *c* (including *c* itself if it is one) in the current state.
|
||||
4. `share` is |*T*| divided by |*L* ∪ *T*|, written as a decimal with four places, rounded half up.
|
||||
5. `level` is `none` when `count` is 0, `high` when `share` is greater than the threshold, and `low` otherwise.
|
||||
|
||||
The threshold is a quarter by default; a project may set another ([20](20-projects.md)).
|
||||
A reader shows a `high` drift as needing review.
|
||||
A change to a section's heading changes its `own_hash`, so a rename is never mistaken for drift.
|
||||
@@ -0,0 +1,64 @@
|
||||
# 7. Geometry
|
||||
|
||||
## 7.1 Page fingerprints
|
||||
|
||||
Several publishers stamp each download with the institution, date and time, so two copies of one paper rarely share a SHA-256.
|
||||
Every page of a witness therefore also has a fingerprint, `scienting-page/1`.
|
||||
Geometry and digitizations point at fingerprints, so a re-download keeps every box and every digitization.
|
||||
|
||||
The input is the page's size after rotation, its text-layer lines in content-stream order (each a string and, where known, a box), and the SHA-256 of each embedded image's stream data as stored.
|
||||
To compute the fingerprint:
|
||||
|
||||
1. Remove every line that a stamp rule matches ([7.2](#72-stamp-rules)).
|
||||
2. Join the remaining lines with U+0020.
|
||||
3. Fold Unicode script characters and dashes before digits as the strict projection does for text ([5.4](05-hashes.md#54-simple-notation)), then apply NFKC.
|
||||
4. Delete every whitespace character and every character of General Category Cc or Cf.
|
||||
5. Round the width and height to whole points, half up.
|
||||
6. The fingerprint is SHA-256 of the canonical JSON of `{"format": "scienting-page/1", "size": [width, height], "text": the text, "images": the image hashes, sorted}`.
|
||||
|
||||
Whitespace is deleted rather than collapsed because text extractors disagree about where they insert spaces.
|
||||
A scanned page with no text layer is fingerprinted by its size and its images alone.
|
||||
Two witnesses whose fingerprints are equal, page for page, are the same edition of the paper.
|
||||
|
||||
How lines are extracted from a PDF is outside this text; the vectors give the lines, so implementations are tested on the function.
|
||||
`fingerprints.jsonl` gives two differently stamped downloads of one ACS-style page and of one Wiley-style page that share their fingerprints, a page with one changed digit that does not, and a scan.
|
||||
|
||||
## 7.2 Stamp rules
|
||||
|
||||
Stamp rules are published in `scienting-stamps/1` files.
|
||||
Each rule has an `id`, a `pattern`, a `where` (`any`, or `margin` with a `margin` in points, default 36) and a note.
|
||||
A rule matches a line when its pattern is found in the line's normalized text, with ASCII letters compared without case; a `margin` rule matches only lines whose box lies wholly within the margin of one edge.
|
||||
Patterns use only syntax that ECMAScript and Python read alike: literals, `.`, `*`, `+`, `?`, `^`, `$`, classes and escaped punctuation.
|
||||
|
||||
[registry/stamps/scienting-stamps-1.json](../registry/stamps/scienting-stamps-1.json) holds the published rules.
|
||||
A team adds rules it learns, when two witnesses of one version differ only in such lines, in files of its own with `learned: true`.
|
||||
A missed stamp splits one edition into two; an over-broad rule hides a real difference.
|
||||
|
||||
## 7.3 `pages.jsonl`
|
||||
|
||||
| Property | Holds |
|
||||
| --- | --- |
|
||||
| `witness` | The witness ID. |
|
||||
| `page` | The page number in the witness, from 1. |
|
||||
| `page_fp` | The fingerprint. |
|
||||
| `size` | `[width, height]` in points, after rotation. |
|
||||
| `page_label` | The printed page label, when there is one. |
|
||||
| `text_layer` | Whether the page has a text layer. |
|
||||
| `stamps_removed` | The IDs of the rules that removed lines, in line order, when any did. |
|
||||
|
||||
Pages sort by witness ID, then page.
|
||||
|
||||
## 7.4 `locations.jsonl`
|
||||
|
||||
| Property | Holds |
|
||||
| --- | --- |
|
||||
| `node` | The node ID. |
|
||||
| `page_fp` | The page's fingerprint, which some page of some witness has. |
|
||||
| `page` | The page number. |
|
||||
| `page_label` | The printed label. |
|
||||
| `boxes` | One or more `[x0, y0, x1, y1]` in PDF points, origin at the top left after rotation. |
|
||||
| `approx` | `true` when the boxes come from OCR line boxes rather than glyphs. |
|
||||
|
||||
A node has one record per page it appears on; a sentence that wraps has several boxes.
|
||||
Locations sort by the node's position in document order, then page, then fingerprint.
|
||||
Because locations point at fingerprinted pages, a reader can rebuild a document from its own download: it fingerprints the pages, finds the matching ones and reads the text at the stored boxes.
|
||||
@@ -0,0 +1,124 @@
|
||||
# 8. Fidelity, receipts and spot-checks
|
||||
|
||||
## 8.1 Levels of fidelity
|
||||
|
||||
| Level | Meaning |
|
||||
| --- | --- |
|
||||
| F0 converted | One reading, not corroborated. |
|
||||
| F1 resolved | Every suspect in the node was answered against the page image and passed the gate. |
|
||||
| F2 corroborated | An independent witness agrees, by `match_hash` and complex keys, or the text is born digital, as a Typst or LaTeX source is. |
|
||||
| F3 proofread | A named person compared the node with the page image, and a valid receipt they signed lists the node at its current `node_hash`. |
|
||||
|
||||
Levels apply to leaves.
|
||||
The pipeline records F0, F1 or F2 in a leaf's `fidelity`; F3 comes only from receipts.
|
||||
A correction that changes a leaf drops it back: the gate's crop evidence makes it F1 at most, until it is corroborated or proofread again.
|
||||
|
||||
## 8.2 Levels
|
||||
|
||||
A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`.
|
||||
A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does.
|
||||
A document reports, in `document.json`, the lowest level among its leaves and the count at each level.
|
||||
|
||||
## 8.3 Receipts
|
||||
|
||||
A paper has thousands of nodes, and nobody signs thousands of times, so people sign receipts: one signature over a Merkle root of everything reviewed in a sitting.
|
||||
|
||||
| Property | Required | Holds |
|
||||
| --- | --- | --- |
|
||||
| `format` | Yes | `scienting-receipt/1`. |
|
||||
| `id` | Yes | The record ID ([4.6](04-identity.md#46-record-ids)). |
|
||||
| `doc` | Yes | The document key. |
|
||||
| `rev` | Yes | The revision reviewed. |
|
||||
| `level` | Yes | `F1`, `F2` or `F3`. |
|
||||
| `scope` | Yes | `all`, `section:<id>`, `page:<n>`, `targeted:<name>` (such as `targeted:lode`), or `qa:<receipt id>` for a spot-check. |
|
||||
| `nodes` | Yes | `[[id, node_hash], …]`, sorted by ID without repeats. |
|
||||
| `records` | No | `[[kind, id, hash], …]` for other things signed in the sitting: `annotation`, `digitization`, `correction`, `layer` or `checklist`. |
|
||||
| `root` | Yes | The Merkle root ([8.5](#85-merkle-roots)). |
|
||||
| `evidence` | Yes | What the reviewer looked at: `{kind, sha256, node?, page_fp?, box?, note?}` with `kind` `crop`, `witness` or `source-file`, or `{kind: "person", by}`. A regenerated crop is different evidence. |
|
||||
| `by` | Yes | The signer's roster key. |
|
||||
| `at` | Yes | When it was signed. |
|
||||
| `qa` | For `qa:` scopes | The spot-check ([8.8](#88-spot-checks)). |
|
||||
| `export_of` | No | The receipt this one re-issues under a keyed scheme. |
|
||||
| `keyed` | No | The scheme its values are keyed under, when it leaves the team. |
|
||||
| `signature` | Yes | `{scheme, key, value}` ([8.4](#84-record-hashes-and-signatures)). |
|
||||
|
||||
A receipt lists nodes, records, or both.
|
||||
A correction invalidates a receipt for the corrected node only, because the node's `node_hash` no longer matches; the rest still counts.
|
||||
|
||||
A receipt shared outside the team cannot carry plain node hashes, which would leak short texts, and its signature covers them.
|
||||
So an export re-issues it: a new receipt with `export_of`, values under the export's key, signed by the original reviewer or a team key with the `export` purpose.
|
||||
|
||||
## 8.4 Record hashes and signatures
|
||||
|
||||
A signed record's record hash is SHA-256 of the canonical JSON of the record without `id` and `signature`.
|
||||
Its `id` is the 13-character letter-first encoding of that hash.
|
||||
Its signing payload is the canonical JSON of `{"format": <the record's format>, "hash": <the record hash>}`.
|
||||
|
||||
| Scheme | `value` |
|
||||
| --- | --- |
|
||||
| `ed25519` | The RFC 8032 signature of the payload, in standard Base64 with padding. |
|
||||
| `ssh` | An SSH signature of the payload in the namespace `scienting`, armored. |
|
||||
| `gpg` | A detached, armored OpenPGP signature of the payload. |
|
||||
| `sigstore` | A Sigstore bundle for the payload, as JSON text. |
|
||||
|
||||
`signature.key` is the signer's roster key and equals `by`.
|
||||
Self-hosted deployments sign with team keys or a private Sigstore deployment; the public Sigstore instance, whose log permanently records each signer's identity, is for public layers only.
|
||||
`signatures.jsonl` verifies one receipt and the RFC 8032 test vector.
|
||||
|
||||
## 8.5 Merkle roots
|
||||
|
||||
A receipt's items are `["node", id, node_hash]` for each listed node, in order, followed by its `records`.
|
||||
The root follows RFC 6962: a leaf hash is SHA-256 of the byte 0x00 and the item's canonical JSON; an interior hash is SHA-256 of 0x01 and the two child hashes; a list of *n* > 1 items splits after the largest power of two less than *n*.
|
||||
A root over one item is that item's leaf hash; a receipt with no items is invalid.
|
||||
The tree allows a proof that one node was covered without revealing the others; `merkle.jsonl` gives roots for one to seven items.
|
||||
|
||||
## 8.6 Receipt status
|
||||
|
||||
A receipt is:
|
||||
|
||||
1. **invalid** when its root, ID or signature does not verify, or its signer's key was not valid when it was signed (`registered_at` ≤ `at` < `revoked_at`);
|
||||
2. otherwise **revoked** when a valid revocation names it, or names its signer with a period that contains its `at`;
|
||||
3. otherwise **suspended** when a valid spot-check of it found an error;
|
||||
4. otherwise **valid**.
|
||||
|
||||
Only valid receipts set levels.
|
||||
A suspended receipt stays suspended; its reviewer reviews the sitting again and signs a new receipt, which needs its own spot-check.
|
||||
|
||||
## 8.7 Revocations and the roster
|
||||
|
||||
A revocation, `scienting-revocation/1`, names one `receipt`, or a `signer` and a `period` `{from, to}`, with a `reason`, and is signed like a receipt.
|
||||
It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or an owner of the project.
|
||||
|
||||
The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
|
||||
A package MAY carry the roster its receipts need as `roster.json`, so its receipts verify without the store.
|
||||
The roster is meant to match the one Lode already verifies; aligning the two is recorded in [DECISIONS.md](../DECISIONS.md).
|
||||
|
||||
## 8.8 Spot-checks
|
||||
|
||||
For each F3 receipt, a second reviewer checks a sample of its nodes and signs a `qa:` receipt.
|
||||
A seed the reviewer controls could be ground, by varying the receipt until the sample falls on careful work, so the seed is a signature nobody can steer:
|
||||
|
||||
1. The seed is the checker's Ed25519 signature over the canonical JSON of `{"format": "scienting-spot-check/1", "receipt": <the original's record hash>}`, made with a key registered in the roster with the `spot-check` purpose before the original was signed.
|
||||
Ed25519 signatures are deterministic, so the checker cannot choose it, and the reviewer cannot predict it.
|
||||
2. The sample size is the smaller of *n* and the larger of `minimum` and ⌈`rate` × *n*⌉, for the original's *n* nodes; the defaults are a rate of `0.05` and a minimum of 10.
|
||||
3. Rank the original's node IDs by the hexadecimal SHA-256 of the seed's 64 bytes followed by the ID's UTF-8, then by ID, and take the first ones.
|
||||
|
||||
The `qa` object records `of`, `key`, `seed` (Base64), `rate`, `minimum`, the `sample` sorted by ID, and the `errors` found, each `{node, kind?, note?}`.
|
||||
The `qa:` receipt's `nodes` are the sampled nodes at the hashes the original lists.
|
||||
Anyone can recompute the sample; `spotcheck.jsonl` does.
|
||||
|
||||
## 8.9 Reviewer error bounds
|
||||
|
||||
One sample proves little; what a buyer wants is a bound on each reviewer's error rate across all their samples.
|
||||
With *k* errors in *n* sampled nodes, the bound is the one-sided Clopper–Pearson upper confidence bound at confidence *c*, by default 0.95: the rate *p* at which the probability of at most *k* errors in *n* is 1 − *c*, or 1 when *k* = *n*.
|
||||
It is reported to six decimal places, rounded up.
|
||||
With no errors the bound is about 3/*n*, so zero errors in 300 sampled nodes bounds a rate below 1%; `bounds.jsonl` gives cases.
|
||||
|
||||
A QA report, `scienting-qa-report/1`, states for each reviewer who signed a delivery the nodes sampled, the errors found, the confidence, the method `clopper-pearson` and the bound.
|
||||
|
||||
## 8.10 Targeted F3
|
||||
|
||||
The target set of a document is every node that a Lode extract, a dataset value, a confirmed `claim-link`, a draft quotation or a pinned citation anchors, plus every display equation, plus every sentence with a digit in a `text` or `math` run.
|
||||
Digits in `cite`, `xref` and `fnmark` runs do not count.
|
||||
A project sets the digit rule to `all`, `anchored-sections` (only sentences in a section that holds an anchored node) or `none`.
|
||||
Neighbors shown for context are not in the set, and nothing new comes to rest on a node below F3 without the node joining the set.
|
||||
@@ -0,0 +1,37 @@
|
||||
# 9. Figures, tables and equations
|
||||
|
||||
## 9.1 Figures and panels
|
||||
|
||||
A figure holds its panels, its caption and its notes.
|
||||
A panel's own content is its label only.
|
||||
Its image and region are evidence, not content: re-extracting the image at native resolution adds an asset and changes no hash.
|
||||
|
||||
| Panel attribute | Holds |
|
||||
| --- | --- |
|
||||
| `image` | `{sha256, media_type, native, dpi?}`: the embedded image, at native resolution when the PDF holds one. |
|
||||
| `crop` | `{sha256, media_type, dpi}`: a crop of the panel's region at a fixed resolution. |
|
||||
| `figure_text` | `[{id, text, role, page_fp?, boxes?}]`: text found inside the panel, with `role` `axis-label`, `tick-label`, `legend`, `annotation` or `other`. |
|
||||
|
||||
Text inside a figure is not prose and is not split into sentences.
|
||||
Its items have IDs ([4.2](04-identity.md#42-minting-node-ids)) so that a digitization can anchor its axis labels to them.
|
||||
A panel's region on the page is its location ([7.4](07-geometry.md#74-locationsjsonl)).
|
||||
|
||||
## 9.2 Tables
|
||||
|
||||
A table holds its caption, an optional head, one body and its notes.
|
||||
Rows and cells are nodes, with all five values, so the gate, the hashes and the database treat them like any other node, and a cell is addressed like a sentence.
|
||||
|
||||
Columns are defined in the table's `columns` attribute, each with an ID and a header path: the IDs of the header cells above it, outermost first.
|
||||
Header text is stored once, in the header cells.
|
||||
A row's `header_path` lists the cells that label it: a row-group header and the row's own header cell.
|
||||
A cell records its column, its spans and its role; the expanded grid is derived, never stored.
|
||||
|
||||
So inserting a row that a reading dropped adds a row node and changes no other row's or cell's ID or hash, and fixing a header cell changes that cell, not the table.
|
||||
A table continued across pages is one table node; its pages are locations.
|
||||
For people, a cell can be shown as "Table 2, row CT–CT, column K_r" from its header paths.
|
||||
|
||||
## 9.3 Equations
|
||||
|
||||
An equation keeps its source as written, `tex` from a conversion or `typst` from a draft, and its printed number as its `label`, so numbering never collides with a renderer's own.
|
||||
An equation that the source numbers line by line (Eq. 3a) has `equation-line` children, each with its own label and source.
|
||||
MathML is derived by a pinned tool and recorded, with the tool's name and version, in `derived/mathml.jsonl`.
|
||||
@@ -0,0 +1,15 @@
|
||||
# 10. References and citations
|
||||
|
||||
Each reference is a `reference` node holding its printed runs, exactly as printed, and its printed number as its `label`.
|
||||
`references.jsonl` holds, for each reference node in document order, a record with `node`, `label`, an optional parsed `csl` record in CSL-JSON, and its `resolution`:
|
||||
|
||||
| Resolution | Means |
|
||||
| --- | --- |
|
||||
| `{type: "doi", doi}` | The reference names a DOI. |
|
||||
| `{type: "document", doc}` | It names a document in the same store, by key. |
|
||||
| `{type: "none"}` | Nothing yet. |
|
||||
|
||||
Every `cite` run links a sentence to one or more references.
|
||||
The citation graph is therefore sentence-level: for any reference, the format gives every sentence that cites it, which is the citation's context.
|
||||
The database derives a `citations` table from the runs ([13](13-database.md)).
|
||||
An article's references and its Supporting Information's are separate documents' references and never mix.
|
||||
@@ -0,0 +1,70 @@
|
||||
# 11. Corrections
|
||||
|
||||
A correction, `scienting-correction/1`, is an operation on an established document, with its evidence and its author.
|
||||
Corrections are added and never edited, so a document's history is its original reading plus its corrections.
|
||||
Each has an `index`, counted from 1 in the order corrections are applied, and the document's revision is the index of the last.
|
||||
|
||||
## 11.1 Operations
|
||||
|
||||
| `op` | Fields | Does | IDs |
|
||||
| --- | --- | --- | --- |
|
||||
| `replace` | `target`, `find`, `replace`, optional `run` | Replaces the one occurrence of `find` among the target's editable strings with `replace`, a string or, inside a text run, a list of runs. | Unchanged. |
|
||||
| `insert` | `parent`, `place_after` (a sibling, or null for first), `node` | Adds a subtree: kinds, labels, attributes and content, with each equation run naming its child as `{"kind": "equation", "child": i}`. | New IDs, minted ([4.2](04-identity.md#42-minting-node-ids)). |
|
||||
| `insert` | `target`, `run`, `offset`, `text` | Inserts text into one run of the target, at a code-point offset in that run's editable string. | Unchanged. |
|
||||
| `delete` | `target` | Removes an artifact and its subtree, such as a running head read as text. | All retired, without successors. |
|
||||
| `split` | `target`, `offset` | Splits a sentence at an offset in its plain projection, which falls between runs or inside a text run; whitespace at the split is removed from both parts, and each equation child goes with its run. | The first part keeps the ID; the second gets a new one. |
|
||||
| `merge` | `target`, `with`, optional `joiner` | Joins a node with its next sibling of the same kind: the runs of two run-bearing nodes, with the joiner (default one space) between, or the children of two `paragraph`, `list`, `item`, `quote`, `caption`, `figure-note` or `table-note` nodes. | `with` is retired, with the target as successor. |
|
||||
| `move` | `target`, `parent`, `place_after` | Changes reading order or containment. | Unchanged. |
|
||||
| `reclassify` | `target`, `kind` | Changes a node's kind, such as a paragraph that is really a caption, to a kind with the same content shape. | Unchanged. |
|
||||
| `relabel` | `target`, `label` | Changes a printed label. | Unchanged. |
|
||||
| `set` | `target`, `attr`, `value` | Sets one structural attribute: a section's `level`, a table's `columns`, a cell's `column`, `rowspan`, `colspan` or `role`, a row's `header_path`, a list's `ordered`, an item's `marker` or a theorem's `variant`. | Unchanged. |
|
||||
|
||||
A leaf's editable strings are the `text` of its `text`, `link`, `cite`, `xref` and `fnmark` runs and the `source` of its `math` runs, each searched on its own; an equation's `source`; a verbatim node's `text`.
|
||||
`set` is an addition to the design's eight operations, needed to fix table structure; see [DECISIONS.md](../DECISIONS.md).
|
||||
Runs are made canonical after every operation, and the result MUST satisfy every rule of [3](03-nodes.md).
|
||||
|
||||
## 11.2 Every correction records
|
||||
|
||||
| Property | Holds |
|
||||
| --- | --- |
|
||||
| `evidence` | At least one item: `crop`, `witness` or `source-file` by SHA-256, or `person`. |
|
||||
| `author` | `{key}` for a person in the roster, or `{model, run}` for a model's session. |
|
||||
| `reason`, `at` | Why, and when. |
|
||||
|
||||
A proposal, such as one an MCP client makes, carries these and `status: "proposed"`, and no applied fields.
|
||||
|
||||
## 11.3 Applied fields
|
||||
|
||||
When a correction is applied, its record gains:
|
||||
|
||||
| Property | Holds |
|
||||
| --- | --- |
|
||||
| `index`, `id` | Its position, and its record ID. |
|
||||
| `before`, `after` | The target's `node_hash` before and after, or null; for an inserted subtree, `target` is its root. |
|
||||
| `path` | The target's ancestors when it was applied, root first; for an insert, the new subtree's. |
|
||||
| `to_path` | For a move, the new ancestors. |
|
||||
| `touched` | The leaves whose content the correction changed, sorted: the target of a `replace`, text `insert`, `relabel` or `set` on a leaf; the new leaves of a node `insert`; the deleted leaves of a `delete`; both parts of a `split`; both nodes of a run `merge`, or the moved-in leaves of a container `merge`; the moved leaves of a `move`; the leaves of a reclassified node. |
|
||||
| `created`, `retired` | IDs minted or retired. |
|
||||
| `from` | The label, kind or attribute value before a `relabel`, `reclassify` or `set`. |
|
||||
|
||||
Retirements are also entered in `ids.jsonl` with the correction's index as `rev`.
|
||||
Drift ([6.6](06-anchors.md#66-drift)) is computed from `path`, `to_path` and `touched` alone, so it needs no replay.
|
||||
|
||||
## 11.4 The gate
|
||||
|
||||
A correction is refused, naming the rule ([22](22-conformance.md)), when:
|
||||
|
||||
1. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified;
|
||||
2. `find` is empty or does not occur exactly once;
|
||||
3. an `insert` has no crop evidence;
|
||||
4. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits;
|
||||
5. any leaf outside `touched` has different digits afterwards;
|
||||
6. a `split` falls outside the sentence, inside a run other than text, or leaves a part empty;
|
||||
7. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings;
|
||||
8. a `move` would put a node inside itself;
|
||||
9. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it; or
|
||||
10. the result breaks a rule of [3](03-nodes.md).
|
||||
|
||||
Digits are characters of General Category Nd in the plain projection.
|
||||
A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, and the last correction of each leaf ends at its current `node_hash`.
|
||||
`corrections.jsonl` in the vectors applies each operation, and each refusal, to a small document.
|
||||
@@ -0,0 +1,70 @@
|
||||
# 12. The package
|
||||
|
||||
A document travels as a package: a folder named by its key, or the same folder zipped.
|
||||
|
||||
## 12.1 Files
|
||||
|
||||
| Path | Required | Holds |
|
||||
| --- | --- | --- |
|
||||
| `MANIFEST.json` | Yes | The manifest ([12.3](#123-the-manifest)). |
|
||||
| `document.json` | Yes | The document ([2.2](02-model.md#22-documentjson)). |
|
||||
| `nodes.jsonl` | Yes | Every node, in document order ([3](03-nodes.md)). |
|
||||
| `witnesses.json` | Yes | Every witness, with its rights ([2.3](02-model.md#23-witnessesjson)). |
|
||||
| `provenance.json` | Yes | Tools and versions, backends, cached readings, stages and where each ran. |
|
||||
| `pages.jsonl` | No | Every page of every witness ([7.3](07-geometry.md#73-pagesjsonl)). |
|
||||
| `locations.jsonl` | No | Geometry per node and page ([7.4](07-geometry.md#74-locationsjsonl)). |
|
||||
| `references.jsonl` | No | Parsed references and their resolution ([10](10-references.md)). |
|
||||
| `corrections.jsonl` | No | Every correction, in order ([11](11-corrections.md)). |
|
||||
| `receipts.jsonl` | No | Receipts and revocations, sorted by `at` and then `id` ([8](08-fidelity.md)). |
|
||||
| `ids.jsonl` | No | Retired IDs and legacy aliases ([4.4](04-identity.md#44-idsjsonl)). |
|
||||
| `suspects.jsonl` | No | Unresolved suspects, sorted by `id` ([5.7](05-hashes.md#57-suspects)). |
|
||||
| `roster.json` | No | The roster its receipts need ([8.7](08-fidelity.md#87-revocations-and-the-roster)). |
|
||||
| `assets/` | No | Images and crops, as `<sha256>.<extension>`, present or listed only. |
|
||||
| `datasets/<id>/` | No | Datasets ([15](15-datasets.md)). |
|
||||
| `derived/` | No | Rebuildable renderings, never authoritative ([12.7](#127-derived-files)). |
|
||||
|
||||
An optional JSON Lines file with no records is omitted.
|
||||
Every file is in the canonical form of [1.9](01-conventions.md#19-files).
|
||||
|
||||
## 12.2 Paths
|
||||
|
||||
Paths use `/`, are relative, have no empty, `.` or `..` segment, no backslash and no NUL, and are already normalized.
|
||||
In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit.
|
||||
A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields.
|
||||
|
||||
## 12.3 The manifest
|
||||
|
||||
`MANIFEST.json`, `scienting-package/1`, is `{format, doc, key, files}`: the package format, `scienting-doc/1`, the document key, and every other file as `{path, sha256, size}`, sorted by the UTF-8 bytes of the path.
|
||||
An asset whose bytes stay in the asset store is listed with `present: false`; nothing else may be.
|
||||
The SHA-256 of `MANIFEST.json` pins the whole package.
|
||||
The manifest is meant to have the same shape as Lode's `lode-cache/1`, so Lode can hold a package as a source cache.
|
||||
|
||||
## 12.4 Provenance
|
||||
|
||||
`provenance.json` holds `tools` (`{name, version}`), and optionally `backends`, `readings` (each cached reading's ID, witness, backend and SHA-256), `stages` (each stage's name, tool, input hashes, output hash, the `processor` where it ran, and the uses it ran under) and a `session`.
|
||||
Every stage that uses a model or an outside service records where it ran ([14](14-rights.md)).
|
||||
|
||||
## 12.5 Text policies
|
||||
|
||||
| Policy | Means |
|
||||
| --- | --- |
|
||||
| `full` | Every leaf keeps its content. |
|
||||
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed. |
|
||||
|
||||
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
|
||||
Its text lives in the team's private asset store, and only key holders can check its values.
|
||||
`fixtures/packages/hale2026harmonic.checks-only` is such an export of `hale2026harmonic` under the team key in `fixtures/keys/`.
|
||||
|
||||
## 12.6 Assets
|
||||
|
||||
Assets are named by SHA-256, so every read is rehashed and a regenerated crop is a different asset.
|
||||
Bytes that live only in the asset store are listed in the manifest with `present: false`, with their size.
|
||||
|
||||
## 12.7 Derived files
|
||||
|
||||
`derived/` holds what a pinned tool makes from the package, such as `derived/mathml.jsonl`, whose lines are `{node, run, tool, mathml}` with `run` null for an equation node.
|
||||
Nothing derived is content, and a consumer may delete and rebuild it.
|
||||
|
||||
## 12.8 Media type and extension
|
||||
|
||||
The proposed extension is `.sct.zip` and the media type `application/vnd.scienting.doc+zip`; both wait for a decision ([DECISIONS.md](../DECISIONS.md)) and are fixed forever once packages are shared.
|
||||
@@ -0,0 +1,57 @@
|
||||
# 13. The database form and the bundle
|
||||
|
||||
## 13.1 Portability
|
||||
|
||||
[sql/doc.sql](../sql/doc.sql) defines the document tables and [sql/store.sql](../sql/store.sql) a team store's tables.
|
||||
Both run unchanged on SQLite, Dolt and PostgreSQL, under these rules:
|
||||
|
||||
1. No foreign keys; references are checked by validators, not by the database.
|
||||
2. Enumerations are `VARCHAR` with `CHECK (… IN (…))`.
|
||||
3. JSON values are canonical JSON in `TEXT` columns; booleans are `INTEGER` 0 or 1.
|
||||
4. Every table has a primary key, and key columns are `VARCHAR` with a length, as MySQL-compatible engines require.
|
||||
5. No table, column or index is named with a word reserved in SQLite, PostgreSQL or MySQL; so the design's `key`, `index`, `at`, `by`, `before` and `after` are `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
|
||||
|
||||
## 13.2 Tables
|
||||
|
||||
| Table | One row per | Notes |
|
||||
| --- | --- | --- |
|
||||
| `works` | work | Its key and `same_as`. |
|
||||
| `documents` | document | Every property of `document.json` as a column, plus `provenance`, `roster` and `package_hash`. |
|
||||
| `witnesses` | witness | With `ord`, its position in `witnesses.json`. |
|
||||
| `nodes` | node | With `seq`, its position in document order. |
|
||||
| `pages`, `locations`, `refs` | page, location, reference | |
|
||||
| `corrections` | correction | The common properties as columns; the rest in `details`. |
|
||||
| `node_ids` | retired ID or alias | |
|
||||
| `receipts`, `revocations` | receipt, revocation | |
|
||||
| `suspects` | suspect | |
|
||||
| `package_files` | asset, dataset file or derived file | Its path, SHA-256, size and whether its bytes were present. Bytes go to the asset store. |
|
||||
| `citations` | cite run and reference | Derived on load. |
|
||||
| `receipt_nodes` | node a receipt lists | Derived on load. |
|
||||
|
||||
A 30-page paper is about 3,000 nodes, so 500 documents are about a million and a half rows, which SQLite and Dolt hold comfortably.
|
||||
|
||||
## 13.3 Load and unload
|
||||
|
||||
Loading a package is a pure function of the package: each record becomes a row, an optional property absent from the record is NULL, and JSON-valued properties are stored as canonical JSON.
|
||||
Unloading writes each file back from its rows in the file's defined order, omitting optional properties that are NULL and optional files that have no rows, copies the other files from the asset store, and writes the manifest.
|
||||
Unloading gives the package back byte for byte, and its manifest hash with it; the suite checks this for every fixture.
|
||||
|
||||
## 13.4 The store tables
|
||||
|
||||
`store.sql` adds what a team adds to its documents: `roster_keys`, `projects`, `members`, `project_documents`, `reading_paths`, `layers`, `annotations`, `annotation_targets` (derived, so a reader finds the annotations on a node), `datasets`, `dataset_values`, `digitizations`, `views`, `policies`, `submissions` and `grades`.
|
||||
Each keeps its record whole in a `record` or `descriptor` column, with the columns a query needs beside it.
|
||||
|
||||
## 13.5 The bundle
|
||||
|
||||
A bundle, `scienting-bundle/1`, is one SQLite file that a reader, a server or an MCP server queries.
|
||||
It holds the tables of `doc.sql` and `store.sql`, then those of [sql/bundle-sqlite.sql](../sql/bundle-sqlite.sql):
|
||||
|
||||
| Table | Holds |
|
||||
| --- | --- |
|
||||
| `bundle_meta` | `name` and `value` pairs: at least `format` (`scienting-bundle/1`), `store`, `built_from` (the store revision) and `spec` (the version of these specifications). |
|
||||
| `search` | An FTS5 index with one row per sentence, cell, display equation and figure-text item: the document, the node, the unit, the context (the kind of a sentence's container) and the normalized plain projection. Withheld text is never indexed. |
|
||||
| `embeddings_meta` | Reserved for embeddings with `sqlite-vec`: the model, its dimensions, and the rights use they were made under. |
|
||||
|
||||
At team scale this is all the search a team needs.
|
||||
A team can download its bundle and use it offline.
|
||||
Because a bundle holds text, it is served only under the rights of [14](14-rights.md), checked again whenever text is served.
|
||||
@@ -0,0 +1,61 @@
|
||||
# 14. Rights
|
||||
|
||||
What a team may do with a paper depends on how it holds the paper, and AI processing is now a use that licenses name.
|
||||
Every witness carries a rights record, every stage declares the uses it needs, and a stage runs only when the rights allow them.
|
||||
Nothing here is legal advice; the defaults are proposals for counsel to set.
|
||||
|
||||
## 14.1 The rights record
|
||||
|
||||
`scienting-rights/1`:
|
||||
|
||||
| Property | Holds |
|
||||
| --- | --- |
|
||||
| `license` | The license as stated, `{spdx?, text?, url?}`, or null. |
|
||||
| `basis` | How the team holds the right: `open-license`, `publisher-tdm`, `collective-license`, `subscription`, `own-work`, `permission` or `unknown`. |
|
||||
| `uses` | Each of `read`, `tdm`, `redistribute`, `quote-public` and `accessible-copy` as `true`, `false` or `"unknown"`, and `ai`: a map from each pair `<purpose>/<processor>` to the same values. |
|
||||
| `asserted_by` | `{by, evidence}`: who or what recorded the basis, and on what: a license URL, a contract's hash, Crossref's license field, a delivery, a statement. |
|
||||
| `at`, `valid_until` | When it was recorded, and when the basis ends, or null. |
|
||||
|
||||
Purposes are `internal` (results stay inside the organization) and `external` (results leave it).
|
||||
Processors are `local` (the team's machines), `tenancy` (the team's cloud account), `outside-no-retention` (an outside service contractually keeping nothing) and `outside` (one that may retain).
|
||||
A use or pair that is absent is `"unknown"`.
|
||||
A producer sets `license` and a provisional `basis` from Crossref for open-access papers, and never sets a use to `true` on its own for anything else.
|
||||
|
||||
## 14.2 Effective uses
|
||||
|
||||
To find a witness's uses at a time *t*:
|
||||
|
||||
1. If `valid_until` is set and *t* is not before it, every use is `false`: an expired basis stops everything at once.
|
||||
2. Otherwise start from the record's uses, and for each policy that applies at *t*, replace each `"unknown"` with the policy's value, where the policy has one.
|
||||
A policy never overrides `true` or `false`.
|
||||
|
||||
A policy, `scienting-policy/1`, is a project owner's signed decision: `applies_to` (any of `basis`, `license`, `doi_prefixes` and `witnesses`, all of which must match), the `uses` it decides, a `statement`, `by`, `at` and `valid_until`.
|
||||
It applies from `at` until `valid_until`.
|
||||
|
||||
A node established from several witnesses takes the most restrictive of their uses: `false` if any is false, else `"unknown"` if any is unknown, else `true`.
|
||||
That combining rule is proposed for counsel to confirm.
|
||||
|
||||
## 14.3 Stages
|
||||
|
||||
`scienting-stages/1` maps each stage to what it needs: a list of `uses`, and optionally `ai`, with a `purpose` and the `processors` it may run on.
|
||||
A stage may run on a processor when every listed use is `true`, the processor is among those allowed, if any are listed, and the pair `ai/<purpose>/<processor>` is `true`.
|
||||
When a check fails, the refusal lists what is lacking: a use or pair that is `false` by name, one that is unknown with a trailing `?`, and a disallowed processor as `processor:<name>`.
|
||||
|
||||
[registry/stages/scienting-stages-default.json](../registry/stages/scienting-stages-default.json) holds the design's proposed defaults; a team replaces the file, not the code.
|
||||
Rights are checked when a stage runs and again whenever text is served.
|
||||
Every derived artifact records the rights record and use it was made under, so it can be listed and purged when a basis ends.
|
||||
|
||||
## 14.4 Backends and profiles
|
||||
|
||||
A backend's code and weights can be licensed differently, and the weights decide whether a paying team may use it.
|
||||
`scienting-backend/1` declares a backend's `code_license`, its `weights` with their licenses, and the `conditions` they impose:
|
||||
|
||||
| Condition | Met when the team profile |
|
||||
| --- | --- |
|
||||
| `revenue-cap` with `usd` | states revenue or funding at most `usd`. |
|
||||
| `no-competition` with `vendor` | states what it competes with, and that does not include the vendor. |
|
||||
| `purpose` with `allowed` | states a purpose among those allowed. |
|
||||
|
||||
`scienting-profile/1` records what conditions ask: the team's `purpose` (`research`, `personal`, `education`, `startup` or `commercial`), `revenue_or_funding_usd` as `{at_most}` or null when unknown, and `competes_with`, or null when unknown.
|
||||
A condition whose answer is unknown is not met, and the gate refuses a backend with any unmet condition.
|
||||
`rights.jsonl` exercises stages, policies, expiry, combining and backends.
|
||||
@@ -0,0 +1,27 @@
|
||||
# 15. Datasets
|
||||
|
||||
A dataset, `scienting-dataset/1`, is a Frictionless Data Package (`datapackage.json` with CSV or Parquet resources) with three additions.
|
||||
|
||||
| Addition | Meaning |
|
||||
| --- | --- |
|
||||
| Units | Every numeric field has a `unit`, a string Pint can parse, as Lode's dimension check expects. |
|
||||
| Uncertainty | A field may give `uncertainty`: `{field?, kind}`, the field that holds it and its kind, `sd`, `se`, `resolution` or `unknown`. |
|
||||
| Origin | Each resource has an `origin`, or `origin_field` and `origins` when it varies by row. |
|
||||
|
||||
| Origin | Holds |
|
||||
| --- | --- |
|
||||
| `reported` | Values anchored to the table cells or sentences that print them, in `anchors.jsonl`. |
|
||||
| `digitized` | Values read off a figure: the digitization's ID, and `anchors.jsonl` naming each value's series and point. |
|
||||
| `computed` | A Lode claim's implementation, by claim and pin. |
|
||||
| `measured` | The team's own data, with its provenance. |
|
||||
|
||||
The descriptor also carries `format` and an `id`, `ds-` and a key.
|
||||
Each line of `anchors.jsonl` names a `resource`, a `row` (by primary key) and a `field`, and either an `anchor` or a `digitization`, `series` and `point`.
|
||||
A value or row is addressed as `ds-…/<row>/<field>`.
|
||||
|
||||
Values keep their digits as printed: CSV cells are never reformatted, and in JSON values are strings, so `1.526` and `1.5260` stay different.
|
||||
A digitized value never passes as reported, and its uncertainty is at least the resolution of its digitization.
|
||||
|
||||
A dataset's content hash is SHA-256 of the canonical JSON of `{"format": "scienting-dataset/1", "files": [[path, sha256, size], …]}` over its files, sorted by path.
|
||||
A dataset can be a Lode source of kind `dataset`, so a claim can rest on a value read from a figure, with that fact visible in its evidence.
|
||||
`fixtures/packages/hale2026harmonic/datasets/ds-hale-bond-params` reports Table 1, and the suite checks that every value resolves to the cell that prints it.
|
||||
@@ -0,0 +1,18 @@
|
||||
# 16. Digitization records
|
||||
|
||||
A digitization, `scienting-digitization/1`, says how numbers were read off a figure panel, completely enough to repeat.
|
||||
|
||||
| Part | Holds |
|
||||
| --- | --- |
|
||||
| `target` | The panel's `anchor`, and the `region` read, as a box on the page given by `page_fp`, with the witness's SHA-256 as evidence; or, for a figure that exists only as an image, the image's SHA-256 with the region in its pixels. |
|
||||
| `axes` | For each axis: its `scale` (`linear`, `log`, `reciprocal` or `date`), `unit`, at least two `calibration` points from page coordinates to value, an optional third `check`, and its `label` anchored to the panel's figure text. |
|
||||
| `series` | For each series: its legend, its mark (color, marker, line), its kind (`points`, `line`, `bars`, `error-bars` or `region`) and its values: each a position, a value, and an uncertainty from the image's resolution, the marker's size and the calibration's residual. |
|
||||
| `method` | `manual`, `traced` or `proposed` (by a model), with the tool and its version. |
|
||||
| `verification` | The overlay check: the image of the values re-plotted under the calibration, by SHA-256, and the mean and largest residual. |
|
||||
| `confirmation` | The receipt of the person who confirmed it after seeing the overlay, or null. |
|
||||
| `dataset`, `rev` | The dataset it yields, and the revision it was made at. |
|
||||
|
||||
Calibration and values are in page coordinates, not an image's pixels, so a digitization does not depend on the raster it was done on.
|
||||
A digitization stays current while the document has a page with its `page_fp` and its panel anchor is not stale.
|
||||
A teammate's re-download, stamped differently, has the same fingerprints, so the digitization stays current on it; a different edition or region has a different fingerprint, so a swapped image never passes.
|
||||
A digitization counts only when a person confirms it: a receipt with the record `["digitization", id, hash]`, where the hash is SHA-256 of the record without `confirmation`.
|
||||
@@ -0,0 +1,8 @@
|
||||
# 17. Views
|
||||
|
||||
A view, `scienting-view/1`, is a Vega-Lite specification whose `usermeta.scienting` pins its data: `{format, id, data, computation?}`.
|
||||
Each entry of `data` names a Vega-Lite data source, the dataset it is, the dataset's content hash and a resource, and the specification refers to it as `{"data": {"name": …}}`.
|
||||
A view is declarative JSON, so it is diffable, renders without studio code, and is easy for a model to write.
|
||||
|
||||
A view can also hold a computation: a Lode claim's implementation, by claim and pin, run in the browser by Pyodide, with Vega-Lite parameters bound to its inputs.
|
||||
That is how a reader drags a bond constant and watches the energy curve change beside the equation that defines it.
|
||||
@@ -0,0 +1,57 @@
|
||||
# 18. Layers and annotations
|
||||
|
||||
Everything people and models add to a document is an annotation, and annotations are grouped in layers.
|
||||
`scienting-layer/1` is the W3C Web Annotation Data Model with the anchor as a selector, so any Web Annotation tool can read it.
|
||||
The JSON-LD context is [context/scienting-1.jsonld](../context/scienting-1.jsonld), published at `https://scient.ing/ns/scienting/1.jsonld`.
|
||||
|
||||
## 18.1 One rule decides what is a claim and what is a layer
|
||||
|
||||
If a statement is true independently of this paper, it belongs in Lode as a claim, where it is checked and leveled.
|
||||
If it is about reading this paper, it belongs in a layer, where it is attributed and signed but not checked.
|
||||
|
||||
## 18.2 The layer
|
||||
|
||||
A layer is a folder with `layer.json` and `annotations.jsonl`.
|
||||
`layer.json` is an `AnnotationCollection` with the two contexts, `format`, an `id` of the form `urn:scienting:layer:<store>/<record id>`, a `label`, a `purpose`, an `audience` (`novice`, `student`, `expert` or `mixed`), a `visibility` (`private`, `team` or `public`), a `license` (CC BY 4.0 by default when public), its `creator`, `created`, its `project`, the `docs` it annotates and its `total`.
|
||||
A reader turns layers on and off, and the audience dial picks the level of glosses.
|
||||
|
||||
## 18.3 The annotation
|
||||
|
||||
Each line of `annotations.jsonl` is a W3C `Annotation`:
|
||||
|
||||
| Property | Holds |
|
||||
| --- | --- |
|
||||
| `@context` | `["http://www.w3.org/ns/anno.jsonld", "https://scient.ing/ns/scienting/1.jsonld"]`. |
|
||||
| `id` | `urn:scienting:annotation:<store>/<record id>` ([4.6](04-identity.md#46-record-ids)). |
|
||||
| `type`, `layer` | `Annotation`, and the layer's ID. |
|
||||
| `motivation` | A W3C motivation: `commenting`, `describing`, `linking`, `questioning`, `assessing`, `replying` and the others. |
|
||||
| `creator`, `created`, `modified` | An agent `{id, type, name}`, with `type` `Person` or `Software`, and times. |
|
||||
| `body` | One body ([18.4](#184-bodies)). |
|
||||
| `target` | A `SpecificResource` whose `source` is `urn:scienting:doc:<store>/<key>` and whose `selector` lists an `AnchorSelector` holding a `scienting-anchor/1`, optionally followed by a W3C `TextQuoteSelector` for tools that do not know ours; or, for a reply, a `SpecificResource` whose source is another annotation. |
|
||||
| `status` | `proposed`, `confirmed`, `rejected` or `withdrawn`. A model's annotation is `proposed` until a person confirms it. |
|
||||
| `receipt` | For a confirmed annotation, the receipt that covers it. |
|
||||
|
||||
An annotation's content hash is SHA-256 of the canonical JSON of the annotation without `status`, `receipt` and `modified`.
|
||||
A confirmed annotation is covered by its confirmer's receipt, which lists `["annotation", <record id>, <content hash>]`.
|
||||
Replies are annotations that target annotations, so discussions are threads anchored to sentences.
|
||||
|
||||
## 18.4 Bodies
|
||||
|
||||
| `kind` | W3C type | Adds | Says |
|
||||
| --- | --- | --- | --- |
|
||||
| `gloss` | `TextualBody` | `audience` | An explanation at `novice`, `student` or `expert` level. |
|
||||
| `claim-link` | `SpecificResource` | `source` (`lode:<tag>`), `pin`, `role` | That the span `states`, `uses` or `disputes` a Lode claim. |
|
||||
| `symbol` | `SpecificResource` | `source`, `pin`, `symbol` | That this occurrence of a symbol is a Lode symbol claim. |
|
||||
| `comment` | `TextualBody` | | A remark, opinion or interpretation. |
|
||||
| `caveat` | `TextualBody` | | A warning a reader should see. |
|
||||
| `question` | `TextualBody` | `for` | A question for a `reader`, a `group` or the `authors`. |
|
||||
| `exercise` | `TextualBody` | `exercise` | A task with a key and a rule ([19](19-exercises.md)). |
|
||||
| `citation-context` | `TextualBody` | `reference`, `verdict`, `reason` | Why a sentence cites a reference, and whether it supports it. |
|
||||
| `prerequisite` | `SpecificResource` | `source` | What to read or know first. |
|
||||
| `crosslink` | `SpecificResource` | `source`, `selector` | That the span corresponds to a span in another document. |
|
||||
| `translation` | `TextualBody` | `language` | The span in another language. |
|
||||
| `generated` | `TextualBody` | `generator` | Text a model wrote about the span; never mistaken for source. |
|
||||
|
||||
A textual body has a `value`, and optionally a `format` (`text/plain` or `text/markdown`) and a `language`.
|
||||
Nothing a model writes is ever stored as source text: figure descriptions and drafted alt text are `generated` annotations.
|
||||
`fixtures/records/layer/` holds a layer with one annotation of each common kind and the receipt that confirms some of them.
|
||||
@@ -0,0 +1,42 @@
|
||||
# 19. Exercises, submissions and grades
|
||||
|
||||
Because every sentence and cell has a permanent address, the answer to a verification exercise is an address, and an address can be graded.
|
||||
|
||||
## 19.1 Exercises
|
||||
|
||||
An exercise is an annotation of kind `exercise` whose `exercise` property holds a `scienting-exercise/1`: an `exercise` kind, a `prompt`, a `key` and a `rule`.
|
||||
|
||||
| Kind | The student | The key |
|
||||
| --- | --- | --- |
|
||||
| `anchor` | Finds the sentence or cell that supports a statement. | `elements`: each a set of acceptable anchors, with points and feedback. |
|
||||
| `check` | Decides whether a cited passage supports the citing sentence, and anchors the passage. | A `verdict` with points, and `elements`. |
|
||||
| `trace` | Follows a value back to its original measurement. | A `chain`: an element per link, in order. |
|
||||
| `reproduce` | Digitizes a plot and recomputes a value. | A `value`, a `tolerance`, a `unit`, the confirmed `digitization` it comes from, and points. |
|
||||
| `verify` | Checks a model's `summary`, written with addresses, in which errors were planted. | `errors`: each a set of anchors and the correct verdict, with points. |
|
||||
|
||||
The rule's `match` is `same-node` (the student's anchor resolves to the node a key anchor resolves to), `overlap` (and, where the key has a span, the student's span covers at least `min_overlap` of it; default `0.5`), or `chain` (each link matches in order, with credit per link).
|
||||
Feedback is `{hit, miss}` per element.
|
||||
|
||||
## 19.2 Verdicts
|
||||
|
||||
| Verdict | Means | As Lode evidence, if a person confirms it |
|
||||
| --- | --- | --- |
|
||||
| `supports` | The passage says what the citing sentence claims. | `supports` |
|
||||
| `partly` | It says some of it, or with a qualification the citing sentence drops. | `qualifies` |
|
||||
| `does-not` | It does not say it, or says the opposite. | `disputes`, or no record when merely irrelevant; the reviewer chooses. |
|
||||
| `cannot-tell` | The passage, or the student's access to it, is not enough. | No record. |
|
||||
|
||||
## 19.3 Submissions
|
||||
|
||||
A submission, `scienting-submission/1`, holds the exercise's ID, the document and revision, a pseudonymous `student` ID, and `answers`: `anchors`, a `verdict`, a `chain` of anchor lists, a `value`, or `flags` (each an anchor and a verdict), as the kind needs.
|
||||
The student's name never enters a submission; the mapping from pseudonym to student stays in the course's records.
|
||||
|
||||
## 19.4 Grading
|
||||
|
||||
Before grading, every anchor in the key is resolved, and if any is stale nothing is graded with it.
|
||||
Each element of the key earns its points when any of the student's anchors matches any of its anchors under the rule; a `check` earns its verdict's points when the verdicts are equal; a `reproduce` earns its points when the value is within the tolerance; a `verify` earns an error's points when a flag matches it with the correct verdict.
|
||||
Points are decimal strings, summed exactly.
|
||||
The result is a `scienting-grade/1`: the submission, the exercise, the `score`, the `max`, and an item per element with its points, its maximum and its feedback.
|
||||
`grading.jsonl` grades a hit, a miss and an overlap.
|
||||
|
||||
Student submissions are educational records, so they live in the course's private store, never on a public log.
|
||||
@@ -0,0 +1,16 @@
|
||||
# 20. Projects
|
||||
|
||||
A project, `scienting-project/1`, is the unit a team works in.
|
||||
|
||||
| Part | Holds |
|
||||
| --- | --- |
|
||||
| `key`, `store`, `name`, `description` | Its identity. |
|
||||
| `members` | People with a `role` (`owner`, `editor`, `reader` or `guest`) and the roster keys they sign with. |
|
||||
| `documents` | The project's documents, each with the `reason` it is included, `tags`, and a reading `status` per member: `unread`, `reading` or `read`. |
|
||||
| `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group. |
|
||||
| `layers`, `datasets`, `views`, `drafts` | What the project holds. |
|
||||
| `claims` | The Lode namespace or bundle it uses. |
|
||||
| `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), and the stage requirements in use. |
|
||||
|
||||
Projects are small on purpose: everything about them is designed for hundreds of documents, not millions.
|
||||
Owners sign policies ([14.2](14-rights.md#142-effective-uses)) and may revoke receipts ([8.7](08-fidelity.md#87-revocations-and-the-roster)).
|
||||
@@ -0,0 +1,17 @@
|
||||
# 21. Drafts
|
||||
|
||||
A draft is a document of kind `draft`, established from a Typst project as a born-digital witness, so every sentence is F2 from the start.
|
||||
Its sentences cite library sentences by address, in `xref` runs whose targets are addresses in other documents, and its works in `cite` runs whose references resolve to documents.
|
||||
|
||||
## 21.1 The snapshot
|
||||
|
||||
A snapshot, `scienting-snapshot/1`, is to a Typst draft what `lode.lock` is to a Quarto document: a pinned, verified file the compile reads, so a draft renders offline and reproducibly.
|
||||
It holds the `store`, when it was made, the `quotes` the draft cites (each an address, an anchor, the text, its fidelity and how the anchor resolved), the `claims` (each a reference and pin, with whatever renderings the draft uses) and the `datasets` (each an ID and content hash).
|
||||
|
||||
## 21.2 Sentence IDs in a draft
|
||||
|
||||
Authors rewrite constantly, and IDs must survive without cluttering the source.
|
||||
IDs live in a sidecar, `ids.jsonl` in the Typst project, `scienting-draft-ids/1`: one line per sentence with its `id`, its `text_hash`, its `section` and its `ord`, and `pinned` when the author fixed the ID to a label.
|
||||
Each conversion aligns the new sentences with the previous revision, exact text first and then by edit distance within the section; an edited sentence keeps its ID and gets a new hash, so its annotations reattach or go stale as [6.5](06-anchors.md#65-resolution) says.
|
||||
|
||||
Both formats are provisional until the Typst package exists.
|
||||
@@ -0,0 +1,53 @@
|
||||
# 22. Conformance
|
||||
|
||||
## 22.1 The suite
|
||||
|
||||
| Path | Holds | A conforming implementation |
|
||||
| --- | --- | --- |
|
||||
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
|
||||
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document. | Accepts each, and recomputes every value it contains. |
|
||||
| `fixtures/records/` | A valid record of every other format. | Accepts each. |
|
||||
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
|
||||
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |
|
||||
|
||||
All fixture text is synthetic and dedicated to the public domain; license fields inside fixtures describe the fictional works.
|
||||
Each part's CI runs this suite against its own implementation, so the parts cannot drift apart silently.
|
||||
|
||||
## 22.2 Validating a package
|
||||
|
||||
A validator checks, in order, and reports each failure as `path:line: rule: message`:
|
||||
|
||||
1. The manifest: present, valid, sorted, every listed file present with its hash and size, nothing unlisted, paths safe, required files present.
|
||||
2. Every file in canonical form, and every record against its schema.
|
||||
3. The tree ([3](03-nodes.md)), runs and references between nodes.
|
||||
4. The policy and hashing scheme ([12.5](12-package.md#125-text-policies)).
|
||||
5. The five values of every node, the root and `meta_hash`, when the text is present and the key, if any, is known.
|
||||
6. The ID table, the corrections, the witnesses, pages, locations and references, and every file's order.
|
||||
7. Receipts: IDs, roots, documents, node order, signatures against the package's roster, spot-check keys, seeds and samples, and the fidelity distribution.
|
||||
|
||||
## 22.3 Schemas
|
||||
|
||||
Schemas use draft 2020-12, restricted to the keywords `$schema`, `$id`, `$ref`, `$defs`, `$comment`, `title`, `description`, `examples`, `default`, `deprecated`, `type`, `enum`, `const`, `properties`, `required`, `additionalProperties`, `patternProperties`, `propertyNames`, `minProperties`, `maxProperties`, `dependentRequired`, `items`, `prefixItems`, `minItems`, `maxItems`, `uniqueItems`, `contains`, `minLength`, `maxLength`, `pattern`, `format`, `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then` and `else`.
|
||||
Patterns avoid `\d`, `\w`, `\s`, `\b` and lookbehind, whose meaning differs between regular-expression dialects.
|
||||
A schema checks shape; the chapters' rules, such as the five values, are checked by validators.
|
||||
|
||||
## 22.4 Rules
|
||||
|
||||
[registry/rules.json](../registry/rules.json) lists every rule ID with its meaning.
|
||||
Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is never reused.
|
||||
|
||||
| Prefix | Covers |
|
||||
| --- | --- |
|
||||
| `package.` | The manifest, paths, files and schemas. |
|
||||
| `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.` | The tree, node content and runs. |
|
||||
| `doc.hash.` | The five values, the root and the metadata hash. |
|
||||
| `doc.id.`, `doc.order.` | The ID table and file order. |
|
||||
| `doc.policy.` | Text policies and keyed hashing. |
|
||||
| `doc.correction.`, `correction.` | A package's corrections, and the gate. |
|
||||
| `doc.receipt.`, `doc.qa.`, `doc.fidelity.` | Receipts, spot-checks and levels. |
|
||||
| `doc.witness.`, `doc.page.`, `doc.location.`, `doc.reference.` | Witnesses and geometry. |
|
||||
| `grade.` | Grading. |
|
||||
|
||||
## 22.5 Freezing 1.0
|
||||
|
||||
These specifications become 1.0 when the formats have been written by at least three producers (conversion, hand-built fixtures, and a structured witness such as JATS) and read by at least three consumers (databases, Lode and the reader), and at least one paying buyer has used a delivery built on them.
|
||||
@@ -0,0 +1,31 @@
|
||||
# The scienting specifications
|
||||
|
||||
These chapters are the normative text of `spec` 0.1.
|
||||
Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are normative too, and the chapters say which.
|
||||
Where a chapter and a vector disagree, the disagreement is a bug in this repository; report it.
|
||||
|
||||
| Chapter | Subject |
|
||||
| --- | --- |
|
||||
| [00](00-introduction.md) | Scope, conformance, versions and status |
|
||||
| [01](01-conventions.md) | Canonical JSON, numbers, hashes, normalization, offsets, times, IDs, keys and files |
|
||||
| [02](02-model.md) | Works, versions, witnesses and documents; `document.json` and `witnesses.json` |
|
||||
| [03](03-nodes.md) | Nodes, their kinds and children, content, attributes and inline runs |
|
||||
| [04](04-identity.md) | Minting IDs, permanence, retirement, aliases and record IDs |
|
||||
| [05](05-hashes.md) | The five content values, the projections, alignment, suspects and keyed hashes |
|
||||
| [06](06-anchors.md) | Addresses, anchors and the five resolutions, with drift |
|
||||
| [07](07-geometry.md) | Page fingerprints, stamp rules, pages and locations |
|
||||
| [08](08-fidelity.md) | Fidelity levels, receipts, the roster, revocations, spot-checks and error bounds |
|
||||
| [09](09-figures-tables-equations.md) | Figures, panels, tables, cells and equations |
|
||||
| [10](10-references.md) | References and citations |
|
||||
| [11](11-corrections.md) | Corrections and the gate |
|
||||
| [12](12-package.md) | The package |
|
||||
| [13](13-database.md) | The database form and the bundle |
|
||||
| [14](14-rights.md) | Rights records, policies, stages and backends |
|
||||
| [15](15-datasets.md) | Datasets |
|
||||
| [16](16-digitization.md) | Digitization records |
|
||||
| [17](17-views.md) | Views |
|
||||
| [18](18-layers.md) | Layers and annotations |
|
||||
| [19](19-exercises.md) | Exercises, submissions and grades |
|
||||
| [20](20-projects.md) | Projects |
|
||||
| [21](21-drafts.md) | Drafts: snapshots and sentence-ID sidecars |
|
||||
| [22](22-conformance.md) | Conformance, the suite and the rule catalog |
|
||||
Reference in new issue
Block a user