infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First implementation (#1)
check / check (push) Successful in 42s
Details
site / site (push) Successful in 3m4s
Details

Browse Source 
Reviewed-on: #1
This commit was merged in pull request #1.
This commit is contained in:
alexm committed 2026-10-15 05:47:05 -04:00
1 parent 34c0abb26a
commit 4fa326dc76
534 files changed
+28280 -37
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

spec/00-introduction.md
+89
View File
@@ -0,0 +1,89 @@
# 0. Introduction
This chapter is for three readers.
A newcomer, who knows what a scientific paper and a citation are, finds what the specification covers and how to read it.
An implementer, who will write a reader, writer, converter or checker, finds the rules for conformance, identifiers, versions and status.
An expert, who knows the neighboring standards, finds the status of each format and how the specification relates to those standards.
## 0.1 Scope
Tools that read, correct, annotate and check papers can work together only if they agree on the files they exchange.
The specification defines that data: documents and their packages, anchors, receipts, rights records, layers, exercises, datasets, digitizations, views, projects and bundles.
A package is a document stored as a set of files; an anchor is a reference to a place in a document that stays checkable when the document changes.
It defines data, not programs.
A conforming file can be read, checked and loaded without any particular program, and every rule is stated so that a second implementation, in any language, computes the same bytes.
The specification does not replace the standards around it.
JATS ([ANSI/NISO Z39.96-2024](https://www.niso.org/publications/z3996-2024-jats)) remains the format for publishers and archives to exchange journal articles; here it is only one kind of input to a document.
Layers are [W3C Web Annotations](https://www.w3.org/TR/annotation-model/) with a custom selector for the anchor. [docs/comparison.md](../docs/comparison.md) compares the specification with the standards near it and says where each is the better choice.
The [guides](../docs/README.md) explain what each format is for, and the [rationale](../docs/rationale.md) why its rules are what they are.
## 0.2 Conformance language
The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY and OPTIONAL are to be interpreted as described in RFC 2119 and RFC 8174 when, and only when, they appear in capitals.
The suite is the set of test vectors (inputs paired with the outputs every implementation must compute) and fixtures (example files, valid and invalid) in the repository. [22](22-conformance.md) describes it.
The specification distinguishes three classes of implementation.
| Class | Does | Must |
| --- | --- | --- |
| Producer | Writes files in a format. | Write only files that validate, with values computed as specified. |
| Consumer | Reads files in a format. | Accept every valid file, and refuse every file the suite lists as invalid, naming the rule. |
| Validator | Decides whether a file is valid. | Check every rule in [22](22-conformance.md) that the file's contents allow it to check. |
An implementation states, for each format, which versions it reads and writes.
## 0.3 Format identifiers and versions
Every format has an identifier of the form `scienting-<name>/<major>`, such as `scienting-doc/1`.
The identifier tells a reader which rules apply to a file.
The prefix `scienting-` is the namespace of the formats.
The namespace is not the name of this repository, paperspec, and formats are named after what they hold.
A file or record that can travel on its own carries its identifier in a `format` property.
The anchor in `fixtures/records/anchor-leaf.json` begins `{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1", ...`.
Records inside a package file whose format the manifest fixes, such as the lines of `nodes.jsonl`, do not repeat it.
A change that makes a valid file invalid, or that changes a computed value, is a breaking change and needs a new major version, with a migration.
Adding an optional property, a node kind, a run kind, or an enumeration value is a minor change.
A consumer of version *n* MUST refuse a major version it does not know, and SHOULD warn on, but accept, enumeration values added in a later minor version.
The specification as a whole is versioned with semantic versioning; this is version 0.1.0. [registry/formats.json](../registry/formats.json) lists every format with its media type, schema and status.
## 0.4 Status
Each format has one of three statuses, listed in [registry/formats.json](../registry/formats.json).
In version 0.1.0 the registry lists 32 formats: 19 are `candidate` and 13 are `provisional`; none is `reserved`.
| Status | Meaning |
| --- | --- |
| `candidate` | Complete, with vectors and fixtures, and ready to implement. |
| `provisional` | Complete, but expected to change as implementations gain experience with it. |
| `reserved` | Named and shaped, but not yet specified in full. |
Some fields matter only to some users: a personal library has no use for spot-checks, and a team that reads only open-access papers rarely needs a policy.
Those fields are defined anyway, so that files move between a personal library and a team's store without conversion.
A consumer that does not use a field MUST still preserve it when it rewrites a file.
## 0.5 How the repository is arranged
A normative folder defines what conforming means.
The other folders help people use the specification.
| Path | Holds | Normative |
| --- | --- | --- |
| `spec/` | These chapters. | Yes |
| `schemas/` | JSON Schemas, draft 2020-12. | Yes |
| `sql/` | Table definitions for SQLite, Dolt and PostgreSQL, and the SQLite bundle. | Yes |
| `context/` | The JSON-LD context for layers. | Yes |
| `registry/` | Formats, rule IDs, stamp rules and the default stage requirements. | Yes |
| `vectors/` | Test vectors, one case per line. | Yes |
| `fixtures/` | Valid packages and records, invalid packages, and the keys that signed them. | Yes |
| `tools/` | The programs that compute and check the vectors and fixtures, and load packages into SQLite. | No |
| `docs/` | Guides for people getting to know the formats. | No |
The programs in `tools/` compute every expected value, so that none is typed by hand.
They are not a library, and implementations conform to the chapters and the vectors.
spec/01-conventions.md
+95
View File
@@ -0,0 +1,95 @@
# 1. Shared conventions
These rules apply to every format.
## 1.1 Canonical JSON
Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with `\b`, `\t`, `\n`, `\f`, `\r`, `\"` and `\\` for those characters and `\u00xx` with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
Strings are not normalized: canonical JSON keeps every code point as given.
A document containing a lone surrogate, `NaN` or an infinity has no canonical form and is invalid.
Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
The vectors in `canonical-json.jsonl` give JSON text to parse as IEEE 754 doubles and the canonical output.
A store may keep its own canonical form for its stored hashes; that form is outside this text.
## 1.2 Numbers
A decimal that carries meaning is a string, so that `310.0` and `1.5260` keep their printed digits.
The schemas call this a `decimal`: an optional minus sign, an integer part without leading zeros, and an optional fraction.
JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
Geometry is in PDF points.
A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
## 1.3 Hashes
A hash is SHA-256, written as 64 lowercase hexadecimal digits.
Six digits are for display and for pins in addresses; no record stores a shortened hash.
`document.json` names the function in `hashing.algorithm`, `sha-256` in this version, and every hash in the package, its records and its proofs uses the one function the document names; a consumer refuses an algorithm it does not implement (`doc.hash.algorithm`).
Property names such as `sha256` and `page_fp` belong to that function, so a later algorithm arrives as a new major version of the formats whose values it changes, with the name in the data letting two versions share a store.
Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as [5.8](05-hashes.md#58-keyed-hashes) defines.
## 1.4 Text normalization for hashing
To normalize a string for hashing:
1. Apply Unicode Normalization Form C.
2. Replace every maximal run of whitespace characters with one U+0020.
3. Remove a leading and a trailing U+0020.
Whitespace means the 25 characters with the Unicode property `White_Space`: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
U+200B ZERO WIDTH SPACE is not whitespace.
Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
The vectors in `normalize.jsonl` cover the cases, including the angstrom sign, which NFC turns into Å.
Normalization forms, General Categories and the `White_Space` property are those of Unicode 16.0.
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference tools refuse to run on another, and moving the formats to a later version is a change to every format that hashes text ([0.3](00-introduction.md#03-format-identifiers-and-versions)).
## 1.5 Character offsets
Offsets count Unicode code points, never UTF-16 code units or bytes.
An offset pair `[start, end]` names the code points from `start` up to but not including `end`.
A browser implementation converts to UTF-16 at its edge; `offsets.jsonl` gives both forms for text with characters outside the Basic Multilingual Plane.
## 1.6 Times
A time is UTC, written `YYYY-MM-DDTHH:MM:SSZ`, with no fractions.
A date is `YYYY`, `YYYY-MM` or `YYYY-MM-DD`.
## 1.7 Determinism
Every value these specifications define is a pure function of its inputs.
Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently.
From the cached readings onward, the same inputs and tool versions MUST give byte-identical files.
No file carries the time it was written, unless the time is part of the record, as a receipt's is.
## 1.8 IDs and keys
| Kind | Form | Pattern |
| --- | --- | --- |
| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$` |
| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | `^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$` |
| Work key | A Scholar-style part, a hyphen and 4 to 51 characters of Crockford base32 ([2.4](02-model.md#24-work-and-document-keys)). At most 128 characters. | `^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}$` |
| Document key | A work key, with a dotted suffix for a version other than the version of record. At most 128 characters. | `^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\.[a-z0-9][a-z0-9-]*)*$` |
| Key | A store, project or source key, or a legacy document key kept as an alias. At most 128 characters. | `^[a-z0-9][a-z0-9._-]*$` |
| Alias | A legacy ID; it always holds a `.`, `_` or `-`, so it never looks like a node ID. | `^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$` |
| Dataset ID | `ds-` and a key. | `^ds-[a-z0-9][a-z0-9._-]*$` |
The Crockford alphabet is `0123456789abcdefghjkmnpqrstvwxyz`; it has no `i`, `l`, `o` or `u`.
Its 22 letters are `abcdefghjkmnpqrstvwxyz`.
Because the first character is a letter, every ID is a valid HTML `id`, CSS selector and URL fragment without escaping.
A key qualified by its store, `smithlab/cornell1995second`, names a document anywhere.
## 1.9 Files
JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A.
JSON files hold one value in canonical JSON followed by one U+000A.
So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence.
Each JSON Lines file has a defined order, given where the file is defined.
## 1.10 Prose
Prose in these formats, and in this repository, is US English, one sentence per line.
spec/02-model.md
+95
View File
@@ -0,0 +1,95 @@
# 2. Works, versions, witnesses and documents
## 2.1 Four objects
| Object | Is | Identified by |
| --- | --- | --- |
| Work | The intellectual work, across all its versions. | A work key ([2.4](#24-work-and-document-keys)), and `same_as` identifiers: `doi:`, `arxiv:`, `isbn:`, `pmid:`, `pmcid:`, `openalex:`, `url:` or `lode:`. |
| Version | One expression of the work: a preprint, an accepted manuscript, the version of record, an erratum, Supporting Information, a draft. | The work's key and a version type, with an optional sequence number and date. |
| Witness | One file of a version: a PDF, JATS XML, a LaTeX source archive, an HTML page, an EPUB, a Typst project. | The SHA-256 of its bytes. |
| Document | The text established for one version from one or more witnesses. | A key, and a root hash per revision. |
A document's key is its work's key, with a suffix for anything other than the version of record: `cornell1995second-saqk`, `cornell1995second-saqk.si`, `dong2026machine-1ard.v2`.
Supporting Information is its own document, with its own pages, numbering and references, related to its article by `supplement-of`.
A textbook is a work whose chapters are documents related by `part-of`.
A draft is a document of kind `draft`.
Keys are immutable once a document is shared.
A renamed document keeps its old keys in `aliases`.
## 2.2 `document.json`
| Property | Required | Holds |
| --- | --- | --- |
| `format` | Yes | `scienting-doc/1`. |
| `key` | Yes | The document key. |
| `store` | No | The store that holds it, when the package says so. |
| `aliases` | No | Earlier keys. |
| `kind` | Yes | `article`, `supplement`, `chapter`, `book`, `thesis`, `report`, `draft`, `course-material` or `other`. |
| `work` | Yes | `{key, same_as}`. |
| `version` | Yes | `{type, seq?, date?}`, where `type` is `preprint`, `submitted`, `accepted`, `version-of-record`, `erratum`, `retraction`, `supplement`, `draft` or `other`. |
| `relations` | Yes | A list of `{type, target}`; `type` is `supplement-of`, `part-of`, `version-of`, `erratum-of`, `retraction-of` or `translation-of`, and `target` a key, optionally store-qualified. |
| `meta` | Yes | The version's bibliographic record, as one CSL-JSON item. |
| `language` | Yes | A BCP 47 tag. |
| `license` | No | The document's license as stated: `{spdx?, text?, url?}`. Rights are per witness ([14](14-rights.md)). |
| `policy` | Yes | The text policy: `full`, or `checks-only` ([12.5](12-package.md#125-text-policies)). |
| `hashing` | Yes | How content values are computed: `{scheme: "plain"}`, a team HMAC or a page-derived key ([5.8](05-hashes.md#58-keyed-hashes)). |
| `root` | Yes | The root node's `node_hash`. |
| `meta_hash` | Yes | The metadata hash ([5.9](05-hashes.md#59-the-metadata-hash)). |
| `history` | Yes | The history root: the root of the tree over the package's corrections ([11.5](11-corrections.md#115-the-history-root)). |
| `rev` | Yes | The revision: the index of the last correction applied, or 0. |
| `counts` | Yes | `{nodes, leaves}`. |
| `fidelity` | Yes | `{lowest, counts}`: the lowest level among leaves, or null when there are none, and the number of leaves at each level, computed as [8.2](08-fidelity.md#82-levels) defines. |
The schema is `document.schema.json`.
## 2.3 `witnesses.json`
`witnesses.json` is an object with one property, `witnesses`, a non-empty list in which each witness is:
| Property | Required | Holds |
| --- | --- | --- |
| `id` | Yes | The witness ID: seven letter-first base32 characters encoding the SHA-256, as [4.6](04-identity.md#46-record-ids) defines. |
| `sha256` | Yes | The SHA-256 of the file. |
| `size` | Yes | Its size in bytes. |
| `kind` | Yes | `pdf`, `jats`, `latex`, `html`, `epub`, `typst`, `xml`, `image` or `other`. |
| `media_type` | Yes | Its media type. |
| `pages` | No | The number of pages, for a witness that has pages. |
| `text_layer` | No | Whether a PDF has a text layer; a scan has none. |
| `version_label` | No | The version as the witness labels it. |
| `origin` | Yes | `{source, url?, doi?, retrieved_at?, note?}`, where `source` is `publisher`, `repository`, `preprint-server`, `author`, `team`, `licensed-delivery`, `library` or `other`. |
| `rights` | Yes | The rights record, `scienting-rights/1` ([14](14-rights.md)). |
The witness's bytes are not part of the package; they live in the team's asset store, by SHA-256.
The SHA-256 is the exact identity of one file.
Because publishers stamp each download, two copies of one paper rarely share it; page fingerprints identify their pages instead ([7](07-geometry.md)).
## 2.4 Work and document keys
A work key names a work the way a reader would, and never collides: a Scholar-style citation key, a hyphen, and the first characters of a hash of the work's identity.
```text
vaswani2017attention-a241
└─── Scholar part ──┘ └hash┘
```
**The Scholar part** is built as Google Scholar builds its BibTeX keys, from the CSL-JSON `meta` of the version:
1. the first author's family name: `family`, or the part of a `literal` name before a comma, else its last word that is not a suffix such as Jr. or III; folded to ASCII with NFKD, lower-cased, letters only, so `Müller` gives `muller` and `O'Brien` gives `obrien`; no author, or a name with no Latin letters, gives `anon`;
2. the year of `issued`, when it has four digits, else nothing;
3. the first word of `title` that is not a stop word, folded the same way, letters and digits only, with apostrophes removed and other punctuation splitting words; no such word gives nothing.
The stop words are `a an the of on in for to and or at by with from as into onto about toward towards via is are be its it this that these those`.
**The identity** is the work's first identifier among `doi`, `arxiv`, `isbn`, `pmid`, `pmcid` and `openalex` in `same_as`, in that order, written `<kind>:<value>` with a DOI's ASCII letters lower-cased.
A work with none has the identity `meta:<title words>|<year>|<family>` from the same folded parts.
`url:` and `lode:` identifiers name copies and claims, not the work, and never enter the identity.
**The hash part** is the leading bits of SHA-256 of `scienting-key/1:` followed by the identity, in UTF-8, written in lowercase Crockford base32, five bits a character: four characters, or, when another work in the store already holds that key, six, eight and so on until the key is free.
A key is minted once, when the work enters a store, and never recomputed: a corrected title or a year revised later leaves it alone, while the hash of an identifier still shows which work the key names.
A work key matches `^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}$`.
**The document key** is the work key for the version of record, and the work key with a suffix for any other version: `.si` for Supporting Information, `.v<seq>` for a preprint, submitted, accepted or draft version with a sequence number, else a dot and the version type: `cornell1995second-saqk`, `cornell1995second-saqk.si`, `dong2026machine-1ard.v2`.
A validator checks that the document key is the work key with the version's suffix (`doc.key.work`), and that a key whose identity is an identifier carries that identifier's hash (`doc.key.suffix`); a metadata identity may have been corrected since the key was minted, so it is not held to the key.
Store, project and source keys keep the form `^[a-z0-9][a-z0-9._-]*$`, as do legacy document keys kept in `aliases`.
`keys.jsonl` pins the Scholar parts, identities, hashes and document keys, with accented names, name suffixes, missing years and authors, and the lengthening that settles a collision.
spec/03-nodes.md
+127
View File
@@ -0,0 +1,127 @@
# 3. Nodes
A document is a tree of nodes.
`nodes.jsonl` holds one node per line, in document order: the root first, then each node before its children, and siblings in order.
## 3.1 The node record
| Property | Required | Holds |
| --- | --- | --- |
| `id` | Yes | The node ID ([4](04-identity.md)). |
| `kind` | Yes | One of the kinds in [3.2](#32-kinds-and-children). |
| `parent` | Yes | The parent's ID, or null for the root. |
| `ord` | Yes | The position among its siblings, from 0. |
| `label` | No | The printed label (`2.3`, `Figure 2`, `(1)`, `a`), for kinds that take one. |
| `attrs` | No | Structural attributes, by kind ([3.4](#34-attributes)). |
| `content` | No | The node's own content, by kind ([3.3](#33-content)). |
| `withheld` | No | `true` when the policy withholds the content ([12.5](12-package.md#125-text-policies)). |
| `fidelity` | No | For leaves, the level the pipeline established: `F0`, `F1` or `F2` ([8.1](08-fidelity.md#81-levels-of-fidelity)). |
| `prov` | No | `{witnesses, backend?, reading?, corroborated_by?}`: the witness IDs the text came from, the backend, and the witnesses that agree. |
| `text_hash`, `match_hash`, `align_key`, `own_hash`, `node_hash` | Yes | The five content values ([5](05-hashes.md)). |
Optional properties are omitted, never written as null.
The schema is `node.schema.json`.
## 3.2 Kinds and children
| Kind | May contain | Content | Label |
| --- | --- | --- | --- |
| `document` | `front`, `section`, `back` | | |
| `front` | `title`, `authors`, `abstract`, `keywords`, `notes` | | |
| `title` | `sentence` | | |
| `authors` | | runs | |
| `abstract` | blocks, `section` | | |
| `keywords` | | runs | |
| `notes` | `footnote`, `paragraph` | | |
| `section` | `heading` first, then blocks and `section` | | Yes |
| `heading` | `sentence` | | |
| `paragraph` | `sentence` | | |
| `sentence` | `equation` | runs | |
| `equation` | `equation-line` | math | Yes |
| `equation-line` | | math | Yes |
| `figure` | `panel`, `caption`, `figure-note` | | Yes |
| `panel` | | | Yes |
| `caption` | `sentence` | | |
| `figure-note` | `sentence` | | |
| `table` | `caption`, `table-head`, `table-body`, `table-note` | | Yes |
| `table-head`, `table-body` | `row` | | |
| `table-note` | `sentence` | | |
| `row` | `cell` | | |
| `cell` | | runs | |
| `list` | `item` | | |
| `item`, `quote` | blocks | | |
| `code`, `algorithm` | | verbatim | Yes |
| `theorem` | blocks | | Yes |
| `footnote` | `sentence` | | Yes |
| `back` | `acknowledgments`, `references`, `appendix`, `notes` | | |
| `acknowledgments` | blocks | | |
| `references` | `reference` | | |
| `reference` | | runs | Yes |
| `appendix` | `section` | | |
Blocks are `paragraph`, `equation`, `figure`, `table`, `list`, `quote`, `code`, `algorithm`, `theorem` and `footnote`.
These rules also hold:
1. Exactly one node is a root, and it is a `document`.
2. A `heading` is a section's first child, and a section has at most one.
3. `front`, `back`, `title`, `authors`, `abstract`, `keywords`, `caption`, `table-head`, `table-body`, `acknowledgments` and `references` occur at most once among their siblings.
4. A `table` has exactly one `table-body`.
5. `title`, `heading`, `paragraph`, `caption`, `figure-note`, `table-note`, `table-head`, `table-body`, `row`, `list`, `item`, `quote`, `theorem`, `footnote` and `appendix` have at least one child.
6. Only the kinds marked in the table take a `label`.
7. A body without headings, as in a letter, is a section with no heading.
Kinds outside this list are refused; a new kind is a minor version.
Page breaks, running heads and columns are not nodes: they are geometry ([7](07-geometry.md)).
## 3.3 Content
| Content | Kinds | Shape |
| --- | --- | --- |
| Runs | `sentence`, `cell`, `reference`, `authors`, `keywords` | A non-empty list of inline runs ([3.5](#35-inline-runs)). |
| Math | `equation`, `equation-line` | `{notation, source}`: `tex` or `typst`, and the source as written. |
| Verbatim | `code`, `algorithm` | `{language?, text}`. |
Other kinds have no `content`.
Derived renderings, such as MathML, are not content; they live in `derived/` ([12.7](12-package.md#127-derived-files)).
## 3.4 Attributes
| Kind | Attribute | Holds |
| --- | --- | --- |
| `section` | `level` (required) | 1 for a section, 2 for a subsection, and so on. |
| `table` | `columns` | The column definitions: `[{id, header_path}]`, where `header_path` lists header cell IDs, outermost first ([9.2](09-figures-tables-equations.md#92-tables)). |
| `cell` | `column`, `rowspan`, `colspan`, `role` | The column's ID, spans (default 1), and `data` (default), `column-header`, `row-header` or `row-group-header`. |
| `row` | `header_path` | The cell IDs that label the row, outermost first. |
| `list` | `ordered` | Whether the list is numbered. |
| `item` | `marker` | The printed marker. |
| `theorem` | `variant` | The printed kind: Theorem, Lemma, Definition, Proof, Remark. |
| `panel` | `image`, `crop`, `figure_text` | The image asset, a crop, and the text found inside the panel ([9.1](09-figures-tables-equations.md#91-figures-and-panels)). |
No other kind carries `attrs`, and each kind carries only its own attributes.
## 3.5 Inline runs
| Run | Properties | Plain projection |
| --- | --- | --- |
| `text` | `text`, and optional `style`: a sorted list of distinct values from `em`, `strong`, `sup`, `sub`, `sc`, `mono`, never both `sup` and `sub`. | The text. |
| `math` | `notation` (`tex` or `typst`), `source`. | `$` + source + `$` |
| `cite` | `refs`: reference node IDs; `text`: the printed form, such as `23`, `[1,2]` or `(Smith, 2019)`. | The printed form. |
| `xref` | `target`: a node ID in this document, an address in another, or null when not yet resolved; `text`. | The printed form. |
| `fnmark` | `footnote`: a footnote's ID, or null; `text`: the mark. | The mark. |
| `link` | `url`, `text`. | The text. |
| `equation` | `node`: the ID of an equation child of the sentence. | `$$` + that equation's source + `$$` |
Runs are canonical: no `text` run is empty, and no two adjacent `text` runs have the same style list.
A sentence's `equation` runs name its `equation` children, each once, in the children's order, and a sentence has no other children.
A display equation the sentence runs through is thus part of the sentence, while keeping its own ID and hashes.
## 3.6 The plain projection
The plain projection of a node with runs is the concatenation of its runs' projections.
That of an equation or an equation line is `$$` + source + `$$`, and that of a verbatim node is its text.
That of any other node is the plain projections of its text leaves joined with one U+0020.
A node's text leaves are found by walking its subtree in document order and stopping at `sentence`, `cell`, `reference`, `authors`, `keywords`, `code` and `algorithm` nodes, and at `equation` nodes whose parent is not a sentence; an `equation-line` is never a text leaf.
Offsets, quotations and the text hash are all defined on the plain projection as stored, before normalization.
spec/04-identity.md
+73
View File
@@ -0,0 +1,73 @@
# 4. Identity
A node's ID is a name, not a position.
It carries no meaning, and once a document is established it is never recomputed, reassigned or reused.
## 4.1 Letter-first base32
To encode a SHA-256 digest *d* (32 bytes) as a letter-first ID of *n* characters:
1. The first character is the letter at index `d[0] mod 22` of `abcdefghjkmnpqrstvwxyz`.
2. Read the bytes `d[1]` to `d[31]` as one big-endian bit string.
3. Each further character is the Crockford character for the next five bits, from the most significant end.
Node IDs have seven characters, about 34.5 bits; derived record IDs have 13.
`ids.jsonl` in the vectors gives digests and their encodings.
## 4.2 Minting node IDs
A node's ID is minted from SHA-256 of the canonical JSON of the array `["scienting-id/1", doc_key, kind, text_hash, occurrence]`, encoded with seven characters.
`text_hash` is the plain, never keyed, text hash of the node ([5.1](05-hashes.md#51-the-five-values)).
When a document is established, nodes are minted in document order.
A node's occurrence starts at the number of earlier nodes with the same kind and text hash; while the resulting ID is already taken, the occurrence increases by one.
So a fresh conversion of the same input mints the same IDs.
A node added later, by an `insert` or a `split`, starts at occurrence 0 and increases while the ID is taken by any live, retired or alias ID of the document.
Columns and figure-text items are not nodes, but they take IDs from the same rule and the same document-wide set, after the nodes:
a column with kind `column` and the text hash of its header cells' plain projections joined with U+0020;
a figure-text item with kind `figure-text` and the text hash of its text.
## 4.3 Permanence, retirement and aliases
| Rule | Detail |
| --- | --- |
| Permanence | Once established, an ID never changes, and no other node ever takes it. |
| Retirement | A `merge` or a `delete` retires IDs ([11](11-corrections.md)). A merge records the surviving node as the successor; a delete records none. |
| Aliases | Legacy IDs, such as the `b-p03-text-004` block IDs an earlier extract format used, are recorded as aliases of the node that now holds that text. |
| Labels | What people say, such as "Eq. 3" or "Table S1", is a `label`, never an ID. |
| Display paths | Tools may show a path such as `§2.3 ¶4 s2`; it is never stored or cited. |
## 4.4 `ids.jsonl`
Each line is one entry.
| Property | Holds |
| --- | --- |
| `id` | The retired node ID, or the alias. |
| `kind` | `retired` or `alias`. |
| `successor` | For a retirement, the successor's ID, or null for a deletion; for an alias, the live node it names. |
| `node_kind` | For a retirement, the kind the node had. |
| `rev` | For a retirement, the index of the correction that retired it. |
Aliases come first, sorted by ID; retirements follow, sorted by `rev` and then ID.
No ID is both live and in the table, every successor is live or itself in the table, and every alias names a live node.
## 4.5 Store-qualified names
A document key qualified by its store, `smithlab/cornell1995second`, names a document anywhere, and an address with a store names a node anywhere ([6.1](06-anchors.md#61-addresses)).
## 4.6 Record IDs
A record that is never edited takes its ID from its own content, as 13 letter-first characters of a hash:
| Record | Hash |
| --- | --- |
| Receipt, revocation, policy | The record hash: SHA-256 of the record without `id` and `signature` ([8.4](08-fidelity.md#84-record-hashes-and-signatures)). |
| Correction | SHA-256 of the record without `id`. |
| Annotation | SHA-256 of the annotation without `id`, `status`, `receipt` and `modified`, as created. |
| Submission, digitization | SHA-256 of the record without `id`. |
A witness ID is the seven-character letter-first encoding of the witness's SHA-256 itself.
Other records, such as layers and qa reports, MAY use any record ID unique in their store.
spec/05-hashes.md
+196
View File
@@ -0,0 +1,196 @@
# 5. Content values
Each node carries five content values, each for one job.
No value covers the node's own ID, its geometry, its provenance or its fidelity.
So re-witnessing a page or proofreading a sentence changes none of them, while changing one character changes the `text_hash` and `node_hash` of that sentence.
In this chapter *H* is SHA-256 written as hexadecimal; under a keyed scheme, *H* is HMAC-SHA-256 under the key ([5.8](#58-keyed-hashes)).
## 5.1 The five values
| Value | Definition | Used for |
| --- | --- | --- |
| `text_hash` | *H*(normalize(plain projection)) | Quotations and verbatim checks, where every character counts. |
| `match_hash` | *H*(strict projection) | Certifying that two witnesses agree, and recognizing the same text elsewhere. |
| `align_key` | *H*(lossy projection) | Proposing alignments. It never certifies anything. |
| `own_hash` | *H*(canonical JSON of the node's own content, [5.2](#52-own-content)) | Anchors on containers that have content of their own. |
| `node_hash` | *H*(canonical JSON of `{"own": own_hash, "children": [each child's node_hash, in order]}`) | Pins on leaves; the root hash; knowing that something below has changed. |
Normalization is that of [1.4](01-conventions.md#14-text-normalization-for-hashing).
Because each `node_hash` covers its children's, the root's `node_hash` covers the whole tree (it is a hash over canonical JSON of `{own, children}`, not an RFC 6962 Merkle root, which [8.5](08-fidelity.md#85-merkle-roots) defines for receipts): a correction to one sentence changes that sentence's values and the `node_hash` of each ancestor, and nothing else.
`hashes.jsonl` gives every value for every node of a small document, with each node's own content.
## 5.2 Own content
A node's own content is an object with its `kind` and these members, and nothing else:
| Kinds | Members |
| --- | --- |
| Kinds that take a label | `label`: the label, or null. |
| `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content. |
| `equation`, `equation-line` | `notation`, `source`. |
| `code`, `algorithm` | `language` (or null), `text`. |
| `section` | `level`, and `heading`: the `text_hash` of its heading, or null. |
| `table` | `columns`, as stored. |
| `cell` | `column`, `rowspan`, `colspan` and `role`, with defaults 1, 1 and `data` when absent. |
| `theorem` | `variant`, or null. |
So paragraphs, rows, lists, items and the other containers without content have the same `own_hash` as every node of their kind, and are never judged by it.
Own content covers stored values exactly; it is not normalized.
A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table.
## 5.3 The strict projection
The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement.
It is computed over a node's runs, or its math, as follows; for other nodes it is the strict projections of the text leaves joined with U+0020, then step 3.
1. Apply NFC to every string, then project each run:
| Run | Strict projection |
| --- | --- |
| `text` with `sup` | `^{` + inner + `}`, or only primes when the content is only primes ([5.4](#54-simple-notation)) |
| `text` with `sub` | `_{` + inner + `}` |
| other `text`, `link` | The text, with Unicode script characters folded and dashes before digits folded ([5.4](#54-simple-notation)) |
| `math`, `equation` | The canonical linear form when the source is simple; U+FFFC when it is complex |
| `cite`, `xref`, `fnmark` | U+FFF9 + the printed form with whitespace removed + U+FFFB |
Concatenate the results.
2. Apply NFKC, which folds ligatures, the micro sign and Greek variant letters.
3. Collapse whitespace to one U+0020 and trim, as [1.4](01-conventions.md#14-text-normalization-for-hashing) does.
Folding scripts before NFKC matters: NFKC alone would turn a superscript ² into a plain 2 and erase the structure the projection exists to keep.
Marking citations matters for the same reason: "10" followed by citation 3 must not agree with the text "103".
Each complex run also yields a complex key, `notation:source`, with the source in NFC and every whitespace character removed.
A node's complex keys are listed in order.
Two readings agree only when their `match_hash` values are equal and their complex keys are equal, run by run; this is stage 1 of canonical math.
A later minor version may add stage 2, which replaces U+FFFC with canonical MathML; that would change `match_hash` and `align_key` values but no `text_hash`, `own_hash` or `node_hash`, so no anchor would go stale.
## 5.4 Simple notation
A character is simple, after folding, when it is:
1. a letter: General Category L in the Latin blocks U+0041–U+024F and U+1E00–U+1EFF, or in the Greek blocks U+0370–U+03FF, without the Coptic letters U+03E2–U+03EF, and U+1F00–U+1FFF; or U+00B5 MICRO SIGN;
2. a digit: General Category Nd; or
3. one of `+ − = × ⋅ / < > ≤ ≥ ± ( ) ′ . ,`.
Inside notation, characters fold: U+002D and U+2013 to U+2212 MINUS SIGN; U+00B7, U+2219 and U+22C5 to U+22C5 DOT OPERATOR; U+0027 and U+2032 to U+2032 PRIME.
**The canonical linear form.**
Notation is a sequence of atoms; an atom has a base (possibly empty), a count of primes, an optional subscript and an optional superscript, each itself a sequence of atoms.
An atom is written as its base, then its primes as U+2032, then `_{`, the subscript and `}`, then `^{`, the superscript and `}`.
A superscript that holds only bare primes becomes primes on its atom.
There are no spaces in the form, and a script with no base attaches to the character before it, so all of `$H_2O$`, `H$_2$O`, `$\mathrm{H}_{2}\mathrm{O}$`, `H<sub>2</sub>O` and `H₂O` project to `H_{2}O`.
**TeX sources** are read as tokens: control words (`\` and ASCII letters, with following whitespace skipped), control symbols (`\` and one other character), and single characters, with whitespace and `~` ignored.
| Token | Meaning |
| --- | --- |
| `{` … `}` | A group; its atoms are spliced in. A script on a group of more than one atom makes the source complex; a script after an empty group applies to the atom before it, or to an empty base, so `{}^{14}C` is `^{14}C`. |
| `_`, `^` | A subscript or superscript of the previous atom (or of an empty base), taking one argument: a group, a command or a character. A second script of the same kind is complex. |
| `'` | A prime on the previous atom. |
| `\mathrm`, `\text`, `\textrm`, `\mathup` | A wrapper: its argument's atoms are spliced in. |
| `\rm` | Ignored. |
| `\,` `\;` `\:` `\!`, a backslash before a space, `\quad` `\qquad` `\enspace` `\thinspace` `\medspace` `\thickspace` | Spacing, ignored. |
| `\alpha` … `\omega`, `\varepsilon`, `\vartheta`, `\varkappa`, `\varpi`, `\varrho`, `\varsigma`, `\varphi`, `\Gamma` … `\Omega` | The Greek letter TeX typesets; `\epsilon` is ϵ and `\phi` is ϕ. |
| `\times` `\cdot` `\pm` `\le` `\leq` `\ge` `\geq` `\lt` `\gt` `\prime` `\AA` | × ⋅ ± ≤ ≤ ≥ ≥ < > ′ Å |
| `\left(`, `\right)` | Parentheses; any other delimiter is complex. |
| A simple character | An atom. |
| Anything else | The source is complex. |
**Typst sources** are read as tokens: strings in double quotes (complex if they contain a backslash), identifiers (runs of letters, with `.` joining modifiers such as `plus.minus`), numbers (digits with an optional fraction), the two-character operators `<=` and `>=`, and single characters, with whitespace ignored.
| Token | Meaning |
| --- | --- |
| `_`, `^` | A script of the previous atom; its argument is a parenthesized group, whose parentheses are dropped, or one token. |
| `(` … `)` elsewhere | Literal parentheses around a group. |
| `'` | A prime. |
| A string | Its characters, with whitespace removed. |
| A one-letter identifier, a number | Its characters. |
| `upright(…)` | A wrapper. |
| `alpha` … `omega`, `Alpha` … `Omega`, and `epsilon.alt`, `theta.alt`, `phi.alt`, `pi.alt`, `rho.alt`, `sigma.alt`, `kappa.alt`, `beta.alt` | The Greek letter Typst renders; `epsilon` is ε and `phi` is φ. |
| `plus` `minus` `times` `dot` `dot.op` `dot.c` `plus.minus` `lt` `gt` `lt.eq` `gt.eq` `eq` `prime` `slash`, and `<=` `>=` | + − × ⋅ ⋅ ⋅ ± < > ≤ ≥ = ′ / ≤ ≥ |
| `thin` `med` `thick` `quad` `wide` `space` | Spacing, ignored. |
| `/`, `*`, `#`, `&`, brackets, braces, any other identifier, and the operators `->`, `=>`, `!=`, `<-`, `==`, `:=` | Complex. In Typst `/` is a fraction. |
NFKC later folds the variant letters (ϵ to ε, ϕ to φ, ϑ to θ), so TeX's and Typst's defaults agree in the projection.
**Styled scripts.**
The inner form of a `sup` or `sub` text run is its text with whitespace removed and characters folded, when every character is simple or a Unicode script character; otherwise it is the text with whitespace collapsed and trimmed.
**Unicode script characters** are those whose decomposition mapping is tagged `<super>` or `<sub>`, such as ², ⁻ and ₂.
In text, each maximal run of script characters of one kind folds to `^{` or `_{`, the folded bases, and `}`, when every base is simple; otherwise the run is left for NFKC to flatten.
**Dashes in text.**
Outside notation, U+002D and U+2013 immediately before a digit become U+2212, so "10-20" and "10–20" agree; "CT–CT" keeps its dash.
`notation.jsonl` covers every rule, and `projections.jsonl` with `projection-groups.jsonl` covers H₂O, cm⁻¹, 10⁻³ and K_r in every spelling, ligatures, the micro sign, citation marks and "103" against 10³.
## 5.5 The lossy projection
The lossy projection starts from the strict projection and:
1. removes each `_{` and `^{` and the `}` that closes it, leaving the script's content in place, while other braces stay;
2. removes each mark, from U+FFF9 to the next U+FFFB;
3. applies Unicode full case folding, then NFKC;
4. deletes every whitespace character, every character of General Category Pd, and U+00AD SOFT HYPHEN.
So "103" and 10³ share an `align_key` while their `match_hash` values differ, and "bond- length", "bond-length" and "bond length" share one too.
A node's prose length is the number of code points in its lossy projection other than U+FFFC.
## 5.6 Alignment
Two readings of a document are aligned as whole sentence sequences, never paragraph by paragraph, because backends disagree about paragraphs at page and column breaks.
Each reading is a list of sentences in document order, each with its strict projection, lossy projection, complex keys, container and context (the kind of its container: paragraph, heading, caption and so on).
1. **Anchors.** A sentence is a candidate when its lossy projection occurs exactly once in each reading and its prose length is at least 20.
Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections.
Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
3. **Compare** each pair.
They agree when their strict projections and complex keys are equal.
Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`.
The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`.
4. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
5. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted.
`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order.
## 5.7 Suspects
A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note.
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved.
## 5.8 Keyed hashes
A hash of a short text is not a secret: a withheld cell reading "310.0" is recovered by hashing guesses.
So every content value that leaves a team's store is keyed.
| Where the value goes | `hashing` | Key |
| --- | --- | --- |
| The team's own store and bundles | `{"scheme": "plain"}` | None. |
| An export to another machine of the team, whatever its text policy | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
| A public layer, or an export outside the team on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
The destination chooses the scheme, not the text policy.
A checks-only package ([12.5](12-package.md#125-text-policies)) sent between the team's own machines is keyed with the team key, as `hale2026harmonic-damb.checks-only` is; sent to anyone outside the team, it is keyed with the page key.
A keyed scheme replaces SHA-256 with HMAC-SHA-256 under its key in every content value, including the Merkle construction of `node_hash`, and in `meta_hash`.
**The team key** is HKDF-SHA-256 (RFC 5869) with the team secret, at least 32 random bytes, as input keying material, the salt `scienting-team-key/1`, the info `scienting-hash/1` and length 32.
Its `key_id` is the first 16 hexadecimal digits of SHA-256 of the bytes `scienting-key-id/1` followed by the key.
**The page key** is HKDF-SHA-256 with the concatenated 32-byte page fingerprints of the witness, in page order, as input keying material, the salt `scienting-page-key/1`, the UTF-8 of the context as info, and length 32.
The context names the export, such as `layer:smithlab/kp4m2x7q`, so two exports never share a key.
The export lists each page as its page hash, SHA-256 of `scienting-page-hash/1` followed by the fingerprint's 32 bytes, never the fingerprint itself.
A reader who holds the paper computes the fingerprints from their own copy, checks them against the page hashes and derives the key; someone who does not hold it sees only hashes of hashes.
In keyed anchors, quotations become MACs over the normalized quotation and context texts become MACs over their normalized text ([6.3](06-anchors.md#63-keyed-anchors)).
Open-access documents under a license that allows redistribution need none of this.
`keyed.jsonl` gives the derivations and a case a non-holder cannot verify.
## 5.9 The metadata hash
`meta_hash` is *H* of the canonical JSON of `{work, version, relations, meta, language, license}` from `document.json`, with `license` null when absent.
Fixing an affiliation changes `meta_hash` and not the root.
spec/06-anchors.md
+123
View File
@@ -0,0 +1,123 @@
# 6. Addresses and anchors
## 6.1 Addresses
An address names a node, optionally pinned to a hash and narrowed to a span:
```abnf
address = [store "/"] key "#" node ["@" pin] ["[" start ":" end "]"]
store = lowalnum *(lowalnum / "." / "_" / "-")
key = workkey *("." lowalnum *(lowalnum / "-")) ; a document key (2.4)
workkey = LOWER *lowalnum "-" 4*51BASE32
node = lowalnum *(lowalnum / "." / "_" / "-") ; a node ID or an alias
pin = 6HEXLOWER ; the first six digits of a node_hash
start = "0" / (%x31-39 *DIGIT)
end = "0" / (%x31-39 *DIGIT)
lowalnum = LOWER / DIGIT
LOWER = %x61-7A
BASE32 = DIGIT / %x61-68 / %x6A-6B / %x6D-6E / %x70-74 / %x76-7A ; Crockford, lowercase
```
`start` is not greater than `end`, and both are code-point offsets into the plain projection of the node in the pinned state.
| Example | Names |
| --- | --- |
| `cornell1995second-saqk#k3m9q2a` | A sentence, whatever its current text. |
| `cornell1995second-saqk#k3m9q2a@4f1a0c` | That sentence as it read when its `node_hash` began `4f1a0c`. |
| `cornell1995second-saqk#k3m9q2a@4f1a0c[12:57]` | Code points 12 to 57 of its plain projection in that state. |
| `smithlab/cornell1995second#k3m9q2a` | The same sentence in the `smithlab` store. |
On the web the node ID is the fragment: `…/smithlab/cornell1995second#k3m9q2a` lands on the sentence.
`addresses.jsonl` gives valid and invalid addresses and how each parses.
## 6.2 The anchor
An anchor, `scienting-anchor/1`, is the stored form of a pinned address, with enough context to survive change.
It is what extracts, annotations, dataset values and draft citations use.
| Property | Required | Holds |
| --- | --- | --- |
| `format` | Yes | `scienting-anchor/1`. |
| `store` | No | The store. |
| `doc` | Yes | The document key. |
| `node` | Yes | The node ID, or a legacy alias. |
| `kind` | No, RECOMMENDED | The node's kind when the anchor was made. |
| `own_hash`, `node_hash` | Yes | The node's values when the anchor was made. |
| `span` | No | `[start, end]` in code points; present for span anchors. |
| `quote` | With a span, or on a leaf | `{exact, prefix?, suffix?}`, or `{exact_hash, length, prefix?, suffix?}` for quotations longer than 200 code points. |
| `context` | On a cell | `{table_label, row_path, column_path}`: the table's label and the normalized plain projections of the row's and column's header cells, outermost first. |
| `leaves` | On a container without content | `[[id, match_hash], …]`: every leaf below the node, in document order. |
| `rev` | Yes | The document's revision when the anchor was made. |
| `keyed` | No | The hashing scheme, when the anchor leaves the team ([6.3](#63-keyed-anchors)). |
`exact` is the substring of the stored plain projection at the span, or the whole projection for a leaf anchor.
`prefix` and `suffix` are up to 32 code points before and after it, omitted when empty.
`exact_hash` is SHA-256 of the normalized quotation, and `length` its length in code points.
Revisions are counted, not timed, because anchors are made on different machines and clocks disagree.
## 6.3 Keyed anchors
When an anchor leaves the team, in a public layer or on a work whose text is withheld, it carries `keyed`, and:
1. `own_hash` and `node_hash` are computed under the key;
2. `quote` becomes `{mac, length, prefix_mac?, suffix_mac?}`, each MAC over the normalized text; and
3. `context` becomes `{table_label_mac, row_path_mac, column_path_mac}` in the same way.
Such an anchor can be verified, and its span shown, only over a copy of the paper the reader holds.
## 6.4 Four kinds of anchor
| Kind | Points at | Judged by |
| --- | --- | --- |
| Span | Part of a leaf's text: the anchor has a `span`. | The quote, then `node_hash`. |
| Leaf | A whole `sentence`, `cell`, `reference`, `authors`, `keywords`, `equation`, `equation-line`, `code` or `algorithm`. | `node_hash`. |
| Container with content | A `section`, `figure`, `panel`, `table`, `theorem` or `footnote`. | `own_hash`, with `node_hash` reported. |
| Container without content | Any other node. | Its ID and its leaves. |
## 6.5 Resolution
To resolve an anchor against a document's current state (its nodes, its ID table and its corrections):
**Follow the ID.**
If the anchor's node is live, use it.
Otherwise follow the ID table: an alias leads to its node, a retirement to its successor, repeatedly, until a live node is reached, a deletion ends the chain, or an ID is unknown or repeats.
The anchor is *retired* when the chain passed a retirement; an alias alone does not retire it.
The kind is the anchor's `kind`, else the live node's, else the retired entry's `node_kind`.
**Find the quote.**
The quote's hits in a text are the positions where `exact` occurs, or, for a hashed quote, where a window of `length` code points has that hash.
When there is more than one hit and the anchor has a prefix or suffix, keep the hits preceded by the prefix and followed by the suffix.
**Judge**, by kind:
| Kind | Current | Changed below | Moved | Reattached | Otherwise |
| --- | --- | --- | --- | --- | --- |
| Span or leaf | Not retired, and `node_hash` equal. | Never. | Retired, and the successor's text has exactly one hit (a span) or at least one (a leaf); a span takes the new position. | Not retired, changed, and exactly one hit, with the new position; or a changed node whose only changes since `rev` were `relabel` corrections, and whose own content with the earliest of those corrections' `from` label has the anchored `own_hash`. | Stale. |
| Container with content | Not retired, and `own_hash` and `node_hash` equal. | Not retired, `own_hash` equal, `node_hash` not. | Retired, and the successor's `own_hash` equals the anchored one. | Not retired, and only `relabel` corrections changed it, judged as for leaves. | Stale. |
| Container without content | Live and `node_hash` equal. | Live and `node_hash` not equal. | Retired, and the successor's leaves hold at least half of the anchored `leaves`, each matched by ID or else by `match_hash`. | Never. | Stale. |
**The cell rule.**
A cell anchor whose ID led to no live node (a deletion, or an unknown ID), or whose successor fails the leaf rule, is reattached when exactly one cell in the document has the anchored `context` and exactly one hit of the quote.
This finds a value again even in a table with multi-level headers or row labels repeated across parameter sets.
The result is `{result, node, node_hash?, span?, drift?, via?, reason?}`: the outcome, the live node and its `node_hash`, the new span, the drift of a `changed-below` result, the IDs followed, and a reason for `stale` and relabel results.
A `moved` or `reattached` anchor is re-pinned to the live node, and the move is recorded.
Checks are stricter than readers: an extract is a span or leaf anchor and never accepts `changed-below`, because a check is about exact words.
`resolve.jsonl` resolves seventeen anchors made at revision 0 against `fixtures/packages/hale2026harmonic-damb` at revision 12, with every outcome.
## 6.6 Drift
`changed-below` must not last forever.
For a container *c* and an anchor made at revision *r*:
1. The corrections that count are those with `index` greater than *r* whose `path` or `to_path` contains *c* and whose `touched` list is not empty.
2. `count` is their number, and *T* the union of their `touched` lists.
3. *L* is the set of leaves below *c* (including *c* itself if it is one) in the current state.
4. `share` is |*T*| divided by |*L* ∪ *T*|, written as a decimal with four places, rounded half up.
5. `level` is `none` when `count` is 0, `high` when `share` is greater than the threshold, and `low` otherwise.
The threshold is a quarter by default; a project may set another ([20](20-projects.md)).
A reader shows a `high` drift as needing review.
A change to a section's heading changes its `own_hash`, so a rename is never mistaken for drift.
spec/07-geometry.md
+64
View File
@@ -0,0 +1,64 @@
# 7. Geometry
## 7.1 Page fingerprints
Several publishers stamp each download with the institution, date and time, so two copies of one paper rarely share a SHA-256.
Every page of a witness therefore also has a fingerprint, `scienting-page/1`.
Geometry and digitizations point at fingerprints, so a re-download keeps every box and every digitization.
The input is the page's size after rotation, its text-layer lines in content-stream order (each a string and, where known, a box), and the SHA-256 of each embedded image's stream data as stored.
To compute the fingerprint:
1. Remove every line that a stamp rule matches ([7.2](#72-stamp-rules)).
2. Join the remaining lines with U+0020.
3. Delete every whitespace character and every character of General Category Cc or Cf.
4. Fold Unicode script characters and dashes before digits as the strict projection does for text ([5.4](05-hashes.md#54-simple-notation)), then apply NFKC.
5. Round the width and height to whole points, half up.
6. The fingerprint is SHA-256 of the canonical JSON of `{"format": "scienting-page/1", "size": [width, height], "text": the text, "images": the image hashes, sorted}`.
Whitespace is deleted rather than collapsed because text extractors disagree about where they insert spaces, and it is deleted before folding so that "10 -20" and "10-20" fold alike.
A scanned page with no text layer is fingerprinted by its size and its images alone.
Two witnesses whose fingerprints are equal, page for page, are the same edition of the paper.
How lines are extracted from a PDF is outside this text; the vectors give the lines, so implementations are tested on the function.
`fingerprints.jsonl` gives two differently stamped downloads of one ACS-style page and of one Wiley-style page that share their fingerprints, a page with one changed digit that does not, and a scan.
## 7.2 Stamp rules
Stamp rules are published in `scienting-stamps/1` files.
Each rule has an `id`, a `pattern`, a `where` (`any`, or `margin` with a `margin` in points, default 36) and a note.
A rule matches a line when its pattern is found in the line's normalized text, with ASCII letters compared without case; a `margin` rule matches only lines whose box lies wholly within the margin of one edge.
Patterns use only syntax that ECMAScript and Python read alike: literals, `.`, `*`, `+`, `?`, `^`, `$`, classes and escaped punctuation.
[registry/stamps/scienting-stamps-1.json](../registry/stamps/scienting-stamps-1.json) holds the published rules.
A team adds rules it learns, when two witnesses of one version differ only in such lines, in files of its own with `learned: true`.
A missed stamp splits one edition into two; an over-broad rule hides a real difference.
## 7.3 `pages.jsonl`
| Property | Holds |
| --- | --- |
| `witness` | The witness ID. |
| `page` | The page number in the witness, from 1. |
| `page_fp` | The fingerprint. |
| `size` | `[width, height]` in points, after rotation. |
| `page_label` | The printed page label, when there is one. |
| `text_layer` | Whether the page has a text layer. |
| `stamps_removed` | The IDs of the rules that removed lines, in line order, when any did. |
Pages sort by witness ID, then page.
## 7.4 `locations.jsonl`
| Property | Holds |
| --- | --- |
| `node` | The node ID. |
| `page_fp` | The page's fingerprint, which some page of some witness has. |
| `page` | The page number. |
| `page_label` | The printed label. |
| `boxes` | One or more `[x0, y0, x1, y1]` in PDF points, origin at the top left after rotation. |
| `approx` | `true` when the boxes come from OCR line boxes rather than glyphs. |
A node has one record per page it appears on; a sentence that wraps has several boxes.
Locations sort by the node's position in document order, then page, then fingerprint.
Because locations point at fingerprinted pages, a reader can rebuild a document from its own download: it fingerprints the pages, finds the matching ones and reads the text at the stored boxes.
spec/08-fidelity.md
+161
View File
@@ -0,0 +1,161 @@
# 8. Fidelity, receipts and spot-checks
## 8.1 Levels of fidelity
| Level | Meaning |
| --- | --- |
| F0 converted | One reading, not corroborated. |
| F1 resolved | Every suspect in the node was answered against the page image and passed the gate. |
| F2 corroborated | An independent witness agrees, by `match_hash` and complex keys, or the text is born digital, as a Typst or LaTeX source is. |
| F3 proofread | A named person compared the node with the page image, and a valid receipt they signed lists the node at its current `node_hash`. |
Levels apply to leaves.
The pipeline records F0, F1 or F2 in a leaf's `fidelity`; F3 comes only from receipts.
A correction that changes a leaf drops it back: the gate's crop evidence makes it F1 at most, until it is corroborated or proofread again.
## 8.2 Levels
A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`.
A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does.
A spot-check's `qa:` receipt ([8.8](#88-spot-checks)) raises the nodes it sampled, except those its `errors` name.
A document reports, in `document.json`, the lowest level among its leaves and the count at each level.
## 8.3 Receipts
A paper has thousands of nodes, and nobody signs thousands of times, so people sign receipts: one signature over a Merkle root of everything reviewed in a sitting.
| Property | Required | Holds |
| --- | --- | --- |
| `format` | Yes | `scienting-receipt/1`. |
| `id` | Yes | The record ID ([4.6](04-identity.md#46-record-ids)). |
| `doc` | Yes | The document key. |
| `rev` | Yes | The revision reviewed. |
| `level` | Yes | `F1`, `F2` or `F3`. |
| `scope` | Yes | `all`, `section:<id>`, `page:<n>`, `targeted:<name>` (such as `targeted:lode`), or `qa:<receipt id>` for a spot-check. |
| `nodes` | Yes | `[[id, node_hash], …]`, sorted by ID without repeats. |
| `records` | No | `[[kind, id, hash], …]` for other things signed in the sitting: `annotation`, `digitization`, `correction`, `layer` or `checklist`. |
| `root` | Yes | The Merkle root ([8.5](#85-merkle-roots)). |
| `evidence` | Yes | What the reviewer looked at: `{kind, sha256, node?, page_fp?, box?, note?}` with `kind` `crop`, `witness` or `source-file`, or `{kind: "person", by}`. A regenerated crop is different evidence. |
| `by` | Yes | The signer's roster key. |
| `at` | Yes | When it was signed. |
| `qa` | For `qa:` scopes | The spot-check ([8.8](#88-spot-checks)). |
| `export_of` | No | The receipt this one re-issues under a keyed scheme. |
| `keyed` | No | The scheme its values are keyed under, when it leaves the team. |
| `signature` | Yes | `{scheme, key, value}` ([8.4](#84-record-hashes-and-signatures)). |
A receipt lists nodes, records, or both.
A correction invalidates a receipt for the corrected node only, because the node's `node_hash` no longer matches; the rest still counts.
A receipt shared outside the team cannot carry plain node hashes, which would leak short texts, and its signature covers them.
So an export re-issues it: a new receipt with `export_of`, values under the export's key, signed by the original reviewer or a team key with the `export` purpose.
## 8.4 Record hashes and signatures
A signed record's record hash is SHA-256 of the canonical JSON of the record without `id` and `signature`.
Its `id` is the 13-character letter-first encoding of that hash.
Its signing payload is the canonical JSON of `{"format": <the record's format>, "hash": <the record hash>}`.
| Scheme | `value` |
| --- | --- |
| `ed25519` | The RFC 8032 signature of the payload, in standard Base64 with padding. |
| `ssh` | An SSH signature of the payload in the namespace `scienting`, armored. |
| `gpg` | A detached, armored OpenPGP signature of the payload. |
| `sigstore` | A Sigstore bundle for the payload, as JSON text. |
| `cose` | Reserved: a COSE_Sign1 message over the payload, in Base64, as the IETF SCITT work shapes signed statements, so other transparency services can verify a receipt. No vector yet. |
| `ml-dsa-65` | An ML-DSA-65 signature (FIPS 204, the pure variant with an empty context) of the payload, in Base64, for the decades after Ed25519 is no longer considered safe; the roster's `public` is the 1952-byte key in Base64. The reference implementation reproduces NIST's ACVP vectors; `signatures.jsonl` pins a key generation and a deterministic signature. |
A consumer treats a record signed with a scheme it does not implement as invalid, so a reserved scheme can be adopted by a new minor version without any format changing.
`signature.key` is the signer's roster key and equals `by`.
Self-hosted deployments sign with team keys or a private Sigstore deployment; the public Sigstore instance, whose log permanently records each signer's identity, is for public layers only.
`signatures.jsonl` verifies one receipt and the RFC 8032 test vector.
## 8.5 Merkle roots
A receipt's items are `["node", id, node_hash]` for each listed node, in order, followed by its `records`.
The root follows RFC 6962: a leaf hash is SHA-256 of the byte 0x00 and the item's canonical JSON; an interior hash is SHA-256 of 0x01 and the two child hashes; a list of *n* > 1 items splits after the largest power of two less than *n*.
RFC 6962 has since been replaced by RFC 9162, which defines the same tree.
A root over one item is that item's leaf hash; a receipt with no items is invalid.
`merkle.jsonl` gives roots for one to seven items.
The tree allows a proof that one node was covered without revealing the others.
An inclusion proof, `scienting-proof/1` with `kind` `inclusion`, holds the `receipt`, the `item`, its `index` among the receipt's items, the `size` of the tree, the `root` and the `path`: the audit path of RFC 9162 §2.1.3.1, as hex hashes, leaf-most first.
A verifier recomputes the root from the item's leaf hash and the path as RFC 9162 §2.1.3.2 describes and compares it with the receipt's root.
`proofs.jsonl` gives the path for every item of trees of one to seven items, and `fixtures/records/proof-inclusion.json` proves one node of the example paper's first receipt.
## 8.6 Receipt status
A receipt is:
1. **invalid** when its root, ID or signature does not verify, or its signer's key was not valid when it was signed (`registered_at` ≤ `at` < `revoked_at`);
2. otherwise **revoked** when a valid revocation names it, or names its signer with a period that contains its `at`;
3. otherwise **suspended** when a valid spot-check of it found an error;
4. otherwise **valid**.
Only valid receipts set levels.
A suspended receipt stays suspended; its reviewer reviews the sitting again and signs a new receipt, which needs its own spot-check.
A spot-check is itself a receipt, so its status may depend on its own spot-check; statuses are settled from the end of each chain, and the receipts of a chain that loops back on itself are invalid.
## 8.7 Revocations and the roster
A revocation, `scienting-revocation/1`, names one `receipt`, or a `signer` and a `period` `{from, to}`, with a `reason`, and is signed like a receipt.
It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or a key whose `role` in the roster is `owner`.
The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `role`, a `scheme`, its `public` key (Base64 for Ed25519 and ML-DSA-65), its `purposes` (`sign`, `spot-check`, `export`, or `witness` for a party that cosigns the team's log checkpoints, [23](23-log.md)), `registered_at` and `revoked_at`.
A key MAY carry its `did`, the `did:key` decentralized identifier the public key determines: `z` and the base58btc of the multicodec prefix (`0xed` for Ed25519, `0x1207` for ML-DSA-65, as varints) followed by the key bytes; a validator refuses a `did` that the key does not determine (`doc.roster.did`), so a signer can be named outside the store without a second name.
The `role` is the holder's role in the team that keeps the roster, `owner`, `editor`, `reader` or `guest`, as in a project ([20](20-projects.md)).
Because the roster carries it, a package that carries its roster carries everything needed to judge its revocations.
A package MAY carry the roster its receipts need as `roster.json`, so its receipts verify without the store.
The roster is meant to be the one a store verifies against; see the [rationale](../docs/rationale.md#known-limits).
## 8.8 Spot-checks
For each F3 receipt, a second reviewer checks a sample of its nodes and signs a `qa:` receipt.
A seed the reviewer controls could be ground, by varying the receipt until the sample falls on careful work, and a seed the checker controls could be ground by the checker, so the seed is a value neither can steer:
1. The seed is the checker's VRF proof, under RFC 9381's ECVRF-EDWARDS25519-SHA512-TAI, for the input that is the canonical JSON of `{"format": "scienting-spot-check/1", "receipt": <the original's record hash>}`, made with an Ed25519 key registered in the roster with the `spot-check` purpose before the original was signed.
A verifiable random function has exactly one valid proof for a key and an input, where an Ed25519 signature has many, so the checker cannot try proofs until the sample suits, and the reviewer cannot predict the proof without the checker's private key.
2. The sample size is the smaller of *n* and the larger of `minimum` and ⌈`rate` × *n*⌉, for the original's *n* nodes; the defaults are a rate of `0.05` and a minimum of 10.
3. Rank the original's node IDs by the hexadecimal SHA-256 of the proof's VRF output, the 64 bytes `ECVRF_proof_to_hash` gives, followed by the ID's UTF-8, then by ID, and take the first ones.
The `qa` object records `of`, `key`, `seed` (the 80-byte proof, Base64), `rate`, `minimum`, the `sample` sorted by ID, and the `errors` found, each `{node, kind?, note?}`.
A verifier checks the proof against the checker's public key and the input before it recomputes the sample.
`vrf.jsonl` pins the VRF on the RFC's own example and on the example paper's seed.
The `qa:` receipt's `nodes` are the sampled nodes at the hashes the original lists, its `scope` is `qa:` and the original's ID, every error names a sampled node, and it is signed at or after the original.
Anyone can recompute the sample; `spotcheck.jsonl` does.
The payload's identifier, `scienting-spot-check/1`, is registered with the other formats so that no file ever carries it by mistake; it names a signing payload, never a file.
## 8.9 Reviewer error bounds
One sample proves little; what someone relying on the work needs is a bound on each reviewer's error rate across all their samples.
With *k* errors in *n* sampled nodes, the bound is the one-sided Clopper–Pearson upper confidence bound at confidence *c*, by default 0.95: the rate *p* at which the probability of at most *k* errors in *n* is 1 − *c*, or 1 when *k* = *n*.
It is reported to six decimal places, rounded up.
With no errors the bound is about 3/*n*, so zero errors in 300 sampled nodes bounds a rate below 1%; `bounds.jsonl` gives cases.
A QA report, `scienting-qa-report/1`, states for each reviewer who signed a delivery the nodes sampled, the errors found, the confidence, the method `clopper-pearson` and the bound.
It is the record a buyer keeps, so it is signed like a receipt ([8.4](#84-record-hashes-and-signatures)): `by` names the roster key of the team that made the delivery, `id` derives from the record hash, and `signature.key` equals `by`.
## 8.10 Targeted F3
Most of a paper is never relied on, so a team proofreads the target set: the nodes its claims, data and quotations rest on.
1. The anchored nodes: every live node that an extract, a dataset value, a confirmed `claim-link`, a draft quotation or a pinned citation anchors, each anchor resolved first ([6.5](06-anchors.md#65-resolution)).
An anchored leaf is a target; an anchored container contributes its leaves.
2. Every `equation` and `equation-line`.
3. Under the project's digit rule ([20](20-projects.md)): with `all`, every `sentence` and `cell` with a digit in a `text` or `math` run; with `anchored-sections`, only those in a section that holds an anchored node; with `none`, none.
Digits are characters of General Category Nd.
Digits in `cite`, `xref` and `fnmark` runs do not count, so citation marks, "Figure 3" and footnote marks pull nothing in.
A node's section is its nearest ancestor-or-self of kind `section`, or the `front` or `back` matter when it has none.
The set is reported in document order, each node with the first reason that applies: `anchored`, `equation` or `digits`.
Neighbors shown for context are not in the set, and nothing new comes to rest on a node below F3 without the node joining the set.
`targets.jsonl` gives the set of `fixtures/packages/hale2026harmonic-damb` under each digit rule.
## 8.11 Checklists
An export of a document, such as accessible HTML, can be checked against a standard, and that check is a record a buyer keeps.
A checklist, `scienting-checklist/1`, holds the `doc` and `rev` checked; the `standard`, `{name, level?, url?}`, such as WCAG 2.1 at level AA; the `export` checked, `{kind, sha256, size?}`, so the record binds to one file; its `items`, each a `criterion`, its `result` (`pass`, `fail` or `not-applicable`), the `method` (`manual`, or `automated` with the `tool` and its version) and an optional `note`; and `by` and `at`.
Its content hash is SHA-256 of the canonical JSON of the record without `id`, and its `id` is the record ID of that hash ([4.6](04-identity.md#46-record-ids)).
A checklist carries no signature of its own: the person who checked signs a receipt that lists it as `["checklist", id, hash]` ([8.3](#83-receipts)), so one sitting covers a checklist together with the nodes read for it.
The receipt covers the checklist only at that hash, so a checklist changed afterwards is uncovered.
`fixtures/records/checklist/` holds a checklist for an accessible HTML export of the example paper and the receipt that covers it.
The format is `provisional` ([0.4](00-introduction.md#04-status)).
spec/09-figures-tables-equations.md
+37
View File
@@ -0,0 +1,37 @@
# 9. Figures, tables and equations
## 9.1 Figures and panels
A figure holds its panels, its caption and its notes.
A panel's own content is its label only.
Its image and region are evidence, not content: re-extracting the image at native resolution adds an asset and changes no hash.
| Panel attribute | Holds |
| --- | --- |
| `image` | `{sha256, media_type, native, dpi?}`: the embedded image, at native resolution when the PDF holds one. |
| `crop` | `{sha256, media_type, dpi}`: a crop of the panel's region at a fixed resolution. |
| `figure_text` | `[{id, text, role, page_fp?, boxes?}]`: text found inside the panel, with `role` `axis-label`, `tick-label`, `legend`, `annotation` or `other`. |
Text inside a figure is not prose and is not split into sentences.
Its items have IDs ([4.2](04-identity.md#42-minting-node-ids)) so that a digitization can anchor its axis labels to them.
A panel's region on the page is its location ([7.4](07-geometry.md#74-locationsjsonl)).
## 9.2 Tables
A table holds its caption, an optional head, one body and its notes.
Rows and cells are nodes, with all five values, so the gate, the hashes and the database treat them like any other node, and a cell is addressed like a sentence.
Columns are defined in the table's `columns` attribute, each with an ID and a header path: the IDs of the header cells above it, outermost first.
Header text is stored once, in the header cells.
A row's `header_path` lists the cells that label it: a row-group header and the row's own header cell.
A cell records its column, its spans and its role; the expanded grid is derived, never stored.
So inserting a row that a reading dropped adds a row node and changes no other row's or cell's ID or hash, and fixing a header cell changes that cell, not the table.
A table continued across pages is one table node; its pages are locations.
For people, a cell can be shown as "Table 2, row CT–CT, column K_r" from its header paths.
## 9.3 Equations
An equation keeps its source as written, `tex` from a conversion or `typst` from a draft, and its printed number as its `label`, so numbering never collides with a renderer's own.
An equation that the source numbers line by line (Eq. 3a) has `equation-line` children, each with its own label and source.
MathML is derived by a pinned tool and recorded, with the tool's name and version, in `derived/mathml.jsonl`.
spec/10-references.md
+15
View File
@@ -0,0 +1,15 @@
# 10. References and citations
Each reference is a `reference` node holding its printed runs, exactly as printed, and its printed number as its `label`.
`references.jsonl` holds, for each reference node in document order, a record with `node`, `label`, an optional parsed `csl` record in CSL-JSON, and its `resolution`:
| Resolution | Means |
| --- | --- |
| `{type: "doi", doi}` | The reference names a DOI. |
| `{type: "document", doc}` | It names a document in the same store, by key. |
| `{type: "none"}` | Nothing yet. |
Every `cite` run links a sentence to one or more references.
The citation graph is therefore sentence-level: for any reference, the format gives every sentence that cites it, which is the citation's context.
The database derives a `citations` table from the runs ([13](13-database.md)).
An article's references and its Supporting Information's are separate documents' references and never mix.
spec/11-corrections.md
+82
View File
@@ -0,0 +1,82 @@
# 11. Corrections
A correction, `scienting-correction/1`, is an operation on an established document, with its evidence and its author.
Corrections are added and never edited, so a document's history is its original reading plus its corrections.
Each has an `index`, counted from 1 in the order corrections are applied, and the document's revision is the index of the last.
## 11.1 Operations
| `op` | Fields | Does | IDs |
| --- | --- | --- | --- |
| `replace` | `target`, `find`, `replace`, optional `run` | Replaces the one occurrence of `find` among the target's editable strings with `replace`, a string or, inside a text run, a list of runs. | Unchanged. |
| `insert` | `parent`, `place_after` (a sibling, or null for first), `node` | Adds a subtree: kinds, labels, attributes and content, with each equation run naming its child as `{"kind": "equation", "child": i}`. | New IDs, minted ([4.2](04-identity.md#42-minting-node-ids)). |
| `insert` | `target`, `run`, `offset`, `text` | Inserts text into one run of the target, at a code-point offset in that run's editable string. | Unchanged. |
| `delete` | `target` | Removes an artifact and its subtree, such as a running head read as text. | All retired, without successors. |
| `split` | `target`, `offset` | Splits a sentence at an offset in its plain projection, which falls between runs or inside a text run; whitespace at the split is removed from both parts, and each equation child goes with its run. | The first part keeps the ID; the second gets a new one. |
| `merge` | `target`, `with`, optional `joiner` | Joins a node with its next sibling of the same kind: the runs of two run-bearing nodes, with the joiner (default one space) between, or the children of two `paragraph`, `list`, `item`, `quote`, `caption`, `figure-note` or `table-note` nodes. | `with` is retired, with the target as successor. |
| `move` | `target`, `parent`, `place_after` | Changes reading order or containment. | Unchanged. |
| `reclassify` | `target`, `kind` | Changes a node's kind, such as a paragraph that is really a caption, to a kind with the same content shape. | Unchanged. |
| `relabel` | `target`, `label` | Changes a printed label. | Unchanged. |
| `set` | `target`, `attr`, `value` | Sets one structural attribute: a section's `level`, a table's `columns`, a cell's `column`, `rowspan`, `colspan` or `role`, a row's `header_path`, a list's `ordered`, an item's `marker` or a theorem's `variant`. | Unchanged. |
A leaf's editable strings are the `text` of its `text`, `link`, `cite`, `xref` and `fnmark` runs and the `source` of its `math` runs, each searched on its own; an equation's `source`; a verbatim node's `text`.
`set` exists because readings drop spans, roles and header paths; see the [rationale](../docs/rationale.md#corrections).
Runs are made canonical after every operation, and the result MUST satisfy every rule of [3](03-nodes.md).
## 11.2 Every correction records
| Property | Holds |
| --- | --- |
| `evidence` | At least one item: `crop`, `witness` or `source-file` by SHA-256, or `person`. |
| `author` | `{key}` for a person in the roster, or `{model, run}` for a model's session, where `model` is `{name, provider?, version?, weights?, prompt?, parameters?, processor?}`: the model's name, who served it, its version, and SHA-256 of the weights, the instructions and the sampling parameters it ran with, as far as the team can obtain them, and where it ran ([14](14-rights.md)). |
| `reason`, `at` | Why, and when. |
A proposal, such as one a tool makes, carries these and `status: "proposed"`, and no applied fields.
## 11.3 Applied fields
When a correction is applied, its record gains:
| Property | Holds |
| --- | --- |
| `index`, `id` | Its position, and its record ID. |
| `before`, `after` | The target's `node_hash` before and after, or null; for an inserted subtree, `target` is its root. |
| `path` | The target's ancestors when it was applied, root first; for an insert, the new subtree's. |
| `to_path` | For a move, the new ancestors. |
| `touched` | The leaves whose content the correction changed, sorted: the target of a `replace`, text `insert`, `relabel` or `set` on a leaf; the new leaves of a node `insert`; the deleted leaves of a `delete`; both parts of a `split`; both nodes of a run `merge`, or the moved-in leaves of a container `merge`; the moved leaves of a `move`; the leaves of a reclassified node. |
| `created`, `retired` | IDs minted or retired. |
| `from` | The label, kind or attribute value before a `relabel`, `reclassify` or `set`. |
Retirements are also entered in `ids.jsonl` with the correction's index as `rev`.
Drift ([6.6](06-anchors.md#66-drift)) is computed from `path`, `to_path` and `touched` alone, so it needs no replay.
## 11.4 The gate
A correction is refused, naming the rule ([22](22-conformance.md)), when:
1. it does not match the correction schema, `schemas/correction.schema.json`;
2. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified;
3. `find` is empty or does not occur exactly once;
4. an `insert` has no crop evidence, or names a run or offset its target does not have;
5. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits;
6. any leaf outside `touched` has different digits afterwards;
7. a `split` falls outside the sentence, inside a run other than text, or leaves a part empty;
8. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings;
9. a `move` would put a node inside itself;
10. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it; or
11. the result breaks a rule of [3](03-nodes.md).
The gate needs only the document's current state, so a tool can run it on a proposal before applying anything, and get back the record the correction would append or the rule it breaks.
`tools/validate.py --correction` does this against a package.
Digits are characters of General Category Nd in the plain projection.
A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, and the last correction of each leaf ends at its current `node_hash`.
`corrections.jsonl` in the vectors applies each operation, and each refusal, to a small document.
## 11.5 The history root
`document.json` carries `history`: the root of the tree of [8.5](08-fidelity.md#85-merkle-roots) whose items are `["correction", index, id]` for each correction in order, or SHA-256 of the empty string when there are none, which is how RFC 9162 hashes the empty tree.
Because corrections are only appended, a consistency proof (RFC 9162 §2.1.4) between the roots at two revisions shows that the later history extends the earlier one rather than rewriting it.
The proof is a `scienting-proof/1` with `kind` `consistency`, holding the `doc`, `from_rev`, `to_rev`, `from_root`, `to_root` and the `path` as hex hashes; a verifier applies RFC 9162 §2.1.4.2.
A checks-only package carries no corrections, so its history is the empty root.
`proofs.jsonl` gives consistency proofs for every prefix of small trees and for the example paper between revisions 8 and 12; `fixtures/records/proof-consistency.json` is the latter.
spec/12-package.md
+71
View File
@@ -0,0 +1,71 @@
# 12. The package
A document travels as a package: a folder named by its key, or the same folder zipped.
## 12.1 Files
| Path | Required | Holds |
| --- | --- | --- |
| `MANIFEST.json` | Yes | The manifest ([12.3](#123-the-manifest)). |
| `document.json` | Yes | The document ([2.2](02-model.md#22-documentjson)). |
| `nodes.jsonl` | Yes | Every node, in document order ([3](03-nodes.md)). |
| `witnesses.json` | Yes | Every witness, with its rights ([2.3](02-model.md#23-witnessesjson)). |
| `provenance.json` | Yes | Tools and versions, backends, cached readings, stages and where each ran. |
| `pages.jsonl` | No | Every page of every witness ([7.3](07-geometry.md#73-pagesjsonl)). |
| `locations.jsonl` | No | Geometry per node and page ([7.4](07-geometry.md#74-locationsjsonl)). |
| `references.jsonl` | No | Parsed references and their resolution ([10](10-references.md)). |
| `corrections.jsonl` | No | Every correction, in order ([11](11-corrections.md)). |
| `receipts.jsonl` | No | Receipts and revocations, sorted by `at` and then `id` ([8](08-fidelity.md)). |
| `ids.jsonl` | No | Retired IDs and legacy aliases ([4.4](04-identity.md#44-idsjsonl)). |
| `suspects.jsonl` | No | Unresolved suspects, sorted by `id` ([5.7](05-hashes.md#57-suspects)). |
| `roster.json` | No | The roster its receipts need ([8.7](08-fidelity.md#87-revocations-and-the-roster)). |
| `assets/` | No | Images and crops, as `<sha256>.<extension>`, present or listed only. |
| `datasets/<id>/` | No | Datasets ([15](15-datasets.md)). |
| `derived/` | No | Rebuildable renderings, never authoritative ([12.7](#127-derived-files)). |
An optional JSON Lines file with no records is omitted.
Every file is in the canonical form of [1.9](01-conventions.md#19-files).
## 12.2 Paths
Paths use `/`, are relative, have no empty, `.` or `..` segment, no backslash and no NUL, and are already normalized.
No two paths are the same once case-folded, so a package reads alike on every file system.
A package folder carries no symbolic links, and a consumer refuses one that does, as it refuses a package whose files exceed its own size limit.
In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit.
A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields; `fixtures/packages/tiny2026tiny-j4qm.sct.zip` is such an archive.
## 12.3 The manifest
`MANIFEST.json`, `scienting-package/1`, is `{format, doc, key, files}`: the package format, `scienting-doc/1`, the document key, and every other file as `{path, sha256, size}`, sorted by the UTF-8 bytes of the path.
An asset whose bytes stay in the asset store is listed with `present: false`; nothing else may be.
The SHA-256 of `MANIFEST.json` pins the whole package.
## 12.4 Provenance
`provenance.json` holds `tools` (`{name, version}`), and optionally `backends`, `readings` (each cached reading's ID, witness, backend and SHA-256), `stages` (each stage's name, tool, input hashes, output hash, the `processor` where it ran, and the uses it ran under) and a `session`.
Every stage that uses a model or an outside service records where it ran ([14](14-rights.md)).
## 12.5 Text policies
| Policy | Means |
| --- | --- |
| `full` | Every leaf keeps its content. |
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). |
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
Its text lives in the team's private asset store, and only key holders can check its values.
`fixtures/packages/hale2026harmonic-damb.checks-only` is such an export of `hale2026harmonic-damb` to another machine of the team, under the team key in `fixtures/keys/`.
## 12.6 Assets
Assets are named by SHA-256, so every read is rehashed and a regenerated crop is a different asset.
Bytes that live only in the asset store are listed in the manifest with `present: false`, with their size.
## 12.7 Derived files
`derived/` holds what a pinned tool makes from the package, such as `derived/mathml.jsonl`, whose lines are `{node, run, tool, mathml}` with `run` null for an equation node.
Nothing derived is content, and a consumer may delete and rebuild it.
## 12.8 Media type and extension
A zipped package has the extension `.sct.zip` and the media type `application/vnd.scienting.doc+zip`.
spec/13-database.md
+62
View File
@@ -0,0 +1,62 @@
# 13. The database form and the bundle
## 13.1 Portability
[sql/doc.sql](../sql/doc.sql) defines the document tables and [sql/store.sql](../sql/store.sql) a team store's tables.
Both run unchanged on SQLite, Dolt and PostgreSQL, under these rules:
1. No foreign keys; references are checked by validators, not by the database.
2. Enumerations are `VARCHAR` with `CHECK (… IN (…))`.
3. JSON values are canonical JSON in `TEXT` columns; booleans are `INTEGER` 0 or 1.
4. Every table has a primary key, and key columns are `VARCHAR` with a length, as MySQL-compatible engines require.
5. No table, column or index is named with a word reserved in SQLite, PostgreSQL or MySQL; so the formats' `key`, `index`, `at`, `by`, `before` and `after` are `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
## 13.2 Tables
| Table | One row per | Notes |
| --- | --- | --- |
| `works` | work | Its key and `same_as`. |
| `documents` | document | Every property of `document.json` as a column, plus `provenance`, `roster` and `package_hash`. |
| `witnesses` | witness | With `ord`, its position in `witnesses.json`. |
| `nodes` | node | With `seq`, its position in document order. |
| `pages`, `locations`, `refs` | page, location, reference | |
| `corrections` | correction | The common properties as columns; the rest in `details`. |
| `node_ids` | retired ID or alias | |
| `receipts`, `revocations` | receipt, revocation | |
| `suspects` | suspect | |
| `package_files` | asset, dataset file or derived file | Its path, SHA-256, size and whether its bytes were present. Bytes go to the asset store. |
| `citations` | cite run and reference | Derived on load. |
| `receipt_nodes` | node a receipt lists | Derived on load. |
A 30-page paper is about 3,000 nodes, so 500 documents are about a million and a half rows, which SQLite and Dolt hold comfortably.
## 13.3 Load and unload
Loading a package is a pure function of the package: each record becomes a row, an optional property absent from the record is NULL, and JSON-valued properties are stored as canonical JSON.
Unloading writes each file back from its rows in the file's defined order, omitting optional properties that are NULL and optional files that have no rows, copies the other files from the asset store, and writes the manifest.
Unloading gives the package back byte for byte, and its manifest hash with it; the suite checks this for every fixture.
## 13.4 The store tables
`store.sql` adds what a team adds to its documents: `roster_keys`, `projects`, `members`, `project_documents`, `reading_paths`, `layers`, `annotations`, `annotation_targets` (derived, so a reader finds the annotations on a node), `datasets`, `dataset_values`, `digitizations`, `views`, `policies`, `submissions` and `grades`.
Each keeps its record whole in a `record` or `descriptor` column, with the columns a query needs beside it.
## 13.5 The bundle
A bundle, `scienting-bundle/1`, is one SQLite file that a reader or a server queries.
It holds the tables of `doc.sql` and `store.sql`, then those of [sql/bundle-sqlite.sql](../sql/bundle-sqlite.sql):
| Table | Holds |
| --- | --- |
| `bundle_meta` | `name` and `value` pairs: at least `format` (`scienting-bundle/1`), `store`, `built_from` (the store revision) and `spec` (the version of these specifications). |
| `search` | An FTS5 index with one row per sentence, cell, equation outside a sentence and figure-text item, in document order: the document, the node, the unit, the context and the text. Withheld text is never indexed. |
| `embeddings_meta` | Reserved for embeddings with `sqlite-vec`: the model, its dimensions, and the rights use they were made under. |
A search row's `unit` is `sentence`, `cell`, `equation` or `figure-text`.
For a node, `context` is the kind of its parent, such as `paragraph`, `heading`, `caption` or `row`, and `text` is its normalized plain projection ([3.6](03-nodes.md#36-the-plain-projection)).
An equation inside a sentence is searched as part of that sentence, so it has no row of its own.
A figure-text item names its panel as the node, its `role` as the context and its normalized `text` as the text.
At team scale this is all the search a team needs.
A team can download its bundle and use it offline.
Because a bundle holds text, it is served only under the rights of [14](14-rights.md), checked again whenever text is served.
spec/14-rights.md
+61
View File
@@ -0,0 +1,61 @@
# 14. Rights
What a team may do with a paper depends on how it holds the paper, and AI processing is now a use that licenses name.
Every witness carries a rights record, every stage declares the uses it needs, and a stage runs only when the rights allow them.
Nothing here is legal advice: the defaults are a starting point that each team reviews with its own counsel.
## 14.1 The rights record
`scienting-rights/1`:
| Property | Holds |
| --- | --- |
| `license` | The license as stated, `{spdx?, text?, url?}`, or null. |
| `basis` | How the team holds the right: `open-license`, `publisher-tdm`, `collective-license`, `subscription`, `own-work`, `permission` or `unknown`. |
| `uses` | Each of `read`, `tdm`, `redistribute`, `quote-public` and `accessible-copy` as `true`, `false` or `"unknown"`, and `ai`: a map from each pair `<purpose>/<processor>` to the same values. |
| `asserted_by` | `{by, evidence}`: who or what recorded the basis, and on what: a license URL, a contract's hash, Crossref's license field, a delivery, a statement. |
| `at`, `valid_until` | When it was recorded, and when the basis ends, or null. |
Purposes are `internal` (results stay inside the organization) and `external` (results leave it).
Processors are `local` (the team's machines), `tenancy` (the team's cloud account), `outside-no-retention` (an outside service contractually keeping nothing) and `outside` (one that may retain).
A use or pair that is absent is `"unknown"`.
A producer sets `license` and a provisional `basis` from Crossref for open-access papers, and never sets a use to `true` on its own for anything else.
## 14.2 Effective uses
To find a witness's uses at a time *t*:
1. If `valid_until` is set and *t* is not before it, every use is `false`: an expired basis stops everything at once.
2. Otherwise start from the record's uses, and for each policy that applies at *t*, replace each `"unknown"` with the policy's value, where the policy has one.
A policy never overrides `true` or `false`.
A policy, `scienting-policy/1`, is a project owner's signed decision: `applies_to` (any of `basis`, `license`, `doi_prefixes` and `witnesses`, all of which must match), the `uses` it decides, a `statement`, `by`, `at` and `valid_until`.
It applies from `at` until `valid_until`.
A node established from several witnesses takes the most restrictive of their uses: `false` if any is false, else `"unknown"` if any is unknown, else `true`.
This is the default combining rule.
## 14.3 Stages
`scienting-stages/1` maps each stage to what it needs: a list of `uses`, and optionally `ai`, with a `purpose` and the `processors` it may run on.
A stage may run on a processor when every listed use is `true`, the processor is among those allowed, if any are listed, and the pair `ai/<purpose>/<processor>` is `true`.
When a check fails, the refusal lists what is lacking: a use or pair that is `false` by name, one that is unknown with a trailing `?`, and a disallowed processor as `processor:<name>`.
[registry/stages/scienting-stages-default.json](../registry/stages/scienting-stages-default.json) holds a default set; a team replaces the file, not the code.
Rights are checked when a stage runs and again whenever text is served.
Every derived artifact records the rights record and use it was made under, so it can be listed and purged when a basis ends.
## 14.4 Backends and profiles
A backend's code and weights can be licensed differently, and the weight license may exclude commercial or competing use.
`scienting-backend/1` declares a backend's `code_license`, its `weights` with their licenses, and the `conditions` they impose:
| Condition | Met when the team profile |
| --- | --- |
| `revenue-cap` with `usd` | states revenue or funding at most `usd`. |
| `no-competition` with `vendor` | states what it competes with, and that does not include the vendor. |
| `purpose` with `allowed` | states a purpose among those allowed. |
`scienting-profile/1` records what conditions ask: the team's `purpose` (`research`, `personal`, `education`, `startup` or `commercial`), `revenue_or_funding_usd` as `{at_most}` or null when unknown, and `competes_with`, or null when unknown.
A condition whose answer is unknown is not met, and the gate refuses a backend with any unmet condition.
`rights.jsonl` exercises stages, policies, expiry, combining and backends.
spec/15-datasets.md
+27
View File
@@ -0,0 +1,27 @@
# 15. Datasets
A dataset, `scienting-dataset/1`, is a Frictionless Data Package (`datapackage.json` with CSV or Parquet resources) with three additions.
| Addition | Meaning |
| --- | --- |
| Units | Every numeric field has a `unit`, a string Pint can parse, so a checker can test dimensions. |
| Uncertainty | A field may give `uncertainty`: `{field?, kind}`, the field that holds it and its kind, `sd`, `se`, `resolution` or `unknown`. |
| Origin | Each resource has an `origin`, or `origin_field` and `origins` when it varies by row. |
| Origin | Holds |
| --- | --- |
| `reported` | Values anchored to the table cells or sentences that print them, in `anchors.jsonl`. |
| `digitized` | Values read off a figure: the digitization's ID, and `anchors.jsonl` naming each value's series and point. |
| `computed` | A claim's implementation, by claim and pin. |
| `measured` | The team's own data, with its provenance. |
The descriptor also carries `format` and an `id`, `ds-` and a key.
Each line of `anchors.jsonl` names a `resource`, a `row` (by primary key) and a `field`, and either an `anchor` or a `digitization`, `series` and `point`.
A value or row is addressed as `ds-…/<row>/<field>`.
Values keep their digits as printed: CSV cells are never reformatted, and in JSON values are strings, so `1.526` and `1.5260` stay different.
A digitized value never passes as reported, and its uncertainty is at least the resolution of its digitization.
A dataset's content hash is SHA-256 of the canonical JSON of `{"format": "scienting-dataset/1", "files": [[path, sha256, size], …]}` over its files, sorted by path.
A dataset can be a source of kind `dataset`, so a claim can rest on a value read from a figure, with that fact visible in its evidence.
`fixtures/packages/hale2026harmonic-damb/datasets/ds-hale-bond-params` reports Table 1, and the suite checks that every value resolves to the cell that prints it.
spec/16-digitization.md
+18
View File
@@ -0,0 +1,18 @@
# 16. Digitization records
A digitization, `scienting-digitization/1`, says how numbers were read off a figure panel, completely enough to repeat.
| Part | Holds |
| --- | --- |
| `target` | The panel's `anchor`, and the `region` read, as a box on the page given by `page_fp`, with the witness's SHA-256 as evidence; or, for a figure that exists only as an image, the image's SHA-256 with the region in its pixels. |
| `axes` | For each axis: its `scale` (`linear`, `log`, `reciprocal` or `date`), `unit`, at least two `calibration` points from page coordinates to value, an optional third `check`, and its `label` anchored to the panel's figure text. |
| `series` | For each series: its legend, its mark (color, marker, line), its kind (`points`, `line`, `bars`, `error-bars` or `region`) and its values: each a position, a value, and an uncertainty from the image's resolution, the marker's size and the calibration's residual. |
| `method` | `manual`, `traced` or `proposed` (by a model), with the tool and its version. |
| `verification` | The overlay check: the image of the values re-plotted under the calibration, by SHA-256, and the mean and largest residual. |
| `confirmation` | The receipt of the person who confirmed it after seeing the overlay, or null. |
| `dataset`, `rev` | The dataset it yields, and the revision it was made at. |
Calibration and values are in page coordinates, not an image's pixels, so a digitization does not depend on the raster it was done on.
A digitization stays current while the document has a page with its `page_fp` and its panel anchor is not stale.
A teammate's re-download, stamped differently, has the same fingerprints, so the digitization stays current on it; a different edition or region has a different fingerprint, so a swapped image never passes.
A digitization counts only when a person confirms it: a receipt with the record `["digitization", id, hash]`, where the hash is SHA-256 of the record without `confirmation`.
spec/17-views.md
+8
View File
@@ -0,0 +1,8 @@
# 17. Views
A view, `scienting-view/1`, is a Vega-Lite specification whose `usermeta.scienting` pins its data: `{format, id, data, computation?}`.
Each entry of `data` names a Vega-Lite data source, the dataset it is, the dataset's content hash and a resource, and the specification refers to it as `{"data": {"name": …}}`.
A view is declarative JSON, so it is diffable, renders without custom code, and is easy for a model to write.
A view can also hold a computation: a claim's implementation, by claim and pin, run in the browser, with Vega-Lite parameters bound to its inputs.
That is how a reader drags a bond constant and watches the energy curve change beside the equation that defines it.
spec/18-layers.md
+63
View File
@@ -0,0 +1,63 @@
# 18. Layers and annotations
Everything people and models add to a document is an annotation, and annotations are grouped in layers.
`scienting-layer/1` is the W3C Web Annotation Data Model with the anchor as a selector, so any Web Annotation tool can read it.
The JSON-LD context is [context/scienting-1.jsonld](../context/scienting-1.jsonld), published at `https://paperspec.scient.ing/context/scienting-1.jsonld`.
## 18.1 One rule decides what is a claim and what is a layer
If a statement is true independently of this paper, it belongs in a claim store as a claim, where it is checked and leveled.
If it is about reading this paper, it belongs in a layer, where it is attributed and signed but not checked.
## 18.2 The layer
A layer is a folder with `layer.json` and `annotations.jsonl`.
`layer.json` is an `AnnotationCollection` with the two contexts, `format`, an `id` of the form `tag:scient.ing,2026:layer:<store>/<record id>`, a `label`, a `purpose`, an `audience` (`novice`, `student`, `expert` or `mixed`), a `visibility` (`private`, `team` or `public`), a `license` (CC BY 4.0 by default when public), its `creator`, `created`, its `project`, the `docs` it annotates and its `total`.
A reader turns layers on and off, and the audience dial picks the level of glosses.
## 18.3 The annotation
Each line of `annotations.jsonl` is a W3C `Annotation`:
| Property | Holds |
| --- | --- |
| `@context` | `["http://www.w3.org/ns/anno.jsonld", "https://paperspec.scient.ing/context/scienting-1.jsonld"]`. |
| `id` | `tag:scient.ing,2026:annotation:<store>/<record id>` ([4.6](04-identity.md#46-record-ids)). |
| `type`, `layer` | `Annotation`, and the layer's ID. |
| `motivation` | A W3C motivation: `commenting`, `describing`, `linking`, `questioning`, `assessing`, `replying` and the others. |
| `creator`, `created`, `modified` | An agent `{id, type, name}`, with `type` `Person` or `Software`, and times. |
| `body` | One body ([18.4](#184-bodies)). |
| `target` | A `SpecificResource` whose `source` is `tag:scient.ing,2026:doc:<store>/<key>` and whose `selector` lists an `AnchorSelector` holding a `scienting-anchor/1`, optionally followed by a W3C `TextQuoteSelector` for tools that do not know ours; or, for a reply, a `SpecificResource` whose source is another annotation. |
| `status` | `proposed`, `confirmed`, `rejected` or `withdrawn`. A model's annotation is `proposed` until a person confirms it. |
| `receipt` | For a confirmed annotation, the receipt that covers it. |
The `TextQuoteSelector`, when present, repeats the anchor quote's `exact`, `prefix` and `suffix`, with an empty string for a `prefix` or `suffix` the quote lacks.
It is present only when the anchor's quote is plain text; an anchor whose quote is hashed or keyed ([6.3](06-anchors.md#63-keyed-anchors)) has none, so the fallback never shows text the anchor withholds.
An annotation's content hash is SHA-256 of the canonical JSON of the annotation without `status`, `receipt` and `modified`.
A confirmed annotation is covered by its confirmer's receipt, which lists `["annotation", <record id>, <content hash>]`.
Replies are annotations that target annotations, so discussions are threads anchored to sentences.
## 18.4 Bodies
| `kind` | W3C type | Adds | Says |
| --- | --- | --- | --- |
| `gloss` | `TextualBody` | `audience` | An explanation at `novice`, `student` or `expert` level. |
| `claim-link` | `SpecificResource` | `source` (`lode:<tag>`), `pin`, `role` | That the span `states`, `uses` or `disputes` a claim. |
| `symbol` | `SpecificResource` | `source`, `pin`, `symbol` | That this occurrence of a symbol is a symbol claim. |
| `comment` | `TextualBody` | | A remark, opinion or interpretation. |
| `caveat` | `TextualBody` | | A warning a reader should see. |
| `question` | `TextualBody` | `for` | A question for a `reader`, a `group` or the `authors`. |
| `exercise` | `TextualBody` | `exercise` | A task with a key and a rule ([19](19-exercises.md)). |
| `citation-context` | `TextualBody` | `reference`, `verdict`, `reason` | Why a sentence cites a reference, and whether it supports it. |
| `prerequisite` | `SpecificResource` | `source` | What to read or know first. |
| `crosslink` | `SpecificResource` | `source`, `selector` | That the span corresponds to a span in another document. |
| `translation` | `TextualBody` | `language` | The span in another language. |
| `description` | `TextualBody` | `role` | A description of a figure, panel or table for readers who cannot see it: `alt`, a short text equivalent, or `long`, a full description. |
| `generated` | `TextualBody` | `generator` | Text a model wrote about the span; never mistaken for source. `generator` is `{model, run}` as a correction's model author is ([11.2](11-corrections.md#112-every-correction-records)). |
A textual body has a `value`, and optionally a `format` (`text/plain` or `text/markdown`) and a `language`.
Nothing a model writes is ever stored as source text.
A model may propose a `description`, and like any annotation it stays `proposed` until a person confirms it; an accessible export uses confirmed descriptions only, and a figure with none has no description.
Other text a model writes about a span, such as a summary, is a `generated` annotation, which names its `generator`.
`fixtures/records/layer/` holds a layer with one annotation of each common kind and the receipt that confirms some of them.
spec/19-exercises.md
+43
View File
@@ -0,0 +1,43 @@
# 19. Exercises, submissions and grades
Because every sentence and cell has a permanent address, the answer to a verification exercise is an address, and an address can be graded.
The three formats of this chapter, exercise, submission and grade, are `provisional` ([0.4](00-introduction.md#04-status)): no course has used them yet, and the first that does will shape them.
## 19.1 Exercises
An exercise is an annotation of kind `exercise` whose `exercise` property holds a `scienting-exercise/1`: an `exercise` kind, a `prompt`, a `key` and a `rule`.
| Kind | The student | The key |
| --- | --- | --- |
| `anchor` | Finds the sentence or cell that supports a statement. | `elements`: each a set of acceptable anchors, with points and feedback. |
| `check` | Decides whether a cited passage supports the citing sentence, and anchors the passage. | A `verdict` with points, and `elements`. |
| `trace` | Follows a value back to its original measurement. | A `chain`: an element per link, in order. |
| `reproduce` | Digitizes a plot and recomputes a value. | A `value`, a `tolerance`, a `unit`, the confirmed `digitization` it comes from, and points. |
| `verify` | Checks a model's `summary`, written with addresses, in which errors were planted. | `errors`: each a set of anchors and the correct verdict, with points. |
The rule's `match` is `same-node` (the student's anchor resolves to the node a key anchor resolves to), `overlap` (and, where the key has a span, the student's span covers at least `min_overlap` of it; default `0.5`), or `chain` (each link matches in order, with credit per link).
Feedback is `{hit, miss}` per element.
## 19.2 Verdicts
| Verdict | Means | As claim evidence, if a person confirms it |
| --- | --- | --- |
| `supports` | The passage says what the citing sentence claims. | `supports` |
| `partly` | It says some of it, or with a qualification the citing sentence drops. | `qualifies` |
| `does-not` | It does not say it, or says the opposite. | `disputes`, or no record when merely irrelevant; the reviewer chooses. |
| `cannot-tell` | The passage, or the student's access to it, is not enough. | No record. |
## 19.3 Submissions
A submission, `scienting-submission/1`, holds the exercise's ID, the document and revision, a pseudonymous `student` ID, and `answers`: `anchors`, a `verdict`, a `chain` of anchor lists, a `value`, or `flags` (each an anchor and a verdict), as the kind needs.
The student's name never enters a submission; the mapping from pseudonym to student stays in the course's records.
## 19.4 Grading
Before grading, every anchor in the key is resolved, and if any is stale nothing is graded with it.
Each element of the key earns its points when any of the student's anchors matches any of its anchors under the rule; a `check` earns its verdict's points when the verdicts are equal; a `reproduce` earns its points when the value is within the tolerance; a `verify` earns an error's points when a flag matches it with the correct verdict.
Points are decimal strings, summed exactly.
The result is a `scienting-grade/1`: the submission, the exercise, the `score`, the `max`, and an item per element with its points, its maximum and its feedback.
`grading.jsonl` grades a hit, a miss and an overlap.
Student submissions are educational records, so they live in the course's private store, never on a public log.
spec/20-projects.md
+17
View File
@@ -0,0 +1,17 @@
# 20. Projects
A project, `scienting-project/1`, is the unit a team works in.
| Part | Holds |
| --- | --- |
| `key`, `store`, `name`, `description` | Its identity. |
| `members` | People with a `role` (`owner`, `editor`, `reader` or `guest`) and the roster keys they sign with. |
| `documents` | The project's documents, each with the `reason` it is included, `tags`, and a reading `status` per member: `unread`, `reading` or `read`. |
| `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group. |
| `layers`, `datasets`, `views`, `drafts` | What the project holds. |
| `claims` | The claim namespace or bundle it uses, as the `lode` setting. |
| `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), the stage requirements in use, as the `id` of their stage file ([14](14-rights.md)), and the project's transparency `log`: its `origin`, the roster keys that `witness` it, and the `threshold` of witnesses a checkpoint needs ([23](23-log.md)). |
Projects are small on purpose: everything about them is designed for hundreds of documents, not millions.
Owners sign policies ([14.2](14-rights.md#142-effective-uses)).
Who may revoke receipts follows the roster instead: a key whose `role` in the roster is `owner` ([8.7](08-fidelity.md#87-revocations-and-the-roster)), so a project's owners hold keys with that role.
spec/21-drafts.md
+17
View File
@@ -0,0 +1,17 @@
# 21. Drafts
A draft is a document of kind `draft`, established from a Typst project as a born-digital witness, so every sentence is F2 from the start.
Its sentences cite library sentences by address, in `xref` runs whose targets are addresses in other documents, and its works in `cite` runs whose references resolve to documents.
## 21.1 The snapshot
A snapshot, `scienting-snapshot/1`, is to a Typst draft what a lock file is to a document: a pinned, verified file the compile reads, so a draft renders offline and reproducibly.
It holds the `store`, when it was made, the `quotes` the draft cites (each an address, an anchor, the text, its fidelity and how the anchor resolved), the `claims` (each a reference and pin, with whatever renderings the draft uses) and the `datasets` (each an ID and content hash).
## 21.2 Sentence IDs in a draft
Authors rewrite constantly, and IDs must survive without cluttering the source.
IDs live in a sidecar, `ids.jsonl` in the Typst project, `scienting-draft-ids/1`: one line per sentence with its `id`, its `text_hash`, its `section` and its `ord`, and `pinned` when the author fixed the ID to a label.
Each conversion aligns the new sentences with the previous revision, exact text first and then by edit distance within the section; an edited sentence keeps its ID and gets a new hash, so its annotations reattach or go stale as [6.5](06-anchors.md#65-resolution) says.
Both formats are `provisional` ([0.4](00-introduction.md#04-status)).
spec/22-conformance.md
+56
View File
@@ -0,0 +1,56 @@
# 22. Conformance
## 22.1 The suite
| Path | Holds | A conforming implementation |
| --- | --- | --- |
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document, as a folder and as the zip of [12.2](12-package.md#122-paths). | Accepts each, and recomputes every value it contains. |
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `tools/load.py --bundle`. | Accepts each. |
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |
All fixture text is synthetic and dedicated to the public domain; license fields inside fixtures describe the fictional works.
Each part's CI runs this suite against its own implementation, so the parts cannot drift apart silently.
## 22.2 Validating a package
A validator checks, in order, and reports each failure as `path:line: rule: message`:
1. The manifest: present, valid, sorted, every listed file present with its hash and size, nothing unlisted, paths safe and no symbolic links, required files present.
2. Every file in canonical form, byte for byte, and every record against its schema.
3. The tree ([3](03-nodes.md)), runs and references between nodes, the counts, and the keys ([2.4](02-model.md#24-work-and-document-keys)).
4. The policy and hashing scheme ([12.5](12-package.md#125-text-policies)).
5. The five values of every node, the root and `meta_hash`, when the text is present and the key, if any, is known.
6. The ID table, the corrections and the history root, the witnesses, pages, locations and references, and every file's order.
7. Receipts: IDs, roots, documents, node order, signatures against the package's roster, spot-check keys, seeds and samples, and the fidelity distribution.
## 22.3 Schemas
Schemas use draft 2020-12, restricted to the keywords `$schema`, `$id`, `$ref`, `$defs`, `$comment`, `title`, `description`, `examples`, `default`, `deprecated`, `type`, `enum`, `const`, `properties`, `required`, `additionalProperties`, `patternProperties`, `propertyNames`, `minProperties`, `maxProperties`, `dependentRequired`, `items`, `prefixItems`, `minItems`, `maxItems`, `uniqueItems`, `contains`, `minLength`, `maxLength`, `pattern`, `format`, `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then` and `else`.
Patterns avoid `\d`, `\w`, `\s`, `\b` and lookbehind, whose meaning differs between regular-expression dialects.
A schema checks shape; the chapters' rules, such as the five values, are checked by validators.
## 22.4 Rules
[registry/rules.json](../registry/rules.json) lists every rule ID with its meaning.
Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is never reused.
| Prefix | Covers |
| --- | --- |
| `package.` | The manifest, paths, files and schemas. |
| `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.` | The tree, node content and runs. |
| `doc.hash.` | The five values, the root and the metadata hash. |
| `doc.id.`, `doc.order.` | The ID table and file order. |
| `doc.policy.` | Text policies and keyed hashing. |
| `doc.key.`, `doc.counts`, `doc.history` | The keys, the counts and the history root of `document.json`. |
| `doc.correction.`, `correction.` | A package's corrections, and the gate. |
| `doc.receipt.`, `doc.qa.`, `doc.fidelity.` | Receipts, spot-checks and levels. |
| `doc.witness.`, `doc.page.`, `doc.location.`, `doc.reference.` | Witnesses and geometry. |
| `grade.` | Grading. |
## 22.5 Stability
Within a major version of a format, a valid file stays valid, and every value the chapters define is computed the same way.
Rule IDs, enumeration values and format identifiers are never reused with another meaning.
A `candidate` format changes only with a recorded reason and a new or changed vector; a `provisional` one may change more freely, and says so in [registry/formats.json](../registry/formats.json).
spec/23-log.md
+68
View File
@@ -0,0 +1,68 @@
# 23. The transparency log
A receipt's `at` is what its signer wrote, and a roster is a file a package carries.
Neither proves that a receipt existed before the key that signed it was revoked, or that the roster a verifier holds is the one the team published.
A team's transparency log settles both: an append-only record of every receipt, revocation and roster the team publishes, whose state is a Merkle tree root that the team signs and that witnesses may cosign.
The log is the same tree as receipts and the correction history ([8.5](08-fidelity.md#85-merkle-roots), [11.5](11-corrections.md#115-the-history-root)), with the same proofs.
Both formats of this chapter are `provisional` ([0.4](00-introduction.md#04-status)): the design follows the specifications that transparency logs share, and the first deployment will shape the details.
## 23.1 Entries
An entry, `scienting-log-entry/1`, is one item of the tree, `[kind, id, hash]`:
| `kind` | `id` | `hash` |
| --- | --- | --- |
| `receipt` | The receipt's record ID. | Its record hash ([8.4](08-fidelity.md#84-record-hashes-and-signatures)). |
| `revocation` | The revocation's record ID. | Its record hash. |
| `roster` | The store the roster is for. | SHA-256 of the canonical JSON of the roster. |
Entries are appended in the order the team publishes them and never removed.
The leaf hash of an entry is SHA-256 of the byte 0x00 and its canonical JSON, the interior hashes follow RFC 9162, and the root over no entries is SHA-256 of the empty string, as for the history root.
A roster is logged before any receipt a key it adds may sign, so a verifier can find the roster that governed a receipt by looking at the entries before it.
## 23.2 Checkpoints
A checkpoint, `scienting-checkpoint/1`, is a signed note in the format transparency logs share: an origin line, the tree size, the root hash in Base64, an empty line, and one signature line per signer.
```text
scient.ing/log/smithlab
41
Q2xlYXIgdGV4dCBpcyBub3QgYSByb290IGhhc2guLi4uLi4uLi4=
— ana AAAAAAHZ…
```
The origin names the log, and a team's logs are named `scient.ing/log/<store>` or under a domain the team controls.
A signature line is an em dash (U+2014), a space, the signer's roster key, a space, and Base64 of the signer's four-byte key hash followed by the signature.
The key hash is the first four bytes of SHA-256 of the signer's name, a newline, the byte 0x01 and the Ed25519 public key; the signature is Ed25519 over the note's first three lines, each ending in a newline.
A note parsed into JSON has `format`, `origin`, `size`, `root` as hex, `signatures` as `{name, key_hash, value}` and the `note` text itself; the schema is `checkpoint.schema.json`.
`checkpoint.jsonl` gives a checkpoint over the example paper's receipts and its verification.
A checkpoint may carry further signature lines from witnesses: roster keys with the `witness` purpose, held by independent parties, that cosign only a checkpoint consistent with every earlier one they saw.
A verifier that holds two checkpoints of one log checks a consistency proof between them ([11.5](11-corrections.md#115-the-history-root)); a log that cannot produce one has rewritten its past.
A project accepts a checkpoint when its origin is the project's log, a key with the role `owner` and the `sign` purpose signed it, and at least the project's `threshold` of its listed witnesses cosigned it ([20](20-projects.md)); `fixtures/records/log/checkpoint` is signed by the team and one witness.
## 23.3 What a log settles
When a project names a log in its settings ([20](20-projects.md)), a receipt counts toward a level only when the log holds it, and the log's order replaces the records' own times in two rules:
1. A revocation of a signer's period revokes the receipts logged before the revocation that fall in the period; a receipt logged after a revocation of its key never counts, whatever its `at` says.
2. The roster that governs a receipt is the latest roster entry before it.
A reader fetches the latest checkpoint, verifies its signatures against the roster and its consistency with the checkpoint it last saw, then verifies inclusion proofs for the receipts it relies on.
Without a log the rules of [8.6](08-fidelity.md#86-receipt-status) stand, and the `at` of each record is taken at its word.
## 23.4 Serving a log
A log is served as static files in the tile layout transparency logs share, so that a client fetches only the subtrees it needs and any file server can host a log.
| Path | Holds |
| --- | --- |
| `checkpoint` | The latest checkpoint note ([23.2](#232-checkpoints)). |
| `tile/<L>/<N>` | Hash tile N of level L: the 32-byte hashes of the 256 complete subtrees of 256^L entries each that the tile covers, in order. Level 0 holds the leaf hashes. |
| `tile/<L>/<N>.p/<W>` | The rightmost, partial tile of level L, holding W hashes. |
| `tile/entries/<N>` and `tile/entries/<N>.p/<W>` | The entries themselves, 256 to a tile, as canonical JSON Lines. |
N is written as 3-digit groups, every group but the last prefixed `x`: tile 1234067 is `tile/0/x001/x234/067`.
A tile at level L holds only the hashes of complete subtrees, so the hashes a partial tree still needs are computed from lower levels, as RFC 9162 computes a root from its subtrees.
`fixtures/records/log/` is the example paper's log in this layout, and `checkpoint.jsonl` lists its tile paths.
spec/README.md
+52
View File
@@ -0,0 +1,52 @@
# The paperspec specification
This specification states exactly what each paperspec file holds and how every value in it is computed.
The schemas, SQL, JSON-LD context, registries, vectors and fixtures are normative too, and each chapter says which of them it relies on.
A chapter and a vector that disagree signal a bug, so please report it in the repository, <https://git.scient.ing/infra/paperspec>.
Two readers will get the most from it: implementers who build a reader, writer, converter or checker, and experts who want the precise rules.
Newcomers should start with the [guides](../docs/README.md), which explain what the formats are for.
This text assumes you know why and tells you how.
## Reading paths
Find the row for the job you are doing and read its chapters after those in the first row.
Chapter numbers are those in the table below.
| If you are writing | Read |
| --- | --- |
| Anything at all | 0, 1, 3, 4 and 5: introduction, conventions, nodes, identity and the content values. |
| A reader of packages | Then 2, 6, 7 and 12: the model, anchors, geometry and the package. |
| A converter | Then 9, 10 and 11: figures and tables, references, and corrections with the gate. |
| A store or a server | Then 8, 13, 14 and 23: receipts, the database form, rights, and the transparency log. |
| A writer or reader of annotation, teaching or data files | Then 15 to 21. |
| A validator | Chapter 22, with [registry/rules.json](../registry/rules.json). |
## Chapters
| Chapter | Subject |
| --- | --- |
| [0. Introduction](00-introduction.md) | Scope, conformance, versions and status, and how the repository is arranged |
| [1. Shared conventions](01-conventions.md) | Canonical JSON, numbers, hashes, normalization, offsets, times, determinism, IDs, keys, files and prose |
| [2. Works, versions, witnesses and documents](02-model.md) | The four objects; `document.json` and `witnesses.json` |
| [3. Nodes](03-nodes.md) | Nodes, their kinds and children, content, attributes, inline runs and the plain projection |
| [4. Identity](04-identity.md) | Minting IDs, permanence, retirement, aliases, `ids.jsonl`, store-qualified names and record IDs |
| [5. Content values](05-hashes.md) | The five content values, the projections, alignment, suspects, keyed hashes and the metadata hash |
| [6. Addresses and anchors](06-anchors.md) | Addresses, anchors and the five resolutions, with drift |
| [7. Geometry](07-geometry.md) | Page fingerprints, stamp rules, `pages.jsonl` and `locations.jsonl` |
| [8. Fidelity, receipts and spot-checks](08-fidelity.md) | Fidelity levels, receipts, signatures, Merkle roots, the roster, revocations, spot-checks and error bounds |
| [9. Figures, tables and equations](09-figures-tables-equations.md) | Figures, panels, tables, cells and equations |
| [10. References and citations](10-references.md) | Reference nodes, `references.jsonl` and its resolutions, and sentence-level citations |
| [11. Corrections](11-corrections.md) | Corrections and the gate |
| [12. The package](12-package.md) | The files of a package, paths, manifest, provenance, text policies and assets |
| [13. The database form and the bundle](13-database.md) | The database form and the SQLite bundle |
| [14. Rights](14-rights.md) | Rights records, policies, stages and backends |
| [15. Datasets](15-datasets.md) | Frictionless Data Packages with units, uncertainty and origin; `anchors.jsonl`; content hash |
| [16. Digitization records](16-digitization.md) | How values were read off a figure panel: target, axes, series, method, verification and confirmation |
| [17. Views](17-views.md) | Vega-Lite specifications that pin their data, and optional computations |
| [18. Layers and annotations](18-layers.md) | Layers and annotations |
| [19. Exercises, submissions and grades](19-exercises.md) | Exercises, submissions and grades |
| [20. Projects](20-projects.md) | Members and roles, documents, reading paths, held layers, datasets, views and drafts, and settings |
| [21. Drafts](21-drafts.md) | Drafts: snapshots and sentence-ID sidecars |
| [22. Conformance](22-conformance.md) | Conformance, the suite and the rule catalog |
| [23. The transparency log](23-log.md) | A team's log of receipts, revocations and rosters, its checkpoints, and what it settles |