feat: documents and session tokens

spec/00-introduction.md

@@ -2,13 +2,11 @@
 
 ## 0.1 Scope
 
-These specifications define the data the Scientific Computing Studio's parts write and read: documents and their packages, anchors, receipts, rights records, layers, exercises, datasets, digitizations, views, projects and bundles.
+These specifications define the data that tools for reading, correcting, annotating and checking scientific papers exchange: documents and their packages, anchors, receipts, rights records, layers, exercises, datasets, digitizations, views, projects and bundles.
 They define data, not programs.
-A conforming file can be read, checked and loaded without any studio code, and every rule here is stated so that a second implementation, in any language, computes the same bytes.
+A conforming file can be read, checked and loaded without any particular program, and every rule is stated so that a second implementation, in any language, computes the same bytes.
 
-The design document, *The Scientific Computing Studio: design of Lode, paperpack and Varve*, explains why each format exists.
-This text says exactly what each format is.
-Where the two differ, this text governs, and [DECISIONS.md](../DECISIONS.md) records why it differs.
+The [guides](../docs/README.md) explain what each format is for, and the [rationale](../docs/rationale.md) why its rules are what they are.
 
 ## 0.2 Conformance language
 
@@ -39,16 +37,16 @@ The specification as a whole is versioned with semantic versioning; this is vers
 
 ## 0.4 Status
 
-Each format has one of three statuses.
+Each format has one of three statuses, listed in [registry/formats.json](../registry/formats.json).
 
 | Status | Meaning |
 | --- | --- |
-| `candidate` | Complete, with vectors and fixtures, and ready to implement. Changes before 1.0 need a recorded decision. |
-| `provisional` | Complete, but its milestone has not come, and it may change with less ceremony. |
+| `candidate` | Complete, with vectors and fixtures, and ready to implement. |
+| `provisional` | Complete, but expected to change as implementations gain experience with it. |
 | `reserved` | Named and shaped, but not yet specified in full. |
 
-The design builds in levels, and level 1 implements a subset.
-Fields that level 1 does not use, such as rights beyond the license, receipts and spot-checks, are defined now so that formats never change when a team moves up a level.
+Some fields matter only to some users: a personal library has no use for spot-checks, and a team that reads only open-access papers rarely needs a policy.
+Those fields are defined all the same, so that files move between a personal library and a team's store without conversion.
 A consumer that does not use a field MUST still preserve it when it rewrites a file.
 
 ## 0.5 How the repository is arranged
@@ -62,7 +60,8 @@ A consumer that does not use a field MUST still preserve it when it rewrites a f
 | `registry/` | Formats, rule IDs, stamp rules and the default stage requirements. | Yes |
 | `vectors/` | Test vectors, one case per line. | Yes |
 | `fixtures/` | Valid packages and records, invalid packages, and the keys that signed them. | Yes |
-| `tools/` | The oracle that computed the vectors, the builder and the checker. | No |
+| `tools/` | The programs that compute and check the vectors and fixtures, and load packages into SQLite. | No |
+| `docs/` | Guides for people getting to know the formats. | No |
 
-The oracle exists so that no vector is typed by hand.
-It is not a library: implementations conform to the chapters and the vectors, not to the oracle, and `scienting-core` is the reference implementation.
+The programs in `tools/` compute every expected value, so that none is typed by hand.
+They are not a library: implementations conform to the chapters and the vectors, and `scienting-core` is the reference implementation of the shared rules.

spec/05-hashes.md

@@ -64,7 +64,7 @@ Marking citations matters for the same reason: "10" followed by citation 3 must
 Each complex run also yields a complex key, `notation:source`, with the source in NFC and every whitespace character removed.
 A node's complex keys are listed in order.
 Two readings agree only when their `match_hash` values are equal and their complex keys are equal, run by run; this is stage 1 of canonical math.
-Stage 2, a later minor version, will replace U+FFFC with canonical MathML; it changes `match_hash` and `align_key` values but no `text_hash`, `own_hash` or `node_hash`, so no anchor goes stale.
+A later minor version may add stage 2, which replaces U+FFFC with canonical MathML; that would change `match_hash` and `align_key` values but no `text_hash`, `own_hash` or `node_hash`, so no anchor would go stale.
 
 ## 5.4 Simple notation
 

spec/08-fidelity.md

@@ -91,7 +91,7 @@ It is valid when its signature verifies and its signer is the revoked receipt's
 
 The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
 A package MAY carry the roster its receipts need as `roster.json`, so its receipts verify without the store.
-The roster is meant to match the one Lode already verifies; aligning the two is recorded in [DECISIONS.md](../DECISIONS.md).
+The roster is meant to be the same one Lode verifies; see the [rationale](../docs/rationale.md#limits-worth-knowing).
 
 ## 8.8 Spot-checks
 
@@ -109,7 +109,7 @@ Anyone can recompute the sample; `spotcheck.jsonl` does.
 
 ## 8.9 Reviewer error bounds
 
-One sample proves little; what a buyer wants is a bound on each reviewer's error rate across all their samples.
+One sample proves little; what someone relying on the work needs is a bound on each reviewer's error rate across all their samples.
 With *k* errors in *n* sampled nodes, the bound is the one-sided Clopper–Pearson upper confidence bound at confidence *c*, by default 0.95: the rate *p* at which the probability of at most *k* errors in *n* is 1 − *c*, or 1 when *k* = *n*.
 It is reported to six decimal places, rounded up.
 With no errors the bound is about 3/*n*, so zero errors in 300 sampled nodes bounds a rate below 1%; `bounds.jsonl` gives cases.

spec/11-corrections.md

@@ -20,7 +20,7 @@ Each has an `index`, counted from 1 in the order corrections are applied, and th
 | `set` | `target`, `attr`, `value` | Sets one structural attribute: a section's `level`, a table's `columns`, a cell's `column`, `rowspan`, `colspan` or `role`, a row's `header_path`, a list's `ordered`, an item's `marker` or a theorem's `variant`. | Unchanged. |
 
 A leaf's editable strings are the `text` of its `text`, `link`, `cite`, `xref` and `fnmark` runs and the `source` of its `math` runs, each searched on its own; an equation's `source`; a verbatim node's `text`.
-`set` is an addition to the design's eight operations, needed to fix table structure; see [DECISIONS.md](../DECISIONS.md).
+`set` exists because readings drop spans, roles and header paths; see the [rationale](../docs/rationale.md#corrections).
 Runs are made canonical after every operation, and the result MUST satisfy every rule of [3](03-nodes.md).
 
 ## 11.2 Every correction records

spec/12-package.md

@@ -67,4 +67,4 @@ Nothing derived is content, and a consumer may delete and rebuild it.
 
 ## 12.8 Media type and extension
 
-The proposed extension is `.sct.zip` and the media type `application/vnd.scienting.doc+zip`; both wait for a decision ([DECISIONS.md](../DECISIONS.md)) and are fixed forever once packages are shared.
+A zipped package has the extension `.sct.zip` and the media type `application/vnd.scienting.doc+zip`.

spec/13-database.md

@@ -9,7 +9,7 @@ Both run unchanged on SQLite, Dolt and PostgreSQL, under these rules:
 2. Enumerations are `VARCHAR` with `CHECK (… IN (…))`.
 3. JSON values are canonical JSON in `TEXT` columns; booleans are `INTEGER` 0 or 1.
 4. Every table has a primary key, and key columns are `VARCHAR` with a length, as MySQL-compatible engines require.
-5. No table, column or index is named with a word reserved in SQLite, PostgreSQL or MySQL; so the design's `key`, `index`, `at`, `by`, `before` and `after` are `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
+5. No table, column or index is named with a word reserved in SQLite, PostgreSQL or MySQL; so the formats' `key`, `index`, `at`, `by`, `before` and `after` are `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
 
 ## 13.2 Tables
 

spec/14-rights.md

@@ -2,7 +2,7 @@
 
 What a team may do with a paper depends on how it holds the paper, and AI processing is now a use that licenses name.
 Every witness carries a rights record, every stage declares the uses it needs, and a stage runs only when the rights allow them.
-Nothing here is legal advice; the defaults are proposals for counsel to set.
+Nothing here is legal advice: the defaults are a starting point that each team reviews with its own counsel.
 
 ## 14.1 The rights record
 
@@ -33,7 +33,7 @@ A policy, `scienting-policy/1`, is a project owner's signed decision: `applies_t
 It applies from `at` until `valid_until`.
 
 A node established from several witnesses takes the most restrictive of their uses: `false` if any is false, else `"unknown"` if any is unknown, else `true`.
-That combining rule is proposed for counsel to confirm.
+This is the default combining rule.
 
 ## 14.3 Stages
 
@@ -41,13 +41,13 @@ That combining rule is proposed for counsel to confirm.
 A stage may run on a processor when every listed use is `true`, the processor is among those allowed, if any are listed, and the pair `ai/<purpose>/<processor>` is `true`.
 When a check fails, the refusal lists what is lacking: a use or pair that is `false` by name, one that is unknown with a trailing `?`, and a disallowed processor as `processor:<name>`.
 
-[registry/stages/scienting-stages-default.json](../registry/stages/scienting-stages-default.json) holds the design's proposed defaults; a team replaces the file, not the code.
+[registry/stages/scienting-stages-default.json](../registry/stages/scienting-stages-default.json) holds a default set; a team replaces the file, not the code.
 Rights are checked when a stage runs and again whenever text is served.
 Every derived artifact records the rights record and use it was made under, so it can be listed and purged when a basis ends.
 
 ## 14.4 Backends and profiles
 
-A backend's code and weights can be licensed differently, and the weights decide whether a paying team may use it.
+A backend's code and weights can be licensed differently, and the weight license may exclude commercial or competing use.
 `scienting-backend/1` declares a backend's `code_license`, its `weights` with their licenses, and the `conditions` they impose:
 
 | Condition | Met when the team profile |

spec/21-drafts.md

@@ -14,4 +14,4 @@ Authors rewrite constantly, and IDs must survive without cluttering the source.
 IDs live in a sidecar, `ids.jsonl` in the Typst project, `scienting-draft-ids/1`: one line per sentence with its `id`, its `text_hash`, its `section` and its `ord`, and `pinned` when the author fixed the ID to a label.
 Each conversion aligns the new sentences with the previous revision, exact text first and then by edit distance within the section; an edited sentence keeps its ID and gets a new hash, so its annotations reattach or go stale as [6.5](06-anchors.md#65-resolution) says.
 
-Both formats are provisional until the Typst package exists.
+Both formats are `provisional` ([0.4](00-introduction.md#04-status)).

spec/22-conformance.md

@@ -48,6 +48,8 @@ Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is
 | `doc.witness.`, `doc.page.`, `doc.location.`, `doc.reference.` | Witnesses and geometry. |
 | `grade.` | Grading. |
 
-## 22.5 Freezing 1.0
+## 22.5 Stability
 
-These specifications become 1.0 when the formats have been written by at least three producers (conversion, hand-built fixtures, and a structured witness such as JATS) and read by at least three consumers (databases, Lode and the reader), and at least one paying buyer has used a delivery built on them.
+Within a major version of a format, a valid file stays valid, and every value the chapters define is computed the same way.
+Rule IDs, enumeration values and format identifiers are never reused with another meaning.
+A `candidate` format changes only with a recorded reason and a new or changed vector; a `provisional` one may change more freely, and says so in [registry/formats.json](../registry/formats.json).

spec/README.md

@@ -1,8 +1,22 @@
 # The scienting specifications
 
-These chapters are the normative text of `spec` 0.1.
+This is the specification of the scienting formats: exactly what each file holds and how every value in it is computed.
+For what the formats are for, start with the [guides](../docs/README.md); this text assumes you know why and tells you how.
 Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are normative too, and the chapters say which.
-Where a chapter and a vector disagree, the disagreement is a bug in this repository; report it.
+Where a chapter and a vector disagree, the disagreement is a bug; please report it.
+
+## Reading paths
+
+| If you are writing | Read |
+| --- | --- |
+| Anything at all | 0, 1, 3, 4 and 5: conventions, nodes, identity and the content values. |
+| A reader of packages | Then 2, 6, 7 and 12: the model, anchors, geometry and the package. |
+| A converter | Then 9, 10 and 11: figures and tables, references, and corrections with the gate. |
+| A store or a server | Then 8, 13 and 14: receipts, the database form, and rights. |
+| A tool for annotation, teaching or data | Then 15 to 21. |
+| A validator | Chapter 22, with [registry/rules.json](../registry/rules.json). |
+
+## Chapters
 
 | Chapter | Subject |
 | --- | --- |