feat: validator, bundle builder and release checks for 0.1.0

- Add tools/validate.py: validates packages, and with --correction runs
  proposed corrections through the gate against a package without
  writing anything (spec/11 §11.4). The gate now refuses a proposal that
  fails the correction schema, rule correction.schema.
- tools/load.py validates every package before loading, and with
  --bundle --built-from builds a scienting-bundle/1 with the FTS5 search
  index and bundle_meta (spec/13 §13.5). Search rows are defined per
  unit (sentence, cell, equation outside a sentence, figure-text) with
  the parent kind as context.
- Roster keys carry a role, and a key with the role owner may revoke any
  receipt (spec/08 §8.7, spec/20); roster schema, store table and the
  fixture roster updated. The checker signs a trial revocation to prove
  the rule both ways.
- A qa: receipt raises the nodes it sampled except those in its errors
  (spec/08 §8.2); the example paper's fidelity counts become 27 F3,
  13 F2, 4 F1 and 1 F0.
- The destination chooses the keyed scheme, not the text policy:
  hmac-team between the team's machines, hkdf-pages outside the team
  (spec/05 §5.8, spec/12 §12.5).
- A TextQuoteSelector must repeat the anchor's plain quote and is absent
  for hashed or keyed anchors (spec/18 §18.3).
- Add fixtures/records/suspect.json, a scienting-suspect/1 record.
- Oracle: hashes are kept incrementally and checked after every
  correction against a full recomputation; load and write refuse unsafe
  manifest paths; Ed25519, HKDF and schema validation come from the
  cryptography and jsonschema packages; canonical JSON uses the built-in
  encoder with an exact RFC 8785 fallback.
- The tools need Python 3.14 with cryptography and jsonschema; the py310
  CI environment is dropped and the workflow runs the default only.
- Resolve every [CHECK] note in the docs, READMEs and context; add
  CONTRIBUTING.md; every tool answers --help and refuses unknown options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 18:58:00 -04:00
1 parent b357af84d8
commit 8c8b51cc3f
89 files changed
+1141 -513

No files matched your search

+5 -2
View File
@@ -170,8 +170,11 @@ So every content value that leaves a team's store is keyed.
| Where the value goes | `hashing` | Key |
| --- | --- | --- |
| The team's own store and bundles | `{"scheme": "plain"}` | None. |
| An export to another machine of the team | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
| A public layer, or any export on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
| An export to another machine of the team, whatever its text policy | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
| A public layer, or an export outside the team on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
The destination chooses the scheme, not the text policy.
A checks-only package ([12.5](12-package.md#125-text-policies)) sent between the team's own machines is keyed with the team key, as `hale2026harmonic.checks-only` is; sent to anyone outside the team, it is keyed with the page key.
A keyed scheme replaces SHA-256 with HMAC-SHA-256 under its key in every content value, including the Merkle construction of `node_hash`, and in `meta_hash`.
+5 -2
View File
@@ -17,6 +17,7 @@ A correction that changes a leaf drops it back: the gate's crop evidence makes i
A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`.
A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does.
A spot-check's `qa:` receipt ([8.8](#88-spot-checks)) raises the nodes it sampled, except those its `errors` name.
A document reports, in `document.json`, the lowest level among its leaves and the count at each level.
## 8.3 Receipts
@@ -88,9 +89,11 @@ A suspended receipt stays suspended; its reviewer reviews the sitting again and
## 8.7 Revocations and the roster
A revocation, `scienting-revocation/1`, names one `receipt`, or a `signer` and a `period` `{from, to}`, with a `reason`, and is signed like a receipt.
It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or an owner of the project.
It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or a key whose `role` in the roster is `owner`.
The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `role`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
The `role` is the holder's role in the team that keeps the roster, `owner`, `editor`, `reader` or `guest`, as in a project ([20](20-projects.md)).
Because the roster carries it, a package that carries its roster carries everything needed to judge its revocations.
A package MAY carry the roster its receipts need as `roster.json`, so its receipts verify without the store.
The roster is meant to be the one a store verifies against; see the [rationale](../docs/rationale.md#known-limits).
+14 -10
View File
@@ -54,16 +54,20 @@ Drift ([6.6](06-anchors.md#66-drift)) is computed from `path`, `to_path` and `to
A correction is refused, naming the rule ([22](22-conformance.md)), when:
1. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified;
2. `find` is empty or does not occur exactly once;
3. an `insert` has no crop evidence;
4. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits;
5. any leaf outside `touched` has different digits afterwards;
6. a `split` falls outside the sentence, inside a run other than text, or leaves a part empty;
7. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings;
8. a `move` would put a node inside itself;
9. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it; or
10. the result breaks a rule of [3](03-nodes.md).
1. it does not match the correction schema, `schemas/correction.schema.json`;
2. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified;
3. `find` is empty or does not occur exactly once;
4. an `insert` has no crop evidence, or names a run or offset its target does not have;
5. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits;
6. any leaf outside `touched` has different digits afterwards;
7. a `split` falls outside the sentence, inside a run other than text, or leaves a part empty;
8. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings;
9. a `move` would put a node inside itself;
10. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it; or
11. the result breaks a rule of [3](03-nodes.md).
The gate needs only the document's current state, so a tool can run it on a proposal before applying anything, and get back the record the correction would append or the rule it breaks.
`tools/validate.py --correction` does this against a package.
Digits are characters of General Category Nd in the plain projection.
A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, and the last correction of each leaf ends at its current `node_hash`.
+2 -2
View File
@@ -48,11 +48,11 @@ Every stage that uses a model or an outside service records where it ran ([14](1
| Policy | Means |
| --- | --- |
| `full` | Every leaf keeps its content. |
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed. |
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). |
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
Its text lives in the team's private asset store, and only key holders can check its values.
`fixtures/packages/hale2026harmonic.checks-only` is such an export of `hale2026harmonic` under the team key in `fixtures/keys/`.
`fixtures/packages/hale2026harmonic.checks-only` is such an export of `hale2026harmonic` to another machine of the team, under the team key in `fixtures/keys/`.
## 12.6 Assets
+6 -1
View File
@@ -49,9 +49,14 @@ It holds the tables of `doc.sql` and `store.sql`, then those of [sql/bundle-sqli
| Table | Holds |
| --- | --- |
| `bundle_meta` | `name` and `value` pairs: at least `format` (`scienting-bundle/1`), `store`, `built_from` (the store revision) and `spec` (the version of these specifications). |
| `search` | An FTS5 index with one row per sentence, cell, display equation and figure-text item: the document, the node, the unit, the context (the kind of a sentence's container) and the normalized plain projection. Withheld text is never indexed. |
| `search` | An FTS5 index with one row per sentence, cell, equation outside a sentence and figure-text item, in document order: the document, the node, the unit, the context and the text. Withheld text is never indexed. |
| `embeddings_meta` | Reserved for embeddings with `sqlite-vec`: the model, its dimensions, and the rights use they were made under. |
A search row's `unit` is `sentence`, `cell`, `equation` or `figure-text`.
For a node, `context` is the kind of its parent, such as `paragraph`, `heading`, `caption` or `row`, and `text` is its normalized plain projection ([3.6](03-nodes.md#36-the-plain-projection)).
An equation inside a sentence is searched as part of that sentence, so it has no row of its own.
A figure-text item names its panel as the node, its `role` as the context and its normalized `text` as the text.
At team scale this is all the search a team needs.
A team can download its bundle and use it offline.
Because a bundle holds text, it is served only under the rights of [14](14-rights.md), checked again whenever text is served.
+3
View File
@@ -31,6 +31,9 @@ Each line of `annotations.jsonl` is a W3C `Annotation`:
| `status` | `proposed`, `confirmed`, `rejected` or `withdrawn`. A model's annotation is `proposed` until a person confirms it. |
| `receipt` | For a confirmed annotation, the receipt that covers it. |
The `TextQuoteSelector`, when present, repeats the anchor quote's `exact`, `prefix` and `suffix`, with an empty string for a `prefix` or `suffix` the quote lacks.
It is present only when the anchor's quote is plain text; an anchor whose quote is hashed or keyed ([6.3](06-anchors.md#63-keyed-anchors)) has none, so the fallback never shows text the anchor withholds.
An annotation's content hash is SHA-256 of the canonical JSON of the annotation without `status`, `receipt` and `modified`.
A confirmed annotation is covered by its confirmer's receipt, which lists `["annotation", <record id>, <content hash>]`.
Replies are annotations that target annotations, so discussions are threads anchored to sentences.
+3 -2
View File
@@ -10,7 +10,8 @@ A project, `scienting-project/1`, is the unit a team works in.
| `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group. |
| `layers`, `datasets`, `views`, `drafts` | What the project holds. |
| `claims` | The claim namespace or bundle it uses, as the `lode` setting. |
| `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), and the stage requirements in use. |
| `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), and the stage requirements in use, as the `id` of their stage file ([14](14-rights.md)). |
Projects are small on purpose: everything about them is designed for hundreds of documents, not millions.
Owners sign policies ([14.2](14-rights.md#142-effective-uses)) and may revoke receipts ([8.7](08-fidelity.md#87-revocations-and-the-roster)).
Owners sign policies ([14.2](14-rights.md#142-effective-uses)).
Who may revoke receipts follows the roster instead: a key whose `role` in the roster is `owner` ([8.7](08-fidelity.md#87-revocations-and-the-roster)), so a project's owners hold keys with that role.
+1 -1
View File
@@ -6,7 +6,7 @@
| --- | --- | --- |
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document. | Accepts each, and recomputes every value it contains. |
| `fixtures/records/` | A valid record of every other format. | Accepts each. |
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `tools/load.py --bundle`. | Accepts each. |
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |