feat: validator, bundle builder and release checks for 0.1.0
- Add tools/validate.py: validates packages, and with --correction runs proposed corrections through the gate against a package without writing anything (spec/11 §11.4). The gate now refuses a proposal that fails the correction schema, rule correction.schema. - tools/load.py validates every package before loading, and with --bundle --built-from builds a scienting-bundle/1 with the FTS5 search index and bundle_meta (spec/13 §13.5). Search rows are defined per unit (sentence, cell, equation outside a sentence, figure-text) with the parent kind as context. - Roster keys carry a role, and a key with the role owner may revoke any receipt (spec/08 §8.7, spec/20); roster schema, store table and the fixture roster updated. The checker signs a trial revocation to prove the rule both ways. - A qa: receipt raises the nodes it sampled except those in its errors (spec/08 §8.2); the example paper's fidelity counts become 27 F3, 13 F2, 4 F1 and 1 F0. - The destination chooses the keyed scheme, not the text policy: hmac-team between the team's machines, hkdf-pages outside the team (spec/05 §5.8, spec/12 §12.5). - A TextQuoteSelector must repeat the anchor's plain quote and is absent for hashed or keyed anchors (spec/18 §18.3). - Add fixtures/records/suspect.json, a scienting-suspect/1 record. - Oracle: hashes are kept incrementally and checked after every correction against a full recomputation; load and write refuse unsafe manifest paths; Ed25519, HKDF and schema validation come from the cryptography and jsonschema packages; canonical JSON uses the built-in encoder with an exact RFC 8785 fallback. - The tools need Python 3.14 with cryptography and jsonschema; the py310 CI environment is dropped and the workflow runs the default only. - Resolve every [CHECK] note in the docs, READMEs and context; add CONTRIBUTING.md; every tool answers --help and refuses unknown options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
89 files changed
+1141
-513
No files matched your search
+5
-2
@@ -170,8 +170,11 @@ So every content value that leaves a team's store is keyed.
|
||||
| Where the value goes | `hashing` | Key |
|
||||
| --- | --- | --- |
|
||||
| The team's own store and bundles | `{"scheme": "plain"}` | None. |
|
||||
| An export to another machine of the team | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
|
||||
| A public layer, or any export on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
|
||||
| An export to another machine of the team, whatever its text policy | `{"scheme": "hmac-team", "key_id": …}` | The team key. |
|
||||
| A public layer, or an export outside the team on a work whose text is withheld | `{"scheme": "hkdf-pages", "context": …, "pages": […]}` | A key derived from the witness's page fingerprints. |
|
||||
|
||||
The destination chooses the scheme, not the text policy.
|
||||
A checks-only package ([12.5](12-package.md#125-text-policies)) sent between the team's own machines is keyed with the team key, as `hale2026harmonic.checks-only` is; sent to anyone outside the team, it is keyed with the page key.
|
||||
|
||||
A keyed scheme replaces SHA-256 with HMAC-SHA-256 under its key in every content value, including the Merkle construction of `node_hash`, and in `meta_hash`.
|
||||
|
||||
|
||||
+5
-2
@@ -17,6 +17,7 @@ A correction that changes a leaf drops it back: the gate's crop evidence makes i
|
||||
|
||||
A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`.
|
||||
A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does.
|
||||
A spot-check's `qa:` receipt ([8.8](#88-spot-checks)) raises the nodes it sampled, except those its `errors` name.
|
||||
A document reports, in `document.json`, the lowest level among its leaves and the count at each level.
|
||||
|
||||
## 8.3 Receipts
|
||||
@@ -88,9 +89,11 @@ A suspended receipt stays suspended; its reviewer reviews the sitting again and
|
||||
## 8.7 Revocations and the roster
|
||||
|
||||
A revocation, `scienting-revocation/1`, names one `receipt`, or a `signer` and a `period` `{from, to}`, with a `reason`, and is signed like a receipt.
|
||||
It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or an owner of the project.
|
||||
It is valid when its signature verifies and its signer is the revoked receipt's signer, the signer it names, or a key whose `role` in the roster is `owner`.
|
||||
|
||||
The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
|
||||
The roster, `scienting-roster/1`, lists the keys that may sign: each with an `id`, a `person`, a `role`, a `scheme`, its `public` key (Base64 for Ed25519), its `purposes` (`sign`, `spot-check`, `export`), `registered_at` and `revoked_at`.
|
||||
The `role` is the holder's role in the team that keeps the roster, `owner`, `editor`, `reader` or `guest`, as in a project ([20](20-projects.md)).
|
||||
Because the roster carries it, a package that carries its roster carries everything needed to judge its revocations.
|
||||
A package MAY carry the roster its receipts need as `roster.json`, so its receipts verify without the store.
|
||||
The roster is meant to be the one a store verifies against; see the [rationale](../docs/rationale.md#known-limits).
|
||||
|
||||
|
||||
+14
-10
@@ -54,16 +54,20 @@ Drift ([6.6](06-anchors.md#66-drift)) is computed from `path`, `to_path` and `to
|
||||
|
||||
A correction is refused, naming the rule ([22](22-conformance.md)), when:
|
||||
|
||||
1. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified;
|
||||
2. `find` is empty or does not occur exactly once;
|
||||
3. an `insert` has no crop evidence;
|
||||
4. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits;
|
||||
5. any leaf outside `touched` has different digits afterwards;
|
||||
6. a `split` falls outside the sentence, inside a run other than text, or leaves a part empty;
|
||||
7. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings;
|
||||
8. a `move` would put a node inside itself;
|
||||
9. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it; or
|
||||
10. the result breaks a rule of [3](03-nodes.md).
|
||||
1. it does not match the correction schema, `schemas/correction.schema.json`;
|
||||
2. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified;
|
||||
3. `find` is empty or does not occur exactly once;
|
||||
4. an `insert` has no crop evidence, or names a run or offset its target does not have;
|
||||
5. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits;
|
||||
6. any leaf outside `touched` has different digits afterwards;
|
||||
7. a `split` falls outside the sentence, inside a run other than text, or leaves a part empty;
|
||||
8. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings;
|
||||
9. a `move` would put a node inside itself;
|
||||
10. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it; or
|
||||
11. the result breaks a rule of [3](03-nodes.md).
|
||||
|
||||
The gate needs only the document's current state, so a tool can run it on a proposal before applying anything, and get back the record the correction would append or the rule it breaks.
|
||||
`tools/validate.py --correction` does this against a package.
|
||||
|
||||
Digits are characters of General Category Nd in the plain projection.
|
||||
A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, and the last correction of each leaf ends at its current `node_hash`.
|
||||
|
||||
+2
-2
@@ -48,11 +48,11 @@ Every stage that uses a model or an outside service records where it ran ([14](1
|
||||
| Policy | Means |
|
||||
| --- | --- |
|
||||
| `full` | Every leaf keeps its content. |
|
||||
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed. |
|
||||
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). |
|
||||
|
||||
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
|
||||
Its text lives in the team's private asset store, and only key holders can check its values.
|
||||
`fixtures/packages/hale2026harmonic.checks-only` is such an export of `hale2026harmonic` under the team key in `fixtures/keys/`.
|
||||
`fixtures/packages/hale2026harmonic.checks-only` is such an export of `hale2026harmonic` to another machine of the team, under the team key in `fixtures/keys/`.
|
||||
|
||||
## 12.6 Assets
|
||||
|
||||
|
||||
+6
-1
@@ -49,9 +49,14 @@ It holds the tables of `doc.sql` and `store.sql`, then those of [sql/bundle-sqli
|
||||
| Table | Holds |
|
||||
| --- | --- |
|
||||
| `bundle_meta` | `name` and `value` pairs: at least `format` (`scienting-bundle/1`), `store`, `built_from` (the store revision) and `spec` (the version of these specifications). |
|
||||
| `search` | An FTS5 index with one row per sentence, cell, display equation and figure-text item: the document, the node, the unit, the context (the kind of a sentence's container) and the normalized plain projection. Withheld text is never indexed. |
|
||||
| `search` | An FTS5 index with one row per sentence, cell, equation outside a sentence and figure-text item, in document order: the document, the node, the unit, the context and the text. Withheld text is never indexed. |
|
||||
| `embeddings_meta` | Reserved for embeddings with `sqlite-vec`: the model, its dimensions, and the rights use they were made under. |
|
||||
|
||||
A search row's `unit` is `sentence`, `cell`, `equation` or `figure-text`.
|
||||
For a node, `context` is the kind of its parent, such as `paragraph`, `heading`, `caption` or `row`, and `text` is its normalized plain projection ([3.6](03-nodes.md#36-the-plain-projection)).
|
||||
An equation inside a sentence is searched as part of that sentence, so it has no row of its own.
|
||||
A figure-text item names its panel as the node, its `role` as the context and its normalized `text` as the text.
|
||||
|
||||
At team scale this is all the search a team needs.
|
||||
A team can download its bundle and use it offline.
|
||||
Because a bundle holds text, it is served only under the rights of [14](14-rights.md), checked again whenever text is served.
|
||||
@@ -31,6 +31,9 @@ Each line of `annotations.jsonl` is a W3C `Annotation`:
|
||||
| `status` | `proposed`, `confirmed`, `rejected` or `withdrawn`. A model's annotation is `proposed` until a person confirms it. |
|
||||
| `receipt` | For a confirmed annotation, the receipt that covers it. |
|
||||
|
||||
The `TextQuoteSelector`, when present, repeats the anchor quote's `exact`, `prefix` and `suffix`, with an empty string for a `prefix` or `suffix` the quote lacks.
|
||||
It is present only when the anchor's quote is plain text; an anchor whose quote is hashed or keyed ([6.3](06-anchors.md#63-keyed-anchors)) has none, so the fallback never shows text the anchor withholds.
|
||||
|
||||
An annotation's content hash is SHA-256 of the canonical JSON of the annotation without `status`, `receipt` and `modified`.
|
||||
A confirmed annotation is covered by its confirmer's receipt, which lists `["annotation", <record id>, <content hash>]`.
|
||||
Replies are annotations that target annotations, so discussions are threads anchored to sentences.
|
||||
|
||||
+3
-2
@@ -10,7 +10,8 @@ A project, `scienting-project/1`, is the unit a team works in.
|
||||
| `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group. |
|
||||
| `layers`, `datasets`, `views`, `drafts` | What the project holds. |
|
||||
| `claims` | The claim namespace or bundle it uses, as the `lode` setting. |
|
||||
| `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), and the stage requirements in use. |
|
||||
| `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), and the stage requirements in use, as the `id` of their stage file ([14](14-rights.md)). |
|
||||
|
||||
Projects are small on purpose: everything about them is designed for hundreds of documents, not millions.
|
||||
Owners sign policies ([14.2](14-rights.md#142-effective-uses)) and may revoke receipts ([8.7](08-fidelity.md#87-revocations-and-the-roster)).
|
||||
Owners sign policies ([14.2](14-rights.md#142-effective-uses)).
|
||||
Who may revoke receipts follows the roster instead: a key whose `role` in the roster is `owner` ([8.7](08-fidelity.md#87-revocations-and-the-roster)), so a project's owners hold keys with that role.
|
||||
@@ -6,7 +6,7 @@
|
||||
| --- | --- | --- |
|
||||
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
|
||||
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document. | Accepts each, and recomputes every value it contains. |
|
||||
| `fixtures/records/` | A valid record of every other format. | Accepts each. |
|
||||
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `tools/load.py --bundle`. | Accepts each. |
|
||||
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
|
||||
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |
|
||||
|
||||
|
||||
Reference in new issue
Block a user