alexmandClaude
8c8b51cc3f
- Add tools/validate.py: validates packages, and with --correction runs proposed corrections through the gate against a package without writing anything (spec/11 §11.4). The gate now refuses a proposal that fails the correction schema, rule correction.schema. - tools/load.py validates every package before loading, and with --bundle --built-from builds a scienting-bundle/1 with the FTS5 search index and bundle_meta (spec/13 §13.5). Search rows are defined per unit (sentence, cell, equation outside a sentence, figure-text) with the parent kind as context. - Roster keys carry a role, and a key with the role owner may revoke any receipt (spec/08 §8.7, spec/20); roster schema, store table and the fixture roster updated. The checker signs a trial revocation to prove the rule both ways. - A qa: receipt raises the nodes it sampled except those in its errors (spec/08 §8.2); the example paper's fidelity counts become 27 F3, 13 F2, 4 F1 and 1 F0. - The destination chooses the keyed scheme, not the text policy: hmac-team between the team's machines, hkdf-pages outside the team (spec/05 §5.8, spec/12 §12.5). - A TextQuoteSelector must repeat the anchor's plain quote and is absent for hashed or keyed anchors (spec/18 §18.3). - Add fixtures/records/suspect.json, a scienting-suspect/1 record. - Oracle: hashes are kept incrementally and checked after every correction against a full recomputation; load and write refuse unsafe manifest paths; Ed25519, HKDF and schema validation come from the cryptography and jsonschema packages; canonical JSON uses the built-in encoder with an exact RFC 8785 fallback. - The tools need Python 3.14 with cryptography and jsonschema; the py310 CI environment is dropped and the workflow runs the default only. - Resolve every [CHECK] note in the docs, READMEs and context; add CONTRIBUTING.md; every tool answers --help and refuses unknown options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
63 lines
4.3 KiB
Markdown
63 lines
4.3 KiB
Markdown
# 13. The database form and the bundle
|
|
|
|
## 13.1 Portability
|
|
|
|
[sql/doc.sql](../sql/doc.sql) defines the document tables and [sql/store.sql](../sql/store.sql) a team store's tables.
|
|
Both run unchanged on SQLite, Dolt and PostgreSQL, under these rules:
|
|
|
|
1. No foreign keys; references are checked by validators, not by the database.
|
|
2. Enumerations are `VARCHAR` with `CHECK (… IN (…))`.
|
|
3. JSON values are canonical JSON in `TEXT` columns; booleans are `INTEGER` 0 or 1.
|
|
4. Every table has a primary key, and key columns are `VARCHAR` with a length, as MySQL-compatible engines require.
|
|
5. No table, column or index is named with a word reserved in SQLite, PostgreSQL or MySQL; so the formats' `key`, `index`, `at`, `by`, `before` and `after` are `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
|
|
|
|
## 13.2 Tables
|
|
|
|
| Table | One row per | Notes |
|
|
| --- | --- | --- |
|
|
| `works` | work | Its key and `same_as`. |
|
|
| `documents` | document | Every property of `document.json` as a column, plus `provenance`, `roster` and `package_hash`. |
|
|
| `witnesses` | witness | With `ord`, its position in `witnesses.json`. |
|
|
| `nodes` | node | With `seq`, its position in document order. |
|
|
| `pages`, `locations`, `refs` | page, location, reference | |
|
|
| `corrections` | correction | The common properties as columns; the rest in `details`. |
|
|
| `node_ids` | retired ID or alias | |
|
|
| `receipts`, `revocations` | receipt, revocation | |
|
|
| `suspects` | suspect | |
|
|
| `package_files` | asset, dataset file or derived file | Its path, SHA-256, size and whether its bytes were present. Bytes go to the asset store. |
|
|
| `citations` | cite run and reference | Derived on load. |
|
|
| `receipt_nodes` | node a receipt lists | Derived on load. |
|
|
|
|
A 30-page paper is about 3,000 nodes, so 500 documents are about a million and a half rows, which SQLite and Dolt hold comfortably.
|
|
|
|
## 13.3 Load and unload
|
|
|
|
Loading a package is a pure function of the package: each record becomes a row, an optional property absent from the record is NULL, and JSON-valued properties are stored as canonical JSON.
|
|
Unloading writes each file back from its rows in the file's defined order, omitting optional properties that are NULL and optional files that have no rows, copies the other files from the asset store, and writes the manifest.
|
|
Unloading gives the package back byte for byte, and its manifest hash with it; the suite checks this for every fixture.
|
|
|
|
## 13.4 The store tables
|
|
|
|
`store.sql` adds what a team adds to its documents: `roster_keys`, `projects`, `members`, `project_documents`, `reading_paths`, `layers`, `annotations`, `annotation_targets` (derived, so a reader finds the annotations on a node), `datasets`, `dataset_values`, `digitizations`, `views`, `policies`, `submissions` and `grades`.
|
|
Each keeps its record whole in a `record` or `descriptor` column, with the columns a query needs beside it.
|
|
|
|
## 13.5 The bundle
|
|
|
|
A bundle, `scienting-bundle/1`, is one SQLite file that a reader or a server queries.
|
|
It holds the tables of `doc.sql` and `store.sql`, then those of [sql/bundle-sqlite.sql](../sql/bundle-sqlite.sql):
|
|
|
|
| Table | Holds |
|
|
| --- | --- |
|
|
| `bundle_meta` | `name` and `value` pairs: at least `format` (`scienting-bundle/1`), `store`, `built_from` (the store revision) and `spec` (the version of these specifications). |
|
|
| `search` | An FTS5 index with one row per sentence, cell, equation outside a sentence and figure-text item, in document order: the document, the node, the unit, the context and the text. Withheld text is never indexed. |
|
|
| `embeddings_meta` | Reserved for embeddings with `sqlite-vec`: the model, its dimensions, and the rights use they were made under. |
|
|
|
|
A search row's `unit` is `sentence`, `cell`, `equation` or `figure-text`.
|
|
For a node, `context` is the kind of its parent, such as `paragraph`, `heading`, `caption` or `row`, and `text` is its normalized plain projection ([3.6](03-nodes.md#36-the-plain-projection)).
|
|
An equation inside a sentence is searched as part of that sentence, so it has no row of its own.
|
|
A figure-text item names its panel as the node, its `role` as the context and its normalized `text` as the text.
|
|
|
|
At team scale this is all the search a team needs.
|
|
A team can download its bundle and use it offline.
|
|
Because a bundle holds text, it is served only under the rights of [14](14-rights.md), checked again whenever text is served.
|