58 lines
3.9 KiB
Markdown
58 lines
3.9 KiB
Markdown
# 13. The database form and the bundle
|
|
|
|
## 13.1 Portability
|
|
|
|
[sql/doc.sql](../sql/doc.sql) defines the document tables and [sql/store.sql](../sql/store.sql) a team store's tables.
|
|
Both run unchanged on SQLite, Dolt and PostgreSQL, under these rules:
|
|
|
|
1. No foreign keys; references are checked by validators, not by the database.
|
|
2. Enumerations are `VARCHAR` with `CHECK (… IN (…))`.
|
|
3. JSON values are canonical JSON in `TEXT` columns; booleans are `INTEGER` 0 or 1.
|
|
4. Every table has a primary key, and key columns are `VARCHAR` with a length, as MySQL-compatible engines require.
|
|
5. No table, column or index is named with a word reserved in SQLite, PostgreSQL or MySQL; so the formats' `key`, `index`, `at`, `by`, `before` and `after` are `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
|
|
|
|
## 13.2 Tables
|
|
|
|
| Table | One row per | Notes |
|
|
| --- | --- | --- |
|
|
| `works` | work | Its key and `same_as`. |
|
|
| `documents` | document | Every property of `document.json` as a column, plus `provenance`, `roster` and `package_hash`. |
|
|
| `witnesses` | witness | With `ord`, its position in `witnesses.json`. |
|
|
| `nodes` | node | With `seq`, its position in document order. |
|
|
| `pages`, `locations`, `refs` | page, location, reference | |
|
|
| `corrections` | correction | The common properties as columns; the rest in `details`. |
|
|
| `node_ids` | retired ID or alias | |
|
|
| `receipts`, `revocations` | receipt, revocation | |
|
|
| `suspects` | suspect | |
|
|
| `package_files` | asset, dataset file or derived file | Its path, SHA-256, size and whether its bytes were present. Bytes go to the asset store. |
|
|
| `citations` | cite run and reference | Derived on load. |
|
|
| `receipt_nodes` | node a receipt lists | Derived on load. |
|
|
|
|
A 30-page paper is about 3,000 nodes, so 500 documents are about a million and a half rows, which SQLite and Dolt hold comfortably.
|
|
|
|
## 13.3 Load and unload
|
|
|
|
Loading a package is a pure function of the package: each record becomes a row, an optional property absent from the record is NULL, and JSON-valued properties are stored as canonical JSON.
|
|
Unloading writes each file back from its rows in the file's defined order, omitting optional properties that are NULL and optional files that have no rows, copies the other files from the asset store, and writes the manifest.
|
|
Unloading gives the package back byte for byte, and its manifest hash with it; the suite checks this for every fixture.
|
|
|
|
## 13.4 The store tables
|
|
|
|
`store.sql` adds what a team adds to its documents: `roster_keys`, `projects`, `members`, `project_documents`, `reading_paths`, `layers`, `annotations`, `annotation_targets` (derived, so a reader finds the annotations on a node), `datasets`, `dataset_values`, `digitizations`, `views`, `policies`, `submissions` and `grades`.
|
|
Each keeps its record whole in a `record` or `descriptor` column, with the columns a query needs beside it.
|
|
|
|
## 13.5 The bundle
|
|
|
|
A bundle, `scienting-bundle/1`, is one SQLite file that a reader or a server queries.
|
|
It holds the tables of `doc.sql` and `store.sql`, then those of [sql/bundle-sqlite.sql](../sql/bundle-sqlite.sql):
|
|
|
|
| Table | Holds |
|
|
| --- | --- |
|
|
| `bundle_meta` | `name` and `value` pairs: at least `format` (`scienting-bundle/1`), `store`, `built_from` (the store revision) and `spec` (the version of these specifications). |
|
|
| `search` | An FTS5 index with one row per sentence, cell, display equation and figure-text item: the document, the node, the unit, the context (the kind of a sentence's container) and the normalized plain projection. Withheld text is never indexed. |
|
|
| `embeddings_meta` | Reserved for embeddings with `sqlite-vec`: the model, its dimensions, and the rights use they were made under. |
|
|
|
|
At team scale this is all the search a team needs.
|
|
A team can download its bundle and use it offline.
|
|
Because a bundle holds text, it is served only under the rights of [14](14-rights.md), checked again whenever text is served.
|