First implementation (#1)

Reviewed-on: #1
2026-10-15 05:47:05 -04:00
1 parent 34c0abb26a
commit 4fa326dc76
@@ -0,0 +1,66 @@
+# SQL
+
+This folder holds the table definitions for the database form of packages, a team's store, and the bundle.
+
+## What the database form is for
+
+A package is a folder of canonical JSON files, one document per folder ([chapter 12](../spec/12-package.md)).
+Canonical JSON is JSON with exactly one valid byte sequence, so a file's hash changes only when its content does.
+
+Handing over a folder is easy.
+Querying across hundreds of documents in folders is hard.
+The database form holds the same data as rows so that a team can ask SQL questions of it, and a package loaded into the tables unloads back to the same bytes.
+About 500 documents come to about a million and a half rows, and SQLite and Dolt hold that comfortably ([chapter 13](../spec/13-database.md#132-tables)).
+
+## The three files
+
+| File | Holds | Runs on |
+| --- | --- | --- |
+| `doc.sql` | Documents: works, documents, witnesses, nodes, pages, locations, references, citations, corrections, IDs, receipts, revocations, suspects and package files. | SQLite, Dolt, PostgreSQL |
+| `store.sql` | What a team adds: roster keys, projects, members, reading paths, layers, annotations, datasets, digitizations, views, policies, submissions and grades. Load `doc.sql` first. | SQLite, Dolt, PostgreSQL |
+| `bundle-sqlite.sql` | The bundle's extra tables: metadata, a full-text index (FTS5, SQLite's search extension) and room for embeddings. Load both files above first. | SQLite |
+
+## Creating a database
+
+```sh
+sqlite3 lab.db < sql/doc.sql && sqlite3 lab.db < sql/store.sql
+psql -d lab -f sql/doc.sql && psql -d lab -f sql/store.sql
+dolt sql < sql/doc.sql && dolt sql < sql/store.sql
+```
+
+The PostgreSQL database `lab` and the Dolt repository must already exist.
+The `sqlite3` line runs as shown; the `psql` and `dolt` lines follow the portability rules of [chapter 13](../spec/13-database.md#131-portability) but have not been run on those engines in this repository.
+
+To load packages into SQLite without pixi, run `python3 tools/load.py --sqlite lab.db PACKAGE…` with Python 3.14 and two packages, `jsonschema` and `cryptography` installed.
+The file [docs/examples.md](../docs/examples.md#load-packages-into-sqlite) has queries to try.
+The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into `lab.db.assets/`.
+
+## A first query
+
+The query below finds every sentence that cites a given reference, which is the citation's context.
+It returns sentence IDs, and `nodes.content` holds each sentence's text.
+
+```sql
+SELECT c.sentence
+FROM citations c
+JOIN refs r ON r.doc_key = c.doc_key AND r.node = c.ref_node
+WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2';
+```
+
+On `fixtures/packages/hale2026harmonic-damb` it returns one row, `hca6dc5`: the sentence ending "served as the first test case, with a bending band near 1595 cm⁻¹."
+
+## Conventions
+
+- Foreign keys: none.
+  Validators check references, so the same file runs everywhere.
+- Types: enumerations are `VARCHAR` with a `CHECK`, JSON values are canonical JSON in `TEXT`, and booleans are `INTEGER` 0 or 1.
+- Names: no table, column or index name is reserved in SQLite, PostgreSQL or MySQL (Dolt is MySQL-compatible).
+  The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
+- Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash.
+  Unloading copies the files with no table from the asset store, which must travel with the database. `tools/check.py` tests this for every fixture package.
+- Derived tables: `citations`, `receipt_nodes` and `annotation_targets` are built on load for queries and are not part of the package.
+
+## Status
+
+The version is 0.1.0.
+In `registry/formats.json`, `scienting-doc/1` is `candidate`, and the bundle and the project, dataset, digitization and view formats are `provisional`. [Chapter 13](../spec/13-database.md) defines the mapping from files to tables.
@@ -0,0 +1,32 @@
+-- scienting-bundle/1: the SQLite file a reader or a server reads.
+-- Normative: spec/13-database.md. SQLite only: it uses FTS5.
+-- A bundle holds sql/doc.sql and sql/store.sql, then these tables.
+-- License: CC0-1.0.
+
+CREATE TABLE bundle_meta (
+  name VARCHAR(64) NOT NULL,
+  value TEXT NOT NULL,
+  PRIMARY KEY (name)
+);
+
+-- One row per searchable unit: a sentence, a cell, an equation outside a sentence, or a figure-text item.
+-- `unit` is sentence, cell, equation or figure-text. `context` is the kind of the node's parent:
+-- paragraph, heading, caption, footnote, row and so on. A figure-text item names its panel as `node`
+-- and its role as `context`. `text` is the normalized plain projection; withheld text is never indexed.
+CREATE VIRTUAL TABLE search USING fts5(
+  doc_key UNINDEXED,
+  node UNINDEXED,
+  unit UNINDEXED,
+  context UNINDEXED,
+  text,
+  tokenize = 'unicode61 remove_diacritics 2'
+);
+
+-- Reserved for embeddings with sqlite-vec; see spec/13-database.md.
+CREATE TABLE embeddings_meta (
+  model VARCHAR(128) NOT NULL,
+  dimensions INTEGER NOT NULL,
+  rights_use VARCHAR(64) NOT NULL,
+  made_at VARCHAR(20) NOT NULL,
+  PRIMARY KEY (model)
+);
@@ -0,0 +1,219 @@
+-- scienting-doc/1: the database form of packages.
+-- Normative: spec/13-database.md.
+-- Portable to SQLite 3.37+, Dolt and PostgreSQL 12+:
+-- no foreign keys, enumerations as VARCHAR with CHECK, JSON as canonical JSON text,
+-- booleans as INTEGER 0 or 1, and no column name reserved in any of the three.
+-- License: CC0-1.0.
+
+CREATE TABLE works (
+  work_key VARCHAR(128) NOT NULL,
+  same_as TEXT NOT NULL,
+  PRIMARY KEY (work_key)
+);
+
+CREATE TABLE documents (
+  doc_key VARCHAR(128) NOT NULL,
+  store VARCHAR(128),
+  aliases TEXT,
+  kind VARCHAR(32) NOT NULL CHECK (kind IN ('article', 'supplement', 'chapter', 'book', 'thesis', 'report', 'draft', 'course-material', 'other')),
+  work TEXT NOT NULL,
+  version TEXT NOT NULL,
+  relations TEXT NOT NULL,
+  meta TEXT NOT NULL,
+  language VARCHAR(35) NOT NULL,
+  license TEXT,
+  policy VARCHAR(16) NOT NULL CHECK (policy IN ('full', 'checks-only')),
+  hashing TEXT NOT NULL,
+  root_hash VARCHAR(64) NOT NULL,
+  meta_hash VARCHAR(64) NOT NULL,
+  history VARCHAR(64) NOT NULL,
+  rev INTEGER NOT NULL CHECK (rev >= 0),
+  counts TEXT NOT NULL,
+  fidelity TEXT NOT NULL,
+  provenance TEXT NOT NULL,
+  roster TEXT,
+  package_hash VARCHAR(64) NOT NULL,
+  PRIMARY KEY (doc_key)
+);
+
+CREATE TABLE witnesses (
+  doc_key VARCHAR(128) NOT NULL,
+  id VARCHAR(32) NOT NULL,
+  ord INTEGER NOT NULL,
+  sha256 VARCHAR(64) NOT NULL,
+  size BIGINT NOT NULL,
+  kind VARCHAR(16) NOT NULL CHECK (kind IN ('pdf', 'jats', 'latex', 'html', 'epub', 'typst', 'xml', 'image', 'other')),
+  media_type VARCHAR(128) NOT NULL,
+  pages INTEGER,
+  text_layer INTEGER CHECK (text_layer IN (0, 1)),
+  version_label VARCHAR(128),
+  origin TEXT NOT NULL,
+  rights TEXT NOT NULL,
+  PRIMARY KEY (doc_key, id)
+);
+CREATE INDEX witnesses_sha256 ON witnesses (sha256);
+
+CREATE TABLE nodes (
+  doc_key VARCHAR(128) NOT NULL,
+  id VARCHAR(16) NOT NULL,
+  seq INTEGER NOT NULL,
+  kind VARCHAR(24) NOT NULL CHECK (kind IN (
+    'document', 'front', 'title', 'authors', 'abstract', 'keywords', 'notes', 'section', 'heading',
+    'paragraph', 'sentence', 'equation', 'equation-line', 'figure', 'panel', 'caption', 'figure-note',
+    'table', 'table-head', 'table-body', 'table-note', 'row', 'cell', 'list', 'item', 'quote', 'code',
+    'algorithm', 'theorem', 'footnote', 'back', 'acknowledgments', 'references', 'reference', 'appendix')),
+  parent VARCHAR(16),
+  ord INTEGER NOT NULL CHECK (ord >= 0),
+  label VARCHAR(256),
+  attrs TEXT,
+  content TEXT,
+  withheld INTEGER CHECK (withheld IN (0, 1)),
+  fidelity VARCHAR(2) CHECK (fidelity IN ('F0', 'F1', 'F2', 'F3')),
+  prov TEXT,
+  text_hash VARCHAR(64) NOT NULL,
+  match_hash VARCHAR(64) NOT NULL,
+  align_key VARCHAR(64) NOT NULL,
+  own_hash VARCHAR(64) NOT NULL,
+  node_hash VARCHAR(64) NOT NULL,
+  PRIMARY KEY (doc_key, id)
+);
+CREATE INDEX nodes_parent ON nodes (doc_key, parent, ord);
+CREATE INDEX nodes_seq ON nodes (doc_key, seq);
+CREATE INDEX nodes_match_hash ON nodes (match_hash);
+
+CREATE TABLE pages (
+  doc_key VARCHAR(128) NOT NULL,
+  witness VARCHAR(32) NOT NULL,
+  page INTEGER NOT NULL CHECK (page >= 1),
+  page_fp VARCHAR(64) NOT NULL,
+  size TEXT NOT NULL,
+  page_label VARCHAR(64),
+  text_layer INTEGER NOT NULL CHECK (text_layer IN (0, 1)),
+  stamps_removed TEXT,
+  PRIMARY KEY (doc_key, witness, page)
+);
+CREATE INDEX pages_page_fp ON pages (page_fp);
+
+CREATE TABLE locations (
+  doc_key VARCHAR(128) NOT NULL,
+  node VARCHAR(16) NOT NULL,
+  page_fp VARCHAR(64) NOT NULL,
+  page INTEGER NOT NULL CHECK (page >= 1),
+  page_label VARCHAR(64),
+  boxes TEXT NOT NULL,
+  approx INTEGER CHECK (approx IN (0, 1)),
+  PRIMARY KEY (doc_key, node, page, page_fp)
+);
+
+CREATE TABLE refs (
+  doc_key VARCHAR(128) NOT NULL,
+  node VARCHAR(16) NOT NULL,
+  label VARCHAR(64),
+  csl TEXT,
+  resolution TEXT NOT NULL,
+  PRIMARY KEY (doc_key, node)
+);
+
+-- Derived on load from cite runs; not part of the package.
+CREATE TABLE citations (
+  doc_key VARCHAR(128) NOT NULL,
+  sentence VARCHAR(16) NOT NULL,
+  ref_node VARCHAR(16) NOT NULL,
+  ord INTEGER NOT NULL,
+  PRIMARY KEY (doc_key, sentence, ord, ref_node)
+);
+CREATE INDEX citations_ref ON citations (doc_key, ref_node);
+
+CREATE TABLE corrections (
+  doc_key VARCHAR(128) NOT NULL,
+  idx INTEGER NOT NULL CHECK (idx >= 1),
+  id VARCHAR(32) NOT NULL,
+  op VARCHAR(16) NOT NULL CHECK (op IN ('replace', 'insert', 'delete', 'split', 'merge', 'move', 'reclassify', 'relabel', 'set')),
+  target VARCHAR(16),
+  evidence TEXT NOT NULL,
+  author TEXT NOT NULL,
+  reason TEXT NOT NULL,
+  made_at VARCHAR(20) NOT NULL,
+  before_hash VARCHAR(64),
+  after_hash VARCHAR(64),
+  path TEXT,
+  touched TEXT NOT NULL,
+  details TEXT NOT NULL,
+  PRIMARY KEY (doc_key, idx)
+);
+CREATE INDEX corrections_target ON corrections (doc_key, target);
+
+CREATE TABLE node_ids (
+  doc_key VARCHAR(128) NOT NULL,
+  id VARCHAR(128) NOT NULL,
+  kind VARCHAR(8) NOT NULL CHECK (kind IN ('retired', 'alias')),
+  successor VARCHAR(16),
+  node_kind VARCHAR(24),
+  rev INTEGER,
+  PRIMARY KEY (doc_key, id)
+);
+
+CREATE TABLE receipts (
+  doc_key VARCHAR(128) NOT NULL,
+  id VARCHAR(32) NOT NULL,
+  level VARCHAR(2) NOT NULL CHECK (level IN ('F1', 'F2', 'F3')),
+  scope VARCHAR(64) NOT NULL,
+  rev INTEGER NOT NULL,
+  root VARCHAR(64) NOT NULL,
+  nodes TEXT NOT NULL,
+  records TEXT,
+  evidence TEXT NOT NULL,
+  signer VARCHAR(128) NOT NULL,
+  made_at VARCHAR(20) NOT NULL,
+  qa TEXT,
+  export_of VARCHAR(32),
+  keyed TEXT,
+  signature TEXT NOT NULL,
+  PRIMARY KEY (doc_key, id)
+);
+CREATE INDEX receipts_doc ON receipts (doc_key, made_at);
+
+-- Derived on load from receipts.nodes.
+CREATE TABLE receipt_nodes (
+  receipt VARCHAR(32) NOT NULL,
+  doc_key VARCHAR(128) NOT NULL,
+  node VARCHAR(16) NOT NULL,
+  node_hash VARCHAR(64) NOT NULL,
+  PRIMARY KEY (doc_key, receipt, node)
+);
+CREATE INDEX receipt_nodes_node ON receipt_nodes (doc_key, node);
+
+CREATE TABLE revocations (
+  doc_key VARCHAR(128) NOT NULL,
+  id VARCHAR(32) NOT NULL,
+  receipt VARCHAR(32),
+  revoked_signer VARCHAR(128),
+  period_from VARCHAR(20),
+  period_to VARCHAR(20),
+  reason TEXT NOT NULL,
+  signer VARCHAR(128) NOT NULL,
+  made_at VARCHAR(20) NOT NULL,
+  signature TEXT NOT NULL,
+  -- A revocation of a signer's period carries no document, so two packages may carry the same record.
+  PRIMARY KEY (doc_key, id)
+);
+
+CREATE TABLE suspects (
+  doc_key VARCHAR(128) NOT NULL,
+  id VARCHAR(32) NOT NULL,
+  kind VARCHAR(16) NOT NULL CHECK (kind IN ('word', 'number', 'equation', 'structure', 'join', 'added', 'dropped', 'boundary', 'reading')),
+  node VARCHAR(16),
+  record TEXT NOT NULL,
+  PRIMARY KEY (doc_key, id)
+);
+
+-- Files a package holds that have no table of their own: assets, datasets and derived files.
+-- Their bytes live in the asset store, by SHA-256.
+CREATE TABLE package_files (
+  doc_key VARCHAR(128) NOT NULL,
+  path VARCHAR(512) NOT NULL,
+  sha256 VARCHAR(64) NOT NULL,
+  size BIGINT NOT NULL,
+  present INTEGER NOT NULL CHECK (present IN (0, 1)),
+  PRIMARY KEY (doc_key, path)
+);
@@ -0,0 +1,155 @@
+-- The team store: everything a team adds to its documents.
+-- Normative: spec/13-database.md. Load sql/doc.sql first.
+-- Portable to SQLite 3.37+, Dolt and PostgreSQL 12+, under the same rules as sql/doc.sql.
+-- License: CC0-1.0.
+
+CREATE TABLE roster_keys (
+  store VARCHAR(128) NOT NULL,
+  id VARCHAR(128) NOT NULL,
+  person TEXT NOT NULL,
+  role VARCHAR(16) NOT NULL CHECK (role IN ('owner', 'editor', 'reader', 'guest')),
+  scheme VARCHAR(16) NOT NULL CHECK (scheme IN ('ed25519', 'ssh', 'gpg', 'sigstore', 'cose', 'ml-dsa-65')),
+  public_key TEXT NOT NULL,
+  did VARCHAR(128),
+  purposes TEXT NOT NULL,
+  registered_at VARCHAR(20) NOT NULL,
+  revoked_at VARCHAR(20),
+  PRIMARY KEY (store, id)
+);
+
+CREATE TABLE projects (
+  project_key VARCHAR(128) NOT NULL,
+  store VARCHAR(128),
+  name TEXT NOT NULL,
+  description TEXT,
+  claims TEXT,
+  settings TEXT,
+  record TEXT NOT NULL,
+  PRIMARY KEY (project_key)
+);
+
+CREATE TABLE members (
+  project_key VARCHAR(128) NOT NULL,
+  person VARCHAR(256) NOT NULL,
+  role VARCHAR(8) NOT NULL CHECK (role IN ('owner', 'editor', 'reader', 'guest')),
+  signing_keys TEXT NOT NULL,
+  PRIMARY KEY (project_key, person)
+);
+
+CREATE TABLE project_documents (
+  project_key VARCHAR(128) NOT NULL,
+  doc_key VARCHAR(257) NOT NULL,
+  reason TEXT NOT NULL,
+  tags TEXT,
+  reading_status TEXT,
+  PRIMARY KEY (project_key, doc_key)
+);
+
+CREATE TABLE reading_paths (
+  project_key VARCHAR(128) NOT NULL,
+  id VARCHAR(128) NOT NULL,
+  title TEXT NOT NULL,
+  steps TEXT NOT NULL,
+  PRIMARY KEY (project_key, id)
+);
+
+CREATE TABLE layers (
+  id VARCHAR(256) NOT NULL,
+  project_key VARCHAR(128),
+  label TEXT NOT NULL,
+  purpose TEXT NOT NULL,
+  audience VARCHAR(8) CHECK (audience IN ('novice', 'student', 'expert', 'mixed')),
+  visibility VARCHAR(8) NOT NULL CHECK (visibility IN ('private', 'team', 'public')),
+  license TEXT NOT NULL,
+  record TEXT NOT NULL,
+  PRIMARY KEY (id)
+);
+
+CREATE TABLE annotations (
+  id VARCHAR(256) NOT NULL,
+  layer VARCHAR(256) NOT NULL,
+  body_kind VARCHAR(24) NOT NULL CHECK (body_kind IN ('gloss', 'claim-link', 'symbol', 'comment', 'caveat', 'question', 'exercise', 'citation-context', 'prerequisite', 'crosslink', 'translation', 'description', 'generated')),
+  status VARCHAR(10) NOT NULL CHECK (status IN ('proposed', 'confirmed', 'rejected', 'withdrawn')),
+  receipt VARCHAR(32),
+  creator VARCHAR(256) NOT NULL,
+  created VARCHAR(20) NOT NULL,
+  record TEXT NOT NULL,
+  PRIMARY KEY (id)
+);
+CREATE INDEX annotations_layer ON annotations (layer);
+
+-- Derived from annotation targets, so a reader finds the annotations on a node.
+CREATE TABLE annotation_targets (
+  annotation VARCHAR(256) NOT NULL,
+  ord INTEGER NOT NULL,
+  doc_key VARCHAR(257),
+  node VARCHAR(128),
+  target_annotation VARCHAR(256),
+  PRIMARY KEY (annotation, ord)
+);
+CREATE INDEX annotation_targets_node ON annotation_targets (doc_key, node);
+
+CREATE TABLE datasets (
+  id VARCHAR(128) NOT NULL,
+  doc_key VARCHAR(257),
+  content_hash VARCHAR(64) NOT NULL,
+  descriptor TEXT NOT NULL,
+  PRIMARY KEY (id)
+);
+
+CREATE TABLE dataset_values (
+  dataset VARCHAR(128) NOT NULL,
+  resource VARCHAR(128) NOT NULL,
+  row_key VARCHAR(128) NOT NULL,
+  field VARCHAR(128) NOT NULL,
+  printed VARCHAR(64),
+  origin_kind VARCHAR(10) NOT NULL CHECK (origin_kind IN ('reported', 'digitized', 'computed', 'measured')),
+  anchor TEXT,
+  PRIMARY KEY (dataset, resource, row_key, field)
+);
+
+CREATE TABLE digitizations (
+  id VARCHAR(32) NOT NULL,
+  doc_key VARCHAR(257) NOT NULL,
+  panel VARCHAR(16) NOT NULL,
+  page_fp VARCHAR(64),
+  method VARCHAR(8) NOT NULL CHECK (method IN ('manual', 'traced', 'proposed')),
+  confirmation VARCHAR(32),
+  dataset VARCHAR(128),
+  record TEXT NOT NULL,
+  PRIMARY KEY (id)
+);
+
+CREATE TABLE views (
+  id VARCHAR(128) NOT NULL,
+  project_key VARCHAR(128),
+  spec TEXT NOT NULL,
+  PRIMARY KEY (id)
+);
+
+CREATE TABLE policies (
+  id VARCHAR(32) NOT NULL,
+  project_key VARCHAR(128),
+  signer VARCHAR(128) NOT NULL,
+  made_at VARCHAR(20) NOT NULL,
+  valid_until VARCHAR(20),
+  record TEXT NOT NULL,
+  PRIMARY KEY (id)
+);
+
+CREATE TABLE submissions (
+  id VARCHAR(32) NOT NULL,
+  exercise VARCHAR(256) NOT NULL,
+  student VARCHAR(64) NOT NULL,
+  submitted_at VARCHAR(20) NOT NULL,
+  record TEXT NOT NULL,
+  PRIMARY KEY (id)
+);
+
+CREATE TABLE grades (
+  submission VARCHAR(32) NOT NULL,
+  score VARCHAR(32) NOT NULL,
+  max_score VARCHAR(32) NOT NULL,
+  record TEXT NOT NULL,
+  PRIMARY KEY (submission)
+);