paperspec/sql

SQL

This folder holds the table definitions for the database form of packages, a team's store, and the bundle.

What the database form is for

A package is a folder of canonical JSON files, one document per folder (chapter 12). Canonical JSON is JSON with exactly one valid byte sequence, so a file's hash changes only when its content does.

Handing over a folder is easy. Querying across hundreds of documents in folders is hard. The database form holds the same data as rows so that a team can ask SQL questions of it, and a package loaded into the tables unloads back to the same bytes. About 500 documents come to about a million and a half rows, and SQLite and Dolt hold that comfortably (chapter 13).

The three files

File	Holds	Runs on
doc.sql	Documents: works, documents, witnesses, nodes, pages, locations, references, citations, corrections, IDs, receipts, revocations, suspects and package files.	SQLite, Dolt, PostgreSQL
store.sql	What a team adds: roster keys, projects, members, reading paths, layers, annotations, datasets, digitizations, views, policies, submissions and grades. Load doc.sql first.	SQLite, Dolt, PostgreSQL
bundle-sqlite.sql	The bundle's extra tables: metadata, a full-text index (FTS5, SQLite's search extension) and room for embeddings. Load both files above first.	SQLite

Creating a database

sqlite3 lab.db < sql/doc.sql && sqlite3 lab.db < sql/store.sql
psql -d lab -f sql/doc.sql && psql -d lab -f sql/store.sql
dolt sql < sql/doc.sql && dolt sql < sql/store.sql

The PostgreSQL database lab and the Dolt repository must already exist. The sqlite3 line runs as shown; the psql and dolt lines follow the portability rules of chapter 13 but have not been run on those engines in this repository.

To load packages into SQLite without pixi, install the package with its tools extra on Python 3.14, then run paperspec load --sqlite lab.db PACKAGE…. The file docs/examples.md has queries to try. The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into lab.db.assets/.

A first query

The query below finds every sentence that cites a given reference, which is the citation's context. It returns sentence IDs, and nodes.content holds each sentence's text.

SELECT c.sentence
FROM citations c
JOIN refs r ON r.doc_key = c.doc_key AND r.node = c.ref_node
WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2';

On fixtures/packages/hale2026harmonic-damb it returns one row, hca6dc5: the sentence ending "served as the first test case, with a bending band near 1595 cm⁻¹."

Conventions

• Foreign keys: none. Validators check references, so the same file runs everywhere.
• Types: enumerations are VARCHAR with a CHECK, JSON values are canonical JSON in TEXT, and booleans are INTEGER 0 or 1.
• Names: no table, column or index name is reserved in SQLite, PostgreSQL or MySQL (Dolt is MySQL-compatible). The formats' key, index, at, by, before and after become doc_key, idx, made_at, signer, before_hash and after_hash.
• Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash. Unloading copies the files with no table from the asset store, which must travel with the database. paperspec check tests this for every fixture package.
• Derived tables: citations, receipt_nodes and annotation_targets are built on load for queries and are not part of the package.

Status

The version is 0.1.0. In registry/formats.json, scienting-doc/1 is candidate, and the bundle and the project, dataset, digitization and view formats are provisional. Chapter 13 defines the mapping from files to tables.