alexmandClaude
a09d158e16
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
openalex in same_as, else the folded title, year and family name. A
collision lengthens the later key by two characters; a key is minted
once and never recomputed. A document key adds .si, .v<seq> or .<type>
for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
every node and record ID was re-minted and the docs rewritten to match.
The validator checks the document key against the work key and version
(doc.key.work) and the hash part against the identifier (doc.key.suffix).
vectors/keys.jsonl pins the cases paperstack's tests pin.
Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
implemented in tools/oracle/vrf.py in plain Python and reproduced
against the RFC's Example 16; the sample ranks by the VRF output. The
earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
verify_inclusion, consistency_proof, verify_consistency) with
scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
checkpoints with key hashes as transparency logs share them, the rule
that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).
Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
the checker refuses another version.
Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
checkpoint records; a check that every ID and hash the prose quotes
exists in the fixtures or vectors; three more broken packages
(history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
section, media-type registration and OCI distribution notes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
67 lines
3.9 KiB
Markdown
67 lines
3.9 KiB
Markdown
# SQL
|
|
|
|
This folder holds the table definitions for the database form of packages, a team's store, and the bundle.
|
|
|
|
## What the database form is for
|
|
|
|
A package is a folder of canonical JSON files, one document per folder ([chapter 12](../spec/12-package.md)).
|
|
Canonical JSON is JSON with exactly one valid byte sequence, so a file's hash changes only when its content does.
|
|
|
|
Handing over a folder is easy.
|
|
Querying across hundreds of documents in folders is hard.
|
|
The database form holds the same data as rows so that a team can ask SQL questions of it, and a package loaded into the tables unloads back to the same bytes.
|
|
About 500 documents come to about a million and a half rows, and SQLite and Dolt hold that comfortably ([chapter 13](../spec/13-database.md#132-tables)).
|
|
|
|
## The three files
|
|
|
|
| File | Holds | Runs on |
|
|
| --- | --- | --- |
|
|
| `doc.sql` | Documents: works, documents, witnesses, nodes, pages, locations, references, citations, corrections, IDs, receipts, revocations, suspects and package files. | SQLite, Dolt, PostgreSQL |
|
|
| `store.sql` | What a team adds: roster keys, projects, members, reading paths, layers, annotations, datasets, digitizations, views, policies, submissions and grades. Load `doc.sql` first. | SQLite, Dolt, PostgreSQL |
|
|
| `bundle-sqlite.sql` | The bundle's extra tables: metadata, a full-text index (FTS5, SQLite's search extension) and room for embeddings. Load both files above first. | SQLite |
|
|
|
|
## Creating a database
|
|
|
|
```sh
|
|
sqlite3 lab.db < sql/doc.sql && sqlite3 lab.db < sql/store.sql
|
|
psql -d lab -f sql/doc.sql && psql -d lab -f sql/store.sql
|
|
dolt sql < sql/doc.sql && dolt sql < sql/store.sql
|
|
```
|
|
|
|
The PostgreSQL database `lab` and the Dolt repository must already exist.
|
|
The `sqlite3` line runs as shown; the `psql` and `dolt` lines follow the portability rules of [chapter 13](../spec/13-database.md#131-portability) but have not been run on those engines in this repository.
|
|
|
|
To load packages into SQLite without pixi, run `python3 tools/load.py --sqlite lab.db PACKAGE…` with Python 3.14 and two packages, `jsonschema` and `cryptography` installed.
|
|
The file [docs/examples.md](../docs/examples.md#load-packages-into-sqlite) has queries to try.
|
|
The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into `lab.db.assets/`.
|
|
|
|
## A first query
|
|
|
|
The query below finds every sentence that cites a given reference, which is the citation's context.
|
|
It returns sentence IDs, and `nodes.content` holds each sentence's text.
|
|
|
|
```sql
|
|
SELECT c.sentence
|
|
FROM citations c
|
|
JOIN refs r ON r.doc_key = c.doc_key AND r.node = c.ref_node
|
|
WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2';
|
|
```
|
|
|
|
On `fixtures/packages/hale2026harmonic-damb` it returns one row, `hca6dc5`: the sentence ending "served as the first test case, with a bending band near 1595 cm⁻¹."
|
|
|
|
## Conventions
|
|
|
|
- Foreign keys: none.
|
|
Validators check references, so the same file runs everywhere.
|
|
- Types: enumerations are `VARCHAR` with a `CHECK`, JSON values are canonical JSON in `TEXT`, and booleans are `INTEGER` 0 or 1.
|
|
- Names: no table, column or index name is reserved in SQLite, PostgreSQL or MySQL (Dolt is MySQL-compatible).
|
|
The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
|
|
- Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash.
|
|
Unloading copies the files with no table from the asset store, which must travel with the database. `tools/check.py` tests this for every fixture package.
|
|
- Derived tables: `citations`, `receipt_nodes` and `annotation_targets` are built on load for queries and are not part of the package.
|
|
|
|
## Status
|
|
|
|
The version is 0.1.0.
|
|
In `registry/formats.json`, `scienting-doc/1` is `candidate`, and the bundle and the project, dataset, digitization and view formats are `provisional`. [Chapter 13](../spec/13-database.md) defines the mapping from files to tables.
|