varve-spec
The specifications of the Scientific Computing Studio's data formats: documents and their packages, anchors, receipts, rights, layers, exercises, datasets, digitizations, views, projects and bundles.
This is the repository the design calls spec, the baseline every other part implements.
spec defines data, and data outlives code.
Every file any part writes is defined here, and a conforming file can be read without the part that wrote it.
Formats are named scienting-<name>/<major>, never after a product, so a product can be renamed without touching data.
What is here
| Path | Holds |
|---|---|
| spec/ | The normative chapters. Start with spec/00-introduction.md. |
| schemas/ | JSON Schemas (draft 2020-12) for every file and record. |
| sql/ | Tables for SQLite, Dolt and PostgreSQL, and the SQLite bundle with full-text search. |
| context/ | The JSON-LD context that makes layers W3C Web Annotations. |
| registry/ | Formats and media types, validation rule IDs, download-stamp rules and the default stage requirements. |
| vectors/ | Test vectors for every computed value. |
| fixtures/ | Valid packages and records, invalid packages with the rule each breaks, and the test keys that signed them. |
| tools/ | The non-normative oracle that computed the vectors, the builder and the checker. |
| DECISIONS.md | Where this text refines or departs from the design, and the decisions still open. |
Check it
Python 3.10 or later, with nothing installed:
python3 tools/check.py
It validates every schema, record and package, refuses every invalid package for the stated rule, recomputes every vector, rebuilds the fixtures and vectors and compares them byte for byte, loads every package into SQLite and unloads it byte for byte, and lints the prose.
With jsonschema installed it also validates every record a second time with that library.
make check does the same and runs markdownlint when it is available.
Implement it
An implementation conforms to the chapters and the vectors.
scienting-core (Python, standard library only) is the reference implementation of the shared rules, and core-ts the browser's; each runs this suite in its CI.
The oracle in tools/ is not a library and is not meant to be imported: it exists so that no vector is typed by hand.
Status
Version 0.1.0.
The formats of milestone M1 are candidate; datasets, digitizations, views, projects, bundles and the draft formats are provisional until their milestones; see registry/formats.json.
Licenses
The chapters and other prose are licensed CC BY 4.0. Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are dedicated to the public domain under CC0 1.0, so anyone may implement the formats. The tools are licensed Apache-2.0. See LICENSE.md.