4.4 KiB
varve-spec
The data formats behind Lode, paperpack and Varve: how a team's scientific papers are stored once, corrected once, addressed down to the sentence, and kept.
A research team usually cares about a few dozen papers it understands completely, not the world's literature. Those papers deserve better than a folder of PDFs: every sentence, table cell and equation should have a permanent address; every correction should be recorded with its evidence; every note, claim and extracted number should point at exactly what it came from; and everyone should be able to see how far each piece was checked, and by whom.
This repository defines the files that make that possible. It holds no application. paperpack writes these files, Lode reads them to check claims, and Varve stores, renders and annotates them; any other program can do the same, because every format is documented here and readable without the studio's code.
cornell1995second#k3m9q2a@4f1a0c[12:57]
└──── document ──┘└ node ┘└ pin ┘└span┘
That string names code points 12 to 57 of one sentence, as it read when its hash began 4f1a0c.
It survives reconversion with a better tool, a typo fixed elsewhere in the paper, and a teammate's differently stamped download of the same PDF.
Start here
| If you want to | Read |
|---|---|
| Understand what the project is and how the parts fit | docs/overview.md |
| Learn the ideas one at a time, each with an example | docs/concepts.md |
| Follow one paper from its first reading to a signed record | docs/walkthrough.md |
| Run something in the next five minutes | docs/examples.md |
| Implement a reader or writer | spec/, then vectors/ |
| Know why a rule is the way it is | docs/rationale.md |
| Look up a term | docs/glossary.md |
What is in the repository
| Folder | Holds |
|---|---|
| spec/ | The specification, in 23 chapters. Normative. |
| schemas/ | JSON Schemas for every file and record. |
| sql/ | Tables for SQLite, Dolt and PostgreSQL, and the SQLite bundle with full-text search. |
| context/ | The JSON-LD context that makes annotation layers W3C Web Annotations. |
| registry/ | The list of formats, the rule IDs validators report, download-stamp rules, and default rights requirements. |
| vectors/ | Test vectors: inputs and the exact outputs every implementation must compute. |
| fixtures/ | Example packages and records, deliberately broken packages, and the test keys that signed them. |
| tools/ | The programs that build and check everything above. |
| docs/ | Guides for people getting to know the project. |
Try it
With pixi, one command sets up a pinned environment and checks everything:
pixi run check
The checker validates every schema, record and package, confirms that every broken package is refused for the stated reason, recomputes every test vector, rebuilds everything and compares it byte for byte, loads every package into SQLite and back, and lints the prose and the links.
It ends with All checks pass.
pixi task list shows the other tasks: quick, build, lint and load.
The tools need nothing but Python 3.10 or later, so without pixi the same check is python3 tools/check.py; pixi adds the jsonschema package as a second schema validator and markdownlint for the Markdown.
To poke at a real document, load the example paper into SQLite and look around; docs/examples.md has queries to start from.
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic
Formats
Every format is named scienting-<name>/<major>, such as scienting-doc/1 or scienting-anchor/1, and never after a product, so products can be renamed without touching data.
registry/formats.json lists them all with their media types, schemas and chapters.
Licenses
The specification and other prose are CC BY 4.0, so anyone may implement the formats. Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0. The tools are Apache-2.0. See LICENSE.md.