Reorganize paperspec commands and pin dependency versions (#2)
check / check (push) Successful in 52s
site / site (push) Successful in 1m45s

Reviewed-on: #2
This commit was merged in pull request #2.
This commit is contained in:
alexm committed 2026-10-15 09:53:05 -04:00
1 parent 4fa326dc76
commit 78101dc0db
65 files changed
+2360 -298

No files matched your search

+2 -2
View File
@@ -428,8 +428,8 @@ Keep using it as a reading and distribution format.
*Status.*
The specification is version 0.1.0.
`registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `tools/check.py`.
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; tools Apache-2.0 ([README.md](../README.md#licenses)).
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`.
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; the code in `src/` Apache-2.0 ([README.md](../README.md#licenses)).
*No adoption claim.*
The repository names no outside adopter and no independent implementation, and this page claims none.
+6 -6
View File
@@ -9,7 +9,7 @@ The formats are version 0.1.0, and each is `candidate` or `provisional` in `regi
Run every command from the root of the repository, <https://git.scient.ing/infra/paperspec>.
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
Every command also works without pixi, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
Every command also works without pixi once the checkout is installed with `pip install -e ".[tools]"` on Python 3.14, given standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
## Run the checks
@@ -19,10 +19,10 @@ This shows how to confirm that the repository's specification, fixtures and test
pixi run check
```
`pixi run check` lints the Markdown, then runs `python tools/check.py`, the repository's reference checker.
`pixi run check` lints the Markdown, then runs `paperspec check`, the repository's reference checker.
Each section prints how many checks it ran, and the last line is `All checks pass.`
`pixi run quick`, or `python3 tools/check.py --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `tools/check.py` does not run it.
`pixi run quick`, or `paperspec check --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `paperspec check` does not run it.
## Recompute a hash by hand
@@ -146,7 +146,7 @@ This shows that the same packages can live in a database and be queried with SQL
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
```
Without pixi, run `python3 tools/load.py` with the same arguments.
Without pixi, run `paperspec load` with the same arguments.
```text
hale2026harmonic-damb: 75 nodes
@@ -154,7 +154,7 @@ okafor2025solvent-acth: 15 nodes
writer2027fixed-ynr7.v3: 15 nodes
```
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
```python
+6 -5
View File
@@ -107,7 +107,7 @@ A `candidate` format is complete, with vectors and fixtures, and ready to implem
A `provisional` one is complete and expected to change as implementations gain experience.
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint.
The repository holds a reference checker (`tools/check.py`), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
The repository holds a reference checker (`paperspec check`), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
## What does it not do?
@@ -124,8 +124,9 @@ Propose either with the vector or fixture that shows it.
A new kind is a minor version ([spec/00 §0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
See [CONTRIBUTING.md](../CONTRIBUTING.md).
## What are the files in `tools/` for?
## What are the `paperspec` commands for?
`build.py` builds the fixtures and vectors. `check.py` recomputes and checks every one, and `load.py` loads packages into SQLite.
The `oracle/` modules compute every expected value, so that none is typed by hand.
They are not a library; implementations follow the specification ([tools/README.md](../tools/README.md)).
`paperspec build` builds the fixtures and vectors.
`paperspec check` recomputes and checks every one, and `paperspec load` loads packages into SQLite.
The oracle's modules in `src/paperspec/` compute every expected value, so that none is typed by hand.
The other repositories of the project install them as the Python package `paperspec`; independent implementations follow the specification ([tools/README.md](../tools/README.md)).
+4 -3
View File
@@ -54,7 +54,8 @@ It appears inside hashed data and test vectors, and it is separate from the repo
| [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. |
| [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. |
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
| [tools/](../tools/README.md) | Reference programs: `check.py` checks everything, `build.py` rebuilds the fixtures and vectors, `load.py` loads packages into SQLite, and `oracle/` is a compact implementation of every rule. |
| [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. |
| [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. |
| [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
@@ -72,8 +73,8 @@ The repository ships reference tools only for the checker and store roles.
| Reader | Opens a package, resolves addresses and anchors, and shows text, annotations and each passage's fidelity level. |
| Writer | Produces files in these formats: nodes, corrections, receipts, annotations, datasets. |
| Converter | Turns a witness (a PDF, JATS XML, the article format publishers use, a LaTeX source archive, an HTML page, an EPUB or a Typst project) into a package and raises doubtful readings as suspects (flagged disagreements awaiting review). |
| Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `tools/check.py` does this for the repository. |
| Store | Keeps packages and records in a database a team can clone. `tools/load.py` loads packages into SQLite. |
| Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `paperspec check` does this for the repository. |
| Store | Keeps packages and records in a database a team can clone. `paperspec load` loads packages into SQLite. |
The [specification's reading paths](../spec/README.md) say which chapters each role needs.
+1 -1
View File
@@ -112,7 +112,7 @@ Only the three text values, `text_hash`, `match_hash` and `align_key`, normalize
### The character data is pinned to one Unicode version
Normalization and General Categories change between Unicode versions, and both enter every hash.
The chapters pin Unicode 16.0, the reference tools refuse to run on another version, and moving to a later one is a change to every text-hashing format ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
The chapters pin Unicode 16.0, the reference implementation refuses to import on another version, and moving to a later one is a change to every text-hashing format ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
### Hashes that leave the team are keyed
+2 -2
View File
@@ -13,7 +13,7 @@ It has nothing to do with the repository's name.
Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it.
A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line).
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`tools/build.py`](../tools/build.py) expands it into the package.
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`paperspec build`](../src/paperspec/cli/build.py) expands it into the package.
Each step says what a converter, a person or a checker does.
## 1. A PDF arrives
@@ -166,7 +166,7 @@ It is a [Frictionless Data Package](https://datapackage.org/standard/data-packag
Its CSV keeps every digit as printed (`1.526` and `1.5260` stay different), and every value has an anchor to the cell that prints it, so a claim that rests on a value can show the cell.
The descriptor types the fields as numbers; how a generic Data Package tool treats `1.5260` was not verified.
All twelve value anchors resolve as `current`.
The reference checker, [`tools/check.py`](../tools/check.py), confirms each CSV value equals its cell's text.
The reference checker, [`paperspec check`](../src/paperspec/cli/check.py), confirms each CSV value equals its cell's text.
## 8. Sharing without the text