Reorganize paperspec commands and pin dependency versions #2
No files matched your search
+2
-1
@@ -28,6 +28,7 @@ Packaging, commands and lazy imports only; the formats stay at 0.1.0, and no com
|
|||||||
- The package version comes from git tags through setuptools-scm.
|
- The package version comes from git tags through setuptools-scm.
|
||||||
- The base package has no dependencies and requires Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import under any other Unicode version; the extras `paperspec[schema]` (`jsonschema`, `referencing`), `paperspec[crypto]` (`cryptography`) and `paperspec[tools]` (both) cover `schema` and the HKDF and Ed25519 functions of `crypto`, which now import their packages on first use.
|
- The base package has no dependencies and requires Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import under any other Unicode version; the extras `paperspec[schema]` (`jsonschema`, `referencing`), `paperspec[crypto]` (`cryptography`) and `paperspec[tools]` (both) cover `schema` and the HKDF and Ed25519 functions of `crypto`, which now import their packages on first use.
|
||||||
- `paperspec.resources.path` finds the schemas, SQL and vectors in an installed wheel and in an editable install alike.
|
- `paperspec.resources.path` finds the schemas, SQL and vectors in an installed wheel and in an editable install alike.
|
||||||
- The programs in `tools/` become the package's commands in `src/paperspec/cli/`: `paperspec-check`, `paperspec-build`, `paperspec-validate`, `paperspec-load`, `paperspec-stress` and `paperspec-site`, which the pixi tasks and the workflow run.
|
- The programs in `tools/` become subcommands of one command, `paperspec`, in `src/paperspec/cli/`: `paperspec validate`, `load`, `check`, `build`, `stress` and `site`, which the pixi tasks and the workflow run, and which `python -m paperspec` also runs.
|
||||||
|
- `paperspec --help` says which commands work anywhere and which need a checkout, and each command's `--help` gives advice and examples; a misspelled command gets a suggestion.
|
||||||
- `src/` is Apache-2.0, as `tools/` was.
|
- `src/` is Apache-2.0, as `tools/` was.
|
||||||
- The pixi workspace installs the package editable, and a `dev` environment builds the wheel with `pixi run -e dev wheel`.
|
- The pixi workspace installs the package editable, and a `dev` environment builds the wheel with `pixi run -e dev wheel`.
|
||||||
+2
-2
@@ -17,7 +17,7 @@ The schemas, SQL, registries, vectors and fixtures then follow the chapter ([spe
|
|||||||
Then run `pixi run build`, which rewrites `fixtures/` and `vectors/`.
|
Then run `pixi run build`, which rewrites `fixtures/` and `vectors/`.
|
||||||
4. **Run the checks.**
|
4. **Run the checks.**
|
||||||
`pixi run check` must end with `All checks pass.`
|
`pixi run check` must end with `All checks pass.`
|
||||||
Without pixi, install the checkout with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec-check`.
|
Without pixi, install the checkout with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec check`.
|
||||||
5. **Record the change** in [CHANGELOG.md](CHANGELOG.md).
|
5. **Record the change** in [CHANGELOG.md](CHANGELOG.md).
|
||||||
|
|
||||||
## Versions
|
## Versions
|
||||||
@@ -39,7 +39,7 @@ Before tagging:
|
|||||||
|
|
||||||
1. `pixi run check` passes on Linux and macOS, which the workflow in `.gitea/workflows/` runs on every push.
|
1. `pixi run check` passes on Linux and macOS, which the workflow in `.gitea/workflows/` runs on every push.
|
||||||
The check also confirms that every count the prose states (formats, schemas, vectors, broken packages) matches the repository, and that `pixi.toml`, the registries and `src/paperspec/cli/build.py` state the same version.
|
The check also confirms that every count the prose states (formats, schemas, vectors, broken packages) matches the repository, and that `pixi.toml`, the registries and `src/paperspec/cli/build.py` state the same version.
|
||||||
2. `paperspec-stress` passes with its default seed, and with one or two others.
|
2. `paperspec stress` passes with its default seed, and with one or two others.
|
||||||
3. The website is published from the release ([deploy/](deploy/README.md)), and the JSON-LD context at `https://paperspec.scient.ing/context/scienting-1.jsonld` and a schema at its `$id` both answer with their media types.
|
3. The website is published from the release ([deploy/](deploy/README.md)), and the JSON-LD context at `https://paperspec.scient.ing/context/scienting-1.jsonld` and a schema at its `$id` both answer with their media types.
|
||||||
Layers claim to be W3C Web Annotations, and a JSON-LD processor that cannot fetch the context cannot read them.
|
Layers claim to be W3C Web Annotations, and a JSON-LD processor that cannot fetch the context cannot read them.
|
||||||
4. The decisions that freeze on release are made: enumeration values and format identifiers never change meaning afterwards ([spec/22 §22.5](spec/22-conformance.md#225-stability)), so a name worth changing is changed now.
|
4. The decisions that freeze on release are made: enumeration values and format identifiers never change meaning afterwards ([spec/22 §22.5](spec/22-conformance.md#225-stability)), so a name worth changing is changed now.
|
||||||
|
|||||||
@@ -84,7 +84,7 @@ The run ends with `All checks pass.`
|
|||||||
`pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)).
|
`pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)).
|
||||||
|
|
||||||
The tools need Python 3.14 and two packages, `jsonschema` and `cryptography`, which pixi installs along with markdownlint for the Markdown.
|
The tools need Python 3.14 and two packages, `jsonschema` and `cryptography`, which pixi installs along with markdownlint for the Markdown.
|
||||||
Without pixi, install the checkout as the package with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec-check`.
|
Without pixi, install the checkout as the package with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec check`.
|
||||||
|
|
||||||
To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from.
|
To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from.
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -96,7 +96,7 @@ Three of the nine fixture annotations have only an `AnchorSelector` (the section
|
|||||||
- Reader: Follow [chapter 6](../spec/06-anchors.md) to resolve an anchor.
|
- Reader: Follow [chapter 6](../spec/06-anchors.md) to resolve an anchor.
|
||||||
The file `vectors/resolve.jsonl` resolves seventeen anchors against `fixtures/packages/hale2026harmonic-damb` and gives every outcome.
|
The file `vectors/resolve.jsonl` resolves seventeen anchors against `fixtures/packages/hale2026harmonic-damb` and gives every outcome.
|
||||||
- JSON-LD processor: Use a JSON-LD 1.1 processor; the context relies on `@json` and `@version` 1.1.
|
- JSON-LD processor: Use a JSON-LD 1.1 processor; the context relies on `@json` and `@version` 1.1.
|
||||||
- Checker: `schemas/annotation.schema.json` and `schemas/layer.schema.json` validate the files. `paperspec-check` also checks that each annotation's ID derives from its content and that the receipt at `fixtures/records/layer/receipt.json` covers each `confirmed` annotation.
|
- Checker: `schemas/annotation.schema.json` and `schemas/layer.schema.json` validate the files. `paperspec check` also checks that each annotation's ID derives from its content and that the receipt at `fixtures/records/layer/receipt.json` covers each `confirmed` annotation.
|
||||||
|
|
||||||
The traps:
|
The traps:
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1,7 +1,7 @@
|
|||||||
# Deploying the website
|
# Deploying the website
|
||||||
|
|
||||||
This folder holds what it takes to serve the paperspec website at <https://paperspec.scient.ing>.
|
This folder holds what it takes to serve the paperspec website at <https://paperspec.scient.ing>.
|
||||||
The site is static: `paperspec-site` builds it into `_site/`, and any web server can serve that folder.
|
The site is static: `paperspec site` builds it into `_site/`, and any web server can serve that folder.
|
||||||
|
|
||||||
## What the site serves
|
## What the site serves
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -428,7 +428,7 @@ Keep using it as a reading and distribution format.
|
|||||||
*Status.*
|
*Status.*
|
||||||
The specification is version 0.1.0.
|
The specification is version 0.1.0.
|
||||||
`registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
|
`registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
|
||||||
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec-check`.
|
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`.
|
||||||
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; tools Apache-2.0 ([README.md](../README.md#licenses)).
|
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; tools Apache-2.0 ([README.md](../README.md#licenses)).
|
||||||
|
|
||||||
*No adoption claim.*
|
*No adoption claim.*
|
||||||
|
|||||||
+5
-5
@@ -19,10 +19,10 @@ This shows how to confirm that the repository's specification, fixtures and test
|
|||||||
pixi run check
|
pixi run check
|
||||||
```
|
```
|
||||||
|
|
||||||
`pixi run check` lints the Markdown, then runs `paperspec-check`, the repository's reference checker.
|
`pixi run check` lints the Markdown, then runs `paperspec check`, the repository's reference checker.
|
||||||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||||||
`pixi run quick`, or `paperspec-check --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
|
`pixi run quick`, or `paperspec check --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
|
||||||
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `paperspec-check` does not run it.
|
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `paperspec check` does not run it.
|
||||||
|
|
||||||
## Recompute a hash by hand
|
## Recompute a hash by hand
|
||||||
|
|
||||||
@@ -146,7 +146,7 @@ This shows that the same packages can live in a database and be queried with SQL
|
|||||||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
|
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
|
||||||
```
|
```
|
||||||
|
|
||||||
Without pixi, run `paperspec-load` with the same arguments.
|
Without pixi, run `paperspec load` with the same arguments.
|
||||||
|
|
||||||
```text
|
```text
|
||||||
hale2026harmonic-damb: 75 nodes
|
hale2026harmonic-damb: 75 nodes
|
||||||
@@ -154,7 +154,7 @@ okafor2025solvent-acth: 15 nodes
|
|||||||
writer2027fixed-ynr7.v3: 15 nodes
|
writer2027fixed-ynr7.v3: 15 nodes
|
||||||
```
|
```
|
||||||
|
|
||||||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec-check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
|
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
|
||||||
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
|
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
|||||||
+2
-2
@@ -107,7 +107,7 @@ A `candidate` format is complete, with vectors and fixtures, and ready to implem
|
|||||||
A `provisional` one is complete and expected to change as implementations gain experience.
|
A `provisional` one is complete and expected to change as implementations gain experience.
|
||||||
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint.
|
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint.
|
||||||
|
|
||||||
The repository holds a reference checker (`paperspec-check`), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
|
The repository holds a reference checker (`paperspec check`), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
|
||||||
|
|
||||||
## What does it not do?
|
## What does it not do?
|
||||||
|
|
||||||
@@ -126,6 +126,6 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md).
|
|||||||
|
|
||||||
## What are the files in `tools/` for?
|
## What are the files in `tools/` for?
|
||||||
|
|
||||||
`paperspec-build` builds the fixtures and vectors. `paperspec-check` recomputes and checks every one, and `paperspec-load` loads packages into SQLite.
|
`paperspec build` builds the fixtures and vectors. `paperspec check` recomputes and checks every one, and `paperspec load` loads packages into SQLite.
|
||||||
The oracle's modules in `src/paperspec/` compute every expected value, so that none is typed by hand.
|
The oracle's modules in `src/paperspec/` compute every expected value, so that none is typed by hand.
|
||||||
The other repositories of the project install them as the Python package `paperspec`; independent implementations follow the specification ([tools/README.md](../tools/README.md)).
|
The other repositories of the project install them as the Python package `paperspec`; independent implementations follow the specification ([tools/README.md](../tools/README.md)).
|
||||||
+3
-3
@@ -54,7 +54,7 @@ It appears inside hashed data and test vectors, and it is separate from the repo
|
|||||||
| [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. |
|
| [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. |
|
||||||
| [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. |
|
| [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. |
|
||||||
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
|
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
|
||||||
| [tools/](../tools/README.md) | How to run the reference commands: `paperspec-check` checks everything, `paperspec-build` rebuilds the fixtures and vectors, and `paperspec-load` loads packages into SQLite. |
|
| [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. |
|
||||||
| [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. |
|
| [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. |
|
||||||
| [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. |
|
| [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. |
|
||||||
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
|
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
|
||||||
@@ -73,8 +73,8 @@ The repository ships reference tools only for the checker and store roles.
|
|||||||
| Reader | Opens a package, resolves addresses and anchors, and shows text, annotations and each passage's fidelity level. |
|
| Reader | Opens a package, resolves addresses and anchors, and shows text, annotations and each passage's fidelity level. |
|
||||||
| Writer | Produces files in these formats: nodes, corrections, receipts, annotations, datasets. |
|
| Writer | Produces files in these formats: nodes, corrections, receipts, annotations, datasets. |
|
||||||
| Converter | Turns a witness (a PDF, JATS XML, the article format publishers use, a LaTeX source archive, an HTML page, an EPUB or a Typst project) into a package and raises doubtful readings as suspects (flagged disagreements awaiting review). |
|
| Converter | Turns a witness (a PDF, JATS XML, the article format publishers use, a LaTeX source archive, an HTML page, an EPUB or a Typst project) into a package and raises doubtful readings as suspects (flagged disagreements awaiting review). |
|
||||||
| Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `paperspec-check` does this for the repository. |
|
| Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `paperspec check` does this for the repository. |
|
||||||
| Store | Keeps packages and records in a database a team can clone. `paperspec-load` loads packages into SQLite. |
|
| Store | Keeps packages and records in a database a team can clone. `paperspec load` loads packages into SQLite. |
|
||||||
|
|
||||||
The [specification's reading paths](../spec/README.md) say which chapters each role needs.
|
The [specification's reading paths](../spec/README.md) say which chapters each role needs.
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -13,7 +13,7 @@ It has nothing to do with the repository's name.
|
|||||||
|
|
||||||
Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it.
|
Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it.
|
||||||
A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line).
|
A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line).
|
||||||
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`paperspec-build`](../src/paperspec/cli/build.py) expands it into the package.
|
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`paperspec build`](../src/paperspec/cli/build.py) expands it into the package.
|
||||||
Each step says what a converter, a person or a checker does.
|
Each step says what a converter, a person or a checker does.
|
||||||
|
|
||||||
## 1. A PDF arrives
|
## 1. A PDF arrives
|
||||||
@@ -166,7 +166,7 @@ It is a [Frictionless Data Package](https://datapackage.org/standard/data-packag
|
|||||||
Its CSV keeps every digit as printed (`1.526` and `1.5260` stay different), and every value has an anchor to the cell that prints it, so a claim that rests on a value can show the cell.
|
Its CSV keeps every digit as printed (`1.526` and `1.5260` stay different), and every value has an anchor to the cell that prints it, so a claim that rests on a value can show the cell.
|
||||||
The descriptor types the fields as numbers; how a generic Data Package tool treats `1.5260` was not verified.
|
The descriptor types the fields as numbers; how a generic Data Package tool treats `1.5260` was not verified.
|
||||||
All twelve value anchors resolve as `current`.
|
All twelve value anchors resolve as `current`.
|
||||||
The reference checker, [`paperspec-check`](../src/paperspec/cli/check.py), confirms each CSV value equals its cell's text.
|
The reference checker, [`paperspec check`](../src/paperspec/cli/check.py), confirms each CSV value equals its cell's text.
|
||||||
|
|
||||||
## 8. Sharing without the text
|
## 8. Sharing without the text
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -35,7 +35,7 @@ A conforming reader accepts each one and recomputes every hash it contains.
|
|||||||
## Records
|
## Records
|
||||||
|
|
||||||
`records/` holds a valid example of every other format that is a JSON file of its own.
|
`records/` holds a valid example of every other format that is a JSON file of its own.
|
||||||
Three formats have their example elsewhere: the stage requirements (`scienting-stages/1`) and the stamp rules (`scienting-stamps/1`) are in [registry/](../registry/README.md), and a bundle (`scienting-bundle/1`) is a SQLite file that `paperspec-load --bundle` builds from the packages.
|
Three formats have their example elsewhere: the stage requirements (`scienting-stages/1`) and the stamp rules (`scienting-stamps/1`) are in [registry/](../registry/README.md), and a bundle (`scienting-bundle/1`) is a SQLite file that `paperspec load --bundle` builds from the packages.
|
||||||
Each record is a single file, or a folder where a format needs several files.
|
Each record is a single file, or a folder where a format needs several files.
|
||||||
|
|
||||||
| Files | What they are for |
|
| Files | What they are for |
|
||||||
@@ -109,4 +109,4 @@ Use them for nothing else.
|
|||||||
## Sources
|
## Sources
|
||||||
|
|
||||||
`src/` holds the hand-written descriptions of the three main documents, as first read; see [src/README.md](src/README.md).
|
`src/` holds the hand-written descriptions of the three main documents, as first read; see [src/README.md](src/README.md).
|
||||||
Everything else here is generated by `paperspec-build`, so change the sources or the builder, never the generated files.
|
Everything else here is generated by `paperspec build`, so change the sources or the builder, never the generated files.
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Fixture sources
|
# Fixture sources
|
||||||
|
|
||||||
Each source describes a document as a converter first read it, mistakes included, in a shorthand that `paperspec-build` expands.
|
Each source describes a document as a converter first read it, mistakes included, in a shorthand that `paperspec build` expands.
|
||||||
The build mints IDs, applies the listed corrections, lays out synthetic geometry, signs the listed receipts and writes the package.
|
The build mints IDs, applies the listed corrections, lays out synthetic geometry, signs the listed receipts and writes the package.
|
||||||
The shorthand is tooling, not a format.
|
The shorthand is tooling, not a format.
|
||||||
|
|
||||||
|
|||||||
@@ -39,13 +39,13 @@ default = { solve-group = "default" }
|
|||||||
dev = { features = ["dev"], solve-group = "default" }
|
dev = { features = ["dev"], solve-group = "default" }
|
||||||
|
|
||||||
[tasks]
|
[tasks]
|
||||||
check = { cmd = "paperspec-check", depends-on = [
|
check = { cmd = "paperspec check", depends-on = [
|
||||||
"lint",
|
"lint",
|
||||||
], description = "Check schemas, records, packages, vectors, a byte-for-byte rebuild, the SQL round trip, prose and links" }
|
], description = "Check schemas, records, packages, vectors, a byte-for-byte rebuild, the SQL round trip, prose and links" }
|
||||||
quick = { cmd = "paperspec-check --quick", description = "Check everything except the byte-for-byte rebuild" }
|
quick = { cmd = "paperspec check --quick", description = "Check everything except the byte-for-byte rebuild" }
|
||||||
build = { cmd = "paperspec-build", description = "Rebuild fixtures/ and vectors/ from fixtures/src/ and the oracle" }
|
build = { cmd = "paperspec build", description = "Rebuild fixtures/ and vectors/ from fixtures/src/ and the oracle" }
|
||||||
lint = { cmd = "markdownlint-cli2", description = "Lint every Markdown file with markdownlint" }
|
lint = { cmd = "markdownlint-cli2", description = "Lint every Markdown file with markdownlint" }
|
||||||
load = { cmd = "paperspec-load", description = "Load packages into SQLite: pixi run load --sqlite lab.db PACKAGE..." }
|
load = { cmd = "paperspec load", description = "Load packages into SQLite: pixi run load --sqlite lab.db PACKAGE..." }
|
||||||
validate = { cmd = "paperspec-validate", description = "Validate packages, or run proposed corrections through the gate" }
|
validate = { cmd = "paperspec validate", description = "Validate packages, or run proposed corrections through the gate" }
|
||||||
stress = { cmd = "paperspec-stress", description = "Stress the oracle with random inputs: pixi run stress --seed N" }
|
stress = { cmd = "paperspec stress", description = "Stress the oracle with random inputs: pixi run stress --seed N" }
|
||||||
site = { cmd = "paperspec-site", description = "Build the website into _site/" }
|
site = { cmd = "paperspec site", description = "Build the website into _site/" }
|
||||||
+3
-7
@@ -29,14 +29,10 @@ schema = ["jsonschema>=4.26", "referencing>=0.35"]
|
|||||||
crypto = ["cryptography>=46"]
|
crypto = ["cryptography>=46"]
|
||||||
tools = ["paperspec[crypto,schema]"]
|
tools = ["paperspec[crypto,schema]"]
|
||||||
|
|
||||||
# validate and load work wherever the package is installed with [tools]; the other four work on a checkout.
|
# One command with subcommands: paperspec validate, load, check, build, stress and site; `paperspec --help`
|
||||||
|
# says which work anywhere and which need a checkout.
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
paperspec-build = "paperspec.cli.build:cli"
|
paperspec = "paperspec.cli.main:main"
|
||||||
paperspec-check = "paperspec.cli.check:main"
|
|
||||||
paperspec-load = "paperspec.cli.load:main"
|
|
||||||
paperspec-site = "paperspec.cli.site:main"
|
|
||||||
paperspec-stress = "paperspec.cli.stress:main"
|
|
||||||
paperspec-validate = "paperspec.cli.validate:main"
|
|
||||||
|
|
||||||
[project.urls]
|
[project.urls]
|
||||||
Homepage = "https://paperspec.scient.ing"
|
Homepage = "https://paperspec.scient.ing"
|
||||||
|
|||||||
+1
-1
@@ -43,7 +43,7 @@ A rule's meaning never changes, and a retired rule's ID is never reused, so tool
|
|||||||
A tool can match on the `rule` field, whose meaning is looked up in `rules.json`, and treat the message as text for people.
|
A tool can match on the `rule` field, whose meaning is looked up in `rules.json`, and treat the message as text for people.
|
||||||
[Chapter 22, section 22.4](../spec/22-conformance.md#224-rules) lists the ID prefixes.
|
[Chapter 22, section 22.4](../spec/22-conformance.md#224-rules) lists the ID prefixes.
|
||||||
Each directory in `fixtures/invalid/` has an `expect.json` that names the rule it breaks.
|
Each directory in `fixtures/invalid/` has an `expect.json` that names the rule it breaks.
|
||||||
The reference checker (`paperspec-check`) fails if that rule is not in `rules.json`.
|
The reference checker (`paperspec check`) fails if that rule is not in `rules.json`.
|
||||||
|
|
||||||
## Adding stamp rules
|
## Adding stamp rules
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -64,7 +64,7 @@ The `Registry` maps that name to the file's contents, so nothing is fetched over
|
|||||||
Never load a schema alone.
|
Never load a schema alone.
|
||||||
|
|
||||||
Any draft 2020-12 validator works the same way: give it every schema under its `$id`, then validate against the one you need.
|
Any draft 2020-12 validator works the same way: give it every schema under its `$id`, then validate against the one you need.
|
||||||
`paperspec-check` validates every fixture with `jsonschema` this way, and checks that every schema keeps to the keyword subset.
|
`paperspec check` validates every fixture with `jsonschema` this way, and checks that every schema keeps to the keyword subset.
|
||||||
|
|
||||||
## What a schema does not check
|
## What a schema does not check
|
||||||
|
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ A correction is refused, naming the rule ([22](22-conformance.md)), when:
|
|||||||
11. the result breaks a rule of [3](03-nodes.md).
|
11. the result breaks a rule of [3](03-nodes.md).
|
||||||
|
|
||||||
The gate needs only the document's current state, so a tool can run it on a proposal before applying anything, and get back the record the correction would append or the rule it breaks.
|
The gate needs only the document's current state, so a tool can run it on a proposal before applying anything, and get back the record the correction would append or the rule it breaks.
|
||||||
`paperspec-validate --correction` does this against a package.
|
`paperspec validate --correction` does this against a package.
|
||||||
|
|
||||||
Digits are characters of General Category Nd in the plain projection.
|
Digits are characters of General Category Nd in the plain projection.
|
||||||
A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, and the last correction of each leaf ends at its current `node_hash`.
|
A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, and the last correction of each leaf ends at its current `node_hash`.
|
||||||
|
|||||||
@@ -6,7 +6,7 @@
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
|
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
|
||||||
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document, as a folder and as the zip of [12.2](12-package.md#122-paths). | Accepts each, and recomputes every value it contains. |
|
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document, as a folder and as the zip of [12.2](12-package.md#122-paths). | Accepts each, and recomputes every value it contains. |
|
||||||
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `paperspec-load --bundle`. | Accepts each. |
|
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `paperspec load --bundle`. | Accepts each. |
|
||||||
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
|
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
|
||||||
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |
|
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -31,7 +31,7 @@ dolt sql < sql/doc.sql && dolt sql < sql/store.sql
|
|||||||
The PostgreSQL database `lab` and the Dolt repository must already exist.
|
The PostgreSQL database `lab` and the Dolt repository must already exist.
|
||||||
The `sqlite3` line runs as shown; the `psql` and `dolt` lines follow the portability rules of [chapter 13](../spec/13-database.md#131-portability) but have not been run on those engines in this repository.
|
The `sqlite3` line runs as shown; the `psql` and `dolt` lines follow the portability rules of [chapter 13](../spec/13-database.md#131-portability) but have not been run on those engines in this repository.
|
||||||
|
|
||||||
To load packages into SQLite without pixi, install the package with its `tools` extra on Python 3.14, then run `paperspec-load --sqlite lab.db PACKAGE…`.
|
To load packages into SQLite without pixi, install the package with its `tools` extra on Python 3.14, then run `paperspec load --sqlite lab.db PACKAGE…`.
|
||||||
The file [docs/examples.md](../docs/examples.md#load-packages-into-sqlite) has queries to try.
|
The file [docs/examples.md](../docs/examples.md#load-packages-into-sqlite) has queries to try.
|
||||||
The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into `lab.db.assets/`.
|
The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into `lab.db.assets/`.
|
||||||
|
|
||||||
@@ -57,7 +57,7 @@ On `fixtures/packages/hale2026harmonic-damb` it returns one row, `hca6dc5`: the
|
|||||||
- Names: no table, column or index name is reserved in SQLite, PostgreSQL or MySQL (Dolt is MySQL-compatible).
|
- Names: no table, column or index name is reserved in SQLite, PostgreSQL or MySQL (Dolt is MySQL-compatible).
|
||||||
The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
|
The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
|
||||||
- Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash.
|
- Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash.
|
||||||
Unloading copies the files with no table from the asset store, which must travel with the database. `paperspec-check` tests this for every fixture package.
|
Unloading copies the files with no table from the asset store, which must travel with the database. `paperspec check` tests this for every fixture package.
|
||||||
- Derived tables: `citations`, `receipt_nodes` and `annotation_targets` are built on load for queries and are not part of the package.
|
- Derived tables: `citations`, `receipt_nodes` and `annotation_targets` are built on load for queries and are not part of the package.
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
"""`python -m paperspec COMMAND`: the same as the `paperspec` command."""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from paperspec.cli.main import main
|
||||||
|
|
||||||
|
sys.exit(main())
|
||||||
@@ -1,21 +1,28 @@
|
|||||||
"""The paperspec commands, installed as console scripts; see tools/README.md.
|
"""The `paperspec` command's subcommands, one module each; the parser and the help are in `main.py`.
|
||||||
|
|
||||||
`paperspec-validate` and `paperspec-load` work anywhere the package is installed with `paperspec[tools]`.
|
`paperspec validate` and `paperspec load` work anywhere the package is installed with `paperspec[tools]`.
|
||||||
`paperspec-check`, `paperspec-build`, `paperspec-stress` and `paperspec-site` work on a checkout of the
|
`paperspec check`, `build`, `stress` and `site` work on a checkout of the paperspec repository: the one an
|
||||||
paperspec repository: the one an editable install points at, or else the one around the current folder.
|
editable install points at, or else the one around the current folder.
|
||||||
Each is also a module, so `python -m paperspec.cli.check` runs the same program.
|
Each module has `run(args)`, which takes the parsed arguments and returns the exit status.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import importlib.util
|
import importlib.util
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
|
REPOSITORY = "ssh://git@git.scient.ing/infra/paperspec.git"
|
||||||
|
|
||||||
|
|
||||||
|
class UsageError(Exception):
|
||||||
|
"""Arguments that parse but do not fit together; `paperspec` prints the command's usage and exits 2."""
|
||||||
|
|
||||||
|
|
||||||
def require(*modules):
|
def require(*modules):
|
||||||
"""Exit with a message naming the extra to install when a package a command needs is missing."""
|
"""Exit with what to install when a package a command needs is missing."""
|
||||||
missing = [m for m in modules if importlib.util.find_spec(m) is None]
|
missing = [m for m in modules if importlib.util.find_spec(m) is None]
|
||||||
if missing:
|
if missing:
|
||||||
sys.exit("This command needs %s %s; install paperspec[tools], or run it through pixi."
|
sys.exit("This command needs %s %s.\nInstall them with: pip install \"paperspec[tools]\"\n"
|
||||||
|
"In a checkout of the repository, pixi installs them: pixi run COMMAND"
|
||||||
% ("the package" if len(missing) == 1 else "the packages", " and ".join(missing)))
|
% ("the package" if len(missing) == 1 else "the packages", " and ".join(missing)))
|
||||||
|
|
||||||
|
|
||||||
@@ -24,10 +31,10 @@ def _is_checkout(d):
|
|||||||
|
|
||||||
|
|
||||||
def checkout():
|
def checkout():
|
||||||
"""The root of the paperspec checkout a repository command works on, or exit with a message.
|
"""The root of the paperspec checkout a repository command works on, or exit with what to do.
|
||||||
|
|
||||||
An editable install (and `python -m` from `src/`) runs the checkout this file is in, so the code and the
|
An editable install (and `python -m paperspec` from `src/`) runs the checkout this file is in, so the code
|
||||||
files it checks always match; an installed wheel uses the nearest checkout above the current folder.
|
and the files it checks always match; an installed wheel uses the nearest checkout above the current folder.
|
||||||
"""
|
"""
|
||||||
own = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
|
own = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
|
||||||
if _is_checkout(own):
|
if _is_checkout(own):
|
||||||
@@ -38,5 +45,9 @@ def checkout():
|
|||||||
return d
|
return d
|
||||||
parent = os.path.dirname(d)
|
parent = os.path.dirname(d)
|
||||||
if parent == d:
|
if parent == d:
|
||||||
sys.exit("This command works on a checkout of the paperspec repository; run it inside one.")
|
sys.exit("This command works on a checkout of the paperspec repository, and %s is not inside one.\n"
|
||||||
|
"Clone it and run the command there:\n"
|
||||||
|
" git clone %s\n"
|
||||||
|
" cd paperspec && pixi run check\n"
|
||||||
|
"paperspec validate and paperspec load work anywhere." % (os.getcwd(), REPOSITORY))
|
||||||
d = parent
|
d = parent
|
||||||
@@ -1,9 +1,8 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Build every fixture and test vector from fixtures/src and the cases below.
|
"""Build every fixture and test vector from fixtures/src and the cases below.
|
||||||
|
|
||||||
Non-normative tooling.
|
Non-normative tooling.
|
||||||
The build is deterministic: run it twice and the output is byte-identical; `paperspec-check` verifies that.
|
The build is deterministic: run it twice and the output is byte-identical; `paperspec check` verifies that.
|
||||||
Usage: paperspec-build [--out DIR] (default: the repository itself)
|
Usage: paperspec build [--out DIR] (default: the repository itself)
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import base64
|
import base64
|
||||||
@@ -1853,21 +1852,16 @@ def main(out):
|
|||||||
invalid(out, tiny)
|
invalid(out, tiny)
|
||||||
|
|
||||||
|
|
||||||
def cli():
|
def run(args):
|
||||||
import argparse
|
out = os.path.abspath(args.out or ROOT)
|
||||||
ap = argparse.ArgumentParser(
|
|
||||||
description="Build every fixture and test vector from fixtures/src and the oracle. The build is "
|
|
||||||
"deterministic, and paperspec-check rebuilds into a temporary folder to prove it.")
|
|
||||||
ap.add_argument("--out", metavar="DIR", default=ROOT,
|
|
||||||
help="build into DIR instead of rewriting fixtures/ and vectors/ in this repository")
|
|
||||||
out = os.path.abspath(ap.parse_args().out)
|
|
||||||
if out != ROOT:
|
if out != ROOT:
|
||||||
for sub in ("registry", "fixtures/src"):
|
for sub in ("registry", "fixtures/src"):
|
||||||
dst = os.path.join(out, sub)
|
dst = os.path.join(out, sub)
|
||||||
if not os.path.exists(dst):
|
if not os.path.exists(dst):
|
||||||
shutil.copytree(os.path.join(ROOT, sub), dst)
|
shutil.copytree(os.path.join(ROOT, sub), dst)
|
||||||
main(out)
|
main(out)
|
||||||
|
if out == ROOT:
|
||||||
|
print("Rebuilt fixtures/ and vectors/ in %s.\n"
|
||||||
if __name__ == "__main__":
|
"Review the changes with git status and git diff, then commit them with the change that caused them."
|
||||||
cli()
|
% ROOT)
|
||||||
|
return 0
|
||||||
+10
-16
@@ -1,8 +1,7 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Check the whole repository. Exit status 0 means every check passed.
|
"""Check the whole repository. Exit status 0 means every check passed.
|
||||||
|
|
||||||
Non-normative tooling; it needs the packages `jsonschema` and `cryptography`, which pixi and paperspec[tools] install.
|
Non-normative tooling; it needs the packages `jsonschema` and `cryptography`, which pixi and paperspec[tools] install.
|
||||||
Usage: paperspec-check [--quick] (--quick skips the determinism rebuild)
|
Usage: paperspec check [--quick] (--quick skips the determinism rebuild)
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import base64
|
import base64
|
||||||
@@ -710,7 +709,7 @@ def check_vectors():
|
|||||||
|
|
||||||
def check_determinism():
|
def check_determinism():
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
subprocess.run([sys.executable, "-m", "paperspec.cli.build", "--out", tmp], check=True)
|
subprocess.run([sys.executable, "-m", "paperspec", "build", "--out", tmp], check=True)
|
||||||
for sub in ("fixtures", "vectors"):
|
for sub in ("fixtures", "vectors"):
|
||||||
# Files the repository ignores, such as macOS's .DS_Store, are not part of the build.
|
# Files the repository ignores, such as macOS's .DS_Store, are not part of the build.
|
||||||
cmp = filecmp.dircmp(p(sub), os.path.join(tmp, sub), ignore=filecmp.DEFAULT_IGNORES + [".DS_Store"])
|
cmp = filecmp.dircmp(p(sub), os.path.join(tmp, sub), ignore=filecmp.DEFAULT_IGNORES + [".DS_Store"])
|
||||||
@@ -1166,14 +1165,7 @@ def check_site():
|
|||||||
ok("site", t[4:] in pages["vocab/index.html"].ids, "the vocabulary page has no #%s" % t[4:])
|
ok("site", t[4:] in pages["vocab/index.html"].ids, "the vocabulary page has no #%s" % t[4:])
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def run(args):
|
||||||
import argparse
|
|
||||||
ap = argparse.ArgumentParser(
|
|
||||||
description="Check the whole repository: schemas, records, packages, zipped packages, broken packages, "
|
|
||||||
"vectors, a byte-for-byte rebuild, the SQL round trip and the bundle, counts, prose and links. "
|
|
||||||
"Exit status 0 means every check passed.")
|
|
||||||
ap.add_argument("--quick", action="store_true", help="skip the byte-for-byte rebuild, the slowest check")
|
|
||||||
args = ap.parse_args()
|
|
||||||
check_schemas()
|
check_schemas()
|
||||||
check_registry()
|
check_registry()
|
||||||
check_records()
|
check_records()
|
||||||
@@ -1200,9 +1192,11 @@ def main():
|
|||||||
print("\n%d failures:" % len(FAILS))
|
print("\n%d failures:" % len(FAILS))
|
||||||
for f in FAILS[:60]:
|
for f in FAILS[:60]:
|
||||||
print(" " + f)
|
print(" " + f)
|
||||||
sys.exit(1)
|
if len(FAILS) > 60:
|
||||||
|
print(" and %d more" % (len(FAILS) - 60))
|
||||||
|
if any(f.startswith("determinism") for f in FAILS):
|
||||||
|
print("\nThe rebuild differs from fixtures/ or vectors/: if you changed a rule, a fixture source or a"
|
||||||
|
"\nbuild case, run paperspec build, review the diff and commit the regenerated files.")
|
||||||
|
return 1
|
||||||
print("\nAll checks pass.")
|
print("\nAll checks pass.")
|
||||||
|
return 0
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
+12
-22
@@ -1,9 +1,8 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Load packages into a SQLite database with the tables of sql/doc.sql and sql/store.sql.
|
"""Load packages into a SQLite database with the tables of sql/doc.sql and sql/store.sql.
|
||||||
|
|
||||||
Non-normative tooling.
|
Non-normative tooling.
|
||||||
Usage: paperspec-load --sqlite lab.db PACKAGE [PACKAGE ...]
|
Usage: paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]
|
||||||
paperspec-load --sqlite lab.db --bundle --built-from REV PACKAGE [PACKAGE ...]
|
paperspec load --sqlite lab.db --bundle --built-from REV PACKAGE [PACKAGE ...]
|
||||||
A package is a folder, or a zipped package (`.sct.zip`), unpacked under the rules of spec/12 §12.2 first.
|
A package is a folder, or a zipped package (`.sct.zip`), unpacked under the rules of spec/12 §12.2 first.
|
||||||
With --bundle the new file is a scienting-bundle/1: it also gets the tables of sql/bundle-sqlite.sql,
|
With --bundle the new file is a scienting-bundle/1: it also gets the tables of sql/bundle-sqlite.sql,
|
||||||
the full-text search index and bundle_meta (spec/13 §13.5).
|
the full-text search index and bundle_meta (spec/13 §13.5).
|
||||||
@@ -12,13 +11,12 @@ Files without tables of their own (assets, datasets, derived files) are copied b
|
|||||||
asset store folder beside the database, named <database>.assets/.
|
asset store folder beside the database, named <database>.assets/.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
|
||||||
import os
|
import os
|
||||||
import sqlite3
|
import sqlite3
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
from paperspec.cli import require
|
from paperspec.cli import UsageError, require
|
||||||
|
|
||||||
require("cryptography", "jsonschema")
|
require("cryptography", "jsonschema")
|
||||||
|
|
||||||
@@ -26,18 +24,13 @@ from paperspec import db, package, resources # noqa: E402
|
|||||||
from paperspec.schema import Store # noqa: E402
|
from paperspec.schema import Store # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def run(a):
|
||||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
if a.bundle and not a.built_from:
|
||||||
ap.add_argument("--sqlite", required=True, help="the database file; created with the tables if new")
|
raise UsageError("--bundle needs --built-from REV, the store revision the packages come from")
|
||||||
ap.add_argument("--bundle", action="store_true", help="build a new scienting-bundle/1 file with search")
|
if a.bundle and os.path.exists(a.sqlite):
|
||||||
ap.add_argument("--built-from", help="with --bundle: the store revision the packages come from")
|
raise UsageError("--bundle builds a new file, and %s exists; give a new name or remove it" % a.sqlite)
|
||||||
ap.add_argument("--store", help="with --bundle: the store's name (default: the packages' common store)")
|
if not a.bundle and (a.built_from or a.store):
|
||||||
ap.add_argument("--max-size", metavar="BYTES", type=int, default=package.ZIP_LIMIT,
|
raise UsageError("--built-from and --store go with --bundle")
|
||||||
help="refuse a zipped package that expands past this many bytes (default %d)" % package.ZIP_LIMIT)
|
|
||||||
ap.add_argument("packages", nargs="+", help="package folders or zipped packages")
|
|
||||||
a = ap.parse_args()
|
|
||||||
if a.bundle and (os.path.exists(a.sqlite) or not a.built_from):
|
|
||||||
ap.error("--bundle needs --built-from and a database file that does not exist yet")
|
|
||||||
store, bad = Store(resources.path("schemas")), False
|
store, bad = Store(resources.path("schemas")), False
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
folders = []
|
folders = []
|
||||||
@@ -55,7 +48,7 @@ def main():
|
|||||||
print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
|
print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
|
||||||
bad = True
|
bad = True
|
||||||
if bad:
|
if bad:
|
||||||
sys.exit("Nothing was loaded.")
|
sys.exit("Nothing was loaded. Fix the packages above; paperspec validate checks one without loading it.")
|
||||||
new = not os.path.exists(a.sqlite)
|
new = not os.path.exists(a.sqlite)
|
||||||
conn = sqlite3.connect(a.sqlite)
|
conn = sqlite3.connect(a.sqlite)
|
||||||
if a.bundle:
|
if a.bundle:
|
||||||
@@ -76,7 +69,4 @@ def main():
|
|||||||
sys.exit("The packages name no single store; give one with --store.")
|
sys.exit("The packages name no single store; give one with --store.")
|
||||||
db.write_meta(conn, store_name, a.built_from)
|
db.write_meta(conn, store_name, a.built_from)
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
return 0
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -0,0 +1,248 @@
|
|||||||
|
"""The `paperspec` command: one entry point, with a subcommand for each tool.
|
||||||
|
|
||||||
|
The parser lives here, apart from the commands, so that `paperspec --help` and `paperspec COMMAND --help` work
|
||||||
|
without a checkout or the tools extra; a command's module is imported only when the command runs.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import importlib
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from paperspec import UNICODE_VERSION, __version__
|
||||||
|
from paperspec.cli import UsageError
|
||||||
|
from paperspec.package import ZIP_LIMIT
|
||||||
|
|
||||||
|
SITE = "https://paperspec.scient.ing"
|
||||||
|
|
||||||
|
DESCRIPTION = """\
|
||||||
|
The reference implementation of the scienting formats: validate packages, load them into SQLite,
|
||||||
|
and check, rebuild and stress the paperspec repository itself.
|
||||||
|
"""
|
||||||
|
|
||||||
|
EPILOG = """\
|
||||||
|
examples:
|
||||||
|
paperspec validate paper.sct.zip check a package against every rule
|
||||||
|
paperspec validate --correction fix.json paper/ try a proposed correction without writing it
|
||||||
|
paperspec load --sqlite lab.db paper/ other/ load packages into a SQLite file to query
|
||||||
|
paperspec check --quick in a checkout: check everything but the rebuild
|
||||||
|
|
||||||
|
where each command works:
|
||||||
|
validate and load work wherever paperspec is installed with its tools extra:
|
||||||
|
pip install "paperspec[tools]"
|
||||||
|
check, build, stress and site work on a checkout of the paperspec repository,
|
||||||
|
where pixi also runs them as tasks: pixi run check
|
||||||
|
|
||||||
|
Run "paperspec COMMAND --help" for a command's options, advice and examples.
|
||||||
|
"""
|
||||||
|
|
||||||
|
PACKAGE_ADVICE = """\
|
||||||
|
A package is a folder, or a zipped package (.sct.zip) that is unpacked under the rules of
|
||||||
|
spec/12 §12.2 before anything else happens.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _validate(sub):
|
||||||
|
p = sub.add_parser(
|
||||||
|
"validate", help="check packages against every rule, or try proposed corrections",
|
||||||
|
description="Validate packages, or run proposed corrections through the gate against a package.\n\n"
|
||||||
|
+ PACKAGE_ADVICE + """
|
||||||
|
Each package is checked as spec/22 §22.2 describes, and each failure is printed as
|
||||||
|
`package: where: rule: message`. The rule ID is listed in """ + SITE + """/registry/rules.json
|
||||||
|
with the chapter that states the rule, which is where to read why it failed.
|
||||||
|
|
||||||
|
With --correction, PROPOSALS is a JSON file holding one proposed correction, or a JSON Lines
|
||||||
|
file of several, applied in order. Each runs through the gate (spec/11 §11.4) against the
|
||||||
|
package's current state, and the record it would append to corrections.jsonl is printed,
|
||||||
|
until one is refused. Nothing is written, so it is safe to try a correction before making it.
|
||||||
|
|
||||||
|
Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.""",
|
||||||
|
epilog="""\
|
||||||
|
examples:
|
||||||
|
paperspec validate fixtures/packages/hale2026harmonic-damb
|
||||||
|
paperspec validate paper/ other.sct.zip several at once; each failure names its package
|
||||||
|
paperspec validate --correction fix.json paper/ one proposal; prints the record it would append
|
||||||
|
paperspec validate --correction fixes.jsonl paper.sct.zip
|
||||||
|
several, in order, stopping at the first refusal""")
|
||||||
|
p.add_argument("--correction", metavar="PROPOSALS",
|
||||||
|
help="a JSON or JSON Lines file of proposed corrections to run through the gate")
|
||||||
|
p.add_argument("--max-size", metavar="BYTES", type=int, default=ZIP_LIMIT,
|
||||||
|
help="refuse a zipped package that expands past this many bytes (default %d, 1 GiB)" % ZIP_LIMIT)
|
||||||
|
p.add_argument("packages", nargs="+", metavar="PACKAGE", help="a package folder or a zipped package")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def _load(sub):
|
||||||
|
p = sub.add_parser(
|
||||||
|
"load", help="validate packages and load them into a SQLite database",
|
||||||
|
description="Load packages into a SQLite database with the tables of sql/doc.sql and sql/store.sql.\n\n"
|
||||||
|
+ PACKAGE_ADVICE + """
|
||||||
|
Every package is validated first, and nothing is loaded if any fails; each failure is printed as
|
||||||
|
`package: where: rule: message`, as `paperspec validate` prints it. A new file gets the tables;
|
||||||
|
an existing one takes more packages. Files with no table of their own (assets, datasets, derived
|
||||||
|
files) are copied by SHA-256 into an asset store folder beside the database, DATABASE.assets/,
|
||||||
|
which must travel with it. The full and checks-only forms of one document share its key, so
|
||||||
|
load them into separate databases.
|
||||||
|
|
||||||
|
With --bundle the new file is a scienting-bundle/1 (spec/13 §13.5): it also gets the tables of
|
||||||
|
sql/bundle-sqlite.sql, the full-text search index and bundle_meta. A bundle is built once, into
|
||||||
|
a file that does not exist yet, and names the store revision it was built from.""",
|
||||||
|
epilog="""\
|
||||||
|
examples:
|
||||||
|
paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
||||||
|
paperspec load --sqlite lab.db more/ other.sct.zip add packages to the same database later
|
||||||
|
paperspec load --sqlite search.db --bundle --built-from 4fa326d paper/ other/
|
||||||
|
a new bundle with full-text search
|
||||||
|
sqlite3 lab.db "SELECT doc_key, COUNT(*) FROM nodes GROUP BY doc_key"
|
||||||
|
|
||||||
|
Queries to start from: %s/docs/examples.html""" % SITE)
|
||||||
|
p.add_argument("--sqlite", required=True, metavar="DATABASE",
|
||||||
|
help="the database file; created with the tables if it does not exist")
|
||||||
|
p.add_argument("--bundle", action="store_true", help="build a new scienting-bundle/1 file with search")
|
||||||
|
p.add_argument("--built-from", metavar="REV", help="with --bundle: the store revision the packages come from")
|
||||||
|
p.add_argument("--store", help="with --bundle: the store's name (default: the packages' common store)")
|
||||||
|
p.add_argument("--max-size", metavar="BYTES", type=int, default=ZIP_LIMIT,
|
||||||
|
help="refuse a zipped package that expands past this many bytes (default %d, 1 GiB)" % ZIP_LIMIT)
|
||||||
|
p.add_argument("packages", nargs="+", metavar="PACKAGE", help="a package folder or a zipped package")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def _check(sub):
|
||||||
|
p = sub.add_parser(
|
||||||
|
"check", help="check the whole repository (in a checkout)",
|
||||||
|
description="""\
|
||||||
|
Check the whole paperspec repository: schemas, records, packages and the zipped package, the
|
||||||
|
broken packages, every vector, a byte-for-byte rebuild, the SQL round trip and the bundle, the
|
||||||
|
counts the prose states, the prose, the links and the website.
|
||||||
|
|
||||||
|
Run it before every commit. Each section prints how many checks it ran; a failure names its
|
||||||
|
section and the file and line, and the run ends with `All checks pass.` or the failures.
|
||||||
|
After you change a rule, a fixture source or a case in the builder, run `paperspec build` and
|
||||||
|
commit the regenerated files, or the rebuild check fails on the out-of-date ones.
|
||||||
|
`pixi run check` also lints the Markdown first, which this command does not.
|
||||||
|
|
||||||
|
It works on a checkout of the repository and needs the tools extra.
|
||||||
|
Exit status: 0 when every check passed, 1 otherwise.""",
|
||||||
|
epilog="""\
|
||||||
|
examples:
|
||||||
|
paperspec check everything
|
||||||
|
paperspec check --quick everything except the rebuild, the slowest part
|
||||||
|
pixi run check lint the Markdown, then everything""")
|
||||||
|
p.add_argument("--quick", action="store_true", help="skip the byte-for-byte rebuild, the slowest check")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def _build(sub):
|
||||||
|
p = sub.add_parser(
|
||||||
|
"build", help="rebuild fixtures/ and vectors/ from their sources (in a checkout)",
|
||||||
|
description="""\
|
||||||
|
Build every fixture and test vector from the hand-written sources in fixtures/src/ and the
|
||||||
|
cases the builder defines, computing each expected value with the oracle.
|
||||||
|
|
||||||
|
Without --out it rewrites fixtures/ and vectors/ in place: run it after you change a rule, a
|
||||||
|
fixture source or a case, review the diff, and commit the regenerated files with the change.
|
||||||
|
The build is deterministic, and `paperspec check` rebuilds into a temporary folder and compares
|
||||||
|
every file byte for byte to prove it.
|
||||||
|
|
||||||
|
It works on a checkout of the repository and needs the tools extra.""",
|
||||||
|
epilog="""\
|
||||||
|
examples:
|
||||||
|
paperspec build rewrite fixtures/ and vectors/ in this checkout
|
||||||
|
paperspec build --out /tmp/rebuilt build elsewhere, then compare:
|
||||||
|
diff -r /tmp/rebuilt/vectors vectors""")
|
||||||
|
p.add_argument("--out", metavar="DIR",
|
||||||
|
help="build into DIR instead of rewriting fixtures/ and vectors/ in the checkout")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def _stress(sub):
|
||||||
|
p = sub.add_parser(
|
||||||
|
"stress", help="stress the oracle with random inputs (in a checkout)",
|
||||||
|
description="""\
|
||||||
|
Stress the reference implementation with random inputs from a seeded generator. The vectors pin
|
||||||
|
values; this checks the invariants the chapters state, such as canonical JSON round trips,
|
||||||
|
corrections keeping the five values equal to a full recomputation, anchors always resolving to
|
||||||
|
one of the five outcomes, the zip reader never writing outside its folder, and a mutated package
|
||||||
|
always being refused.
|
||||||
|
|
||||||
|
A failure prints the seed, so it reproduces exactly: run it again with that --seed.
|
||||||
|
Before a release, run it with the default seed and with one or two others.
|
||||||
|
|
||||||
|
It works on a checkout of the repository and needs the tools extra.
|
||||||
|
Exit status: 0 when every property held, 1 otherwise.""",
|
||||||
|
epilog="""\
|
||||||
|
examples:
|
||||||
|
paperspec stress the default seed, 200 cases per property
|
||||||
|
paperspec stress --seed 7 --rounds 1000 another seed, more cases
|
||||||
|
paperspec stress --rounds 5 a quick smoke test""")
|
||||||
|
p.add_argument("--seed", type=int, default=20261014, help="the random seed (default 20261014)")
|
||||||
|
p.add_argument("--rounds", type=int, default=200, help="cases per property (default 200)")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def _site(sub):
|
||||||
|
p = sub.add_parser(
|
||||||
|
"site", help="build the website into _site/ (in a checkout)",
|
||||||
|
description="""\
|
||||||
|
Build the website. The output mirrors the repository, so each file is served at its own path:
|
||||||
|
a schema at the URL in its `$id`, the JSON-LD context at the address layers name, and every
|
||||||
|
vector, fixture and table as it is. Beside each Markdown file it writes a page, with a vocabulary
|
||||||
|
page, folder listings, a search index and a 404 page; deploy/README.md says how it is served.
|
||||||
|
|
||||||
|
The output folder must not be the repository or contain it. It needs only the standard library
|
||||||
|
and works on a checkout of the repository.""",
|
||||||
|
epilog="""\
|
||||||
|
examples:
|
||||||
|
paperspec site write _site/
|
||||||
|
paperspec site --out /tmp/site --base-url http://localhost:8000
|
||||||
|
python -m http.server --directory /tmp/site 8000 preview it at http://localhost:8000""")
|
||||||
|
p.add_argument("--out", metavar="DIR", help="the output folder (default _site/ in the checkout)")
|
||||||
|
p.add_argument("--base-url", metavar="URL",
|
||||||
|
help="the site's address, for canonical links (default https://paperspec.scient.ing)")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
COMMANDS = {"validate": _validate, "load": _load, "check": _check, "build": _build, "stress": _stress,
|
||||||
|
"site": _site}
|
||||||
|
|
||||||
|
|
||||||
|
def parser():
|
||||||
|
"""The `paperspec` parser and a map from each command to its own parser."""
|
||||||
|
fmt = argparse.RawDescriptionHelpFormatter
|
||||||
|
top = argparse.ArgumentParser(prog="paperspec", description=DESCRIPTION, epilog=EPILOG, formatter_class=fmt,
|
||||||
|
suggest_on_error=True)
|
||||||
|
top.add_argument("--version", action="version",
|
||||||
|
version="paperspec %s (Unicode %s)" % (__version__, UNICODE_VERSION))
|
||||||
|
sub = top.add_subparsers(dest="command", title="commands", metavar="COMMAND")
|
||||||
|
subs = {}
|
||||||
|
for name, add in COMMANDS.items():
|
||||||
|
subs[name] = add(_Sub(sub, fmt))
|
||||||
|
return top, subs
|
||||||
|
|
||||||
|
|
||||||
|
class _Sub:
|
||||||
|
"""add_parser with the formatter and suggestions every command shares."""
|
||||||
|
|
||||||
|
def __init__(self, sub, fmt):
|
||||||
|
self.sub, self.fmt = sub, fmt
|
||||||
|
|
||||||
|
def add_parser(self, name, **kw):
|
||||||
|
return self.sub.add_parser(name, formatter_class=self.fmt, suggest_on_error=True, **kw)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
top, subs = parser()
|
||||||
|
args = top.parse_args(argv)
|
||||||
|
if args.command is None:
|
||||||
|
top.print_help()
|
||||||
|
return 2
|
||||||
|
module = importlib.import_module("paperspec.cli." + args.command)
|
||||||
|
try:
|
||||||
|
return module.run(args) or 0
|
||||||
|
except UsageError as e:
|
||||||
|
subs[args.command].error(str(e))
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
return 130
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -1,8 +1,7 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Build the paperspec website: every file of the repository, and its Markdown rendered as documentation.
|
"""Build the paperspec website: every file of the repository, and its Markdown rendered as documentation.
|
||||||
|
|
||||||
Non-normative tooling, standard library only.
|
Non-normative tooling, standard library only.
|
||||||
Usage: paperspec-site [--out DIR] [--base-url URL]
|
Usage: paperspec site [--out DIR] [--base-url URL]
|
||||||
The output folder (default `_site/`) mirrors the repository, so each file is served at its own path: a schema at
|
The output folder (default `_site/`) mirrors the repository, so each file is served at its own path: a schema at
|
||||||
the URL in its `$id`, the JSON-LD context at the address layers name, and every vector, fixture and table as it is.
|
the URL in its `$id`, the JSON-LD context at the address layers name, and every vector, fixture and table as it is.
|
||||||
Beside each Markdown file the builder writes a page, `README.md` becoming its folder's `index.html` and `X.md`
|
Beside each Markdown file the builder writes a page, `README.md` becoming its folder's `index.html` and `X.md`
|
||||||
@@ -11,15 +10,13 @@ term IRIs point at, a search index, a 404 page and one stylesheet. Links between
|
|||||||
the pages, so the site works from any host, and `deploy/Caddyfile` serves it with the right media types.
|
the pages, so the site works from any host, and `deploy/Caddyfile` serves it with the right media types.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
|
||||||
import html
|
import html
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import sys
|
|
||||||
|
|
||||||
from paperspec.cli import checkout
|
from paperspec.cli import UsageError, checkout
|
||||||
|
|
||||||
ROOT = checkout()
|
ROOT = checkout()
|
||||||
SKIP_DIRS = {".git", ".pixi", ".claude", "_site", "__pycache__", "node_modules", ".gitea", ".vscode", ".idea"}
|
SKIP_DIRS = {".git", ".pixi", ".claude", "_site", "__pycache__", "node_modules", ".gitea", ".vscode", ".idea"}
|
||||||
@@ -38,7 +35,7 @@ def version():
|
|||||||
# --- inline Markdown ------------------------------------------------------------------------------------------
|
# --- inline Markdown ------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
def slug(heading):
|
def slug(heading):
|
||||||
"""A heading's anchor as GitHub, Gitea and Forgejo compute it (the same rule paperspec-check uses)."""
|
"""A heading's anchor as GitHub, Gitea and Forgejo compute it (the same rule paperspec check uses)."""
|
||||||
t = re.sub(r"[`*]", "", heading.strip().lower())
|
t = re.sub(r"[`*]", "", heading.strip().lower())
|
||||||
t = "".join(ch for ch in t if ch.isalnum() or ch in " -_")
|
t = "".join(ch for ch in t if ch.isalnum() or ch in " -_")
|
||||||
return t.replace(" ", "-")
|
return t.replace(" ", "-")
|
||||||
@@ -622,17 +619,10 @@ def build(out, base_url=BASE_URL):
|
|||||||
return len(mds)
|
return len(mds)
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def run(a):
|
||||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
out = os.path.abspath(a.out or os.path.join(ROOT, "_site"))
|
||||||
ap.add_argument("--out", default=os.path.join(ROOT, "_site"), help="the output folder (default _site/)")
|
|
||||||
ap.add_argument("--base-url", default=BASE_URL, help="the site's address, for canonical links (default %s)" % BASE_URL)
|
|
||||||
a = ap.parse_args()
|
|
||||||
out = os.path.abspath(a.out)
|
|
||||||
if out == ROOT or ROOT.startswith(out + os.sep):
|
if out == ROOT or ROOT.startswith(out + os.sep):
|
||||||
ap.error("the output folder must not be the repository or contain it")
|
raise UsageError("the output folder must not be the repository or contain it; the default is _site/")
|
||||||
n = build(out, a.base_url)
|
n = build(out, a.base_url or BASE_URL)
|
||||||
print("Wrote %d pages and every file of the repository to %s" % (n, out))
|
print("Wrote %d pages and every file of the repository to %s" % (n, out))
|
||||||
|
return 0
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,8 +1,7 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Stress the reference implementation with random inputs. Exit status 0 means every property held.
|
"""Stress the reference implementation with random inputs. Exit status 0 means every property held.
|
||||||
|
|
||||||
Non-normative tooling; it needs the packages `jsonschema` and `cryptography`, which pixi and paperspec[tools] install.
|
Non-normative tooling; it needs the packages `jsonschema` and `cryptography`, which pixi and paperspec[tools] install.
|
||||||
Usage: paperspec-stress [--seed N] [--rounds N]
|
Usage: paperspec stress [--seed N] [--rounds N]
|
||||||
The vectors pin values; this program checks the invariants the chapters state, over inputs drawn from a seeded
|
The vectors pin values; this program checks the invariants the chapters state, over inputs drawn from a seeded
|
||||||
random generator, so a failure is reproducible from the seed it prints. Canonical JSON round-trips and sorts keys
|
random generator, so a failure is reproducible from the seed it prints. Canonical JSON round-trips and sorts keys
|
||||||
by UTF-16 units; normalization is idempotent; notation folding never raises; random corrections keep the five
|
by UTF-16 units; normalization is idempotent; notation folding never raises; random corrections keep the five
|
||||||
@@ -14,7 +13,6 @@ input without raising and never write outside their folders; rights combine the
|
|||||||
spot-check chains hundreds deep settle without recursion.
|
spot-check chains hundreds deep settle without recursion.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
|
||||||
import copy
|
import copy
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
@@ -692,11 +690,7 @@ def stress_mldsa(rng, rounds):
|
|||||||
"garbage verifies")
|
"garbage verifies")
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def run(a):
|
||||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
|
||||||
ap.add_argument("--seed", type=int, default=20261014, help="the random seed (default 20261014)")
|
|
||||||
ap.add_argument("--rounds", type=int, default=200, help="cases per property (default 200)")
|
|
||||||
a = ap.parse_args()
|
|
||||||
rng = random.Random(a.seed)
|
rng = random.Random(a.seed)
|
||||||
print("seed %d, %d rounds" % (a.seed, a.rounds))
|
print("seed %d, %d rounds" % (a.seed, a.rounds))
|
||||||
stress_canonical_json(rng, a.rounds)
|
stress_canonical_json(rng, a.rounds)
|
||||||
@@ -726,9 +720,9 @@ def main():
|
|||||||
print("\n%d failures (seed %d):" % (len(FAILS), a.seed))
|
print("\n%d failures (seed %d):" % (len(FAILS), a.seed))
|
||||||
for f in FAILS[:60]:
|
for f in FAILS[:60]:
|
||||||
print(" " + f)
|
print(" " + f)
|
||||||
sys.exit(1)
|
if len(FAILS) > 60:
|
||||||
|
print(" and %d more" % (len(FAILS) - 60))
|
||||||
|
print("\nReproduce them with: paperspec stress --seed %d --rounds %d" % (a.seed, a.rounds))
|
||||||
|
return 1
|
||||||
print("\nEvery property held.")
|
print("\nEvery property held.")
|
||||||
|
return 0
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,9 +1,8 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Validate packages, or run the correction gate on proposals against a package.
|
"""Validate packages, or run the correction gate on proposals against a package.
|
||||||
|
|
||||||
Non-normative tooling.
|
Non-normative tooling.
|
||||||
Usage: paperspec-validate PACKAGE [PACKAGE ...]
|
Usage: paperspec validate PACKAGE [PACKAGE ...]
|
||||||
paperspec-validate --correction PROPOSALS PACKAGE
|
paperspec validate --correction PROPOSALS PACKAGE
|
||||||
A package is a folder, or a zipped package (`.sct.zip`), which is unpacked under the rules of spec/12 §12.2
|
A package is a folder, or a zipped package (`.sct.zip`), which is unpacked under the rules of spec/12 §12.2
|
||||||
before it is checked. Each is checked as spec/22 §22.2 describes, and each failure printed as
|
before it is checked. Each is checked as spec/22 §22.2 describes, and each failure printed as
|
||||||
`package: where: rule: message`.
|
`package: where: rule: message`.
|
||||||
@@ -13,13 +12,12 @@ it would append to corrections.jsonl is printed, until one is refused. Nothing i
|
|||||||
Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.
|
Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
from paperspec.cli import require
|
from paperspec.cli import UsageError, require
|
||||||
|
|
||||||
require("cryptography", "jsonschema")
|
require("cryptography", "jsonschema")
|
||||||
|
|
||||||
@@ -44,18 +42,9 @@ def validate_one(path, store, limit):
|
|||||||
return package.validate(path, store)
|
return package.validate(path, store)
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def run(a):
|
||||||
lines = __doc__.splitlines()
|
|
||||||
ap = argparse.ArgumentParser(description=lines[0], epilog="\n".join(lines[5:]),
|
|
||||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
ap.add_argument("--correction", metavar="PROPOSALS",
|
|
||||||
help="a JSON or JSON Lines file of proposed corrections to run through the gate")
|
|
||||||
ap.add_argument("--max-size", metavar="BYTES", type=int, default=package.ZIP_LIMIT,
|
|
||||||
help="refuse a zipped package that expands past this many bytes (default %d)" % package.ZIP_LIMIT)
|
|
||||||
ap.add_argument("packages", nargs="+", metavar="PACKAGE", help="package folders or zipped packages")
|
|
||||||
a = ap.parse_args()
|
|
||||||
if a.correction and len(a.packages) != 1:
|
if a.correction and len(a.packages) != 1:
|
||||||
ap.error("--correction checks proposals against exactly one package")
|
raise UsageError("--correction checks proposals against exactly one package")
|
||||||
store, bad = Store(resources.path("schemas")), False
|
store, bad = Store(resources.path("schemas")), False
|
||||||
for p in a.packages:
|
for p in a.packages:
|
||||||
for e in validate_one(p, store, a.max_size).errors:
|
for e in validate_one(p, store, a.max_size).errors:
|
||||||
@@ -85,7 +74,3 @@ def main():
|
|||||||
return 1
|
return 1
|
||||||
print(jcs.dumps(record))
|
print(jcs.dumps(record))
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
+17
-16
@@ -7,7 +7,8 @@ Implementers get the commands and the traps.
|
|||||||
These commands are the repository's reference implementation.
|
These commands are the repository's reference implementation.
|
||||||
They apply every rule of the specification, so the rest of the repository can be checked against them.
|
They apply every rule of the specification, so the rest of the repository can be checked against them.
|
||||||
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
|
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
|
||||||
They are console scripts of the Python package `paperspec`, whose code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
|
They are subcommands of one command, `paperspec`, which the Python package `paperspec` installs; its code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
|
||||||
|
`paperspec --help` lists them and says where each works, and `paperspec COMMAND --help` gives a command's options, advice and examples.
|
||||||
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
|
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
|
||||||
|
|
||||||
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic-damb`.
|
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic-damb`.
|
||||||
@@ -16,14 +17,14 @@ A fixture is an example file or package under `fixtures/`.
|
|||||||
|
|
||||||
| Command | Does |
|
| Command | Does |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `paperspec-check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
|
| `paperspec check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
|
||||||
| `paperspec-build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
| `paperspec build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
||||||
| `paperspec-validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec-validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
| `paperspec validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
||||||
| `paperspec-load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec-load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
| `paperspec load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
||||||
| `paperspec-site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
|
| `paperspec site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
|
||||||
| `paperspec-stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec-stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
|
| `paperspec stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
|
||||||
|
|
||||||
`paperspec-validate` and `paperspec-load` work anywhere the package is installed, with the schemas and SQL it carries.
|
`paperspec validate` and `paperspec load` work anywhere the package is installed, with the schemas and SQL it carries.
|
||||||
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
|
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
|
||||||
|
|
||||||
## When to run what
|
## When to run what
|
||||||
@@ -37,21 +38,21 @@ If you do not use pixi, install the checkout editable with its extras on Python
|
|||||||
|
|
||||||
```sh
|
```sh
|
||||||
pip install -e ".[tools]"
|
pip install -e ".[tools]"
|
||||||
paperspec-check # everything
|
paperspec check # everything
|
||||||
paperspec-check --quick # everything except the rebuild
|
paperspec check --quick # everything except the rebuild
|
||||||
paperspec-build --out /tmp/rebuilt
|
paperspec build --out /tmp/rebuilt
|
||||||
paperspec-validate fixtures/packages/hale2026harmonic-damb
|
paperspec validate fixtures/packages/hale2026harmonic-damb
|
||||||
paperspec-load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
||||||
```
|
```
|
||||||
|
|
||||||
Each command is also a module, so `python -m paperspec.cli.check` runs the same program.
|
`python -m paperspec check` runs the same program as `paperspec check`, which helps where the installed command is not on the path.
|
||||||
|
|
||||||
A passing check ends with `All checks pass.`
|
A passing check ends with `All checks pass.`
|
||||||
The load command prints `hale2026harmonic-damb: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
|
The load command prints `hale2026harmonic-damb: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
|
||||||
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
|
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
|
||||||
The full and checks-only forms of one document share its key, so they go into separate databases.
|
The full and checks-only forms of one document share its key, so they go into separate databases.
|
||||||
Each command prints its options with `--help` and refuses options it does not know.
|
Each command prints its options with `--help` and refuses options it does not know.
|
||||||
`paperspec-build` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
`paperspec build` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
||||||
|
|
||||||
Run `pixi run check` before every commit.
|
Run `pixi run check` before every commit.
|
||||||
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `src/paperspec/cli/build.py`, and commit the regenerated files with the change.
|
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `src/paperspec/cli/build.py`, and commit the regenerated files with the change.
|
||||||
@@ -71,7 +72,7 @@ Canonical JSON goes through Python's built-in JSON encoder, and falls back to an
|
|||||||
|
|
||||||
Hashes are kept incrementally.
|
Hashes are kept incrementally.
|
||||||
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
|
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
|
||||||
`paperspec-check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
||||||
|
|
||||||
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec` (`pip install .`, or a git pin) and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
|
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec` (`pip install .`, or a git pin) and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
|
||||||
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
|
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
|
||||||
|
|||||||
+1
-1
@@ -81,7 +81,7 @@ Several files also mix more than one kind of case, told apart by their fields.
|
|||||||
The reference checker in [tools/](../tools/README.md) runs all twenty-five files, together with the rest of the repository's checks:
|
The reference checker in [tools/](../tools/README.md) runs all twenty-five files, together with the rest of the repository's checks:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
paperspec-check --quick
|
paperspec check --quick
|
||||||
```
|
```
|
||||||
|
|
||||||
It needs Python 3.14 and two packages, `jsonschema` and `cryptography`, and has no option to run one file.
|
It needs Python 3.14 and two packages, `jsonschema` and `cryptography`, and has no option to run one file.
|
||||||
|
|||||||
Reference in new issue
Block a user