From a4eecb4d5bded37420a0092471e30486401871ac Mon Sep 17 00:00:00 2001 From: Alex Maldonado Date: Thu, 15 Oct 2026 09:00:40 -0400 Subject: [PATCH] refactor: move the reference implementation to src/paperspec The oracle had two import names, `tools.oracle` for the tools and `paperspec` for other repositories, which could load the same code twice in one process and needed a gate keyed on the module name. It now lives in the standard src layout under its one name. - git mv tools/oracle -> src/paperspec; the 22 rule modules are unchanged. - The tools import `paperspec`. `tools/__init__.py` now holds the Python 3.14 and cryptography/jsonschema check and puts src/ first on sys.path, so the tools run this checkout's code with or without pixi. The package's own __init__ checks nothing and exposes __version__. - pyproject.toml: version from git tags via setuptools-scm (guess-next-dev, node-and-timestamp, src/paperspec/_version.py, gitignored). The packages stay listed rather than found because schemas/, sql/ and vectors/ stay at the root: the spec links to them and the site serves the schemas at their $id URLs. - pixi.toml: a `dev` feature and environment with setuptools, setuptools-scm and python-build, and `pixi run -e dev wheel`. - check.py scans src/paperspec for rule IDs; README, CONTRIBUTING, the FAQ, the overview, the vectors and tools READMEs and the changelog point at the new path. The FAQ no longer says the oracle is not a library. pixi.lock is not re-solved in this commit: with a dynamic version, pixi builds the package metadata to lock, which needs the conda environment installed. Run `pixi lock` and commit the result before pushing; the site workflow installs with --locked. pixi run check passes; no computed value, vector, fixture, schema or SQL file changed. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 3 ++ CHANGELOG.md | 7 +++-- CONTRIBUTING.md | 4 +-- README.md | 1 + docs/faq.md | 4 +-- docs/overview.md | 3 +- pixi.toml | 17 +++++++++-- pyproject.toml | 20 +++++++++---- src/paperspec/__init__.py | 14 +++++++++ {tools/oracle => src/paperspec}/align.py | 0 {tools/oracle => src/paperspec}/anchors.py | 0 .../oracle => src/paperspec}/corrections.py | 0 {tools/oracle => src/paperspec}/crypto.py | 0 {tools/oracle => src/paperspec}/db.py | 0 .../oracle => src/paperspec}/fingerprint.py | 0 {tools/oracle => src/paperspec}/grade.py | 0 {tools/oracle => src/paperspec}/hashing.py | 0 {tools/oracle => src/paperspec}/jcs.py | 0 {tools/oracle => src/paperspec}/keys.py | 0 {tools/oracle => src/paperspec}/log.py | 0 {tools/oracle => src/paperspec}/mldsa.py | 0 {tools/oracle => src/paperspec}/model.py | 0 {tools/oracle => src/paperspec}/notation.py | 0 {tools/oracle => src/paperspec}/package.py | 0 {tools/oracle => src/paperspec}/receipts.py | 0 {tools/oracle => src/paperspec}/rights.py | 0 {tools/oracle => src/paperspec}/schema.py | 0 {tools/oracle => src/paperspec}/targets.py | 0 {tools/oracle => src/paperspec}/text.py | 0 {tools/oracle => src/paperspec}/tree.py | 0 {tools/oracle => src/paperspec}/vrf.py | 0 tools/README.md | 9 +++--- tools/__init__.py | 24 +++++++++++++++ tools/build.py | 25 ++++++++-------- tools/check.py | 29 ++++++++++--------- tools/load.py | 5 ++-- tools/oracle/__init__.py | 21 -------------- tools/stress.py | 21 +++++++------- tools/validate.py | 7 +++-- vectors/README.md | 4 +-- 40 files changed, 134 insertions(+), 84 deletions(-) create mode 100644 src/paperspec/__init__.py rename {tools/oracle => src/paperspec}/align.py (100%) rename {tools/oracle => src/paperspec}/anchors.py (100%) rename {tools/oracle => src/paperspec}/corrections.py (100%) rename {tools/oracle => src/paperspec}/crypto.py (100%) rename {tools/oracle => src/paperspec}/db.py (100%) rename {tools/oracle => src/paperspec}/fingerprint.py (100%) rename {tools/oracle => src/paperspec}/grade.py (100%) rename {tools/oracle => src/paperspec}/hashing.py (100%) rename {tools/oracle => src/paperspec}/jcs.py (100%) rename {tools/oracle => src/paperspec}/keys.py (100%) rename {tools/oracle => src/paperspec}/log.py (100%) rename {tools/oracle => src/paperspec}/mldsa.py (100%) rename {tools/oracle => src/paperspec}/model.py (100%) rename {tools/oracle => src/paperspec}/notation.py (100%) rename {tools/oracle => src/paperspec}/package.py (100%) rename {tools/oracle => src/paperspec}/receipts.py (100%) rename {tools/oracle => src/paperspec}/rights.py (100%) rename {tools/oracle => src/paperspec}/schema.py (100%) rename {tools/oracle => src/paperspec}/targets.py (100%) rename {tools/oracle => src/paperspec}/text.py (100%) rename {tools/oracle => src/paperspec}/tree.py (100%) rename {tools/oracle => src/paperspec}/vrf.py (100%) delete mode 100644 tools/oracle/__init__.py diff --git a/.gitignore b/.gitignore index 856c31e..f4515de 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,9 @@ __pycache__/ *.db *.db.assets/ +# the package version, written by setuptools-scm when the package is built or installed +/src/paperspec/_version.py + # the website, built by tools/site.py /_site/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 949ced4..7296e9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,8 @@ The first release. Packaging and lazy imports only; the formats stay at 0.1.0, and no computed value, vector, fixture, schema or SQL file changed. -- `tools/oracle` installs as the Python package `paperspec` (`pyproject.toml`, setuptools), with `schemas/`, `sql/` and `vectors/` as the data subpackages `paperspec.schemas`, `paperspec.sql` and `paperspec.vectors`, so other repositories import the reference implementation instead of reimplementing it. +- The oracle moves from `tools/oracle/` to `src/paperspec/` and installs as the Python package `paperspec` (`pyproject.toml`, setuptools), with `schemas/`, `sql/` and `vectors/` as the data subpackages `paperspec.schemas`, `paperspec.sql` and `paperspec.vectors`, so other repositories import the reference implementation instead of reimplementing it. +- The package version comes from git tags through setuptools-scm. - The base package has no dependencies and runs on Python 3.12; the extras `paperspec[schema]` (`jsonschema`, `referencing`), `paperspec[crypto]` (`cryptography`) and `paperspec[tools]` (both) cover `schema` and the HKDF and Ed25519 functions of `crypto`, which now import their packages on first use. -- The Python 3.14 and package gate in `tools/oracle/__init__.py` runs only when the repository's own tools import it as `tools.oracle`. -- The pixi workspace installs the package editable, so the tools and the package resolve to the same code. +- The tools import `paperspec` from `src/`, and their Python 3.14 and package check moves to `tools/__init__.py`, so the package itself checks nothing on import. +- The pixi workspace installs the package editable, and a `dev` environment builds the wheel with `pixi run -e dev wheel`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d3d2acb..961c44e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,7 +13,7 @@ The schemas, SQL, registries, vectors and fixtures then follow the chapter ([spe 2. **Change the schemas, SQL and registries** that the rule shapes. A new refusal gets a rule ID in [registry/rules.json](registry/rules.json). 3. **Compute, never type, the expected values.** - Change the oracle in [tools/oracle/](tools/oracle/) to apply the rule, and add the case that shows it to [tools/build.py](tools/build.py): a vector, a fixture or a broken package in `fixtures/invalid/`. + Change the oracle in [src/paperspec/](src/paperspec/) to apply the rule, and add the case that shows it to [tools/build.py](tools/build.py): a vector, a fixture or a broken package in `fixtures/invalid/`. Then run `pixi run build`, which rewrites `fixtures/` and `vectors/`. 4. **Run the checks.** `pixi run check` must end with `All checks pass.` @@ -29,7 +29,7 @@ A rebuild that changes any file under `vectors/` or any hash in `fixtures/` is a ## New node kinds and run kinds Propose a kind with the vector or fixture that shows it. -A node kind needs its row in [chapter 3](spec/03-nodes.md), the node schema, and its entry in the `CHILDREN` table of `tools/oracle/model.py`. +A node kind needs its row in [chapter 3](spec/03-nodes.md), the node schema, and its entry in the `CHILDREN` table of `src/paperspec/model.py`. A run kind needs its row in [section 3.5](spec/03-nodes.md#35-inline-runs) and the node schema, and its text in the plain projection ([3.6](spec/03-nodes.md#36-the-plain-projection)) and in the strict and lossy projections ([5.3](spec/05-hashes.md#53-the-strict-projection), [5.5](spec/05-hashes.md#55-the-lossy-projection)). ## Releasing diff --git a/README.md b/README.md index 9e6d5b3..08da43c 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,7 @@ The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and | [vectors/](vectors/README.md) | Test vectors: inputs and the exact outputs every implementation must compute. | | [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. | | [tools/](tools/README.md) | The programs that build and check everything above. | +| [src/paperspec/](src/paperspec/) | The reference implementation of every rule, installable as the Python package `paperspec`. | | [docs/](docs/README.md) | Guides for people getting to know the project. | | [deploy/](deploy/README.md) | How the website at is built and served. | diff --git a/docs/faq.md b/docs/faq.md index 05331bc..545fa01 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -127,5 +127,5 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md). ## What are the files in `tools/` for? `build.py` builds the fixtures and vectors. `check.py` recomputes and checks every one, and `load.py` loads packages into SQLite. -The `oracle/` modules compute every expected value, so that none is typed by hand. -They are not a library; implementations follow the specification ([tools/README.md](../tools/README.md)). +The oracle's modules in `src/paperspec/` compute every expected value, so that none is typed by hand. +The other repositories of the project install them as the Python package `paperspec`; independent implementations follow the specification ([tools/README.md](../tools/README.md)). diff --git a/docs/overview.md b/docs/overview.md index d06c2f3..c521bbc 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -54,7 +54,8 @@ It appears inside hashed data and test vectors, and it is separate from the repo | [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. | | [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. | | [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. | -| [tools/](../tools/README.md) | Reference programs: `check.py` checks everything, `build.py` rebuilds the fixtures and vectors, `load.py` loads packages into SQLite, and `oracle/` is a compact implementation of every rule. | +| [tools/](../tools/README.md) | Reference programs: `check.py` checks everything, `build.py` rebuilds the fixtures and vectors, and `load.py` loads packages into SQLite. | +| [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. | | [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. | | [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. | | [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. | diff --git a/pixi.toml b/pixi.toml index 74f869b..e0a3bb5 100644 --- a/pixi.toml +++ b/pixi.toml @@ -20,11 +20,24 @@ markdownlint-cli2 = ">=0.23.3" nodejs = ">=22" cryptography = ">=46" -# The tools import the oracle as `tools.oracle`; other programs install it as `paperspec`. The editable -# self-install makes both resolve to the same code. +# The reference implementation in src/paperspec, installed editable so that `import paperspec` works in +# every pixi environment; the tools put src/ first on the path themselves, so they also run without pixi. [pypi-dependencies] paperspec = { path = ".", editable = true } +# What building the package needs: pixi run -e dev wheel +[feature.dev.dependencies] +setuptools = ">=80" +setuptools-scm = ">=8" +python-build = ">=1.2" + +[feature.dev.tasks] +wheel = { cmd = "python -m build --wheel --no-isolation --outdir dist", description = "Build the paperspec wheel into dist/" } + +[environments] +default = { solve-group = "default" } +dev = { features = ["dev"], solve-group = "default" } + [tasks] check = { cmd = "python tools/check.py", depends-on = [ "lint", diff --git a/pyproject.toml b/pyproject.toml index ac5818c..2d9bc38 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,11 +1,10 @@ [build-system] +requires = ["setuptools>=80", "setuptools-scm>=8"] build-backend = "setuptools.build_meta" -requires = ["setuptools>=77"] - [project] name = "paperspec" -version = "0.1.0" +dynamic = ["version"] description = "The scienting formats: the reference implementation of every rule, with the JSON Schemas, SQL and test vectors as package data" readme = "README.md" license = "Apache-2.0" @@ -32,13 +31,15 @@ Homepage = "https://paperspec.scient.ing" Repository = "https://git.scient.ing/infra/paperspec" Documentation = "https://paperspec.scient.ing/docs/" -# `tools/oracle` is the package `paperspec`; the schemas, SQL and vectors travel -# with it as data subpackages, so a consumer reads them with importlib.resources. +# The code is `src/paperspec`. The schemas, SQL and vectors stay at the repository root, where the +# specification links to them and the website serves the schemas at their `$id` URLs, and travel with the +# package as data subpackages, so a consumer reads them with importlib.resources. That mapping is why the +# packages are listed rather than found. [tool.setuptools] packages = ["paperspec", "paperspec.schemas", "paperspec.sql", "paperspec.vectors"] [tool.setuptools.package-dir] -"paperspec" = "tools/oracle" +"" = "src" "paperspec.schemas" = "schemas" "paperspec.sql" = "sql" "paperspec.vectors" = "vectors" @@ -47,3 +48,10 @@ packages = ["paperspec", "paperspec.schemas", "paperspec.sql", "paperspec.vector "paperspec.schemas" = ["*.json"] "paperspec.sql" = ["*.sql"] "paperspec.vectors" = ["*.jsonl"] + +# The package version comes from git tags (v0.1.0 builds as 0.1.0, later commits as 0.1.1.devN). It is the +# version of this code, not of the formats, which CHANGELOG.md and pixi.toml carry. +[tool.setuptools_scm] +version_file = "src/paperspec/_version.py" +version_scheme = "guess-next-dev" +local_scheme = "node-and-timestamp" diff --git a/src/paperspec/__init__.py b/src/paperspec/__init__.py new file mode 100644 index 0000000..7868569 --- /dev/null +++ b/src/paperspec/__init__.py @@ -0,0 +1,14 @@ +"""The reference implementation of the scienting formats: the computations behind every expected value +in this repository; see tools/README.md. + +The modules other programs use (`keys`, `text`, `jcs`, `package`, `db`, `rights`, ...) need only the +standard library and Python 3.12. +`schema` needs `paperspec[schema]` and the HKDF and Ed25519 functions of `crypto` need `paperspec[crypto]`; +both import their packages on first use. +The schemas, SQL and vectors travel as package data: `importlib.resources.files("paperspec") / "sql"`. +""" + +try: + from ._version import __version__ +except ImportError: # a checkout that was never built or installed, as the repository's tools use it + __version__ = "0+unknown" diff --git a/tools/oracle/align.py b/src/paperspec/align.py similarity index 100% rename from tools/oracle/align.py rename to src/paperspec/align.py diff --git a/tools/oracle/anchors.py b/src/paperspec/anchors.py similarity index 100% rename from tools/oracle/anchors.py rename to src/paperspec/anchors.py diff --git a/tools/oracle/corrections.py b/src/paperspec/corrections.py similarity index 100% rename from tools/oracle/corrections.py rename to src/paperspec/corrections.py diff --git a/tools/oracle/crypto.py b/src/paperspec/crypto.py similarity index 100% rename from tools/oracle/crypto.py rename to src/paperspec/crypto.py diff --git a/tools/oracle/db.py b/src/paperspec/db.py similarity index 100% rename from tools/oracle/db.py rename to src/paperspec/db.py diff --git a/tools/oracle/fingerprint.py b/src/paperspec/fingerprint.py similarity index 100% rename from tools/oracle/fingerprint.py rename to src/paperspec/fingerprint.py diff --git a/tools/oracle/grade.py b/src/paperspec/grade.py similarity index 100% rename from tools/oracle/grade.py rename to src/paperspec/grade.py diff --git a/tools/oracle/hashing.py b/src/paperspec/hashing.py similarity index 100% rename from tools/oracle/hashing.py rename to src/paperspec/hashing.py diff --git a/tools/oracle/jcs.py b/src/paperspec/jcs.py similarity index 100% rename from tools/oracle/jcs.py rename to src/paperspec/jcs.py diff --git a/tools/oracle/keys.py b/src/paperspec/keys.py similarity index 100% rename from tools/oracle/keys.py rename to src/paperspec/keys.py diff --git a/tools/oracle/log.py b/src/paperspec/log.py similarity index 100% rename from tools/oracle/log.py rename to src/paperspec/log.py diff --git a/tools/oracle/mldsa.py b/src/paperspec/mldsa.py similarity index 100% rename from tools/oracle/mldsa.py rename to src/paperspec/mldsa.py diff --git a/tools/oracle/model.py b/src/paperspec/model.py similarity index 100% rename from tools/oracle/model.py rename to src/paperspec/model.py diff --git a/tools/oracle/notation.py b/src/paperspec/notation.py similarity index 100% rename from tools/oracle/notation.py rename to src/paperspec/notation.py diff --git a/tools/oracle/package.py b/src/paperspec/package.py similarity index 100% rename from tools/oracle/package.py rename to src/paperspec/package.py diff --git a/tools/oracle/receipts.py b/src/paperspec/receipts.py similarity index 100% rename from tools/oracle/receipts.py rename to src/paperspec/receipts.py diff --git a/tools/oracle/rights.py b/src/paperspec/rights.py similarity index 100% rename from tools/oracle/rights.py rename to src/paperspec/rights.py diff --git a/tools/oracle/schema.py b/src/paperspec/schema.py similarity index 100% rename from tools/oracle/schema.py rename to src/paperspec/schema.py diff --git a/tools/oracle/targets.py b/src/paperspec/targets.py similarity index 100% rename from tools/oracle/targets.py rename to src/paperspec/targets.py diff --git a/tools/oracle/text.py b/src/paperspec/text.py similarity index 100% rename from tools/oracle/text.py rename to src/paperspec/text.py diff --git a/tools/oracle/tree.py b/src/paperspec/tree.py similarity index 100% rename from tools/oracle/tree.py rename to src/paperspec/tree.py diff --git a/tools/oracle/vrf.py b/src/paperspec/vrf.py similarity index 100% rename from tools/oracle/vrf.py rename to src/paperspec/vrf.py diff --git a/tools/README.md b/tools/README.md index 9ad9832..fcfdfe5 100644 --- a/tools/README.md +++ b/tools/README.md @@ -21,7 +21,7 @@ A fixture is an example file or package under `fixtures/`. | `load.py` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). | | `site.py` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). | | `stress.py` | Stresses the oracle with random inputs from a seeded generator: `python3 tools/stress.py [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. | -| `oracle/` | A compact implementation of every rule. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. | +| [src/paperspec/](../src/paperspec/) | The oracle: a compact implementation of every rule, and the Python package `paperspec`. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. The programs put `src/` first on the path, so they run this checkout's code with or without pixi. | ## When to run what @@ -48,7 +48,7 @@ Each program prints its options with `--help` and refuses options it does not kn `build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`. Run `pixi run check` before every commit. -Run `pixi run build` after you change a fixture source, a rule in `oracle/` or a case in `build.py`, and commit the regenerated files with the change. +Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `build.py`, and commit the regenerated files with the change. The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files. ## About the oracle @@ -59,7 +59,7 @@ The oracle exists so that no expected value in this repository is typed by hand. It favors clarity, and leaves the cryptography and schema validation to established packages. Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`. -The VRF of RFC 9381 (`oracle/vrf.py`) and ML-DSA-65 of FIPS 204 (`oracle/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors. +The VRF of RFC 9381 (`src/paperspec/vrf.py`) and ML-DSA-65 of FIPS 204 (`src/paperspec/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors. The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to. Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names. @@ -67,8 +67,9 @@ Hashes are kept incrementally. A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined. `check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree. -Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec` (`pip install .`, or a git pin) and import the oracle, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code. +Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec` (`pip install .`, or a git pin) and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code. The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data (`importlib.resources.files("paperspec") / "sql" / "doc.sql"`). Those modules run on Python 3.12 with the standard library alone, except that `schema` needs `paperspec[schema]` (`jsonschema` and `referencing`) and the HKDF and Ed25519 functions of `crypto` need `paperspec[crypto]`; `paperspec[tools]` installs both. +The package version comes from git tags through setuptools-scm, and `pixi run -e dev wheel` builds the wheel into `dist/`. When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing. Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md). diff --git a/tools/__init__.py b/tools/__init__.py index e69de29..5a63fc9 100644 --- a/tools/__init__.py +++ b/tools/__init__.py @@ -0,0 +1,24 @@ +"""The repository's own programs; see tools/README.md. + +Importing this package checks that the tools can run here, and puts `src/` first on the path, so that +`import paperspec` gets this checkout's reference implementation rather than an installed copy. +The tools need Python 3.14 and the packages `cryptography` and `jsonschema`; the `paperspec` package +itself does not, and other programs install it on its own. +""" + +import importlib.util +import os +import sys + +if sys.version_info < (3, 14): + sys.exit("The paperspec tools need Python 3.14 or later; this is Python %d.%d." % sys.version_info[:2]) + +_missing = [m for m in ("cryptography", "jsonschema") if importlib.util.find_spec(m) is None] +if _missing: + sys.exit("The paperspec tools need %s %s; run them through pixi, or install %s with pip." + % ("the package" if len(_missing) == 1 else "the packages", " and ".join(_missing), + "it" if len(_missing) == 1 else "them")) + +SRC = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "src") +if SRC not in sys.path: + sys.path.insert(0, SRC) diff --git a/tools/build.py b/tools/build.py index 93e33ef..55e77bd 100644 --- a/tools/build.py +++ b/tools/build.py @@ -20,18 +20,19 @@ from fractions import Fraction ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, ROOT) +import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported -from tools.oracle import align as AL # noqa: E402 -from tools.oracle import anchors as AN # noqa: E402 -from tools.oracle import corrections as CO # noqa: E402 -from tools.oracle import crypto, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402 -from tools.oracle import keys as K, log as LG, mldsa, targets as TG, vrf # noqa: E402 -from tools.oracle.model import LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc # noqa: E402 -from tools.oracle.notation import fold_math # noqa: E402 -from tools.oracle.package import ZIP_EXTENSION, history_items, jsonl, line, witness_id, write, write_zip # noqa: E402 -from tools.oracle.schema import Store # noqa: E402 -from tools.oracle.text import letter_first, mint_node_id, norm, record_id, sha256_hex # noqa: E402 -from tools.oracle.tree import Refusal, _flatten, mint_ids, validate_structure # noqa: E402 +from paperspec import align as AL # noqa: E402 +from paperspec import anchors as AN # noqa: E402 +from paperspec import corrections as CO # noqa: E402 +from paperspec import crypto, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402 +from paperspec import keys as K, log as LG, mldsa, targets as TG, vrf # noqa: E402 +from paperspec.model import LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc # noqa: E402 +from paperspec.notation import fold_math # noqa: E402 +from paperspec.package import ZIP_EXTENSION, history_items, jsonl, line, witness_id, write, write_zip # noqa: E402 +from paperspec.schema import Store # noqa: E402 +from paperspec.text import letter_first, mint_node_id, norm, record_id, sha256_hex # noqa: E402 +from paperspec.tree import Refusal, _flatten, mint_ids, validate_structure # noqa: E402 VERSION = "0.1.0" STORE = "smithlab" @@ -903,7 +904,7 @@ def vectors(out, hale, tiny): ("astral", [T("Let "), M("\U0001d465"), T(" vary with "), M("\U0001d466"), T(".")], [4, 7])]: doc, s = sentence_doc(runs) p = hashing.plain(doc, s) - from tools.oracle.text import cp_to_utf16 + from paperspec.text import cp_to_utf16 off.append({"id": cid, "runs": runs, "plain": p, "span": span, "exact": p[span[0]:span[1]], "utf16_span": [cp_to_utf16(p, span[0]), cp_to_utf16(p, span[1])], "code_points": len(p), "utf16_units": cp_to_utf16(p, len(p))}) diff --git a/tools/check.py b/tools/check.py index d43a72e..0f97573 100644 --- a/tools/check.py +++ b/tools/check.py @@ -20,21 +20,22 @@ from fractions import Fraction ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, ROOT) +import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported -from tools.oracle import align as AL # noqa: E402 -from tools.oracle import anchors as AN # noqa: E402 -from tools.oracle import corrections as CO # noqa: E402 -from tools.oracle import crypto, db, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402 -from tools.oracle import package as PK # noqa: E402 -from tools.oracle import keys as K, log as LG, mldsa, targets as TG, vrf # noqa: E402 +from paperspec import align as AL # noqa: E402 +from paperspec import anchors as AN # noqa: E402 +from paperspec import corrections as CO # noqa: E402 +from paperspec import crypto, db, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402 +from paperspec import package as PK # noqa: E402 +from paperspec import keys as K, log as LG, mldsa, targets as TG, vrf # noqa: E402 import unicodedata # noqa: E402 -from tools.oracle.model import Doc # noqa: E402 -from tools.oracle.notation import fold_math # noqa: E402 -from tools.oracle.schema import Store, validate as sv # noqa: E402 +from paperspec.model import Doc # noqa: E402 +from paperspec.notation import fold_math # noqa: E402 +from paperspec.schema import Store, validate as sv # noqa: E402 from jsonschema import Draft202012Validator # noqa: E402 from jsonschema.exceptions import SchemaError # noqa: E402 -from tools.oracle.text import cp_to_utf16, letter_first, mint_node_id, norm, record_id, sha256_hex # noqa: E402 -from tools.oracle.tree import Refusal # noqa: E402 +from paperspec.text import cp_to_utf16, letter_first, mint_node_id, norm, record_id, sha256_hex # noqa: E402 +from paperspec.tree import Refusal # noqa: E402 FAILS = [] # Folders that are not part of the repository: version control, environments and caches. @@ -111,11 +112,11 @@ def check_registry(): for rule in js(p("registry", "stamps", "scienting-stamps-1.json"))["rules"]: re.compile(rule["pattern"]) rules = {r["id"] for r in js(p("registry", "rules.json"))["rules"]} - for f in sorted(os.listdir(p("tools", "oracle"))): + for f in sorted(os.listdir(p("src", "paperspec"))): if f.endswith(".py"): - for rule in re.findall(r'(?:Refusal|rep\.add)\(\s*"([a-z][a-z0-9.-]+)"', open(p("tools", "oracle", f)).read()): + for rule in re.findall(r'(?:Refusal|rep\.add)\(\s*"([a-z][a-z0-9.-]+)"', open(p("src", "paperspec", f)).read()): if not rule.endswith("."): - ok("registry", rule in rules, "tools/oracle/%s raises %s, which is not in rules.json" % (f, rule)) + ok("registry", rule in rules, "src/paperspec/%s raises %s, which is not in rules.json" % (f, rule)) for v in ("text-hash", "match-hash", "align-key", "own-hash", "node-hash"): ok("registry", "doc.hash." + v in rules, "doc.hash.%s is not in rules.json" % v) # The character data the hashes depend on is pinned to one Unicode version (spec/01 §1.4). diff --git a/tools/load.py b/tools/load.py index 3e67f58..24d318f 100644 --- a/tools/load.py +++ b/tools/load.py @@ -20,9 +20,10 @@ import tempfile ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, ROOT) +import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported -from tools.oracle import db, package # noqa: E402 -from tools.oracle.schema import Store # noqa: E402 +from paperspec import db, package # noqa: E402 +from paperspec.schema import Store # noqa: E402 def main(): diff --git a/tools/oracle/__init__.py b/tools/oracle/__init__.py deleted file mode 100644 index 6865ba3..0000000 --- a/tools/oracle/__init__.py +++ /dev/null @@ -1,21 +0,0 @@ -"""The reference computations behind every expected value in this repository; see tools/README.md. - -Imported as `tools.oracle` by the repository's own tools, which need Python 3.14, `cryptography` and -`jsonschema` for everything they check; the gate below exits with a message when one is missing. -Installed and imported as `paperspec` by other programs, the gate is skipped: the standard-library modules -(`keys`, `text`, `jcs`, `package`, `db`, `rights`, ...) run on Python 3.12, and `schema` and `crypto` import -their packages on first use, so `paperspec[schema]` and `paperspec[crypto]` are needed only for those two. -""" - -import importlib.util -import sys - -if __name__ == "tools.oracle": - if sys.version_info < (3, 14): - sys.exit("The paperspec tools need Python 3.14 or later; this is Python %d.%d." % sys.version_info[:2]) - - _missing = [m for m in ("cryptography", "jsonschema") if importlib.util.find_spec(m) is None] - if _missing: - sys.exit("The paperspec tools need %s %s; run them through pixi, or install %s with pip." - % ("the package" if len(_missing) == 1 else "the packages", " and ".join(_missing), - "it" if len(_missing) == 1 else "them")) diff --git a/tools/stress.py b/tools/stress.py index e46c40d..172f236 100644 --- a/tools/stress.py +++ b/tools/stress.py @@ -29,17 +29,18 @@ from math import comb ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, ROOT) +import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported -from tools.oracle import align as AL # noqa: E402 -from tools.oracle import anchors as AN # noqa: E402 -from tools.oracle import corrections as CO # noqa: E402 -from tools.oracle import crypto, fingerprint as FP, hashing, jcs, receipts as RC, rights as RI, targets as TG # noqa: E402 -from tools.oracle import keys as K, log as LG, mldsa, vrf # noqa: E402 -from tools.oracle import package as PK # noqa: E402 -from tools.oracle.model import LABELLED, LEAVES, Doc # noqa: E402 -from tools.oracle.notation import fold_math # noqa: E402 -from tools.oracle.text import NODE_ID_RE, RECORD_ID_RE, letter_first, norm, record_id, sha256_hex, is_ws # noqa: E402 -from tools.oracle.tree import Refusal, validate_structure # noqa: E402 +from paperspec import align as AL # noqa: E402 +from paperspec import anchors as AN # noqa: E402 +from paperspec import corrections as CO # noqa: E402 +from paperspec import crypto, fingerprint as FP, hashing, jcs, receipts as RC, rights as RI, targets as TG # noqa: E402 +from paperspec import keys as K, log as LG, mldsa, vrf # noqa: E402 +from paperspec import package as PK # noqa: E402 +from paperspec.model import LABELLED, LEAVES, Doc # noqa: E402 +from paperspec.notation import fold_math # noqa: E402 +from paperspec.text import NODE_ID_RE, RECORD_ID_RE, letter_first, norm, record_id, sha256_hex, is_ws # noqa: E402 +from paperspec.tree import Refusal, validate_structure # noqa: E402 from tools.build import ROSTER_BY_ID, SCHEMAS, secret, sentence_doc, tiny_package # noqa: E402 from tools.check import HALE, STORE, TINY_KEY, reference_values # noqa: E402 diff --git a/tools/validate.py b/tools/validate.py index abafff3..87d93c0 100755 --- a/tools/validate.py +++ b/tools/validate.py @@ -21,10 +21,11 @@ import tempfile ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, ROOT) +import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported -from tools.oracle import corrections, jcs, package # noqa: E402 -from tools.oracle.schema import Store # noqa: E402 -from tools.oracle.tree import Refusal # noqa: E402 +from paperspec import corrections, jcs, package # noqa: E402 +from paperspec.schema import Store # noqa: E402 +from paperspec.tree import Refusal # noqa: E402 def read_proposals(path): diff --git a/vectors/README.md b/vectors/README.md index 18c288a..d9e5aca 100644 --- a/vectors/README.md +++ b/vectors/README.md @@ -85,7 +85,7 @@ python3 tools/check.py --quick ``` It needs Python 3.14 and two packages, `jsonschema` and `cryptography`, and has no option to run one file. -It tests the reference implementation in `tools/oracle/` against the vectors. +It tests the reference implementation in `src/paperspec/` against the vectors. It does not test yours. ## What each file tests @@ -136,7 +136,7 @@ The specification, and the vectors with it, are at version 0.1.0 ([0.3](../spec/ The specification has 32 formats: 19 are `candidate` and 13 are `provisional` in [registry/formats.json](../registry/formats.json). A candidate format changes only with a recorded reason and a new or changed vector ([22.5](../spec/22-conformance.md#225-stability)). -A passing run shows one thing: for these inputs, your implementation computes the same values as the reference implementation in `tools/oracle/`. +A passing run shows one thing: for these inputs, your implementation computes the same values as the reference implementation in `src/paperspec/`. That code computed the expected outputs, and `pixi run build` regenerates them ([tools/](../tools/README.md)). The vectors hold 311 cases in all, some of them small: `hashes.jsonl` has one document, `alignment.jsonl` one pair of readings, `offsets.jsonl` two cases.