feat!: pin Python 3.14 and Unicode 16.0, and make the tools commands
Python and Unicode
- requires-python is >=3.14,<3.15: 3.14 is the only Python with the
Unicode 16.0 data the specification pins (3.13 has 15.1, 3.15 has 17.0).
- `import paperspec` raises ImportError unless unicodedata reports
paperspec.UNICODE_VERSION (16.0.0), so a source checkout or an install
that bypasses requires-python cannot compute values on other data.
Commands
- tools/{check,build,validate,load,stress,site}.py move with history to
src/paperspec/cli/ and install as paperspec-check, -build, -validate,
-load, -stress and -site; each also runs as `python -m paperspec.cli.X`.
- paperspec.cli.require() exits with "install paperspec[tools]" when
cryptography or jsonschema is missing; paperspec.cli.checkout() finds
the repository the check, build, stress and site commands work on (the
editable install's own checkout, else the one around the current
folder) and exits with a message outside one. tools/__init__.py is gone.
- validate and load read the schemas and SQL through the new
paperspec.resources.path(), which finds them in a wheel and in an
editable install alike, so they work anywhere the package is installed.
- check runs the rebuild as `python -m paperspec.cli.build`.
- build still writes the provenance tool label "paperspec tools/build.py":
it is part of every hashed and signed fixture.
- pixi tasks and the check workflow run the commands; tools/README.md
documents them, and the other docs name the commands instead of paths.
License
- LICENSE.md gives `src/` Apache-2.0 alongside `tools/`. Since the
previous commit moved the code to src/, no row had covered it.
pixi run check passes (run as `python -m paperspec.cli.check` with src/ on
the path, since this environment cannot install the entry points); the
byte-for-byte rebuild matches, so no fixture or vector changed.
pixi.lock still needs `pixi lock`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
33 files changed
+227
-149
No files matched your search
+32
-25
@@ -4,51 +4,57 @@ This page is for anyone who wants to run the repository's checks, rebuild its ex
|
||||
Newcomers get each term defined where it first appears.
|
||||
Implementers get the commands and the traps.
|
||||
|
||||
These programs are the repository's reference implementation.
|
||||
These commands are the repository's reference implementation.
|
||||
They apply every rule of the specification, so the rest of the repository can be checked against them.
|
||||
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
|
||||
They are console scripts of the Python package `paperspec`, whose code is in [src/paperspec/](../src/paperspec/), with the commands in `src/paperspec/cli/`.
|
||||
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
|
||||
|
||||
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
|
||||
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic-damb`.
|
||||
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
|
||||
A fixture is an example file or package under `fixtures/`.
|
||||
|
||||
| Program | Does |
|
||||
| Command | Does |
|
||||
| --- | --- |
|
||||
| `check.py` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
|
||||
| `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
||||
| `validate.py` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
||||
| `load.py` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
||||
| `site.py` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
|
||||
| `stress.py` | Stresses the oracle with random inputs from a seeded generator: `python3 tools/stress.py [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
|
||||
| [src/paperspec/](../src/paperspec/) | The oracle: a compact implementation of every rule, and the Python package `paperspec`. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. The programs put `src/` first on the path, so they run this checkout's code with or without pixi. |
|
||||
| `paperspec-check` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
|
||||
| `paperspec-build` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
|
||||
| `paperspec-validate` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `paperspec-validate PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
|
||||
| `paperspec-load` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `paperspec-load --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
|
||||
| `paperspec-site` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
|
||||
| `paperspec-stress` | Stresses the oracle with random inputs from a seeded generator: `paperspec-stress [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
|
||||
|
||||
`paperspec-validate` and `paperspec-load` work anywhere the package is installed, with the schemas and SQL it carries.
|
||||
The other four work on a checkout of this repository: the one an editable install points at, or else the one around the current folder.
|
||||
|
||||
## When to run what
|
||||
|
||||
Each program is also a pixi task.
|
||||
Pixi sets up a pinned Python and the other packages the repository uses.
|
||||
Each command is also a pixi task.
|
||||
Pixi sets up a pinned Python and the other packages the repository uses, and installs this checkout as the package, editable, so a change to the code takes effect at once.
|
||||
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
|
||||
`pixi run check` lints the Markdown first.
|
||||
|
||||
Run the programs directly from the repository root if you do not use pixi:
|
||||
If you do not use pixi, install the checkout editable with its extras on Python 3.14, then run the commands from the repository root:
|
||||
|
||||
```sh
|
||||
python3 tools/check.py # everything
|
||||
python3 tools/check.py --quick # everything except the rebuild
|
||||
python3 tools/build.py --out /tmp/rebuilt
|
||||
python3 tools/validate.py fixtures/packages/hale2026harmonic
|
||||
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
|
||||
pip install -e ".[tools]"
|
||||
paperspec-check # everything
|
||||
paperspec-check --quick # everything except the rebuild
|
||||
paperspec-build --out /tmp/rebuilt
|
||||
paperspec-validate fixtures/packages/hale2026harmonic-damb
|
||||
paperspec-load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
||||
```
|
||||
|
||||
Each command is also a module, so `python -m paperspec.cli.check` runs the same program.
|
||||
|
||||
A passing check ends with `All checks pass.`
|
||||
The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
|
||||
The load command prints `hale2026harmonic-damb: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
|
||||
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
|
||||
The full and checks-only forms of one document share its key, so they go into separate databases.
|
||||
Each program prints its options with `--help` and refuses options it does not know.
|
||||
`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
||||
Each command prints its options with `--help` and refuses options it does not know.
|
||||
`paperspec-build` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
|
||||
|
||||
Run `pixi run check` before every commit.
|
||||
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `build.py`, and commit the regenerated files with the change.
|
||||
Run `pixi run build` after you change a fixture source, a rule in `src/paperspec/` or a case in `src/paperspec/cli/build.py`, and commit the regenerated files with the change.
|
||||
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
|
||||
|
||||
## About the oracle
|
||||
@@ -65,11 +71,12 @@ Canonical JSON goes through Python's built-in JSON encoder, and falls back to an
|
||||
|
||||
Hashes are kept incrementally.
|
||||
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
|
||||
`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
||||
`paperspec-check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
|
||||
|
||||
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec` (`pip install .`, or a git pin) and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
|
||||
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data (`importlib.resources.files("paperspec") / "sql" / "doc.sql"`).
|
||||
Those modules need Python 3.14, the first with the Unicode 16.0 character data the specification pins, and otherwise only the standard library, except that `schema` needs `paperspec[schema]` (`jsonschema` and `referencing`) and the HKDF and Ed25519 functions of `crypto` need `paperspec[crypto]`; `paperspec[tools]` installs both.
|
||||
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
|
||||
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
|
||||
Otherwise it needs only the standard library, except that `schema` needs `paperspec[schema]` (`jsonschema` and `referencing`) and the HKDF and Ed25519 functions of `crypto` need `paperspec[crypto]`; `paperspec[tools]` installs both.
|
||||
The package version comes from git tags through setuptools-scm, and `pixi run -e dev wheel` builds the wheel into `dist/`.
|
||||
When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
|
||||
Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
|
||||
@@ -1,24 +0,0 @@
|
||||
"""The repository's own programs; see tools/README.md.
|
||||
|
||||
Importing this package checks that the tools can run here, and puts `src/` first on the path, so that
|
||||
`import paperspec` gets this checkout's reference implementation rather than an installed copy.
|
||||
The tools need Python 3.14 and the packages `cryptography` and `jsonschema`; the `paperspec` package
|
||||
itself needs Python 3.14 but neither package, and other programs install it on its own.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
import sys
|
||||
|
||||
if sys.version_info < (3, 14):
|
||||
sys.exit("The paperspec tools need Python 3.14 or later; this is Python %d.%d." % sys.version_info[:2])
|
||||
|
||||
_missing = [m for m in ("cryptography", "jsonschema") if importlib.util.find_spec(m) is None]
|
||||
if _missing:
|
||||
sys.exit("The paperspec tools need %s %s; run them through pixi, or install %s with pip."
|
||||
% ("the package" if len(_missing) == 1 else "the packages", " and ".join(_missing),
|
||||
"it" if len(_missing) == 1 else "them"))
|
||||
|
||||
SRC = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "src")
|
||||
if SRC not in sys.path:
|
||||
sys.path.insert(0, SRC)
|
||||
-1870
File diff suppressed because it is too large.
Load diff
-1207
File diff suppressed because it is too large.
Load diff
@@ -1,82 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Load packages into a SQLite database with the tables of sql/doc.sql and sql/store.sql.
|
||||
|
||||
Non-normative tooling.
|
||||
Usage: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]
|
||||
python3 tools/load.py --sqlite lab.db --bundle --built-from REV PACKAGE [PACKAGE ...]
|
||||
A package is a folder, or a zipped package (`.sct.zip`), unpacked under the rules of spec/12 §12.2 first.
|
||||
With --bundle the new file is a scienting-bundle/1: it also gets the tables of sql/bundle-sqlite.sql,
|
||||
the full-text search index and bundle_meta (spec/13 §13.5).
|
||||
Each package is validated first, and nothing is loaded if any package fails.
|
||||
Files without tables of their own (assets, datasets, derived files) are copied by SHA-256 into an
|
||||
asset store folder beside the database, named <database>.assets/.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, ROOT)
|
||||
import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported
|
||||
|
||||
from paperspec import db, package # noqa: E402
|
||||
from paperspec.schema import Store # noqa: E402
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--sqlite", required=True, help="the database file; created with the tables if new")
|
||||
ap.add_argument("--bundle", action="store_true", help="build a new scienting-bundle/1 file with search")
|
||||
ap.add_argument("--built-from", help="with --bundle: the store revision the packages come from")
|
||||
ap.add_argument("--store", help="with --bundle: the store's name (default: the packages' common store)")
|
||||
ap.add_argument("--max-size", metavar="BYTES", type=int, default=package.ZIP_LIMIT,
|
||||
help="refuse a zipped package that expands past this many bytes (default %d)" % package.ZIP_LIMIT)
|
||||
ap.add_argument("packages", nargs="+", help="package folders or zipped packages")
|
||||
a = ap.parse_args()
|
||||
if a.bundle and (os.path.exists(a.sqlite) or not a.built_from):
|
||||
ap.error("--bundle needs --built-from and a database file that does not exist yet")
|
||||
store, bad = Store(os.path.join(ROOT, "schemas")), False
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
folders = []
|
||||
for i, p in enumerate(a.packages):
|
||||
if package.is_zip(p):
|
||||
try:
|
||||
folders.append(package.read_zip(p, os.path.join(tmp, str(i)), a.max_size))
|
||||
except package.Refusal as e:
|
||||
print("%s: %s: %s" % (p, e.rule, e.message), file=sys.stderr)
|
||||
bad = True
|
||||
continue
|
||||
else:
|
||||
folders.append(p)
|
||||
for e in package.validate(folders[-1], store).errors:
|
||||
print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
|
||||
bad = True
|
||||
if bad:
|
||||
sys.exit("Nothing was loaded.")
|
||||
new = not os.path.exists(a.sqlite)
|
||||
conn = sqlite3.connect(a.sqlite)
|
||||
if a.bundle:
|
||||
db.create_bundle(conn, os.path.join(ROOT, "sql"))
|
||||
elif new:
|
||||
db.create(conn, os.path.join(ROOT, "sql"))
|
||||
for p in folders:
|
||||
key = db.load(conn, p, a.sqlite + ".assets")
|
||||
n = conn.execute("SELECT COUNT(*) FROM nodes WHERE doc_key = ?", (key,)).fetchone()[0]
|
||||
if a.bundle:
|
||||
print("%s: %d nodes, %d search rows" % (key, n, db.index(conn, key)))
|
||||
else:
|
||||
print("%s: %d nodes" % (key, n))
|
||||
if a.bundle:
|
||||
stores = {r[0] for r in conn.execute("SELECT DISTINCT store FROM documents WHERE store IS NOT NULL")}
|
||||
store_name = a.store or (stores.pop() if len(stores) == 1 else None)
|
||||
if store_name is None:
|
||||
sys.exit("The packages name no single store; give one with --store.")
|
||||
db.write_meta(conn, store_name, a.built_from)
|
||||
conn.commit()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
-636
@@ -1,636 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the paperspec website: every file of the repository, and its Markdown rendered as documentation.
|
||||
|
||||
Non-normative tooling, standard library only.
|
||||
Usage: python3 tools/site.py [--out DIR] [--base-url URL]
|
||||
The output folder (default `_site/`) mirrors the repository, so each file is served at its own path: a schema at
|
||||
the URL in its `$id`, the JSON-LD context at the address layers name, and every vector, fixture and table as it is.
|
||||
Beside each Markdown file the builder writes a page, `README.md` becoming its folder's `index.html` and `X.md`
|
||||
becoming `X.html`; a folder with no README gets a listing. It also writes the vocabulary page that the context's
|
||||
term IRIs point at, a search index, a 404 page and one stylesheet. Links between Markdown files are rewritten to
|
||||
the pages, so the site works from any host, and `deploy/Caddyfile` serves it with the right media types.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import html
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
SKIP_DIRS = {".git", ".pixi", ".claude", "_site", "__pycache__", "node_modules", ".gitea", ".vscode", ".idea"}
|
||||
SKIP_FILES = {".DS_Store"}
|
||||
NAME = "paperspec"
|
||||
REPO = "https://git.scient.ing/infra/paperspec"
|
||||
BASE_URL = "https://paperspec.scient.ing"
|
||||
|
||||
|
||||
def version():
|
||||
with open(os.path.join(ROOT, "CHANGELOG.md"), encoding="utf-8") as f:
|
||||
m = re.search(r"^## (\S+)", f.read(), re.M)
|
||||
return m.group(1) if m else "unreleased"
|
||||
|
||||
|
||||
# --- inline Markdown ------------------------------------------------------------------------------------------
|
||||
|
||||
def slug(heading):
|
||||
"""A heading's anchor as GitHub, Gitea and Forgejo compute it (the same rule tools/check.py uses)."""
|
||||
t = re.sub(r"[`*]", "", heading.strip().lower())
|
||||
t = "".join(ch for ch in t if ch.isalnum() or ch in " -_")
|
||||
return t.replace(" ", "-")
|
||||
|
||||
|
||||
def rewrite_href(target):
|
||||
"""A Markdown link target as a site link: `X.md` becomes `X.html`, `README.md` its folder."""
|
||||
if re.match(r"^[a-z][a-z0-9+.-]*:", target) or target.startswith(("/", "#")):
|
||||
return target
|
||||
path, hashmark, frag = target.partition("#")
|
||||
if path.endswith(".md"):
|
||||
head, base = os.path.split(path)
|
||||
if base == "README.md":
|
||||
path = (head + "/") if head else "./"
|
||||
else:
|
||||
path = path[:-3] + ".html"
|
||||
return path + hashmark + frag
|
||||
|
||||
|
||||
def inline(text):
|
||||
"""Render the inline Markdown the documentation uses: code spans, autolinks, links, strong and emphasis."""
|
||||
stash = []
|
||||
|
||||
def keep(s):
|
||||
stash.append(s)
|
||||
return "\x00%d\x00" % (len(stash) - 1)
|
||||
|
||||
text = re.sub(r"(`+)(.+?)\1", lambda m: keep("<code>%s</code>" % html.escape(m.group(2).strip(), quote=False)), text)
|
||||
text = re.sub(r"<(https?://[^>\s]+)>",
|
||||
lambda m: keep('<a href="%s">%s</a>' % (html.escape(m.group(1)), html.escape(m.group(1)))), text)
|
||||
text = html.escape(text, quote=False)
|
||||
|
||||
def link(m):
|
||||
label, target = m.group(1), html.unescape(m.group(2))
|
||||
return keep('<a href="%s">%s</a>' % (html.escape(rewrite_href(target)), emphasis(label)))
|
||||
|
||||
text = re.sub(r"\[([^\]]+)\]\(([^)\s]+)\)", link, text)
|
||||
text = emphasis(text)
|
||||
while "\x00" in text:
|
||||
text = re.sub(r"\x00(\d+)\x00", lambda m: stash[int(m.group(1))], text)
|
||||
return text
|
||||
|
||||
|
||||
def emphasis(text):
|
||||
text = re.sub(r"\*\*(?=\S)(.+?)(?<=\S)\*\*", r"<strong>\1</strong>", text)
|
||||
return re.sub(r"(?<![\w*])\*(?=[^\s*])(.+?)(?<=[^\s*])\*(?![\w*])", r"<em>\1</em>", text)
|
||||
|
||||
|
||||
def plain(text):
|
||||
"""Inline Markdown as plain text, for titles and the search index."""
|
||||
text = re.sub(r"`+([^`]+)`+", r"\1", text)
|
||||
text = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", text)
|
||||
return re.sub(r"\*+([^*]+)\*+", r"\1", text).strip()
|
||||
|
||||
|
||||
# --- block Markdown -------------------------------------------------------------------------------------------
|
||||
|
||||
LIST_RE = re.compile(r"^( *)([-*]|\d+\.)(?: +(.*)|$)")
|
||||
FENCE_RE = re.compile(r"^( *)```(\S*)\s*$")
|
||||
HEADING_RE = re.compile(r"^(#{1,6}) +(.*?)\s*#*\s*$")
|
||||
|
||||
|
||||
def indent_of(line):
|
||||
return len(line) - len(line.lstrip(" "))
|
||||
|
||||
|
||||
def starts_block(line):
|
||||
return bool(HEADING_RE.match(line) or FENCE_RE.match(line) or LIST_RE.match(line) or line.startswith("|")
|
||||
or line.startswith(": "))
|
||||
|
||||
|
||||
class Renderer:
|
||||
def __init__(self):
|
||||
self.ids = {}
|
||||
self.headings = []
|
||||
self.title = None
|
||||
|
||||
def heading_id(self, text):
|
||||
base = slug(text)
|
||||
n = self.ids.get(base, 0)
|
||||
self.ids[base] = n + 1
|
||||
return base if n == 0 else "%s-%d" % (base, n)
|
||||
|
||||
def blocks(self, lines):
|
||||
out, i = [], 0
|
||||
while i < len(lines):
|
||||
line = lines[i]
|
||||
if not line.strip():
|
||||
i += 1
|
||||
continue
|
||||
m = FENCE_RE.match(line)
|
||||
if m:
|
||||
pad = len(m.group(1))
|
||||
body, i = [], i + 1
|
||||
while i < len(lines) and not re.match(r"^ *```\s*$", lines[i]):
|
||||
body.append(lines[i][pad:] if lines[i][:pad].strip() == "" else lines[i])
|
||||
i += 1
|
||||
lang = m.group(2)
|
||||
cls = ' class="language-%s"' % html.escape(lang) if lang else ""
|
||||
out.append("<pre><code%s>%s</code></pre>" % (cls, html.escape("\n".join(body), quote=False)))
|
||||
i += 1
|
||||
continue
|
||||
m = HEADING_RE.match(line)
|
||||
if m:
|
||||
level, text = len(m.group(1)), m.group(2)
|
||||
hid = self.heading_id(text)
|
||||
if level == 1 and self.title is None:
|
||||
self.title = plain(text)
|
||||
self.headings.append({"level": level, "text": plain(text), "id": hid})
|
||||
anchor = '<a class="anchor" href="#%s" aria-label="Link to this section">#</a>' % hid if level > 1 else ""
|
||||
out.append('<h%d id="%s">%s%s</h%d>' % (level, hid, inline(text), anchor, level))
|
||||
i += 1
|
||||
continue
|
||||
if line.startswith("|"):
|
||||
rows = []
|
||||
while i < len(lines) and lines[i].startswith("|"):
|
||||
rows.append(lines[i])
|
||||
i += 1
|
||||
out.append(self.table(rows))
|
||||
continue
|
||||
if i + 1 < len(lines) and lines[i + 1].startswith(": ") and not starts_block(line):
|
||||
items = []
|
||||
while i + 1 < len(lines) and lines[i].strip() and lines[i + 1].startswith(": "):
|
||||
term, i = lines[i], i + 1
|
||||
desc = []
|
||||
while i < len(lines) and lines[i].strip():
|
||||
desc.append(lines[i][4:] if lines[i].startswith(": ") else lines[i].strip())
|
||||
i += 1
|
||||
items.append("<dt>%s</dt><dd>%s</dd>" % (inline(term.strip()), inline("\n".join(desc))))
|
||||
while i < len(lines) and not lines[i].strip():
|
||||
i += 1
|
||||
out.append("<dl>%s</dl>" % "".join(items))
|
||||
continue
|
||||
m = LIST_RE.match(line)
|
||||
if m:
|
||||
html_list, i = self.list(lines, i)
|
||||
out.append(html_list)
|
||||
continue
|
||||
para = []
|
||||
while i < len(lines) and lines[i].strip() and not (para and starts_block(lines[i])):
|
||||
para.append(lines[i].strip())
|
||||
i += 1
|
||||
out.append("<p>%s</p>" % inline("\n".join(para)))
|
||||
return "\n".join(out)
|
||||
|
||||
def list(self, lines, i):
|
||||
first = LIST_RE.match(lines[i])
|
||||
base = len(first.group(1))
|
||||
ordered = first.group(2)[0].isdigit()
|
||||
items = []
|
||||
while i < len(lines):
|
||||
m = LIST_RE.match(lines[i])
|
||||
if not m or len(m.group(1)) != base or m.group(2)[0].isdigit() != ordered:
|
||||
break
|
||||
content_col = base + len(m.group(2)) + 1
|
||||
body = [m.group(3) or ""] if m.group(3) else []
|
||||
i += 1
|
||||
while i < len(lines):
|
||||
ln = lines[i]
|
||||
if not ln.strip():
|
||||
nxt = next((x for x in lines[i + 1:] if x.strip()), None)
|
||||
if nxt is not None and indent_of(nxt) >= content_col - 1 and indent_of(nxt) > base:
|
||||
body.append("")
|
||||
i += 1
|
||||
continue
|
||||
break
|
||||
if indent_of(ln) > base:
|
||||
body.append(ln[min(indent_of(ln), content_col):])
|
||||
i += 1
|
||||
continue
|
||||
if body and not starts_block(ln):
|
||||
body.append(ln.strip()) # a lazy continuation of the item's paragraph
|
||||
i += 1
|
||||
continue
|
||||
break
|
||||
inner = self.blocks(body)
|
||||
if inner.startswith("<p>") and inner.count("<p>") == 1 and inner.endswith("</p>"):
|
||||
inner = inner[3:-4]
|
||||
items.append("<li>%s</li>" % inner)
|
||||
while i < len(lines) and not lines[i].strip():
|
||||
nxt = next((x for x in lines[i:] if x.strip()), None)
|
||||
m2 = LIST_RE.match(nxt) if nxt else None
|
||||
if m2 and len(m2.group(1)) == base:
|
||||
i += 1
|
||||
else:
|
||||
break
|
||||
tag = "ol" if ordered else "ul"
|
||||
start = int(first.group(2)[:-1]) if ordered else 1
|
||||
attr = ' start="%d"' % start if start != 1 else ""
|
||||
return "<%s%s>%s</%s>" % (tag, attr, "".join(items), tag), i
|
||||
|
||||
def table(self, rows):
|
||||
def cells(row):
|
||||
row = row.strip()
|
||||
if row.startswith("|"):
|
||||
row = row[1:]
|
||||
if row.endswith("|"):
|
||||
row = row[:-1]
|
||||
out, cur, tick = [], "", False
|
||||
for ch in row:
|
||||
if ch == "`":
|
||||
tick = not tick
|
||||
if ch == "|" and not tick:
|
||||
out.append(cur.strip())
|
||||
cur = ""
|
||||
else:
|
||||
cur += ch
|
||||
out.append(cur.strip())
|
||||
return out
|
||||
|
||||
head = cells(rows[0])
|
||||
body = [cells(r) for r in rows[2:]] if len(rows) > 1 and re.match(r"^\|[\s:|-]+\|?$", rows[1]) else \
|
||||
[cells(r) for r in rows[1:]]
|
||||
thead = "<tr>%s</tr>" % "".join("<th>%s</th>" % inline(c) for c in head)
|
||||
tbody = "".join("<tr>%s</tr>" % "".join("<td>%s</td>" % inline(c) for c in r) for r in body)
|
||||
return '<div class="table-wrap"><table><thead>%s</thead><tbody>%s</tbody></table></div>' % (thead, tbody)
|
||||
|
||||
|
||||
def render_markdown(text):
|
||||
r = Renderer()
|
||||
body = r.blocks(text.split("\n"))
|
||||
return body, r
|
||||
|
||||
|
||||
# --- pages ----------------------------------------------------------------------------------------------------
|
||||
|
||||
NAV = [
|
||||
("Start", [("README.md", "Home"), ("docs/overview.md", "Overview"), ("docs/concepts.md", "Concepts"),
|
||||
("docs/walkthrough.md", "Walkthrough"), ("docs/examples.md", "Examples"), ("docs/faq.md", "FAQ"),
|
||||
("docs/glossary.md", "Glossary"), ("docs/rationale.md", "Rationale"),
|
||||
("docs/comparison.md", "Comparison")]),
|
||||
("Specification", None),
|
||||
("Reference", [("schemas/README.md", "Schemas"), ("registry/README.md", "Registry"),
|
||||
("vectors/README.md", "Test vectors"), ("fixtures/README.md", "Fixtures"),
|
||||
("sql/README.md", "SQL"), ("context/README.md", "JSON-LD context"), ("vocab", "Vocabulary"),
|
||||
("tools/README.md", "Tools")]),
|
||||
("Project", [("CHANGELOG.md", "Changelog"), ("CONTRIBUTING.md", "Contributing"), ("LICENSE.md", "Licenses")]),
|
||||
]
|
||||
|
||||
|
||||
def page_path(md_rel):
|
||||
"""Where the page for a Markdown file goes, relative to the site root."""
|
||||
head, base = os.path.split(md_rel)
|
||||
if base == "README.md":
|
||||
return os.path.join(head, "index.html")
|
||||
return md_rel[:-3] + ".html"
|
||||
|
||||
|
||||
def page_url(md_rel):
|
||||
if md_rel == "vocab":
|
||||
return "vocab/"
|
||||
p = page_path(md_rel)
|
||||
return p[:-len("index.html")] if p.endswith("index.html") else p
|
||||
|
||||
|
||||
def rel(from_page, to_url):
|
||||
"""A relative link from the page at `from_page` to the site URL `to_url` (both relative to the root)."""
|
||||
start = os.path.dirname(from_page) or "."
|
||||
target = to_url or "./"
|
||||
trailing = target.endswith("/")
|
||||
r = os.path.relpath(target.rstrip("/") or ".", start)
|
||||
if r == ".":
|
||||
return "./"
|
||||
return r + "/" if trailing else r
|
||||
|
||||
|
||||
def spec_nav():
|
||||
out = []
|
||||
for f in sorted(os.listdir(os.path.join(ROOT, "spec"))):
|
||||
if re.match(r"\d\d-.*\.md$", f):
|
||||
with open(os.path.join(ROOT, "spec", f), encoding="utf-8") as fh:
|
||||
m = re.search(r"^# (.*)$", fh.read(), re.M)
|
||||
out.append(("spec/" + f, plain(m.group(1)) if m else f))
|
||||
return [("spec/README.md", "Contents")] + out
|
||||
|
||||
|
||||
def nav_html(page, here_md):
|
||||
"""The sidebar for the page at `page`, marking `here_md` (a Markdown path, `vocab`, or None) as current."""
|
||||
current = page_url(here_md) if here_md else None
|
||||
parts = []
|
||||
for group, entries in NAV:
|
||||
entries = spec_nav() if entries is None else entries
|
||||
links = []
|
||||
for md, label in entries:
|
||||
url = page_url(md)
|
||||
cur = ' aria-current="page"' if current == url else ""
|
||||
links.append('<li><a href="%s"%s>%s</a></li>' % (html.escape(rel(page, url)), cur, html.escape(label)))
|
||||
parts.append('<section><h2>%s</h2><ul>%s</ul></section>' % (html.escape(group), "".join(links)))
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
CSS = """\
|
||||
:root {
|
||||
--bg: #fbfaf8; --surface: #f2efea; --text: #1d1c1a; --muted: #5d5a54; --rule: #dedad3;
|
||||
--accent: #1f5f8b; --accent-soft: #e3edf4; --code-bg: #efece6; --mark: #fff3c4;
|
||||
--sans: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
--mono: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root:not([data-theme="light"]) {
|
||||
--bg: #161615; --surface: #1f1f1d; --text: #e9e6e0; --muted: #a6a29a; --rule: #34332f;
|
||||
--accent: #7fb6dc; --accent-soft: #1d2c36; --code-bg: #24231f; --mark: #4a3f12;
|
||||
}
|
||||
}
|
||||
:root[data-theme="dark"] {
|
||||
--bg: #161615; --surface: #1f1f1d; --text: #e9e6e0; --muted: #a6a29a; --rule: #34332f;
|
||||
--accent: #7fb6dc; --accent-soft: #1d2c36; --code-bg: #24231f; --mark: #4a3f12;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html { -webkit-text-size-adjust: 100%; }
|
||||
body { margin: 0; background: var(--bg); color: var(--text); font: 16px/1.6 var(--sans); }
|
||||
a { color: var(--accent); text-underline-offset: 2px; }
|
||||
header.top { position: sticky; top: 0; z-index: 2; display: flex; gap: 1rem; align-items: center;
|
||||
padding: .6rem 1rem; background: var(--bg); border-bottom: 1px solid var(--rule); }
|
||||
header.top .brand { font-weight: 650; color: var(--text); text-decoration: none; letter-spacing: -.01em; }
|
||||
header.top .version { color: var(--muted); font-size: .85rem; }
|
||||
header.top .spacer { flex: 1; }
|
||||
header.top input { width: min(18rem, 40vw); padding: .35rem .6rem; border: 1px solid var(--rule); border-radius: 6px;
|
||||
background: var(--surface); color: var(--text); font: inherit; font-size: .9rem; }
|
||||
.results { position: absolute; right: 1rem; top: 3rem; width: min(28rem, calc(100vw - 2rem)); max-height: 60vh;
|
||||
overflow: auto; background: var(--bg); border: 1px solid var(--rule); border-radius: 8px; padding: .3rem 0;
|
||||
box-shadow: 0 8px 24px rgba(0,0,0,.12); }
|
||||
.results a { display: block; padding: .35rem .8rem; text-decoration: none; color: var(--text); }
|
||||
.results a:hover, .results a:focus { background: var(--accent-soft); outline: none; }
|
||||
.results small { display: block; color: var(--muted); }
|
||||
.layout { display: grid; grid-template-columns: 16rem minmax(0, 1fr); max-width: 76rem; margin: 0 auto; }
|
||||
nav.side { position: sticky; top: 3.2rem; align-self: start; max-height: calc(100vh - 3.2rem); overflow: auto;
|
||||
padding: 1.2rem 1rem 2rem; font-size: .9rem; border-right: 1px solid var(--rule); }
|
||||
nav.side h2 { font-size: .72rem; text-transform: uppercase; letter-spacing: .08em; color: var(--muted);
|
||||
margin: 1.2rem 0 .3rem; }
|
||||
nav.side ul { list-style: none; margin: 0; padding: 0; }
|
||||
nav.side a { display: block; padding: .15rem .4rem; border-radius: 4px; text-decoration: none; color: var(--text); }
|
||||
nav.side a:hover { background: var(--surface); }
|
||||
nav.side a[aria-current] { background: var(--accent-soft); color: var(--accent); font-weight: 600; }
|
||||
main { padding: 1.5rem 2.5rem 4rem; min-width: 0; }
|
||||
article { max-width: 48rem; }
|
||||
h1, h2, h3, h4 { line-height: 1.25; letter-spacing: -.01em; }
|
||||
h1 { font-size: 2rem; margin: .2rem 0 1rem; }
|
||||
h2 { font-size: 1.4rem; margin: 2.2rem 0 .6rem; padding-top: .4rem; border-top: 1px solid var(--rule); }
|
||||
h3 { font-size: 1.12rem; margin: 1.6rem 0 .4rem; }
|
||||
h1, h2, h3, h4, h5, h6 { scroll-margin-top: 4rem; }
|
||||
.anchor { margin-left: .4rem; color: var(--muted); text-decoration: none; opacity: 0; font-weight: 400; }
|
||||
h2:hover .anchor, h3:hover .anchor, h4:hover .anchor, .anchor:focus { opacity: 1; }
|
||||
:target { background: var(--mark); }
|
||||
p, ul, ol, dl, pre, .table-wrap { margin: 0 0 1rem; }
|
||||
li + li { margin-top: .25rem; }
|
||||
code { font: .88em/1.4 var(--mono); background: var(--code-bg); padding: .1em .3em; border-radius: 4px;
|
||||
overflow-wrap: anywhere; }
|
||||
pre { background: var(--code-bg); padding: .8rem 1rem; border-radius: 6px; overflow-x: auto; }
|
||||
pre code { background: none; padding: 0; font-size: .85rem; overflow-wrap: normal; }
|
||||
.table-wrap { overflow-x: auto; }
|
||||
table { border-collapse: collapse; font-size: .92rem; min-width: 60%; }
|
||||
th, td { text-align: left; vertical-align: top; padding: .45rem .7rem; border-bottom: 1px solid var(--rule); }
|
||||
th { background: var(--surface); font-weight: 600; }
|
||||
dt { font-weight: 650; margin-top: 1rem; }
|
||||
dd { margin: .2rem 0 0 1.2rem; }
|
||||
.meta { margin-top: 3rem; padding-top: 1rem; border-top: 1px solid var(--rule); color: var(--muted); font-size: .85rem; }
|
||||
.meta a { color: var(--muted); }
|
||||
.listing td:last-child { text-align: right; color: var(--muted); font-variant-numeric: tabular-nums; }
|
||||
.crumbs { color: var(--muted); font-size: .9rem; margin-bottom: .5rem; }
|
||||
details.menu { display: none; }
|
||||
@media (max-width: 52rem) {
|
||||
.layout { display: block; }
|
||||
nav.side { display: none; }
|
||||
details.menu { display: block; padding: .5rem 16px; border-bottom: 1px solid var(--rule); }
|
||||
details.menu summary { cursor: pointer; font-weight: 600; }
|
||||
details.menu nav.side { display: block; position: static; max-height: none; border: 0; padding: .5rem 0; }
|
||||
main { padding: 1rem 16px 3rem; }
|
||||
header.top { padding: .5rem 16px; }
|
||||
header.top input { width: 9rem; }
|
||||
h1 { font-size: 1.6rem; }
|
||||
}
|
||||
"""
|
||||
|
||||
SEARCH_JS = """\
|
||||
(function () {
|
||||
var input = document.getElementById("search"), box = document.getElementById("results"), index = null;
|
||||
if (!input) return;
|
||||
var root = document.documentElement.getAttribute("data-root") || "";
|
||||
function load(cb) {
|
||||
if (index) return cb();
|
||||
fetch(root + "search.json").then(function (r) { return r.json(); }).then(function (j) { index = j; cb(); })
|
||||
.catch(function () { index = []; cb(); });
|
||||
}
|
||||
function esc(s) { return s.replace(/[&<>"]/g, function (c) { return {"&": "&", "<": "<", ">": ">", '"': """}[c]; }); }
|
||||
function show() {
|
||||
var q = input.value.trim().toLowerCase();
|
||||
if (q.length < 2) { box.hidden = true; return; }
|
||||
var hits = [];
|
||||
index.forEach(function (p) {
|
||||
if (p.title.toLowerCase().indexOf(q) >= 0) hits.push({url: p.url, text: p.title, page: ""});
|
||||
p.headings.forEach(function (h) {
|
||||
if (h.text.toLowerCase().indexOf(q) >= 0) hits.push({url: p.url + "#" + h.id, text: h.text, page: p.title});
|
||||
});
|
||||
});
|
||||
hits = hits.slice(0, 25);
|
||||
box.innerHTML = hits.length ? hits.map(function (h) {
|
||||
return '<a href="' + root + esc(h.url) + '">' + esc(h.text) + (h.page ? "<small>" + esc(h.page) + "</small>" : "") + "</a>";
|
||||
}).join("") : '<a aria-disabled="true">No matching heading</a>';
|
||||
box.hidden = false;
|
||||
}
|
||||
input.addEventListener("input", function () { load(show); });
|
||||
input.addEventListener("keydown", function (e) {
|
||||
if (e.key === "Escape") { box.hidden = true; input.blur(); }
|
||||
if (e.key === "Enter") { var a = box.querySelector("a[href]"); if (a) location.href = a.href; }
|
||||
});
|
||||
document.addEventListener("click", function (e) { if (!box.contains(e.target) && e.target !== input) box.hidden = true; });
|
||||
document.addEventListener("keydown", function (e) {
|
||||
if (e.key === "/" && document.activeElement !== input) { e.preventDefault(); input.focus(); }
|
||||
});
|
||||
})();
|
||||
"""
|
||||
|
||||
|
||||
def layout(page, here_md, title, body, source=None, base_url=BASE_URL):
|
||||
"""A full page. `page` is the output path relative to the site root; `here_md` the Markdown it renders."""
|
||||
root = rel(page, "")
|
||||
root = "" if root == "./" else root + ("" if root.endswith("/") else "/")
|
||||
if page == "404.html":
|
||||
root = "/" # served at whatever path was missing, so its links start from the site root
|
||||
canonical = base_url.rstrip("/") + "/" + (page[:-len("index.html")] if page.endswith("index.html") else page)
|
||||
nav = nav_html(page, here_md)
|
||||
meta = ""
|
||||
if source:
|
||||
meta = ('<p class="meta">Source: <a href="%s/src/branch/main/%s">%s</a> · <a href="%s%s">raw</a></p>'
|
||||
% (REPO, html.escape(source), html.escape(source), root, html.escape(source)))
|
||||
return """<!doctype html>
|
||||
<html lang="en" data-root="{root}">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>{title}</title>
|
||||
<link rel="canonical" href="{canonical}">
|
||||
<link rel="stylesheet" href="{root}site.css">
|
||||
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%231f5f8b'/%3E%3Cpath d='M4 3h5.5a3 3 0 0 1 0 6H6v4H4z' fill='white'/%3E%3C/svg%3E">
|
||||
</head>
|
||||
<body>
|
||||
<header class="top">
|
||||
<a class="brand" href="{root_or_dot}">{name}</a><span class="version">{version}</span>
|
||||
<span class="spacer"></span>
|
||||
<input id="search" type="search" placeholder="Search headings ( / )" aria-label="Search headings" autocomplete="off">
|
||||
<div id="results" class="results" hidden></div>
|
||||
</header>
|
||||
<details class="menu"><summary>Menu</summary><nav class="side" aria-label="Site">{nav}</nav></details>
|
||||
<div class="layout">
|
||||
<nav class="side" aria-label="Site">{nav}</nav>
|
||||
<main><article>
|
||||
{body}
|
||||
{meta}
|
||||
</article></main>
|
||||
</div>
|
||||
<script src="{root}search.js" defer></script>
|
||||
</body>
|
||||
</html>
|
||||
""".format(root=root, root_or_dot=root or "./", title=html.escape(title), canonical=html.escape(canonical),
|
||||
name=NAME, version=html.escape(version()), nav=nav, body=body, meta=meta)
|
||||
|
||||
|
||||
def size_text(n):
|
||||
for unit in ("B", "KB", "MB"):
|
||||
if n < 1024 or unit == "MB":
|
||||
return ("%d %s" % (n, unit)) if unit == "B" else ("%.1f %s" % (n, unit))
|
||||
n /= 1024.0
|
||||
|
||||
|
||||
def listing(out, d, base_url):
|
||||
"""A page listing the files and folders of `d` (relative to the site root), for folders with no README."""
|
||||
full = os.path.join(out, d)
|
||||
entries = sorted(os.listdir(full))
|
||||
rows = []
|
||||
if d:
|
||||
rows.append('<tr><td><a href="../">../</a></td><td></td></tr>')
|
||||
for e in entries:
|
||||
if e == "index.html":
|
||||
continue
|
||||
p = os.path.join(full, e)
|
||||
if os.path.isdir(p):
|
||||
rows.append('<tr><td><a href="%s/">%s/</a></td><td></td></tr>' % (html.escape(e), html.escape(e)))
|
||||
elif not (e.endswith(".html") and os.path.exists(os.path.join(full, e[:-5] + ".md"))):
|
||||
rows.append('<tr><td><a href="%s">%s</a></td><td>%s</td></tr>'
|
||||
% (html.escape(e), html.escape(e), size_text(os.path.getsize(p))))
|
||||
crumbs = ['<a href="%s">%s</a>' % (rel(os.path.join(d, "index.html"), ""), NAME)]
|
||||
parts = d.split("/") if d else []
|
||||
for i, part in enumerate(parts):
|
||||
crumbs.append('<a href="%s">%s</a>' % (rel(os.path.join(d, "index.html"), "/".join(parts[:i + 1]) + "/"),
|
||||
html.escape(part)))
|
||||
body = ('<p class="crumbs">%s</p><h1>%s/</h1><div class="table-wrap"><table class="listing"><thead><tr>'
|
||||
'<th>Name</th><th>Size</th></tr></thead><tbody>%s</tbody></table></div>'
|
||||
% (" / ".join(crumbs), html.escape(d or NAME), "".join(rows)))
|
||||
with open(os.path.join(full, "index.html"), "w", encoding="utf-8") as f:
|
||||
f.write(layout(os.path.join(d, "index.html"), None, (d or NAME) + "/ · " + NAME, body, base_url=base_url))
|
||||
|
||||
|
||||
def vocab_page(out, base_url):
|
||||
"""The page the context's term IRIs point at: one entry per IRI fragment of the `sct` namespace."""
|
||||
with open(os.path.join(ROOT, "context", "scienting-1.jsonld"), encoding="utf-8") as f:
|
||||
ctx = json.load(f)["@context"]
|
||||
prefix = ctx.get("sct", "")
|
||||
rows = []
|
||||
for term, value in ctx.items():
|
||||
if term.startswith("@") or term in ("sct", "as"):
|
||||
continue
|
||||
iri = value if isinstance(value, str) else value.get("@id", "")
|
||||
if not iri.startswith("sct:"):
|
||||
continue
|
||||
local = iri[4:]
|
||||
notes = []
|
||||
if isinstance(value, dict):
|
||||
if value.get("@type") == "@id":
|
||||
notes.append("an IRI")
|
||||
if value.get("@type") == "@json":
|
||||
notes.append("a JSON literal, kept intact")
|
||||
if value.get("@container"):
|
||||
notes.append("a set")
|
||||
rows.append('<tr id="%s"><td><code>%s</code></td><td><code>%s%s</code></td><td>%s</td></tr>'
|
||||
% (html.escape(local), html.escape(term), html.escape(prefix), html.escape(local),
|
||||
html.escape(", ".join(notes))))
|
||||
body = ("<h1>Vocabulary</h1><p>The terms the JSON-LD context "
|
||||
'<a href="../context/scienting-1.jsonld"><code>scienting-1.jsonld</code></a> adds to the W3C Web '
|
||||
'Annotation vocabulary. Each term's IRI is a fragment of this page; '
|
||||
'<a href="../context/">the context's guide</a> explains how layers use them.</p>'
|
||||
'<div class="table-wrap"><table><thead><tr><th>Term</th><th>IRI</th><th>Value</th></tr></thead><tbody>%s'
|
||||
"</tbody></table></div>" % "".join(rows))
|
||||
os.makedirs(os.path.join(out, "vocab"), exist_ok=True)
|
||||
with open(os.path.join(out, "vocab", "index.html"), "w", encoding="utf-8") as f:
|
||||
f.write(layout("vocab/index.html", "vocab", "Vocabulary · " + NAME, body, base_url=base_url))
|
||||
return {"title": "Vocabulary", "url": "vocab/", "headings": []}
|
||||
|
||||
|
||||
def build(out, base_url=BASE_URL):
|
||||
if os.path.exists(out):
|
||||
shutil.rmtree(out)
|
||||
os.makedirs(out)
|
||||
mds = []
|
||||
for d, dirs, fs in os.walk(ROOT):
|
||||
dirs[:] = sorted(x for x in dirs if x not in SKIP_DIRS and not (x.startswith(".") and x != ".well-known"))
|
||||
reld = os.path.relpath(d, ROOT)
|
||||
reld = "" if reld == "." else reld.replace(os.sep, "/")
|
||||
if os.path.abspath(d) == os.path.abspath(out):
|
||||
continue
|
||||
for f in sorted(fs):
|
||||
if f in SKIP_FILES or f.startswith("."):
|
||||
continue
|
||||
src = os.path.join(d, f)
|
||||
if os.path.islink(src):
|
||||
continue
|
||||
dst = os.path.join(out, reld, f)
|
||||
os.makedirs(os.path.dirname(dst), exist_ok=True)
|
||||
shutil.copyfile(src, dst)
|
||||
if f.endswith(".md"):
|
||||
mds.append((reld + "/" + f) if reld else f)
|
||||
index = []
|
||||
for md in mds:
|
||||
with open(os.path.join(ROOT, md), encoding="utf-8") as f:
|
||||
body, r = render_markdown(f.read())
|
||||
title = r.title or md
|
||||
page = page_path(md)
|
||||
full_title = (title + " · " + NAME) if md != "README.md" else NAME + " · " + title
|
||||
with open(os.path.join(out, page), "w", encoding="utf-8") as f:
|
||||
f.write(layout(page, md, full_title, body, source=md, base_url=base_url))
|
||||
index.append({"title": title, "url": page_url(md),
|
||||
"headings": [{"text": h["text"], "id": h["id"]} for h in r.headings if h["level"] > 1]})
|
||||
index.append(vocab_page(out, base_url))
|
||||
for d, dirs, fs in os.walk(out):
|
||||
reld = os.path.relpath(d, out)
|
||||
reld = "" if reld == "." else reld.replace(os.sep, "/")
|
||||
if "index.html" not in fs:
|
||||
listing(out, reld, base_url)
|
||||
with open(os.path.join(out, "404.html"), "w", encoding="utf-8") as f:
|
||||
f.write(layout("404.html", None, "Not found · " + NAME,
|
||||
'<h1>Not found</h1><p>No file of the specification has this address. '
|
||||
'Start from <a href="%s/">the home page</a> or search the headings above.</p>'
|
||||
% html.escape(base_url.rstrip("/")), base_url=base_url))
|
||||
with open(os.path.join(out, "site.css"), "w", encoding="utf-8") as f:
|
||||
f.write(CSS)
|
||||
with open(os.path.join(out, "search.js"), "w", encoding="utf-8") as f:
|
||||
f.write(SEARCH_JS)
|
||||
with open(os.path.join(out, "search.json"), "w", encoding="utf-8") as f:
|
||||
json.dump(sorted(index, key=lambda p: p["url"]), f, ensure_ascii=False, separators=(",", ":"))
|
||||
f.write("\n")
|
||||
return len(mds)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--out", default=os.path.join(ROOT, "_site"), help="the output folder (default _site/)")
|
||||
ap.add_argument("--base-url", default=BASE_URL, help="the site's address, for canonical links (default %s)" % BASE_URL)
|
||||
a = ap.parse_args()
|
||||
out = os.path.abspath(a.out)
|
||||
if out == ROOT or ROOT.startswith(out + os.sep):
|
||||
ap.error("the output folder must not be the repository or contain it")
|
||||
n = build(out, a.base_url)
|
||||
print("Wrote %d pages and every file of the repository to %s" % (n, out))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
-733
@@ -1,733 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stress the reference implementation with random inputs. Exit status 0 means every property held.
|
||||
|
||||
Non-normative tooling; it needs Python 3.14 and the packages pixi installs (`jsonschema`, `cryptography`).
|
||||
Usage: python3 tools/stress.py [--seed N] [--rounds N]
|
||||
The vectors pin values; this program checks the invariants the chapters state, over inputs drawn from a seeded
|
||||
random generator, so a failure is reproducible from the seed it prints. Canonical JSON round-trips and sorts keys
|
||||
by UTF-16 units; normalization is idempotent; notation folding never raises; random corrections keep the five
|
||||
values equal to a full recomputation, never reuse an ID and apply deterministically; anchors made before those
|
||||
corrections always resolve to one of the five outcomes; Merkle roots agree with a second implementation;
|
||||
spot-check samples do not depend on input order; error bounds are the least six-decimal value with the stated tail;
|
||||
differently stamped pages share a fingerprint; the zip reader and the package validator refuse or accept mutated
|
||||
input without raising and never write outside their folders; rights combine the same in any order; and
|
||||
spot-check chains hundreds deep settle without recursion.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import zipfile
|
||||
from fractions import Fraction
|
||||
from math import comb
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, ROOT)
|
||||
import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported
|
||||
|
||||
from paperspec import align as AL # noqa: E402
|
||||
from paperspec import anchors as AN # noqa: E402
|
||||
from paperspec import corrections as CO # noqa: E402
|
||||
from paperspec import crypto, fingerprint as FP, hashing, jcs, receipts as RC, rights as RI, targets as TG # noqa: E402
|
||||
from paperspec import keys as K, log as LG, mldsa, vrf # noqa: E402
|
||||
from paperspec import package as PK # noqa: E402
|
||||
from paperspec.model import LABELLED, LEAVES, Doc # noqa: E402
|
||||
from paperspec.notation import fold_math # noqa: E402
|
||||
from paperspec.text import NODE_ID_RE, RECORD_ID_RE, letter_first, norm, record_id, sha256_hex, is_ws # noqa: E402
|
||||
from paperspec.tree import Refusal, validate_structure # noqa: E402
|
||||
from tools.build import ROSTER_BY_ID, SCHEMAS, secret, sentence_doc, tiny_package # noqa: E402
|
||||
from tools.check import HALE, STORE, TINY_KEY, reference_values # noqa: E402
|
||||
|
||||
FAILS = []
|
||||
COUNTS = {}
|
||||
RULES = {r["id"] for r in json.load(open(os.path.join(ROOT, "registry", "rules.json"), encoding="utf-8"))["rules"]}
|
||||
|
||||
|
||||
def ok(section, cond, msg):
|
||||
COUNTS[section] = COUNTS.get(section, 0) + 1
|
||||
if not cond and len(FAILS) < 200:
|
||||
FAILS.append("%s: %s" % (section, msg))
|
||||
|
||||
|
||||
TEXT_POOL = list("abcdefghijklmnopqrstuvwxyzABCDEFXYZ0123456789 .,;:-()[]") + [
|
||||
"é", "Å", "fi", "µ", "−", "–", " ", "\t", "\n", "
", "€",
|
||||
"\U0001d49c", "\U0001f600", "é", "H₂O", "10³", "cm⁻¹", " "]
|
||||
KEY_POOL = list("abcxyz") + ["€", "\U0001f600", "fi", "A", "é", "", "a b", "\u0001"]
|
||||
|
||||
|
||||
def rand_text(rng, lo=0, hi=12, pool=TEXT_POOL):
|
||||
return "".join(rng.choice(pool) for _ in range(rng.randint(lo, hi)))
|
||||
|
||||
|
||||
def rand_json(rng, depth=0):
|
||||
r = rng.random()
|
||||
if depth > 3 or r < 0.4:
|
||||
choice = rng.randint(0, 6)
|
||||
if choice == 0:
|
||||
return None
|
||||
if choice == 1:
|
||||
return rng.random() < 0.5
|
||||
if choice == 2:
|
||||
return rng.randint(-2**53, 2**53)
|
||||
if choice == 3:
|
||||
return rng.uniform(-1, 1) * 10.0 ** rng.randint(-12, 24)
|
||||
if choice == 4:
|
||||
return rng.choice([0, -0.0, 1e21, 1e-7, 0.000001, 123456789.125, 5e-324, 1.7976931348623157e308])
|
||||
return rand_text(rng)
|
||||
if r < 0.7:
|
||||
return [rand_json(rng, depth + 1) for _ in range(rng.randint(0, 4))]
|
||||
return {rand_text(rng, 0, 3, KEY_POOL): rand_json(rng, depth + 1) for _ in range(rng.randint(0, 4))}
|
||||
|
||||
|
||||
def _as_doubles(text):
|
||||
"""Parse JSON as an ECMAScript engine does, which is how canonical JSON is read (vectors/README.md)."""
|
||||
return json.loads(text, parse_int=lambda x: int(x) if abs(int(x)) <= 2**53 else float(x))
|
||||
|
||||
|
||||
def stress_canonical_json(rng, rounds):
|
||||
for _ in range(rounds):
|
||||
v = rand_json(rng)
|
||||
try:
|
||||
s = jcs.dumps(v)
|
||||
except ValueError as e:
|
||||
ok("canonical-json", False, "dumps refused a plain value: %r (%s)" % (v, e))
|
||||
continue
|
||||
ok("canonical-json", jcs.dumpb(v) == s.encode("utf-8"), "dumpb is dumps encoded")
|
||||
back = _as_doubles(s)
|
||||
ok("canonical-json", back == v, "round trip changed the value: %r" % (v,))
|
||||
ok("canonical-json", jcs.dumps(back) == s, "canonical form is not idempotent for %r" % (v,))
|
||||
pairs = _as_doubles(s) if not s.startswith(("{", "[")) else json.loads(s, object_pairs_hook=lambda kv: kv)
|
||||
|
||||
def ordered(x):
|
||||
if isinstance(x, list) and x and isinstance(x[0], tuple):
|
||||
keys = [k for k, _ in x]
|
||||
return keys == sorted(keys, key=lambda k: k.encode("utf-16-be")) and all(ordered(v) for _, v in x)
|
||||
if isinstance(x, list):
|
||||
return all(ordered(y) for y in x)
|
||||
return True
|
||||
ok("canonical-json", ordered(pairs), "object members are not sorted by UTF-16 code units in %s" % s[:80])
|
||||
|
||||
|
||||
def stress_normalization(rng, rounds):
|
||||
for _ in range(rounds):
|
||||
s = rand_text(rng, 0, 30)
|
||||
n = norm(s)
|
||||
ok("normalize", norm(n) == n, "normalization is not idempotent for %r" % s)
|
||||
ok("normalize", not n or (not is_ws(n[0]) and not is_ws(n[-1])), "normalized text keeps edge whitespace")
|
||||
ok("normalize", " " not in n and all(ch == " " or not is_ws(ch) for ch in n),
|
||||
"normalized text keeps a whitespace run or a non-space whitespace: %r" % n)
|
||||
|
||||
|
||||
TEX_TOKENS = list("xyzKrHO0123456789+-=()'") + ["_", "^", "{", "}", "\\alpha", "\\mathrm{H}", "\\frac{a}{b}", " ",
|
||||
"\\,", "\\cdot", "H_2O", "^{14}", "{}", "\\left(", "\\right)", "\\\\",
|
||||
"µ", "−", "''", "_{", "^{"]
|
||||
TYPST_TOKENS = list("xyzKr0123456789+-=()'") + ["_", "^", "(", ")", "alpha", "upright(H)", "\"cm\"", " ", "dot",
|
||||
"plus.minus", "<=", "/", "epsilon.alt", "_(", "^(", "\"a\\b\""]
|
||||
|
||||
|
||||
def stress_notation(rng, rounds):
|
||||
for _ in range(rounds):
|
||||
for notation, pool in (("tex", TEX_TOKENS), ("typst", TYPST_TOKENS)):
|
||||
s = "".join(rng.choice(pool) for _ in range(rng.randint(0, 8)))
|
||||
try:
|
||||
a = fold_math(notation, s)
|
||||
b = fold_math(notation, s)
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("notation", False, "fold_math raised %r on %s %r" % (e, notation, s))
|
||||
continue
|
||||
ok("notation", a == b and (a is None or isinstance(a, str)), "fold_math is not deterministic on %r" % s)
|
||||
ok("notation", a is None or not any(is_ws(ch) for ch in a), "a canonical form holds whitespace: %r" % a)
|
||||
ok("notation", a is None or "$" not in a, "a canonical form holds a dollar sign: %r" % a)
|
||||
|
||||
|
||||
def _leaf_ids(doc, kinds):
|
||||
return [nid for nid in doc.preorder() if doc.nodes[nid]["kind"] in kinds]
|
||||
|
||||
|
||||
def random_correction(rng, st, doc):
|
||||
"""One random proposal against `st`, or None."""
|
||||
sentences = _leaf_ids(doc, {"sentence"})
|
||||
cells = _leaf_ids(doc, {"cell"})
|
||||
op = rng.choice(["replace", "replace", "split", "merge", "move", "relabel", "set", "delete", "reclassify",
|
||||
"insert-text", "insert-node"])
|
||||
evidence = [{"kind": "crop", "sha256": "0" * 64}] if rng.random() < 0.8 else [{"kind": "person", "by": "ana"}]
|
||||
c = {"op": op.split("-")[0], "evidence": evidence}
|
||||
if op == "replace" and (sentences or cells):
|
||||
nid = rng.choice(sentences + cells)
|
||||
text = hashing.plain(doc, nid)
|
||||
if not text:
|
||||
return None
|
||||
i = rng.randrange(len(text))
|
||||
j = min(len(text), i + rng.randint(1, 6))
|
||||
c.update(target=nid, find=text[i:j], replace=rand_text(rng, 0, 8))
|
||||
elif op == "split" and sentences:
|
||||
nid = rng.choice(sentences)
|
||||
c.update(target=nid, offset=rng.randint(0, max(1, len(hashing.plain(doc, nid)))))
|
||||
elif op == "merge" and len(sentences) > 1:
|
||||
nid = rng.choice(sentences)
|
||||
sibs = doc.kids(doc.nodes[nid]["parent"])
|
||||
k = sibs.index(nid)
|
||||
other = sibs[k + 1] if k + 1 < len(sibs) else rng.choice(sentences)
|
||||
c.update(target=nid, **{"with": other}, joiner=rng.choice([" ", ""]))
|
||||
elif op == "move" and sentences:
|
||||
paragraphs = _leaf_ids(doc, {"paragraph"})
|
||||
if not paragraphs:
|
||||
return None
|
||||
c.update(target=rng.choice(sentences), parent=rng.choice(paragraphs), place_after=None)
|
||||
elif op == "relabel":
|
||||
labelled = [nid for nid in doc.preorder() if doc.nodes[nid]["kind"] in LABELLED]
|
||||
if not labelled:
|
||||
return None
|
||||
c.update(target=rng.choice(labelled), label=rng.choice([None, "2", "Table 7", "a", rand_text(rng, 1, 5)]))
|
||||
elif op == "set" and cells:
|
||||
c.update(target=rng.choice(cells), attr=rng.choice(["colspan", "rowspan", "role", "column", "level"]),
|
||||
value=rng.choice([1, 2, 3, "2", -1, "data", "row-header", "boss", None]))
|
||||
elif op == "delete" and len(doc.order) > 8:
|
||||
c.update(target=rng.choice(doc.order[1:]))
|
||||
elif op == "reclassify":
|
||||
c.update(target=rng.choice(doc.order), kind=rng.choice(["footnote", "quote", "paragraph", "caption", "section"]))
|
||||
elif op == "insert-text" and sentences:
|
||||
nid = rng.choice(sentences)
|
||||
c.update(target=nid, run=rng.randint(0, len(doc.nodes[nid]["content"])), offset=rng.randint(0, 20),
|
||||
text=rand_text(rng, 1, 6))
|
||||
elif op == "insert-node":
|
||||
paragraphs = _leaf_ids(doc, {"paragraph"})
|
||||
if not paragraphs:
|
||||
return None
|
||||
parent = rng.choice(paragraphs)
|
||||
kids = doc.kids(parent)
|
||||
node = {"kind": "sentence", "content": [{"kind": "text", "text": rand_text(rng, 1, 20) or "x"}]}
|
||||
if rng.random() < 0.2:
|
||||
node["foo"] = 1
|
||||
c.update(parent=parent, place_after=rng.choice(kids + [None]), node=node)
|
||||
else:
|
||||
return None
|
||||
return c
|
||||
|
||||
|
||||
def stress_corrections(rng, rounds):
|
||||
"""A random walk of corrections from the tiny document; returns the states it passed through."""
|
||||
tiny = tiny_package(None, receipts=False)
|
||||
st = tiny["final"]
|
||||
base = {"author": {"key": "ana"}, "reason": "Stress.", "at": "2026-10-11T12:00:00Z"}
|
||||
seen = {n["id"] for n in st.nodes}
|
||||
history, accepted = [st], 0
|
||||
for _ in range(rounds):
|
||||
doc = st.doc()
|
||||
c = random_correction(rng, st, doc)
|
||||
if c is None:
|
||||
continue
|
||||
c = dict(base, **c)
|
||||
try:
|
||||
st1, rec = CO.apply(st, copy.deepcopy(c), store=SCHEMAS)
|
||||
except Refusal as e:
|
||||
ok("corrections", e.rule in RULES, "a refusal names the unregistered rule %s" % e.rule)
|
||||
continue
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("corrections", False, "apply raised %r on %s" % (e, json.dumps(c)[:160]))
|
||||
continue
|
||||
accepted += 1
|
||||
try:
|
||||
validate_structure(st1.doc())
|
||||
ok("corrections", st1.index().values == reference_values(st1.doc()),
|
||||
"incremental values differ from the full computation after %s" % c["op"])
|
||||
except Refusal as e:
|
||||
ok("corrections", False, "an accepted correction left the tree invalid: %s" % e.rule)
|
||||
new_ids = {n["id"] for n in st1.nodes} - {n["id"] for n in st.nodes}
|
||||
ok("corrections", not (new_ids & seen), "an ID was reused: %s" % sorted(new_ids & seen))
|
||||
seen |= new_ids
|
||||
table = {e["id"] for e in st1.ids}
|
||||
ok("corrections", set(rec.get("retired", [])) <= table, "a retired ID is not in the ID table")
|
||||
ok("corrections", st1.ids == sorted(st1.ids, key=CO.id_order), "ids are not in the order of spec/04 §4.4")
|
||||
ok("corrections", rec["id"] == record_id(sha256_hex(jcs.dumpb({k: v for k, v in rec.items() if k != "id"}))),
|
||||
"the record's id does not derive from its content")
|
||||
ok("corrections", rec["index"] == st.rev + 1 == st1.rev, "the record's index is not the next revision")
|
||||
st2, rec2 = CO.apply(st, copy.deepcopy(c), store=SCHEMAS)
|
||||
ok("corrections", rec2 == rec and st2.nodes == st1.nodes and st2.ids == st1.ids, "apply is not deterministic")
|
||||
try:
|
||||
_, rec3 = CO.gate(st, copy.deepcopy(c), SCHEMAS)
|
||||
ok("corrections", rec3 == rec, "the gate records something else than apply")
|
||||
except Refusal as e:
|
||||
ok("corrections", e.rule == "correction.schema", "the gate refused an applied correction with %s" % e.rule)
|
||||
st = st1
|
||||
history.append(st)
|
||||
ok("corrections", accepted >= rounds // 10, "too few random corrections were accepted: %d of %d" % (accepted, rounds))
|
||||
return history
|
||||
|
||||
|
||||
def stress_anchors(rng, history, rounds):
|
||||
st0, final = history[0], history[-1]
|
||||
doc0, vals0 = st0.doc(), hashing.compute(st0.doc())
|
||||
outcomes = {"current", "changed-below", "moved", "reattached", "stale"}
|
||||
seen_outcomes = set()
|
||||
anchors = []
|
||||
for nid in doc0.preorder():
|
||||
anchors.append(AN.make_anchor(doc0, vals0, nid, 0, store="smithlab"))
|
||||
if doc0.nodes[nid]["kind"] in LEAVES:
|
||||
t = hashing.plain(doc0, nid)
|
||||
if len(t) > 2:
|
||||
i = rng.randrange(len(t) - 1)
|
||||
j = rng.randint(i + 1, len(t))
|
||||
anchors.append(AN.make_anchor(doc0, vals0, nid, 0, span=[i, j], store="smithlab"))
|
||||
for _ in range(rounds // 4):
|
||||
a = copy.deepcopy(rng.choice(anchors))
|
||||
a["node"] = letter_first(os.urandom(32), 7) if rng.random() < 0.5 else a["node"]
|
||||
a.pop("kind", None)
|
||||
anchors.append(a)
|
||||
for state in (final, history[len(history) // 2], history[min(1, len(history) - 1)]):
|
||||
values = hashing.compute(state.doc())
|
||||
for a in anchors:
|
||||
try:
|
||||
r = AN.resolve(state, a)
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("anchors", False, "resolve raised %r on %s" % (e, json.dumps(a)[:120]))
|
||||
continue
|
||||
ok("anchors", r["result"] in outcomes, "unknown outcome %s" % r.get("result"))
|
||||
seen_outcomes.add(r["result"])
|
||||
if r["result"] != "stale" or r.get("node"):
|
||||
ok("anchors", r.get("node") in state.doc().nodes, "a non-stale result names no live node")
|
||||
if r["result"] == "current":
|
||||
ok("anchors", values[r["node"]]["node_hash"] == a["node_hash"], "current with another node_hash")
|
||||
if "span" in r:
|
||||
t = hashing.plain(state.doc(), r["node"])
|
||||
ok("anchors", 0 <= r["span"][0] <= r["span"][1] <= len(t), "a span outside the node's text")
|
||||
ok("anchors", {"current", "stale"} <= seen_outcomes, "the random walk produced only %s" % sorted(seen_outcomes))
|
||||
|
||||
|
||||
def _merkle_reference(items):
|
||||
leaves = [sha256_hex(b"\x00" + jcs.dumpb(x)) for x in items]
|
||||
|
||||
def mth(ls):
|
||||
if len(ls) == 1:
|
||||
return ls[0]
|
||||
k = 1
|
||||
while 2 * k < len(ls):
|
||||
k *= 2
|
||||
return sha256_hex(b"\x01" + bytes.fromhex(mth(ls[:k])) + bytes.fromhex(mth(ls[k:])))
|
||||
return mth(leaves)
|
||||
|
||||
|
||||
def stress_merkle(rng, rounds):
|
||||
"""Receipt items are lists, `["node", id, hash]` or `[kind, record id, hash]` (spec/08 §8.5)."""
|
||||
for _ in range(rounds):
|
||||
n = rng.randint(1, 40)
|
||||
items = [[rng.choice(["node", "annotation", "digitization", "checklist"]), letter_first(os.urandom(32), 7),
|
||||
sha256_hex(os.urandom(8))] for _ in range(n)]
|
||||
if rng.random() < 0.3:
|
||||
items.append([rand_text(rng), [rand_text(rng)], rng.randint(0, 2**53)])
|
||||
ok("merkle", crypto.merkle_root(items) == _merkle_reference(items), "Merkle roots differ for %d items" % len(items))
|
||||
|
||||
|
||||
def stress_spotcheck(rng, rounds):
|
||||
for _ in range(rounds):
|
||||
n = rng.randint(1, 60)
|
||||
ids = sorted({letter_first(os.urandom(32), 7) for _ in range(n)})
|
||||
receipt = {"format": "scienting-receipt/1", "nodes": [[i, sha256_hex(i)] for i in ids]}
|
||||
seed = RC.make_seed(receipt, os.urandom(32))
|
||||
rate, minimum = rng.choice(["0.05", "0.1", "0.5", "1"]), rng.randint(1, 15)
|
||||
sample = RC.sample(receipt, seed, rate, minimum)
|
||||
try:
|
||||
RC.sample(receipt, os.urandom(64), rate, minimum)
|
||||
ok("spot-check", False, "random bytes were accepted as a seed")
|
||||
except ValueError:
|
||||
pass
|
||||
size = RC.sample_size(len(ids), rate, minimum)
|
||||
ok("spot-check", len(sample) == size == min(len(ids), max(minimum, -(-Fraction(rate) * len(ids) // 1))),
|
||||
"sample size %d is not the rule's for n=%d" % (len(sample), len(ids)))
|
||||
ok("spot-check", sample == sorted(sample) and set(sample) <= set(ids), "sample is not sorted or not from the receipt")
|
||||
shuffled = {"nodes": rng.sample(receipt["nodes"], len(receipt["nodes"]))}
|
||||
ok("spot-check", RC.sample(shuffled, seed, rate, minimum) == sample, "the sample depends on input order")
|
||||
|
||||
|
||||
def stress_bounds(rng, rounds):
|
||||
alpha = Fraction(5, 100)
|
||||
for _ in range(rounds // 4):
|
||||
n = rng.randint(1, 120)
|
||||
k = rng.randint(0, n)
|
||||
b = RC.bound_string(k, n)
|
||||
ok("bounds", len(b.split(".")[1]) == 6, "bound has not six decimals: %s" % b)
|
||||
if k >= n:
|
||||
ok("bounds", b == "1.000000", "k = n must give 1")
|
||||
continue
|
||||
p = Fraction(b)
|
||||
|
||||
def tail(q):
|
||||
return sum(comb(n, i) * q**i * (1 - q)**(n - i) for i in range(k + 1))
|
||||
ok("bounds", tail(p) <= alpha, "the tail at the bound is above alpha for k=%d n=%d" % (k, n))
|
||||
if p > 0:
|
||||
ok("bounds", tail(p - Fraction(1, 10**6)) > alpha, "the bound is not the least six-decimal value for k=%d n=%d" % (k, n))
|
||||
if k + 1 < n:
|
||||
ok("bounds", Fraction(RC.bound_string(k + 1, n)) >= p, "the bound is not monotone in k")
|
||||
|
||||
|
||||
def stress_fingerprints(rng, rounds):
|
||||
rules = json.load(open(os.path.join(ROOT, "registry", "stamps", "scienting-stamps-1.json"), encoding="utf-8"))["rules"]
|
||||
for _ in range(rounds):
|
||||
lines = [{"text": rand_text(rng, 1, 40)} for _ in range(rng.randint(1, 8))]
|
||||
page = {"size": [612.0, 792.0], "lines": lines, "images": [sha256_hex(rand_text(rng)) for _ in range(rng.randint(0, 2))]}
|
||||
stamped = copy.deepcopy(page)
|
||||
stamped["lines"].insert(rng.randint(0, len(lines)),
|
||||
{"text": "Downloaded via EXAMPLE UNIV on %s" % rand_text(rng, 1, 10, list("abcXYZ0123456789-:"))})
|
||||
fp, removed = FP.fingerprint(page, rules)
|
||||
fp2, removed2 = FP.fingerprint(stamped, rules)
|
||||
ok("fingerprints", fp == fp2 and len(removed2) == len(removed) + 1, "a stamped download has another fingerprint")
|
||||
digit_lines = [i for i, l in enumerate(lines) if any(ch in "0123456789" for ch in l["text"])]
|
||||
if digit_lines:
|
||||
changed = copy.deepcopy(page)
|
||||
i = rng.choice(digit_lines)
|
||||
t = changed["lines"][i]["text"]
|
||||
j = next(j for j, ch in enumerate(t) if ch in "0123456789")
|
||||
changed["lines"][i]["text"] = t[:j] + str((int(t[j]) + 1) % 10) + t[j + 1:]
|
||||
ok("fingerprints", FP.fingerprint(changed, rules)[0] != fp, "a changed digit kept the fingerprint")
|
||||
spaced = copy.deepcopy(page)
|
||||
spaced["lines"] = [{"text": " ".join(l["text"])} for l in spaced["lines"]]
|
||||
ok("fingerprints", FP.fingerprint(spaced, rules)[0] == fp, "inserted spaces changed the fingerprint")
|
||||
|
||||
|
||||
def stress_zip(rng, rounds):
|
||||
src = os.path.join(ROOT, "fixtures", "packages", TINY_KEY + PK.ZIP_EXTENSION)
|
||||
with open(src, "rb") as f:
|
||||
good = f.read()
|
||||
accepted = 0
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
for i in range(rounds):
|
||||
data = bytearray(good)
|
||||
mode = rng.random()
|
||||
if mode < 0.6:
|
||||
for _ in range(rng.randint(1, 3)):
|
||||
data[rng.randrange(len(data))] = rng.randrange(256)
|
||||
elif mode < 0.8:
|
||||
data = data[:rng.randrange(len(data))]
|
||||
else:
|
||||
data = data[:rng.randrange(len(data))] + os.urandom(rng.randint(1, 64)) + data[rng.randrange(len(data)):]
|
||||
path = os.path.join(tmp, "z%d.zip" % i)
|
||||
with open(path, "wb") as f:
|
||||
f.write(data)
|
||||
try:
|
||||
rep = PK.validate_zip(path, STORE)
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("zip", False, "validate_zip raised %r on mutation %d" % (e, i))
|
||||
continue
|
||||
accepted += bool(rep)
|
||||
dest = os.path.join(tmp, "d%d" % i)
|
||||
try:
|
||||
PK.read_zip(path, dest)
|
||||
except Refusal as e:
|
||||
ok("zip", e.rule in RULES, "read_zip names the unregistered rule %s" % e.rule)
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("zip", False, "read_zip raised %r on mutation %d" % (e, i))
|
||||
stray = [f for f in os.listdir(tmp) if not (f.startswith("z") and f.endswith(".zip")) and not f.startswith("d")]
|
||||
ok("zip", not stray, "the zip reader wrote outside its folder: %s" % stray)
|
||||
ok("zip", accepted <= rounds // 20 + 1, "%d of %d mutated archives were accepted" % (accepted, rounds))
|
||||
|
||||
|
||||
def stress_packages(rng, rounds):
|
||||
src = os.path.join(ROOT, "fixtures", "packages", TINY_KEY)
|
||||
files = sorted(os.path.relpath(os.path.join(d, f), src) for d, _, fs in os.walk(src) for f in fs)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
for i in range(rounds):
|
||||
pkg = os.path.join(tmp, "p%d" % i)
|
||||
shutil.copytree(src, pkg)
|
||||
mode = rng.choice(["flip", "flip", "truncate", "delete", "add", "rename-case", "symlink"])
|
||||
target = rng.choice(files)
|
||||
full = os.path.join(pkg, target)
|
||||
if mode == "flip":
|
||||
data = bytearray(open(full, "rb").read())
|
||||
if data:
|
||||
k = rng.randrange(len(data))
|
||||
data[k] = (data[k] + rng.randrange(1, 256)) % 256
|
||||
open(full, "wb").write(data)
|
||||
elif mode == "truncate":
|
||||
data = open(full, "rb").read()
|
||||
open(full, "wb").write(data[:rng.randrange(len(data) + 1)])
|
||||
elif mode == "delete":
|
||||
os.remove(full)
|
||||
elif mode == "add":
|
||||
open(os.path.join(pkg, rand_text(rng, 1, 8, list("abcxyz")) + ".txt"), "wb").write(b"stray\n")
|
||||
elif mode == "rename-case":
|
||||
if target.lower() == target.upper():
|
||||
continue
|
||||
os.rename(full, os.path.join(pkg, target.upper()))
|
||||
else:
|
||||
os.remove(full)
|
||||
os.symlink(os.path.join(src, target), full)
|
||||
try:
|
||||
rep = PK.validate(pkg, STORE)
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("packages", False, "validate raised %r after %s of %s" % (e, mode, target))
|
||||
continue
|
||||
ok("packages", not rep, "a package mutated by %s of %s validated" % (mode, target))
|
||||
ok("packages", all(e["rule"] in RULES for e in rep.errors), "an unregistered rule was reported")
|
||||
shutil.rmtree(pkg)
|
||||
|
||||
|
||||
def stress_rights(rng, rounds):
|
||||
tri = [True, False, "unknown"]
|
||||
keys = list(RI.USES) + ["ai/" + p for p in RI.AI_PAIRS]
|
||||
for _ in range(rounds):
|
||||
flats = [{k: rng.choice(tri) for k in keys} for _ in range(rng.randint(1, 4))]
|
||||
combined = RI.combine(flats)
|
||||
shuffled = RI.combine(rng.sample(flats, len(flats)))
|
||||
ok("rights", combined == shuffled, "combining depends on order")
|
||||
for k in keys:
|
||||
vals = [f[k] for f in flats]
|
||||
want = False if False in vals else ("unknown" if "unknown" in vals else True)
|
||||
ok("rights", combined[k] == want, "combining %s gave %r for %r" % (k, combined[k], vals))
|
||||
|
||||
|
||||
def _align_items(rng, prefix, texts):
|
||||
items = []
|
||||
for i, t in enumerate(texts):
|
||||
runs = [{"kind": "text", "text": t}]
|
||||
doc, s = sentence_doc(runs)
|
||||
strict, cx = hashing.strict(doc, s)
|
||||
items.append({"id": "%s%d" % (prefix, i), "container": "p%d" % (i // 3), "context": "paragraph", "runs": runs,
|
||||
"strict": strict, "lossy": hashing.lossy_from_strict(strict), "complex": cx})
|
||||
return items
|
||||
|
||||
|
||||
def stress_alignment(rng, rounds):
|
||||
for _ in range(rounds // 4):
|
||||
texts = [rand_text(rng, 5, 60) or "x" for _ in range(rng.randint(1, 12))]
|
||||
a = _align_items(rng, "a", texts)
|
||||
try:
|
||||
same = AL.align(a, _align_items(rng, "b", texts))
|
||||
b_texts = [t for t in texts if rng.random() > 0.3] + [rand_text(rng, 5, 30)]
|
||||
diff = AL.align(a, _align_items(rng, "b", b_texts))
|
||||
ok("alignment", AL.align(a, _align_items(rng, "b", b_texts)) == diff, "alignment is not deterministic")
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("alignment", False, "align raised %r" % (e,))
|
||||
continue
|
||||
ok("alignment", len(same["pairs"]) == len(a) and all(p["agree"] for p in same["pairs"])
|
||||
and not [s for s in same["suspects"] if s["kind"] not in ("structure", "join")],
|
||||
"aligning two equal readings left disagreements or unpaired sentences: %s" % same["suspects"][:3])
|
||||
ids_a = {x["id"] for x in a}
|
||||
ok("alignment", all(p["a"] in ids_a for p in diff["pairs"]), "a pair names an unknown sentence")
|
||||
paired = {p["a"] for p in diff["pairs"]}
|
||||
dropped = {s["a"] for s in diff["suspects"] if s["kind"] == "dropped"}
|
||||
ok("alignment", paired | dropped == ids_a and not (paired & dropped),
|
||||
"every sentence of a reading is paired or dropped, not both or neither")
|
||||
|
||||
|
||||
def stress_addresses(rng, rounds):
|
||||
for _ in range(rounds):
|
||||
store = rng.choice([None, "smithlab", "a1", "x.y-z"])
|
||||
key = rng.choice([HALE, "k-0000", "a1-aaaa.si", "x-k3f9qd.v2"])
|
||||
node = letter_first(os.urandom(32), 7)
|
||||
pin = sha256_hex(rand_text(rng))[:6] if rng.random() < 0.5 else None
|
||||
span = sorted([rng.randint(0, 50), rng.randint(0, 50)]) if rng.random() < 0.5 else None
|
||||
s = AN.format_address(store, key, node, pin, span)
|
||||
parsed = AN.parse_address(s)
|
||||
ok("addresses", parsed and AN.format_address(parsed["store"], parsed["key"], parsed["node"], parsed["pin"],
|
||||
parsed["span"]) == s, "address does not round trip: %s" % s)
|
||||
junk = rand_text(rng, 0, 20)
|
||||
try:
|
||||
AN.parse_address(junk)
|
||||
except Exception as e: # noqa: BLE001
|
||||
ok("addresses", False, "parse_address raised %r on %r" % (e, junk))
|
||||
for _ in range(rounds):
|
||||
d = os.urandom(32)
|
||||
ok("ids", NODE_ID_RE.match(letter_first(d, 7)) and RECORD_ID_RE.match(record_id(d.hex())), "ID form")
|
||||
|
||||
|
||||
def stress_targets(rng, rounds):
|
||||
st = PK.read_state(os.path.join(ROOT, "fixtures", "packages", HALE))
|
||||
doc = st.doc()
|
||||
for _ in range(rounds // 4):
|
||||
anchored = rng.sample(doc.order, rng.randint(0, 6))
|
||||
sets = {}
|
||||
for rule in TG.DIGIT_RULES:
|
||||
got = TG.target_set(doc, anchored, rule)
|
||||
ids = [n for n, _ in got]
|
||||
pos = {nid: i for i, nid in enumerate(doc.order)}
|
||||
ok("targets", ids == sorted(ids, key=pos.get) and len(set(ids)) == len(ids), "targets are not in document order")
|
||||
ok("targets", all(doc.nodes[n]["kind"] in LEAVES for n in ids), "a target is not a leaf")
|
||||
sets[rule] = set(ids)
|
||||
ok("targets", sets["none"] <= sets["anchored-sections"] <= sets["all"], "the digit rules do not nest")
|
||||
try:
|
||||
TG.target_set(doc, anchored + ["zzzzzzz"], "all")
|
||||
ok("targets", False, "an unknown anchored node was accepted")
|
||||
except ValueError:
|
||||
ok("targets", True, "")
|
||||
|
||||
|
||||
def stress_receipt_chains(rng, rounds):
|
||||
"""Chains of spot-checks hundreds deep settle without recursion, and alternate as spec/08 §8.6 implies."""
|
||||
depth = max(50, rounds)
|
||||
ids = sorted({letter_first(os.urandom(32), 7) for _ in range(20)})
|
||||
rec = RC.seal({"format": "scienting-receipt/1", "doc": HALE, "rev": 0, "level": "F3", "scope": "all",
|
||||
"nodes": [[i, sha256_hex(i)] for i in ids], "evidence": [], "by": "ana",
|
||||
"at": "2026-10-11T12:00:00Z"}, secret("ana"), "ana")
|
||||
records = [rec]
|
||||
for i in range(depth):
|
||||
orig = records[-1]
|
||||
seed = RC.make_seed(orig, secret("ben-spot"))
|
||||
sample = RC.sample(orig, seed)
|
||||
hashes = dict(orig["nodes"])
|
||||
q = RC.seal({"format": "scienting-receipt/1", "doc": HALE, "rev": 0, "level": "F3",
|
||||
"scope": "qa:" + orig["id"], "nodes": [[n, hashes[n]] for n in sample], "evidence": [],
|
||||
"by": "ben", "at": "2026-10-11T12:%02d:%02dZ" % (i // 60 % 60, i % 60),
|
||||
"qa": {"of": orig["id"], "key": "ben-spot", "seed": __import__("base64").b64encode(seed).decode(),
|
||||
"rate": "0.05", "minimum": 10, "sample": sample, "errors": [{"node": sample[0]}]}},
|
||||
secret("ben"), "ben")
|
||||
records.append(q)
|
||||
t0 = time.time()
|
||||
try:
|
||||
status = RC.statuses(records, ROSTER_BY_ID)
|
||||
except RecursionError:
|
||||
ok("receipts", False, "a chain of %d spot-checks overflowed the stack" % depth)
|
||||
return
|
||||
ok("receipts", time.time() - t0 < 30, "statuses of a %d-deep chain took too long" % depth)
|
||||
# The last spot-check nobody checks is valid; each receipt it checks is suspended; and so on, alternating.
|
||||
want = "valid"
|
||||
for r in reversed(records):
|
||||
ok("receipts", status[r["id"]] == want, "receipt at depth %d is %s, not %s" % (records.index(r), status[r["id"]], want))
|
||||
want = "suspended" if want == "valid" else "valid"
|
||||
|
||||
|
||||
def stress_keys(rng, rounds):
|
||||
"""Keys (spec/02 §2.4): the pattern always holds, the suffix is a prefix of longer ones, lengthening settles a
|
||||
collision without touching the first key, and a key's hash part depends on the identity alone."""
|
||||
words = ["Attention", "On", "the", "Origin", "of", "Species", "H\u2082O", "3D", "Self-consistent", "\u00c9tude", "\u5f20", "don't", "a"]
|
||||
for _ in range(rounds):
|
||||
meta = {"title": " ".join(rng.choice(words) for _ in range(rng.randint(0, 6))),
|
||||
"author": [{"family": rng.choice(["M\u00fcller", "O'Brien", "\u5f20\u4f1f", "", "de la Cruz", "Smith Jr."])}] if rng.random() < 0.8 else [],
|
||||
"issued": {"date-parts": [[rng.choice([2017, 1859, 99, 20261, None])]]} if rng.random() < 0.8 else {}}
|
||||
same_as = [rng.choice(["doi:10.1/X.Y", "doi:10.1/x.y", "arxiv:1706.03762", "url:https://e.org/p", "openalex:W1"])] if rng.random() < 0.6 else []
|
||||
key = K.work_key(same_as, meta)
|
||||
ok("keys", K.WORK_KEY_RE.match(key) and K.DOCUMENT_KEY_RE.match(key), "a minted key breaks the pattern: %r" % key)
|
||||
ident = K.identity(same_as, meta)
|
||||
ok("keys", K.suffix(ident, 8).startswith(K.suffix(ident)), "a longer suffix does not extend the shorter")
|
||||
longer = K.unique_work_key(same_as, meta, {key})
|
||||
ok("keys", longer != key and longer.startswith(key) and len(longer) == len(key) + 2, "lengthening does not extend the key")
|
||||
if same_as and same_as[0].lower().startswith("doi:"):
|
||||
other = dict(meta, title="Another title entirely")
|
||||
ok("keys", K.suffix_of(K.work_key(same_as, other)) == K.suffix_of(key), "the hash part depends on more than the identity")
|
||||
for v in ({"type": "version-of-record"}, {"type": "supplement"}, {"type": "preprint", "seq": rng.randint(1, 9)}, {"type": "erratum"}):
|
||||
dk = K.document_key(key, v)
|
||||
ok("keys", K.DOCUMENT_KEY_RE.match(dk) and K.split_key(dk)[0] == key, "a document key does not split back")
|
||||
|
||||
|
||||
def stress_vrf(rng, rounds):
|
||||
"""The VRF proves and verifies for random keys and inputs, and a single flipped byte never verifies."""
|
||||
for _ in range(max(5, rounds // 20)):
|
||||
sk = os.urandom(32)
|
||||
pk = vrf.public_key(sk)
|
||||
alpha = os.urandom(rng.randint(0, 40))
|
||||
pi = vrf.prove(sk, alpha)
|
||||
beta = vrf.verify(pk, alpha, pi)
|
||||
ok("vrf", beta is not None and beta == vrf.proof_to_hash(pi) and len(pi) == 80 and len(beta) == 64, "a proof does not verify")
|
||||
ok("vrf", vrf.prove(sk, alpha) == pi, "proving is not deterministic")
|
||||
bad = bytearray(pi)
|
||||
bad[rng.randrange(80)] ^= 1 << rng.randrange(8)
|
||||
ok("vrf", vrf.verify(pk, alpha, bytes(bad)) is None, "a tampered proof verifies")
|
||||
ok("vrf", vrf.verify(pk, alpha + b"x", pi) is None and vrf.verify(vrf.public_key(os.urandom(32)), alpha, pi) is None,
|
||||
"a proof verifies for another input or key")
|
||||
ok("vrf", vrf.verify(pk, alpha, os.urandom(80)) is None and vrf.verify(os.urandom(32), alpha, pi) is None, "garbage verifies")
|
||||
|
||||
|
||||
def stress_proofs(rng, rounds):
|
||||
"""RFC 9162 inclusion and consistency proofs verify for every index and prefix of random trees, and fail for a
|
||||
wrong index, root or size."""
|
||||
for _ in range(rounds // 4):
|
||||
n = rng.randint(1, 40)
|
||||
items = [["node", letter_first(os.urandom(32), 7), sha256_hex(os.urandom(4))] for _ in range(n)]
|
||||
root = crypto.history_root(items)
|
||||
for i in range(n):
|
||||
path = crypto.inclusion_path(items, i)
|
||||
leaf = crypto.leaf_hash(items[i])
|
||||
ok("proofs", crypto.verify_inclusion(root, leaf, i, n, path), "inclusion fails for %d of %d" % (i, n))
|
||||
ok("proofs", n == 1 or not crypto.verify_inclusion(root, leaf, (i + 1) % n, n, path), "inclusion passes for another index")
|
||||
ok("proofs", not crypto.verify_inclusion(sha256_hex(b"x"), leaf, i, n, path), "inclusion passes for another root")
|
||||
# A wrong size is not always detectable from a path: RFC 9162 commits to the content through the
|
||||
# root, and two sizes with one tree shape verify alike. The root is what a verifier must trust.
|
||||
for m in range(n + 1):
|
||||
proof = crypto.consistency_proof(items, m)
|
||||
ok("proofs", crypto.verify_consistency(crypto.history_root(items[:m]), root, m, n, proof), "consistency fails %d to %d" % (m, n))
|
||||
if 0 < m < n:
|
||||
ok("proofs", not crypto.verify_consistency(sha256_hex(b"x"), root, m, n, proof)
|
||||
and not crypto.verify_consistency(crypto.history_root(items[:m]), sha256_hex(b"y"), m, n, proof),
|
||||
"a consistency proof passes with a wrong root")
|
||||
ok("proofs", not crypto.verify_consistency(crypto.history_root(items[:1]), root, 1, n, ["zz"]), "garbage path verifies")
|
||||
|
||||
|
||||
def stress_checkpoints(rng, rounds):
|
||||
"""Checkpoint notes parse back to what was signed and reject edits anywhere."""
|
||||
for _ in range(rounds // 10):
|
||||
entries = [[rng.choice(["receipt", "revocation"]), letter_first(os.urandom(32), 13), sha256_hex(os.urandom(4))]
|
||||
for _ in range(rng.randint(0, 20))]
|
||||
note = LG.sign_checkpoint("scient.ing/log/t", entries, [("ana", secret("ana")), ("ben", secret("ben"))])
|
||||
parsed, verified = LG.verify_checkpoint(note, ROSTER_BY_ID)
|
||||
ok("checkpoints", parsed and parsed["size"] == len(entries) and parsed["root"] == LG.root(entries) and verified == ["ana", "ben"],
|
||||
"a checkpoint does not verify")
|
||||
k = rng.randrange(len(note))
|
||||
bad = note[:k] + chr((ord(note[k]) + 1) % 128 or 1) + note[k + 1:]
|
||||
p2, v2 = LG.verify_checkpoint(bad, ROSTER_BY_ID)
|
||||
# An edit in the body loses every signer; an edit in one signature line loses that signer.
|
||||
ok("checkpoints", p2 is None or len(v2) < len(verified), "an edited note still fully verifies: %r" % bad[max(0, k - 5):k + 5])
|
||||
|
||||
|
||||
def stress_mldsa(rng, rounds):
|
||||
"""ML-DSA-65 signs deterministically and verifies; a flipped bit anywhere, another message or another key fails."""
|
||||
for _ in range(max(3, rounds // 50)):
|
||||
seed = os.urandom(32)
|
||||
pk, sk = mldsa.keygen(seed)
|
||||
msg = os.urandom(rng.randint(0, 100))
|
||||
sig = mldsa.sign(sk, msg)
|
||||
ok("ml-dsa", len(pk) == mldsa.PK_SIZE and len(sk) == mldsa.SK_SIZE and len(sig) == mldsa.SIG_SIZE, "sizes")
|
||||
ok("ml-dsa", mldsa.verify(pk, msg, sig) and mldsa.sign(sk, msg) == sig, "a signature does not verify or is not deterministic")
|
||||
bad = bytearray(sig)
|
||||
bad[rng.randrange(len(bad))] ^= 1 << rng.randrange(8)
|
||||
ok("ml-dsa", not mldsa.verify(pk, msg, bytes(bad)), "a tampered signature verifies")
|
||||
ok("ml-dsa", not mldsa.verify(pk, msg + b"x", sig) and not mldsa.verify(mldsa.keygen(os.urandom(32))[0], msg, sig),
|
||||
"a signature verifies for another message or key")
|
||||
ok("ml-dsa", not mldsa.verify(pk, msg, os.urandom(mldsa.SIG_SIZE)) and not mldsa.verify(os.urandom(mldsa.PK_SIZE), msg, sig),
|
||||
"garbage verifies")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--seed", type=int, default=20261014, help="the random seed (default 20261014)")
|
||||
ap.add_argument("--rounds", type=int, default=200, help="cases per property (default 200)")
|
||||
a = ap.parse_args()
|
||||
rng = random.Random(a.seed)
|
||||
print("seed %d, %d rounds" % (a.seed, a.rounds))
|
||||
stress_canonical_json(rng, a.rounds)
|
||||
stress_normalization(rng, a.rounds)
|
||||
stress_notation(rng, a.rounds)
|
||||
history = stress_corrections(rng, a.rounds)
|
||||
stress_anchors(rng, history, a.rounds)
|
||||
stress_merkle(rng, a.rounds)
|
||||
stress_spotcheck(rng, a.rounds)
|
||||
stress_bounds(rng, a.rounds)
|
||||
stress_fingerprints(rng, a.rounds)
|
||||
stress_zip(rng, a.rounds)
|
||||
stress_packages(rng, a.rounds // 4)
|
||||
stress_rights(rng, a.rounds)
|
||||
stress_alignment(rng, a.rounds)
|
||||
stress_addresses(rng, a.rounds)
|
||||
stress_targets(rng, a.rounds)
|
||||
stress_receipt_chains(rng, a.rounds)
|
||||
stress_keys(rng, a.rounds)
|
||||
stress_vrf(rng, a.rounds)
|
||||
stress_proofs(rng, a.rounds)
|
||||
stress_checkpoints(rng, a.rounds)
|
||||
stress_mldsa(rng, a.rounds)
|
||||
for k in sorted(COUNTS):
|
||||
print("%-17s %6d checks" % (k, COUNTS[k]))
|
||||
if FAILS:
|
||||
print("\n%d failures (seed %d):" % (len(FAILS), a.seed))
|
||||
for f in FAILS[:60]:
|
||||
print(" " + f)
|
||||
sys.exit(1)
|
||||
print("\nEvery property held.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,91 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate packages, or run the correction gate on proposals against a package.
|
||||
|
||||
Non-normative tooling.
|
||||
Usage: python3 tools/validate.py PACKAGE [PACKAGE ...]
|
||||
python3 tools/validate.py --correction PROPOSALS PACKAGE
|
||||
A package is a folder, or a zipped package (`.sct.zip`), which is unpacked under the rules of spec/12 §12.2
|
||||
before it is checked. Each is checked as spec/22 §22.2 describes, and each failure printed as
|
||||
`package: where: rule: message`.
|
||||
With --correction, PROPOSALS is a JSON file holding one proposed correction, or a JSON Lines file of several
|
||||
applied in order. Each runs through the gate (spec/11 §11.4) against the package's current state, and the record
|
||||
it would append to corrections.jsonl is printed, until one is refused. Nothing is written.
|
||||
Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, ROOT)
|
||||
import tools # noqa: E402,F401 checks the runtime and puts src/ first on the path, before paperspec is imported
|
||||
|
||||
from paperspec import corrections, jcs, package # noqa: E402
|
||||
from paperspec.schema import Store # noqa: E402
|
||||
from paperspec.tree import Refusal # noqa: E402
|
||||
|
||||
|
||||
def read_proposals(path):
|
||||
with open(path, encoding="utf-8") as f:
|
||||
text = f.read()
|
||||
try:
|
||||
return [json.loads(text)]
|
||||
except json.JSONDecodeError:
|
||||
return [json.loads(ln) for ln in text.splitlines() if ln.strip()]
|
||||
|
||||
|
||||
def validate_one(path, store, limit):
|
||||
"""The validation report of a package folder or a zipped package."""
|
||||
if package.is_zip(path):
|
||||
return package.validate_zip(path, store, limit=limit)
|
||||
return package.validate(path, store)
|
||||
|
||||
|
||||
def main():
|
||||
lines = __doc__.splitlines()
|
||||
ap = argparse.ArgumentParser(description=lines[0], epilog="\n".join(lines[5:]),
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--correction", metavar="PROPOSALS",
|
||||
help="a JSON or JSON Lines file of proposed corrections to run through the gate")
|
||||
ap.add_argument("--max-size", metavar="BYTES", type=int, default=package.ZIP_LIMIT,
|
||||
help="refuse a zipped package that expands past this many bytes (default %d)" % package.ZIP_LIMIT)
|
||||
ap.add_argument("packages", nargs="+", metavar="PACKAGE", help="package folders or zipped packages")
|
||||
a = ap.parse_args()
|
||||
if a.correction and len(a.packages) != 1:
|
||||
ap.error("--correction checks proposals against exactly one package")
|
||||
store, bad = Store(os.path.join(ROOT, "schemas")), False
|
||||
for p in a.packages:
|
||||
for e in validate_one(p, store, a.max_size).errors:
|
||||
print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
|
||||
bad = True
|
||||
if bad:
|
||||
return 1
|
||||
if not a.correction:
|
||||
for p in a.packages:
|
||||
print("%s: valid" % p)
|
||||
return 0
|
||||
pkg = a.packages[0]
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
if package.is_zip(pkg):
|
||||
pkg = package.read_zip(pkg, tmp, a.max_size)
|
||||
state = package.read_state(pkg)
|
||||
if any(n.get("withheld") for n in state.nodes):
|
||||
print("%s: doc.policy.withheld: a package that withholds its text takes no corrections" % a.packages[0],
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
for i, proposal in enumerate(read_proposals(a.correction), 1):
|
||||
try:
|
||||
state, record = corrections.gate(state, proposal, store)
|
||||
except Refusal as e:
|
||||
where = "%s:%d" % (a.correction, i) + (" " + e.node if e.node else "")
|
||||
print("%s: %s: %s" % (where, e.rule, e.message), file=sys.stderr)
|
||||
return 1
|
||||
print(jcs.dumps(record))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user