Publish the package and its documentation, and move to the papers organization
Add the Zensical documentation of the Python package (docs/package, built by the docs workflow), merge the standard publish workflow with the setuptools-scm tag check, add the SPDX header rule to ruff, restructure the README, and point every repository and registry address at the Gitea organization papers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B8cL4zN4XpaQF5cBtASzfq
This commit is contained in:
37 files changed
+1005
-83
No files matched your search
+1
-1
@@ -4,7 +4,7 @@ A newcomer, such as a student who knows what a scientific paper and a citation a
|
||||
An implementer, who will write a reader, writer, converter or checker, gets the chapter that defines each behavior and the places where the formats differ from a neighbor in ways that cause bugs.
|
||||
An expert, such as a researcher, standards author or reviewer who knows the neighboring standards, gets stated limits, the status of every format, and where the other standard is the better choice.
|
||||
|
||||
The repository is at <https://git.scient.ing/infra/paperspec>.
|
||||
The repository is at <https://git.scient.ing/papers/paperspec>.
|
||||
This page was checked against the repository, at the version given under "Limits and status".
|
||||
Every statement about another standard links to the page it was read on, and those pages were read on 2026-10-12, so versions and dates are as of that day and may have moved.
|
||||
Where the checked sources say nothing, this page says "not seen", "not specified" or "not in scope", and each phrase keeps one meaning: "not in scope" means the standard's own stated purpose excludes the feature, "not seen" means it was not found in the pages read (which is not proof it is absent), and "not specified" means the page read does not define it.
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ The provisional ones are the project, bundle, dataset, digitization, view, snaps
|
||||
|
||||
Format identifiers such as `scienting-anchor/1` and `scienting-correction/1` begin with `scienting-`, which is the formats' namespace.
|
||||
The repository's name, paperspec, is a separate matter, and the formats are named after what they hold.
|
||||
The repository is at <https://git.scient.ing/infra/paperspec>.
|
||||
The repository is at <https://git.scient.ing/papers/paperspec>.
|
||||
|
||||
## Hashes and canonical JSON
|
||||
|
||||
|
||||
+3
-3
@@ -7,7 +7,7 @@ The last sections point implementers (anyone writing a reader, writer, converter
|
||||
This page makes no comparison with the neighboring standards.
|
||||
The formats are version 0.1.0, and each is `candidate` or `provisional` in `registry/formats.json`.
|
||||
|
||||
Run every command from the root of the repository, <https://git.scient.ing/infra/paperspec>.
|
||||
Run every command from the root of the repository, <https://git.scient.ing/papers/paperspec>.
|
||||
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
|
||||
Every command also works without pixi once the checkout is installed with `pip install -e ".[tools]"` on Python 3.14, given standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
|
||||
|
||||
@@ -19,10 +19,10 @@ This shows how to confirm that the repository's specification, fixtures and test
|
||||
pixi run check
|
||||
```
|
||||
|
||||
`pixi run check` lints the Markdown, then runs `paperspec check`, the repository's reference checker.
|
||||
`pixi run check` checks the Python code and lints the Markdown, then runs `paperspec check`, the repository's reference checker.
|
||||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||||
`pixi run quick`, or `paperspec check --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
|
||||
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `paperspec check` does not run it.
|
||||
Without pixi, the code checks and the Markdown lint (`markdownlint-cli2`) are separate steps; `paperspec check` does not run them.
|
||||
|
||||
## Recompute a hash by hand
|
||||
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
This page answers questions about paperspec, a set of file formats that give every sentence and table cell of a scientific paper a permanent address.
|
||||
Newcomers, implementers of a reader, writer, converter or checker, and experts who know JATS (Journal Article Tag Suite), TEI (Text Encoding Initiative) or W3C Web Annotations can all use it.
|
||||
Each answer gives the plain statement first, then the precise one, then the chapter that defines it.
|
||||
The repository is <https://git.scient.ing/infra/paperspec>, and the formats are at version 0.1.0.
|
||||
The repository is <https://git.scient.ing/papers/paperspec>, and the formats are at version 0.1.0.
|
||||
|
||||
## Can I read these files without special software?
|
||||
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
# Glossary
|
||||
|
||||
A scientific paper changes after it is published: errata appear, readings get corrected, annotations pile up. paperspec is a set of file formats for scientific papers that keeps each sentence findable through those changes (repository: <https://git.scient.ing/infra/paperspec>).
|
||||
A scientific paper changes after it is published: errata appear, readings get corrected, annotations pile up. paperspec is a set of file formats for scientific papers that keeps each sentence findable through those changes (repository: <https://git.scient.ing/papers/paperspec>).
|
||||
This glossary defines its terms for readers from newcomer to expert, implementers included.
|
||||
Each entry gives the plain meaning, then the precise one, then the chapter that defines it.
|
||||
|
||||
|
||||
+1
-1
@@ -25,7 +25,7 @@ It cannot give you a signed statement that a named person checked a passage, IDs
|
||||
The formats supply those three, and they are worth most at the scale of one team.
|
||||
|
||||
paperspec is a set of file formats for scientific papers that have been converted, split into addressable pieces and checked.
|
||||
The repository is at <https://git.scient.ing/infra/paperspec>.
|
||||
The repository is at <https://git.scient.ing/papers/paperspec>.
|
||||
It is version 0.1.0 and defines 32 formats; in [registry/formats.json](../registry/formats.json), 19 are `candidate` and 13 are `provisional`.
|
||||
|
||||
## Who it is for
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# Changelog
|
||||
|
||||
The changelog is one file, `CHANGELOG.md` at the root of the repository, so there is a single copy to keep in sync.
|
||||
It is published with the format documentation at <https://paperspec.scient.ing/CHANGELOG.html>, and the source is at <https://git.scient.ing/papers/paperspec/src/branch/main/CHANGELOG.md>.
|
||||
|
||||
Its top-level headings are the versions of the formats, and a release of the Python package that changes only the code is a subsection under the current format version, as `### 0.1.1` sits under `## 0.1.0`.
|
||||
The package version comes from the git tag; the format version is the one `CHANGELOG.md` and `pixi.toml` state.
|
||||
When a release is tagged, the publish workflow takes the release notes from the section whose heading contains the version.
|
||||
@@ -0,0 +1,36 @@
|
||||
# About
|
||||
|
||||
## Status
|
||||
|
||||
The formats are at version 0.1.0.
|
||||
Of the 32 formats, 19 are `candidate` and 13 are `provisional`; `registry/formats.json` in the repository lists each with its media type, schema, status and chapter.
|
||||
The Python package is at an alpha stage (the `Development Status :: 3 - Alpha` classifier).
|
||||
Its version comes from git tags, so a tag `v0.1.1` builds as 0.1.1 and later commits as 0.1.2.devN.
|
||||
|
||||
## Design commitments
|
||||
|
||||
- **The code defines the expected values.**
|
||||
No expected value in a vector or fixture is typed by hand: the oracle computes it, and `paperspec check` rebuilds everything and compares it byte for byte.
|
||||
- **Standard library first.**
|
||||
Every module imports with the standard library alone.
|
||||
Schema validation and the Ed25519 and HKDF functions import `jsonschema`, `referencing` and `cryptography` on first use, through the package's extras.
|
||||
- **Clarity over speed.**
|
||||
The oracle is a slow, plain program.
|
||||
The verifiable random function (RFC 9381) and ML-DSA-65 (FIPS 204) are written out in Python and checked against those standards' own test vectors.
|
||||
- **One Python.**
|
||||
The specification pins the Unicode 16.0 character data that normalization and the character classes come from, so the package refuses to import on a Python with another version.
|
||||
Python 3.14 is the only one that matches.
|
||||
|
||||
## Known limits and unfinished parts
|
||||
|
||||
This documentation describes what the code does.
|
||||
Where an older guide, a doc comment or the `--help` text says something different, the code wins, and this section lists the differences.
|
||||
|
||||
- The package is an oracle, not a production reader or writer.
|
||||
It has not been measured for speed or for large corpora; the formats are built for a team-scale corpus of hundreds of documents.
|
||||
- Only `validate` and `load` work on an installed package.
|
||||
`check`, `build`, `stress` and `site` need a checkout of the repository.
|
||||
- `paperspec validate --correction` reports a proposals file that is not valid JSON with a Python traceback, not a one-line message.
|
||||
- The Python API has no generated reference yet: [the API page](../reference/api.md) is written by hand from the code's docstrings.
|
||||
- The package is not on the studio's registry until a version tag is pushed. `[CHECK: no tag exists yet]`
|
||||
- Thirteen formats are `provisional` and may still change.
|
||||
@@ -0,0 +1,14 @@
|
||||
# License
|
||||
|
||||
paperspec is not under the studio's usual license.
|
||||
Because the formats are meant to be implemented by anyone, in any product, without asking, each path of the repository keeps the license that `LICENSE.md` gives it.
|
||||
|
||||
| Paths | License | SPDX |
|
||||
| --- | --- | --- |
|
||||
| `spec/`, `docs/`, `README.md`, `CONTRIBUTING.md`, `CHANGELOG.md` and the other prose | Creative Commons Attribution 4.0 International | `CC-BY-4.0` |
|
||||
| `schemas/`, `sql/`, `context/`, `registry/`, `vectors/`, `fixtures/` | Creative Commons Zero 1.0 Universal | `CC0-1.0` |
|
||||
| `src/` (the `paperspec` package and its commands), `tests/`, `tools/` | Apache License 2.0 | `Apache-2.0` |
|
||||
|
||||
Python source files open with an `SPDX-License-Identifier: Apache-2.0` header, which ruff checks.
|
||||
All text in the fixtures is synthetic, and the keys in `fixtures/keys/` are public test keys that must never sign anything else.
|
||||
The full text is in `LICENSE.md` at the root of the repository, <https://git.scient.ing/papers/paperspec/src/branch/main/LICENSE.md>.
|
||||
@@ -0,0 +1,14 @@
|
||||
# LLM assistance
|
||||
|
||||
Large language models (Anthropic's Claude, through Claude Code) write much of this code, its tests and its documentation, and we do not hide it.
|
||||
The repository is built so that this is safe to rely on.
|
||||
|
||||
- **A person reviews every change.** A model proposes a commit; it does not decide one.
|
||||
- **Checks gate the code.** `pixi run check` runs the formatter, the linter with the full rule set, the type checker, the Markdown lint, the pytest suite and the repository's own checker, `paperspec check`.
|
||||
CI runs the same, with branch coverage that fails under 90%, and the stress test.
|
||||
- **The oracle checks itself.** Every expected value in the vectors and fixtures is computed by the code and rebuilt byte for byte, so a change that alters a computed value shows up as a changed file in the diff.
|
||||
- **Assisted commits are marked.** Each carries a `Co-Authored-By` trailer naming the model, and a `Claude-Session` trailer linking the session.
|
||||
- **Claude's working material stays out of the repository.** Plans, notes and review reports live in an untracked `work/` folder.
|
||||
|
||||
The specification is the authority.
|
||||
Where a model's code and a chapter disagree, the chapter wins and the code is the bug.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Check and rebuild the repository
|
||||
|
||||
Four commands work on a checkout of the paperspec repository, not on an installed package: `check`, `build`, `stress` and `site`.
|
||||
The checkout is the one an editable install points at, or else the one around the current folder.
|
||||
|
||||
## Check everything
|
||||
|
||||
```sh
|
||||
paperspec check
|
||||
```
|
||||
|
||||
The checker validates every schema, record and package, confirms that every broken package is refused for the stated reason, recomputes every test vector, rebuilds everything and compares it byte for byte, loads every package into SQLite and back, and checks the prose, the links and the website.
|
||||
Each section prints how many checks it ran, and a passing run ends with `All checks pass.`
|
||||
The exit status is 0 when every check passed and 1 otherwise.
|
||||
|
||||
`paperspec check --quick` skips the byte-for-byte rebuild, the slowest part.
|
||||
The command does not run the Python code checks or the Markdown lint; `pixi run check` runs those first ([Install](../start/install.md#the-pixi-tasks)).
|
||||
|
||||
## Rebuild the vectors and fixtures
|
||||
|
||||
No expected value in the repository is typed by hand.
|
||||
The vectors and fixtures are computed from the hand-written sources in `fixtures/src/` and the cases the builder defines in `src/paperspec/cli/build.py`.
|
||||
After you change a rule in the code, a fixture source or a case, rebuild and commit the regenerated files with the change:
|
||||
|
||||
```sh
|
||||
paperspec build
|
||||
```
|
||||
|
||||
`paperspec build --out DIR` builds elsewhere, so you can compare with `diff -r DIR/vectors vectors` before overwriting anything.
|
||||
The build is deterministic, and `paperspec check` proves it by rebuilding into a temporary folder and comparing every file.
|
||||
|
||||
## Stress the oracle
|
||||
|
||||
```sh
|
||||
paperspec stress --rounds 5
|
||||
```
|
||||
|
||||
The vectors pin values; `stress` checks the invariants the chapters state, such as canonical JSON round trips and the zip reader never writing outside its folder, on random inputs from a seeded generator.
|
||||
A failure prints its seed, so `paperspec stress --seed SEED` reproduces it.
|
||||
The defaults are the seed 20261014 and 200 cases per property.
|
||||
Before a release, run it with the default seed and with one or two others.
|
||||
|
||||
## Build the website
|
||||
|
||||
```sh
|
||||
paperspec site --out /tmp/paperspec-site
|
||||
```
|
||||
|
||||
The site mirrors the repository: each file is served at its own path, each Markdown file gets a rendered page beside it, and the schemas are at the addresses in their `$id`.
|
||||
It needs only the standard library.
|
||||
The pages you are reading are a separate site, built with Zensical by `pixi run -e docs docs-build`.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Guides
|
||||
|
||||
Each guide takes one task from start to finish, with commands that were run against the repository's synthetic fixtures.
|
||||
|
||||
| Guide | For |
|
||||
| --- | --- |
|
||||
| [Validate a package](validate-a-package.md) | Check a package folder or a zipped package against every rule, and try a proposed correction without writing it. |
|
||||
| [Load packages into SQLite](load-into-sqlite.md) | Put packages in a database you can query, or build a bundle with full-text search. |
|
||||
| [Use the library](use-the-library.md) | Compute hashes and identifiers, read a package and run the gate from Python. |
|
||||
| [Check and rebuild the repository](check-and-build.md) | Run the repository's checks, regenerate the vectors and fixtures, stress the oracle and build the website. |
|
||||
|
||||
The formats themselves are explained in the guides at <https://paperspec.scient.ing/docs/>.
|
||||
@@ -0,0 +1,46 @@
|
||||
# Load packages into SQLite
|
||||
|
||||
`paperspec load` validates packages and loads them into a SQLite database with the tables of `sql/doc.sql` and `sql/store.sql`.
|
||||
A node, a reference, a correction and a receipt each get rows you can join and query.
|
||||
|
||||
## Load a package
|
||||
|
||||
```sh
|
||||
paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
||||
```
|
||||
|
||||
```text
|
||||
hale2026harmonic-damb: 75 nodes
|
||||
```
|
||||
|
||||
- Every package is validated first, and nothing is loaded if any fails.
|
||||
Each failure is printed as `package: where: rule: message`, as `paperspec validate` prints it.
|
||||
- A new file gets the tables, and an existing one takes more packages, so you can add papers to the same database later.
|
||||
- Files with no table of their own (assets, datasets, derived files) are copied by SHA-256 into a folder beside the database, `lab.db.assets/`, which must travel with it.
|
||||
- The full and checks-only forms of one document share its key, so load them into separate databases.
|
||||
|
||||
## Query it
|
||||
|
||||
```sh
|
||||
python -m sqlite3 lab.db "SELECT kind, COUNT(*) FROM nodes GROUP BY kind ORDER BY 2 DESC, kind LIMIT 3"
|
||||
```
|
||||
|
||||
```text
|
||||
('cell', 23)
|
||||
('sentence', 18)
|
||||
('row', 6)
|
||||
```
|
||||
|
||||
The guide at <https://paperspec.scient.ing/docs/examples.html> has queries to start from: the sentences still below fidelity F2, the citation graph at sentence level, and who signed what.
|
||||
|
||||
## Build a bundle with full-text search
|
||||
|
||||
```sh
|
||||
paperspec load --sqlite search.db --bundle --built-from REV paper/ other/
|
||||
```
|
||||
|
||||
A bundle is a `scienting-bundle/1` file: it also gets the tables of `sql/bundle-sqlite.sql`, the full-text search index and `bundle_meta`.
|
||||
It is built once, into a file that does not exist yet.
|
||||
`--bundle` needs `--built-from`, which names the store revision the packages come from, and the command prints the number of search rows it added.
|
||||
`--store` names the store, and defaults to the packages' common store.
|
||||
`--built-from` and `--store` are refused without `--bundle`.
|
||||
@@ -0,0 +1,66 @@
|
||||
# Use the library
|
||||
|
||||
The modules of `paperspec` compute every value the formats define, so a program that writes or reads packages can import them instead of reimplementing the specification.
|
||||
Every module imports with the standard library alone.
|
||||
Schema validation needs `paperspec[schema]`, and Ed25519 and HKDF need `paperspec[crypto]`; `paperspec[tools]` installs both ([Install](../start/install.md)).
|
||||
|
||||
## Canonical JSON and hashes
|
||||
|
||||
`jcs.dumps` writes the canonical JSON the formats hash and store: sorted keys, no spaces, and numbers written as an ECMAScript engine would.
|
||||
`text.sha256_hex` hashes a string, and `text.norm` applies the normalization the five hashes start from.
|
||||
|
||||
```python
|
||||
from paperspec import jcs, text
|
||||
|
||||
print(jcs.dumps({"b": 1.0, "a": [2, "é"]}))
|
||||
print(text.norm("a b\n c"))
|
||||
print(text.sha256_hex("The two parameter sets agree to within 2% for every bond."))
|
||||
```
|
||||
|
||||
```text
|
||||
{"a":[2,"é"],"b":1}
|
||||
a b c
|
||||
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a
|
||||
```
|
||||
|
||||
## Validate a package
|
||||
|
||||
`package.validate` checks a package folder against every rule and returns a report whose `errors` list is empty when the package is valid.
|
||||
The schema store reads the schemas from wherever the package carries them, in an installed wheel or at the root of a checkout.
|
||||
|
||||
```python
|
||||
from paperspec import package, resources
|
||||
from paperspec.schema import Store
|
||||
|
||||
store = Store(resources.path("schemas"))
|
||||
report = package.validate("fixtures/packages/hale2026harmonic-damb", store)
|
||||
print(bool(report), report.errors)
|
||||
```
|
||||
|
||||
```text
|
||||
True []
|
||||
```
|
||||
|
||||
Each entry of `errors` is a dictionary with `rule`, `where` and `message`, the three fields `paperspec validate` prints.
|
||||
|
||||
## Read a package and run the gate
|
||||
|
||||
`package.read_state` reads a package folder into a state that holds its nodes, and `corrections.gate` runs a proposed correction against it without changing it.
|
||||
It returns the state the correction would produce and the record it would append, or raises `tree.Refusal`, which names the rule broken.
|
||||
|
||||
```python
|
||||
from paperspec import corrections, package, resources
|
||||
from paperspec.schema import Store
|
||||
|
||||
state = package.read_state("fixtures/packages/tiny2026tiny-j4qm")
|
||||
store = Store(resources.path("schemas"))
|
||||
new_state, record = corrections.gate(state, proposal, store)
|
||||
```
|
||||
|
||||
Here `proposal` is a parsed correction such as the one in [Validate a package](validate-a-package.md).
|
||||
|
||||
## Check your own code against the vectors
|
||||
|
||||
The vectors travel with the package as the data subpackage `paperspec.vectors`, and `resources.path("vectors")` finds the folder in an installed wheel and in an editable install alike.
|
||||
Four of them, `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl`, name a fixture package, and the fixtures stay in the repository.
|
||||
The [reference](../reference/api.md) lists the modules.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Validate a package
|
||||
|
||||
A package is a folder, or a zipped package (`.sct.zip`), that holds one document.
|
||||
`paperspec validate` checks it against every rule of the specification's conformance chapter and works anywhere the package is installed with its `tools` extra.
|
||||
|
||||
## Check one or several packages
|
||||
|
||||
```sh
|
||||
paperspec validate fixtures/packages/hale2026harmonic-damb
|
||||
paperspec validate fixtures/packages/tiny2026tiny-j4qm.sct.zip fixtures/packages/tiny2026tiny-j4qm
|
||||
```
|
||||
|
||||
Each valid package prints `PACKAGE: valid` and the exit status is 0.
|
||||
A failure goes to standard error as `package: where: rule: message`, the exit status is 1, and nothing else prints.
|
||||
A usage error, such as no package at all, exits with status 2.
|
||||
|
||||
A zipped package is unpacked under the specification's path rules before anything else happens: a path that leaves the folder, a symbolic link and an oversize archive are refused.
|
||||
`--max-size BYTES` sets the largest size an archive may expand to; the default is 1 GiB.
|
||||
|
||||
## Read a refusal
|
||||
|
||||
The rule ID names the rule that failed.
|
||||
`https://paperspec.scient.ing/registry/rules.json` lists every ID with the chapter of the specification that states it, which is where to read why.
|
||||
The folder `fixtures/invalid/` holds one package for each of many rules; its `expect.json` says which rules the package must trigger.
|
||||
|
||||
## Try a correction before making it
|
||||
|
||||
A correction changes one node of a document and records why.
|
||||
Before a tool writes one, it can run the proposal through the gate, which needs only the package's current state:
|
||||
|
||||
```sh
|
||||
paperspec validate --correction fix.json fixtures/packages/tiny2026tiny-j4qm
|
||||
```
|
||||
|
||||
`fix.json` holds one proposal as a JSON object, or several as JSON Lines, applied in order:
|
||||
|
||||
```json
|
||||
{"format":"scienting-correction/1","op":"emend","target":"j663d4h","find":"second","replace":"next","kind":"wording","status":"proposed","evidence":[{"kind":"person","by":"ana","note":"The editor prefers a plainer word."}],"author":{"key":"ana"},"reason":"The authors misuse the ordinal; the paragraph follows the first.","at":"2026-10-03T10:00:00Z"}
|
||||
```
|
||||
|
||||
The command prints the record the correction would append to `corrections.jsonl`, with its computed `index`, `id`, `path`, `before` and `after`, and writes nothing.
|
||||
A refused proposal prints `file:N node: rule: message` on standard error and stops: with this proposal, changing `second` to `2nd` is refused with `correction.digits`, because an emendation changes no digit.
|
||||
`--correction` takes exactly one package, and it refuses a package that withholds its text.
|
||||
@@ -0,0 +1,22 @@
|
||||
# paperspec
|
||||
|
||||
paperspec is the reference implementation of the scienting formats, the files in which a team stores its scientific papers once, corrects them once and addresses them down to the sentence.
|
||||
This site documents the Python package `paperspec` and its command.
|
||||
The formats themselves, with the specification, schemas, test vectors and fixtures, are published at <https://paperspec.scient.ing>.
|
||||
|
||||
The package is an oracle: a plain program whose answers count as the expected ones.
|
||||
Every hash, identifier, canonical JSON string, package check and database form in the repository is computed by this code, and no expected value in a test vector is typed by hand.
|
||||
It favors clarity over speed, and it is not a production reader or writer.
|
||||
|
||||
## Who it is for
|
||||
|
||||
- **A developer writing a reader, writer, converter or store** installs the package to validate their output against every rule, and to compute the values their own code must match.
|
||||
- **A researcher or editor with a package** uses the `paperspec` command to validate it, to try a proposed correction without writing it, and to load it into SQLite and query it.
|
||||
- **A maintainer of the formats** uses the command to rebuild the vectors and fixtures, and to check the whole repository.
|
||||
|
||||
## Where to go next
|
||||
|
||||
- [Get started](start/index.md): install the package and run a first command.
|
||||
- [Guides](guides/index.md): validate a package, load it into SQLite, use the library from Python, and check and rebuild the repository.
|
||||
- [Reference](reference/index.md): every command and option, and the library's modules.
|
||||
- [About](about/index.md): status, design commitments and known limits.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Python API
|
||||
|
||||
The package `paperspec` is a set of small modules, one for each part of the specification, and each opens with a docstring that says what it is for.
|
||||
This page is written by hand from those docstrings and lists the modules and the functions most programs call; `help(paperspec.MODULE)` has the rest.
|
||||
The base package needs only the standard library.
|
||||
`schema` needs `paperspec[schema]`, and the Ed25519 and HKDF functions of `crypto` need `paperspec[crypto]`; both import their packages on first use.
|
||||
|
||||
Two aliases are shared by the type annotations: `jsontypes.Record`, one parsed JSON object, and `jsontypes.StrPath`, a path given as text or as a path object.
|
||||
A function that takes a path returns text unless its docstring says otherwise.
|
||||
A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the rule as the specification's conformance chapter lists it.
|
||||
|
||||
`paperspec.resources.path(name)` returns the folder `schemas`, `sql` or `vectors`, in an installed wheel and in a checkout alike.
|
||||
|
||||
## Text, hashes and canonical JSON
|
||||
|
||||
| Module | For | Main functions |
|
||||
| --- | --- | --- |
|
||||
| `text` | Normalization, IDs and small helpers shared by the rest. | `norm`, `sha256_hex`, `collapse_ws`, `letter_first`, `mint_node_id`, `record_id`, `cp_to_utf16` |
|
||||
| `jcs` | Canonical JSON, RFC 8785. | `dumps`, `dumpb`, `loads_line` |
|
||||
| `hashing` | The plain, strict and lossy projections of a node, and its five content values. | `compute`, `plain`, `strict`, `lossy`, `own_content`, `meta_hash`, `Hasher`, `Index` |
|
||||
| `notation` | Simple math notation and its canonical linear form. | `fold_math`, `complex_key`, `fold_plain_text` |
|
||||
| `keys` | Work and document keys. | `work_key`, `unique_work_key`, `document_key`, `split_key` |
|
||||
|
||||
## The document
|
||||
|
||||
| Module | For | Main functions |
|
||||
| --- | --- | --- |
|
||||
| `model` | The node tree of `scienting-doc/1`. | `Doc`, `anchor_class`, the tables of allowed children and attributes |
|
||||
| `tree` | Establishing a document from a nested tree, and the structural checks. | `establish`, `mint_ids`, `validate_structure`, `validate_references`, `Refusal` |
|
||||
| `anchors` | Addresses, anchors and their resolution. | `parse_address`, `format_address`, `make_anchor`, `resolve` |
|
||||
| `align` | Alignment of two readings of one paper. | `align`, `classify`, `similarity` |
|
||||
| `fingerprint` | Page fingerprints and download-stamp rules. | `fingerprint`, `is_stamp` |
|
||||
| `targets` | The target set of targeted F3. | `target_set` |
|
||||
|
||||
## Packages, corrections and the database
|
||||
|
||||
| Module | For | Main functions |
|
||||
| --- | --- | --- |
|
||||
| `package` | Writing, reading and validating a package folder or a zipped package. | `validate`, `validate_zip`, `read_state`, `write`, `write_zip`, `read_zip`, `read_json`, `read_jsonl` |
|
||||
| `corrections` | The correction gate and the state it works on. | `gate`, `apply`, `State` |
|
||||
| `schema` | JSON Schema validation with the keyword subset the schemas are restricted to. | `Store`, `validate`, `lint` |
|
||||
| `db` | The database form. | `create`, `create_bundle`, `load`, `unload`, `index` |
|
||||
|
||||
## Signatures, receipts and the log
|
||||
|
||||
| Module | For | Main functions |
|
||||
| --- | --- | --- |
|
||||
| `crypto` | Keyed hashing, Merkle roots and proofs, Ed25519, HKDF and `did:key`. | `merkle_root`, `inclusion_path`, `verify_inclusion`, `consistency_proof`, `verify_consistency`, `ed25519_sign`, `ed25519_verify`, `did_key` |
|
||||
| `receipts` | Receipts, revocations, levels and spot-check sampling. | `seal`, `verify_signature`, `statuses`, `effective_levels`, `sample`, `upper_bound` |
|
||||
| `log` | The team's transparency log and its signed checkpoints. | `sign_checkpoint`, `verify_checkpoint`, `write_log`, `read_log` |
|
||||
| `vrf` | ECVRF-EDWARDS25519-SHA512-TAI, RFC 9381. | `prove`, `verify`, `proof_to_hash` |
|
||||
| `mldsa` | ML-DSA-65, FIPS 204, in plain Python. | `keygen`, `sign`, `verify` |
|
||||
|
||||
## Layers, rights and grading
|
||||
|
||||
| Module | For | Main functions |
|
||||
| --- | --- | --- |
|
||||
| `rights` | Rights records, policies and stage checks. | `effective`, `combine`, `check`, `backend_allowed` |
|
||||
| `traces` | Traced citations. | `resolve`, `problems`, `disputed` |
|
||||
| `structures` | Chemical structures tied to words. | `scan`, `problems` |
|
||||
| `sequences` | Biological sequences tied to words. | `residues`, `problems` |
|
||||
| `data` | Data tied to words. | `problems` |
|
||||
| `grade` | Grading a submission against an exercise key. | `grade`, `key_anchors` |
|
||||
|
||||
The `problems` functions of the four annotation modules return the rules an annotation breaks, as a list of `{"rule", "message"}` records in the order checked.
|
||||
|
||||
## The command
|
||||
|
||||
`paperspec.cli` holds the subcommands of the `paperspec` command, one module each, with a `run(args)` function that returns the exit status; see [Command line](cli.md).
|
||||
`cli.checkout()` finds the repository root that `check`, `build`, `stress` and `site` work on.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Command line
|
||||
|
||||
One command, `paperspec`, with six subcommands.
|
||||
`python -m paperspec` runs the same program.
|
||||
`paperspec --version` prints the package version and the Unicode version it pins, and `paperspec COMMAND --help` prints a command's options, advice and examples.
|
||||
|
||||
| Command | Works | Does |
|
||||
| --- | --- | --- |
|
||||
| [`validate`](#validate) | anywhere the package is installed with its `tools` extra | Validate packages, or run proposed corrections through the gate. |
|
||||
| [`load`](#load) | anywhere the package is installed with its `tools` extra | Validate packages and load them into a SQLite database. |
|
||||
| [`check`](#check) | in a checkout | Check the whole repository. |
|
||||
| [`build`](#build) | in a checkout | Rebuild `fixtures/` and `vectors/` from their sources. |
|
||||
| [`stress`](#stress) | in a checkout | Stress the oracle with random inputs. |
|
||||
| [`site`](#site) | in a checkout | Build the website into `_site/`. |
|
||||
|
||||
A misspelled command gets a suggestion, and an option a command does not know is refused.
|
||||
"In a checkout" means the repository an editable install points at, or else the one around the current folder.
|
||||
|
||||
## validate
|
||||
|
||||
```text
|
||||
paperspec validate [-h] [--correction PROPOSALS] [--max-size BYTES] PACKAGE [PACKAGE ...]
|
||||
```
|
||||
|
||||
| Option | Meaning |
|
||||
| --- | --- |
|
||||
| `PACKAGE` | A package folder or a zipped package (`.sct.zip`), unpacked under the specification's path rules first. |
|
||||
| `--correction PROPOSALS` | A JSON file holding one proposed correction, or a JSON Lines file of several, run through the gate against the one package given. Each record that would be appended to `corrections.jsonl` is printed, until one is refused. Nothing is written. |
|
||||
| `--max-size BYTES` | Refuse a zipped package that expands past this many bytes (default 1073741824). |
|
||||
|
||||
Each failure is printed to standard error as `package: where: rule: message`.
|
||||
Exit status: 0 when everything is valid, 1 when something is refused, 2 for a usage error.
|
||||
|
||||
## load
|
||||
|
||||
```text
|
||||
paperspec load [-h] --sqlite DATABASE [--bundle] [--built-from REV] [--store STORE] [--max-size BYTES] PACKAGE [PACKAGE ...]
|
||||
```
|
||||
|
||||
| Option | Meaning |
|
||||
| --- | --- |
|
||||
| `PACKAGE` | A package folder or a zipped package. Every package is validated first, and nothing is loaded if any fails. |
|
||||
| `--sqlite DATABASE` | The database file, created with the tables of `sql/` if it does not exist. Files with no table of their own are copied into `DATABASE.assets/`. |
|
||||
| `--bundle` | Build a new `scienting-bundle/1` file with the full-text search index. Needs `--built-from`. |
|
||||
| `--built-from REV` | With `--bundle`: the store revision the packages come from. |
|
||||
| `--store STORE` | With `--bundle`: the store's name (default: the packages' common store). |
|
||||
| `--max-size BYTES` | As for `validate`. |
|
||||
|
||||
## check
|
||||
|
||||
```text
|
||||
paperspec check [-h] [--quick]
|
||||
```
|
||||
|
||||
`--quick` skips the byte-for-byte rebuild.
|
||||
Exit status: 0 when every check passed, 1 otherwise.
|
||||
|
||||
## build
|
||||
|
||||
```text
|
||||
paperspec build [-h] [--out DIR]
|
||||
```
|
||||
|
||||
Without `--out` it rewrites `fixtures/` and `vectors/` in the checkout; with it, it builds into `DIR` instead.
|
||||
|
||||
## stress
|
||||
|
||||
```text
|
||||
paperspec stress [-h] [--seed SEED] [--rounds ROUNDS]
|
||||
```
|
||||
|
||||
`--seed` sets the random seed (default 20261014) and `--rounds` the cases per property (default 200).
|
||||
Exit status: 0 when every property held, 1 otherwise.
|
||||
|
||||
## site
|
||||
|
||||
```text
|
||||
paperspec site [-h] [--out DIR] [--base-url URL]
|
||||
```
|
||||
|
||||
`--out` sets the output folder, which must not be the repository or contain it (default `_site/` in the checkout).
|
||||
`--base-url` sets the site's address for canonical links (default `https://paperspec.scient.ing`).
|
||||
@@ -0,0 +1,6 @@
|
||||
# Reference
|
||||
|
||||
- [Command line](cli.md): the `paperspec` command, its six subcommands and every option.
|
||||
- [Python API](api.md): the modules of the package and what each is for.
|
||||
|
||||
The formats the code implements are specified at <https://paperspec.scient.ing/spec/>, and the JSON Schemas, SQL and test vectors are at the same address.
|
||||
@@ -0,0 +1,9 @@
|
||||
# Get started
|
||||
|
||||
Two pages take you from nothing to a result.
|
||||
|
||||
1. [Install](install.md) the package from the studio's package registry, or work from a checkout.
|
||||
2. [Quick start](quick-start.md) validates the example paper and loads it into a database.
|
||||
|
||||
The package needs Python 3.14.
|
||||
It is the only version with the Unicode 16.0 character data the specification pins, and the package refuses to import on any other.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Install
|
||||
|
||||
The package needs Python 3.14.
|
||||
Its base install has no dependencies: every module imports with the standard library alone.
|
||||
Three extras add what some modules need:
|
||||
|
||||
| Extra | Adds | Needed by |
|
||||
| --- | --- | --- |
|
||||
| `schema` | `jsonschema`, `referencing` | schema validation, so `package.validate` and `corrections.gate` |
|
||||
| `crypto` | `cryptography` | Ed25519 signatures and HKDF in `crypto`, which the signature and receipt checks use |
|
||||
| `tools` | both of the above | the `paperspec` command, and any program that validates packages |
|
||||
|
||||
## From the studio's package registry
|
||||
|
||||
The registry does not mirror PyPI, so an installer needs two indexes: the registry for `paperspec`, and PyPI for `jsonschema`, `referencing` and `cryptography`.
|
||||
Tie `paperspec` to the registry wherever the installer allows it, so that a package of the same name on PyPI is never chosen instead.
|
||||
`[CHECK: no version tag has been pushed yet, so the registry holds no release until the first tag is published]`
|
||||
|
||||
With pixi, in `pixi.toml`:
|
||||
|
||||
```toml
|
||||
[pypi-dependencies]
|
||||
paperspec = { version = ">=0.1.1", extras = ["tools"], index = "https://git.scient.ing/api/packages/papers/pypi/simple" }
|
||||
```
|
||||
|
||||
With uv, in `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[[tool.uv.index]]
|
||||
name = "scienting"
|
||||
url = "https://git.scient.ing/api/packages/papers/pypi/simple"
|
||||
explicit = true
|
||||
|
||||
[tool.uv.sources]
|
||||
paperspec = { index = "scienting" }
|
||||
```
|
||||
|
||||
With pip, which cannot tie a package to an index and takes the highest version from either:
|
||||
|
||||
```sh
|
||||
pip install --index-url https://git.scient.ing/api/packages/papers/pypi/simple/ --extra-index-url https://pypi.org/simple/ "paperspec[tools]"
|
||||
```
|
||||
|
||||
If the organization or the package is private, the installer also needs a token with the `read:package` scope, for example in a `~/.netrc` entry for `git.scient.ing`.
|
||||
|
||||
## From a checkout
|
||||
|
||||
Clone the repository and let pixi set up a pinned environment, with the package installed editable:
|
||||
|
||||
```sh
|
||||
git clone https://git.scient.ing/papers/paperspec
|
||||
pixi install
|
||||
pixi run check
|
||||
```
|
||||
|
||||
`pixi run check` runs the formatter, the linter, the type checker, the Markdown lint, the tests and the repository's own checker, and ends with `All checks pass.`
|
||||
Without pixi, install the checkout with `pip install -e ".[tools]" --group dev` on Python 3.14.
|
||||
|
||||
## The pixi tasks
|
||||
|
||||
| Task | Does |
|
||||
| --- | --- |
|
||||
| `check` | Every check CI runs, in order: `fmt-check`, `lint`, `typecheck`, `lint-md`, `test`, `verify`. |
|
||||
| `verify` | `paperspec check`, the repository's own checker. |
|
||||
| `quick` | `paperspec check --quick`, which skips the byte-for-byte rebuild. |
|
||||
| `fmt`, `fmt-check` | Format the code with ruff, or check the formatting. |
|
||||
| `lint`, `lint-fix` | Lint with ruff, or lint and apply the safe fixes. |
|
||||
| `typecheck` | Type-check `src/` with ty. |
|
||||
| `test`, `cov` | Run the pytest suite, or run it with branch coverage that fails under 90%. |
|
||||
| `lint-md` | Lint every Markdown file with markdownlint. |
|
||||
| `build`, `load`, `validate`, `stress`, `site` | The commands of the same names. |
|
||||
| `hooks` | Install the git pre-commit hooks. |
|
||||
| `docs-build`, `docs-serve` | Build these pages strictly into `build/docs`, or preview them. |
|
||||
| `dist` | Build the sdist and wheel into `dist/`. |
|
||||
@@ -0,0 +1,59 @@
|
||||
# Quick start
|
||||
|
||||
This page validates the repository's example paper, shows what a refusal looks like, and loads the paper into a database.
|
||||
Run every command from the root of a checkout (see [Install](install.md)); the example is a synthetic chemistry paper in `fixtures/packages/hale2026harmonic-damb`.
|
||||
|
||||
## Validate a package
|
||||
|
||||
```sh
|
||||
paperspec validate fixtures/packages/hale2026harmonic-damb
|
||||
```
|
||||
|
||||
```text
|
||||
fixtures/packages/hale2026harmonic-damb: valid
|
||||
```
|
||||
|
||||
The exit status is 0.
|
||||
A broken package prints one line per failure, as `package: where: rule: message`, and exits with status 1.
|
||||
The folder `fixtures/invalid/` holds packages that each break one named rule:
|
||||
|
||||
```sh
|
||||
paperspec validate fixtures/invalid/text-edited/package
|
||||
```
|
||||
|
||||
```text
|
||||
fixtures/invalid/text-edited/package: s0fn2fj: doc.hash.text-hash: text_hash does not recompute
|
||||
fixtures/invalid/text-edited/package: s0fn2fj: doc.hash.match-hash: match_hash does not recompute
|
||||
…
|
||||
```
|
||||
|
||||
Each rule ID, such as `doc.hash.text-hash`, is listed in <https://paperspec.scient.ing/registry/rules.json> with the chapter of the specification that states it.
|
||||
|
||||
## Load it into SQLite
|
||||
|
||||
```sh
|
||||
paperspec load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
||||
```
|
||||
|
||||
```text
|
||||
hale2026harmonic-damb: 75 nodes
|
||||
```
|
||||
|
||||
The command validates the package first, creates `lab.db` with the tables of `sql/`, and loads the package.
|
||||
Query the result with any SQLite client; Python's own works:
|
||||
|
||||
```sh
|
||||
python -m sqlite3 lab.db "SELECT kind, COUNT(*) FROM nodes GROUP BY kind ORDER BY 2 DESC, kind LIMIT 3"
|
||||
```
|
||||
|
||||
```text
|
||||
('cell', 23)
|
||||
('sentence', 18)
|
||||
('row', 6)
|
||||
```
|
||||
|
||||
## Where next
|
||||
|
||||
- [Validate a package](../guides/validate-a-package.md), including how to try a correction without writing it.
|
||||
- [Load packages into SQLite](../guides/load-into-sqlite.md).
|
||||
- [Use the library](../guides/use-the-library.md) from Python.
|
||||
+1
-1
@@ -10,7 +10,7 @@ Experts find the cost of each choice and, where a neighboring standard chose dif
|
||||
|
||||
The formats are at version 0.1.0.
|
||||
Of the 29 in `registry/formats.json`, 19 are `candidate` and 13 are `provisional`, and the provisional ones are named where they appear.
|
||||
The repository is paperspec, at <https://git.scient.ing/infra/paperspec>.
|
||||
The repository is paperspec, at <https://git.scient.ing/papers/paperspec>.
|
||||
The `scienting-` prefix in format names such as `scienting-anchor/1` is the formats' namespace, separate from the repository's name.
|
||||
|
||||
## Identity
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ Its problems are modeled on the failures that the specification's rules address.
|
||||
Three readers can use the guide, and each gets something different.
|
||||
A newcomer knows what a paper and a citation are, not hashes or JSON, so every term is defined where it first appears.
|
||||
An implementer who will write a reader, writer, converter or checker gets the chapter and test vector for each step.
|
||||
An expert gets the status: [paperspec](https://git.scient.ing/infra/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (19 candidate, 13 provisional).
|
||||
An expert gets the status: [paperspec](https://git.scient.ing/papers/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (19 candidate, 13 provisional).
|
||||
The `scienting-` prefix on format names, as in `scienting-doc/1`, is the formats' namespace.
|
||||
It has nothing to do with the repository's name.
|
||||
|
||||
|
||||
Reference in new issue
Block a user