Publish the package and its documentation, and move to the papers organization

Add the Zensical documentation of the Python package (docs/package, built by the docs
workflow), merge the standard publish workflow with the setuptools-scm tag check, add the
SPDX header rule to ruff, restructure the README, and point every repository and registry
address at the Gitea organization papers.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B8cL4zN4XpaQF5cBtASzfq
This commit is contained in:
alexmandClaude committed 2026-10-19 04:23:49 -04:00
1 parent bdbf03ebee
commit 30d310f64d
37 files changed
+1005 -83

No files matched your search

+138 -42
View File
@@ -6,7 +6,7 @@ This page is for three readers.
A newcomer should read straight through; each term is explained where it first appears.
The tables below point an implementer, who will write a reader, writer, converter or checker, to the chapters, test vectors and fixtures that define exact behavior.
The sections on status and on differences give an expert in the neighboring standards the version, the maturity of each format and the comparison.
The documentation and every file of the specification are published at <https://paperspec.scient.ing>, and the repository is at <https://git.scient.ing/infra/paperspec>.
The specification and every file of the formats are published at <https://paperspec.scient.ing>, the documentation of the Python package is at <https://docs.scient.ing/paperspec/>, and the repository is at <https://git.scient.ing/papers/paperspec>.
A research team usually cares about a few dozen papers it understands completely, not the world's literature.
Those papers deserve better than a folder of PDFs.
@@ -37,19 +37,6 @@ If a different converter reads the paper, each sentence it reads identically kee
Nothing changes until a correction with its evidence is accepted ([spec/04](spec/04-identity.md), [spec/05 §5.6](spec/05-hashes.md#56-alignment)).
The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and [vectors/addresses.jsonl](vectors/addresses.jsonl) holds valid and invalid examples.
## Start here
| If you want to | Read |
| --- | --- |
| Understand what the project is and how the parts fit | [docs/overview.md](docs/overview.md) |
| Learn the ideas one at a time, each with an example | [docs/concepts.md](docs/concepts.md) |
| Follow one paper from its first reading to a signed document | [docs/walkthrough.md](docs/walkthrough.md) |
| Run something in the next five minutes | [docs/examples.md](docs/examples.md) |
| Implement a reader or writer | [spec/](spec/README.md), then [vectors/](vectors/README.md) |
| Know why a rule is the way it is | [docs/rationale.md](docs/rationale.md) |
| Look up a term | [docs/glossary.md](docs/glossary.md) |
| Compare with neighboring standards | [docs/comparison.md](docs/comparison.md) |
## What is in the repository
| Folder | Holds |
@@ -63,36 +50,103 @@ The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and
| [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. |
| [tools/](tools/README.md) | How to run the commands that build and check everything above. |
| [src/paperspec/](src/paperspec/) | The reference implementation of every rule and those commands, installable as the Python package `paperspec`. |
| [docs/](docs/README.md) | Guides for people getting to know the project. |
| [deploy/](deploy/README.md) | How the website at <https://paperspec.scient.ing> is built and served. |
| [tests/](tests/) | The pytest suite for the reference implementation, which takes its data from the vectors and fixtures. |
| [docs/](docs/README.md) | Guides to the formats for people getting to know the project, and the Python package's documentation in `docs/package/`. |
| [deploy/](deploy/README.md) | How the website at <https://paperspec.scient.ing> is built and served, and how the package is published. |
## Try it
## Installing
With [pixi](https://pixi.sh), one command sets up a pinned environment and checks everything:
The formats need no installation: they are files.
The Python package `paperspec` needs Python 3.14 and, for the `paperspec` command, the `tools` extra (`jsonschema`, `referencing` and `cryptography`).
### From the studio's package registry
The registry does not mirror PyPI, so an installer needs two indexes: the registry for `paperspec`, and PyPI for the packages it depends on.
Tie `paperspec` to the registry wherever the installer allows it, so that a package of the same name on PyPI is never chosen instead.
`[CHECK: no version tag has been pushed, so the registry holds no release yet]`
With pixi, in `pixi.toml`:
```toml
[pypi-dependencies]
paperspec = { version = ">=0.1.1", extras = ["tools"], index = "https://git.scient.ing/api/packages/papers/pypi/simple" }
```
With uv, in `pyproject.toml`:
```toml
[[tool.uv.index]]
name = "scienting"
url = "https://git.scient.ing/api/packages/papers/pypi/simple"
explicit = true
[tool.uv.sources]
paperspec = { index = "scienting" }
```
With pip, which cannot tie a package to an index and takes the highest version from either:
```sh
pip install --index-url https://git.scient.ing/api/packages/papers/pypi/simple/ --extra-index-url https://pypi.org/simple/ "paperspec[tools]"
```
[deploy/](deploy/README.md#install-from-the-registry) says what a private registry needs.
### From a checkout
```sh
git clone https://git.scient.ing/papers/paperspec
pixi install
pixi run check
```
Without pixi, install the checkout as the package with `pip install -e ".[tools]" --group dev` on Python 3.14, then run `paperspec check`.
## Quick start
With [pixi](https://pixi.sh), one command sets up a pinned environment and runs every check:
```sh
pixi run check
```
The checker validates every schema, record and package.
It first checks the Python code: formatting and lint with ruff, types with ty, and the pytest suite.
Then it lints the Markdown.
Last it runs `paperspec check`, the repository's own checker, which you can also run alone with `pixi run verify`.
That checker validates every schema, record and package.
It confirms that every broken package is refused for the stated reason.
It recomputes every test vector, and it rebuilds everything and compares it byte for byte.
Last, it loads every package into SQLite and back, and checks the prose and the links.
Pixi lints the Markdown first.
It also loads every package into SQLite and back, and checks the prose and the links.
The run ends with `All checks pass.`
`pixi task list` shows the other tasks: `quick`, `build`, `lint`, `load`, `validate` and `stress`.
`pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)).
The commands need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), which pixi installs along with markdownlint for the Markdown.
Without pixi, install the checkout as the package with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec check`.
`paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)).
To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from.
To validate the example paper, a synthetic chemistry paper with 12 corrections, and load it into SQLite to query it:
```sh
pixi run validate fixtures/packages/hale2026harmonic-damb
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
```
[docs/examples.md](docs/examples.md) has queries to start from.
`paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)).
`pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)).
## Documentation
The documentation of the Python package and its command is at <https://docs.scient.ing/paperspec/>: how to install it, guides for validating a package, loading it into SQLite, using the library and checking the repository, the command line and the modules.
Its source is in `docs/package/`, and `pixi run docs-build` builds it.
The guides to the formats are at <https://paperspec.scient.ing/docs/>:
| If you want to | Read |
| --- | --- |
| Understand what the project is and how the parts fit | [docs/overview.md](docs/overview.md) |
| Learn the ideas one at a time, each with an example | [docs/concepts.md](docs/concepts.md) |
| Follow one paper from its first reading to a signed document | [docs/walkthrough.md](docs/walkthrough.md) |
| Run something in the next five minutes | [docs/examples.md](docs/examples.md) |
| Implement a reader or writer | [spec/](spec/README.md), then [vectors/](vectors/README.md) |
| Know why a rule is the way it is | [docs/rationale.md](docs/rationale.md) |
| Look up a term | [docs/glossary.md](docs/glossary.md) |
| Compare with neighboring standards | [docs/comparison.md](docs/comparison.md) |
## Format names
Every format is named `scienting-<name>/<major>`, such as `scienting-doc/1` or `scienting-anchor/1`.
@@ -101,19 +155,6 @@ A format is named after what it holds, never after a product or a repository, an
Schemas use the identifier base `https://paperspec.scient.ing/schemas/`.
[registry/formats.json](registry/formats.json) lists every format with its media type, schema, status and chapter.
## Status and limits
This is version 0.1.0.
Of the 32 formats, 19 are `candidate` and 13 are `provisional`.
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint; where those are described, they are marked as such.
The repository includes a reference checker, test vectors and fixtures.
The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
They do not replace publisher interchange formats.
They do not fix what a converter reads wrongly; they record it and make it checkable.
Not every addressable thing is a node with five hashes (the content values `text_hash`, `match_hash`, `align_key`, `own_hash` and `node_hash`, explained in [docs/concepts.md](docs/concepts.md)).
Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](spec/04-identity.md), [spec/09](spec/09-figures-tables-equations.md)).
## How it differs from neighboring standards
- JATS is the XML format in which publishers and archives exchange journal content ([NISO Z39.96-2024](https://www.niso.org/publications/z3996-2024-jats)).
@@ -129,9 +170,64 @@ Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](
It identifies items by position in a list, such as `#/texts/12` ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
Here, IDs do not depend on position, and a second converter's reading is aligned with the first and compared.
## Development
The pixi tasks:
| Task | Does |
| --- | --- |
| `fmt`, `fmt-check` | Format the code with ruff, or check the formatting. |
| `lint`, `lint-fix` | Lint with ruff, or lint and apply the safe fixes. |
| `typecheck` | Type-check `src/` with ty. |
| `test`, `cov` | Run the pytest suite, or run it with branch coverage that fails under 90%. |
| `check` | Every check CI runs, then the repository's own checker (`verify`). |
| `hooks` | Install the git pre-commit hooks. |
| `docs-build`, `docs-serve` | Build the package documentation strictly into `build/docs`, or preview it. |
| `dist` | Build the sdist and wheel into `dist/` (the `release` environment). |
Work happens on the `dev` branch.
A release is a tag `vX.Y.Z` on the commit to release, and the package version is the tag, which setuptools-scm reads.
Add the release's section to [CHANGELOG.md](CHANGELOG.md), tag the commit and push the tag; CI checks the repository, builds the package, confirms the built version is the tag's, publishes it to the package registry and creates the release ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), [deploy/](deploy/README.md#publish-the-python-package)).
## Built with LLM assistance
Large language models (Anthropic's Claude, through Claude Code) write much of this code, its tests and its documentation.
A person reviews every change, `pixi run check` gates it, and each assisted commit carries a `Co-Authored-By` trailer.
## Status and known limits
This is version 0.1.0.
Of the 32 formats, 19 are `candidate` and 13 are `provisional`.
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint; where those are described, they are marked as such.
The repository includes a reference checker, test vectors and fixtures.
The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
They do not replace publisher interchange formats.
They do not fix what a converter reads wrongly; they record it and make it checkable.
Not every addressable thing is a node with five hashes (the content values `text_hash`, `match_hash`, `align_key`, `own_hash` and `node_hash`, explained in [docs/concepts.md](docs/concepts.md)).
Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](spec/04-identity.md), [spec/09](spec/09-figures-tables-equations.md)).
The Python package is an oracle, not a production reader or writer: it favors clarity over speed, and its VRF and ML-DSA-65 are written out in plain Python ([tools/](tools/README.md#about-the-oracle)).
It needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
Only `paperspec validate` and `paperspec load` work anywhere the package is installed; `check`, `build`, `stress` and `site` work on a checkout of this repository.
## Licenses
The specification and other prose are CC BY 4.0, so anyone may implement the formats.
Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0.
The code, the `paperspec` package in `src/` with its commands, is Apache-2.0.
The code, the `paperspec` package in `src/` with its commands and its tests, is Apache-2.0, and its Python files open with an SPDX header that says so.
This repository does not use the studio's Prosperity license, because the formats are meant to be implemented by anyone without asking.
See [LICENSE.md](LICENSE.md).
## Credits
The reference implementation leaves the cryptography and the schema validation to established packages: [cryptography](https://cryptography.io) for Ed25519 and HKDF, and [jsonschema](https://python-jsonschema.dev) with [referencing](https://referencing.readthedocs.io) for JSON Schema.
Canonical JSON follows [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785).
The verifiable random function follows [RFC 9381](https://www.rfc-editor.org/rfc/rfc9381), and ML-DSA-65 follows [FIPS 204](https://csrc.nist.gov/pubs/fips/204/final); both are checked against those standards' own test vectors.
The annotation layers build on the [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/).
[pixi](https://pixi.sh), [ruff](https://docs.astral.sh/ruff/), [ty](https://docs.astral.sh/ty/), [pytest](https://pytest.org) and [markdownlint-cli2](https://github.com/DavidAnson/markdownlint-cli2) run the checks, and [Zensical](https://zensical.org) builds the package documentation.
## Contributing
Propose a change as a pull request; [CONTRIBUTING.md](CONTRIBUTING.md) says what a change needs, from the chapter to the vectors.
`[CHECK: the studio's contributor license agreement does not exist yet; CONTRIBUTING.md invites pull requests as it did before]`