Publish the package and its documentation, and move to the papers organization
Add the Zensical documentation of the Python package (docs/package, built by the docs workflow), merge the standard publish workflow with the setuptools-scm tag check, add the SPDX header rule to ruff, restructure the README, and point every repository and registry address at the Gitea organization papers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B8cL4zN4XpaQF5cBtASzfq
This commit is contained in:
37 files changed
+1005
-83
No files matched your search
@@ -6,7 +6,7 @@ This page is for three readers.
|
||||
A newcomer should read straight through; each term is explained where it first appears.
|
||||
The tables below point an implementer, who will write a reader, writer, converter or checker, to the chapters, test vectors and fixtures that define exact behavior.
|
||||
The sections on status and on differences give an expert in the neighboring standards the version, the maturity of each format and the comparison.
|
||||
The documentation and every file of the specification are published at <https://paperspec.scient.ing>, and the repository is at <https://git.scient.ing/infra/paperspec>.
|
||||
The specification and every file of the formats are published at <https://paperspec.scient.ing>, the documentation of the Python package is at <https://docs.scient.ing/paperspec/>, and the repository is at <https://git.scient.ing/papers/paperspec>.
|
||||
|
||||
A research team usually cares about a few dozen papers it understands completely, not the world's literature.
|
||||
Those papers deserve better than a folder of PDFs.
|
||||
@@ -37,19 +37,6 @@ If a different converter reads the paper, each sentence it reads identically kee
|
||||
Nothing changes until a correction with its evidence is accepted ([spec/04](spec/04-identity.md), [spec/05 §5.6](spec/05-hashes.md#56-alignment)).
|
||||
The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and [vectors/addresses.jsonl](vectors/addresses.jsonl) holds valid and invalid examples.
|
||||
|
||||
## Start here
|
||||
|
||||
| If you want to | Read |
|
||||
| --- | --- |
|
||||
| Understand what the project is and how the parts fit | [docs/overview.md](docs/overview.md) |
|
||||
| Learn the ideas one at a time, each with an example | [docs/concepts.md](docs/concepts.md) |
|
||||
| Follow one paper from its first reading to a signed document | [docs/walkthrough.md](docs/walkthrough.md) |
|
||||
| Run something in the next five minutes | [docs/examples.md](docs/examples.md) |
|
||||
| Implement a reader or writer | [spec/](spec/README.md), then [vectors/](vectors/README.md) |
|
||||
| Know why a rule is the way it is | [docs/rationale.md](docs/rationale.md) |
|
||||
| Look up a term | [docs/glossary.md](docs/glossary.md) |
|
||||
| Compare with neighboring standards | [docs/comparison.md](docs/comparison.md) |
|
||||
|
||||
## What is in the repository
|
||||
|
||||
| Folder | Holds |
|
||||
@@ -63,36 +50,103 @@ The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and
|
||||
| [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. |
|
||||
| [tools/](tools/README.md) | How to run the commands that build and check everything above. |
|
||||
| [src/paperspec/](src/paperspec/) | The reference implementation of every rule and those commands, installable as the Python package `paperspec`. |
|
||||
| [docs/](docs/README.md) | Guides for people getting to know the project. |
|
||||
| [deploy/](deploy/README.md) | How the website at <https://paperspec.scient.ing> is built and served. |
|
||||
| [tests/](tests/) | The pytest suite for the reference implementation, which takes its data from the vectors and fixtures. |
|
||||
| [docs/](docs/README.md) | Guides to the formats for people getting to know the project, and the Python package's documentation in `docs/package/`. |
|
||||
| [deploy/](deploy/README.md) | How the website at <https://paperspec.scient.ing> is built and served, and how the package is published. |
|
||||
|
||||
## Try it
|
||||
## Installing
|
||||
|
||||
With [pixi](https://pixi.sh), one command sets up a pinned environment and checks everything:
|
||||
The formats need no installation: they are files.
|
||||
The Python package `paperspec` needs Python 3.14 and, for the `paperspec` command, the `tools` extra (`jsonschema`, `referencing` and `cryptography`).
|
||||
|
||||
### From the studio's package registry
|
||||
|
||||
The registry does not mirror PyPI, so an installer needs two indexes: the registry for `paperspec`, and PyPI for the packages it depends on.
|
||||
Tie `paperspec` to the registry wherever the installer allows it, so that a package of the same name on PyPI is never chosen instead.
|
||||
`[CHECK: no version tag has been pushed, so the registry holds no release yet]`
|
||||
|
||||
With pixi, in `pixi.toml`:
|
||||
|
||||
```toml
|
||||
[pypi-dependencies]
|
||||
paperspec = { version = ">=0.1.1", extras = ["tools"], index = "https://git.scient.ing/api/packages/papers/pypi/simple" }
|
||||
```
|
||||
|
||||
With uv, in `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[[tool.uv.index]]
|
||||
name = "scienting"
|
||||
url = "https://git.scient.ing/api/packages/papers/pypi/simple"
|
||||
explicit = true
|
||||
|
||||
[tool.uv.sources]
|
||||
paperspec = { index = "scienting" }
|
||||
```
|
||||
|
||||
With pip, which cannot tie a package to an index and takes the highest version from either:
|
||||
|
||||
```sh
|
||||
pip install --index-url https://git.scient.ing/api/packages/papers/pypi/simple/ --extra-index-url https://pypi.org/simple/ "paperspec[tools]"
|
||||
```
|
||||
|
||||
[deploy/](deploy/README.md#install-from-the-registry) says what a private registry needs.
|
||||
|
||||
### From a checkout
|
||||
|
||||
```sh
|
||||
git clone https://git.scient.ing/papers/paperspec
|
||||
pixi install
|
||||
pixi run check
|
||||
```
|
||||
|
||||
Without pixi, install the checkout as the package with `pip install -e ".[tools]" --group dev` on Python 3.14, then run `paperspec check`.
|
||||
|
||||
## Quick start
|
||||
|
||||
With [pixi](https://pixi.sh), one command sets up a pinned environment and runs every check:
|
||||
|
||||
```sh
|
||||
pixi run check
|
||||
```
|
||||
|
||||
The checker validates every schema, record and package.
|
||||
It first checks the Python code: formatting and lint with ruff, types with ty, and the pytest suite.
|
||||
Then it lints the Markdown.
|
||||
Last it runs `paperspec check`, the repository's own checker, which you can also run alone with `pixi run verify`.
|
||||
That checker validates every schema, record and package.
|
||||
It confirms that every broken package is refused for the stated reason.
|
||||
It recomputes every test vector, and it rebuilds everything and compares it byte for byte.
|
||||
Last, it loads every package into SQLite and back, and checks the prose and the links.
|
||||
Pixi lints the Markdown first.
|
||||
It also loads every package into SQLite and back, and checks the prose and the links.
|
||||
The run ends with `All checks pass.`
|
||||
`pixi task list` shows the other tasks: `quick`, `build`, `lint`, `load`, `validate` and `stress`.
|
||||
`pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)).
|
||||
|
||||
The commands need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), which pixi installs along with markdownlint for the Markdown.
|
||||
Without pixi, install the checkout as the package with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec check`.
|
||||
`paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)).
|
||||
|
||||
To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from.
|
||||
To validate the example paper, a synthetic chemistry paper with 12 corrections, and load it into SQLite to query it:
|
||||
|
||||
```sh
|
||||
pixi run validate fixtures/packages/hale2026harmonic-damb
|
||||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
||||
```
|
||||
|
||||
[docs/examples.md](docs/examples.md) has queries to start from.
|
||||
`paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)).
|
||||
`pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)).
|
||||
|
||||
## Documentation
|
||||
|
||||
The documentation of the Python package and its command is at <https://docs.scient.ing/paperspec/>: how to install it, guides for validating a package, loading it into SQLite, using the library and checking the repository, the command line and the modules.
|
||||
Its source is in `docs/package/`, and `pixi run docs-build` builds it.
|
||||
The guides to the formats are at <https://paperspec.scient.ing/docs/>:
|
||||
|
||||
| If you want to | Read |
|
||||
| --- | --- |
|
||||
| Understand what the project is and how the parts fit | [docs/overview.md](docs/overview.md) |
|
||||
| Learn the ideas one at a time, each with an example | [docs/concepts.md](docs/concepts.md) |
|
||||
| Follow one paper from its first reading to a signed document | [docs/walkthrough.md](docs/walkthrough.md) |
|
||||
| Run something in the next five minutes | [docs/examples.md](docs/examples.md) |
|
||||
| Implement a reader or writer | [spec/](spec/README.md), then [vectors/](vectors/README.md) |
|
||||
| Know why a rule is the way it is | [docs/rationale.md](docs/rationale.md) |
|
||||
| Look up a term | [docs/glossary.md](docs/glossary.md) |
|
||||
| Compare with neighboring standards | [docs/comparison.md](docs/comparison.md) |
|
||||
|
||||
## Format names
|
||||
|
||||
Every format is named `scienting-<name>/<major>`, such as `scienting-doc/1` or `scienting-anchor/1`.
|
||||
@@ -101,19 +155,6 @@ A format is named after what it holds, never after a product or a repository, an
|
||||
Schemas use the identifier base `https://paperspec.scient.ing/schemas/`.
|
||||
[registry/formats.json](registry/formats.json) lists every format with its media type, schema, status and chapter.
|
||||
|
||||
## Status and limits
|
||||
|
||||
This is version 0.1.0.
|
||||
Of the 32 formats, 19 are `candidate` and 13 are `provisional`.
|
||||
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint; where those are described, they are marked as such.
|
||||
The repository includes a reference checker, test vectors and fixtures.
|
||||
|
||||
The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
|
||||
They do not replace publisher interchange formats.
|
||||
They do not fix what a converter reads wrongly; they record it and make it checkable.
|
||||
Not every addressable thing is a node with five hashes (the content values `text_hash`, `match_hash`, `align_key`, `own_hash` and `node_hash`, explained in [docs/concepts.md](docs/concepts.md)).
|
||||
Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](spec/04-identity.md), [spec/09](spec/09-figures-tables-equations.md)).
|
||||
|
||||
## How it differs from neighboring standards
|
||||
|
||||
- JATS is the XML format in which publishers and archives exchange journal content ([NISO Z39.96-2024](https://www.niso.org/publications/z3996-2024-jats)).
|
||||
@@ -129,9 +170,64 @@ Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](
|
||||
It identifies items by position in a list, such as `#/texts/12` ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
|
||||
Here, IDs do not depend on position, and a second converter's reading is aligned with the first and compared.
|
||||
|
||||
## Development
|
||||
|
||||
The pixi tasks:
|
||||
|
||||
| Task | Does |
|
||||
| --- | --- |
|
||||
| `fmt`, `fmt-check` | Format the code with ruff, or check the formatting. |
|
||||
| `lint`, `lint-fix` | Lint with ruff, or lint and apply the safe fixes. |
|
||||
| `typecheck` | Type-check `src/` with ty. |
|
||||
| `test`, `cov` | Run the pytest suite, or run it with branch coverage that fails under 90%. |
|
||||
| `check` | Every check CI runs, then the repository's own checker (`verify`). |
|
||||
| `hooks` | Install the git pre-commit hooks. |
|
||||
| `docs-build`, `docs-serve` | Build the package documentation strictly into `build/docs`, or preview it. |
|
||||
| `dist` | Build the sdist and wheel into `dist/` (the `release` environment). |
|
||||
|
||||
Work happens on the `dev` branch.
|
||||
A release is a tag `vX.Y.Z` on the commit to release, and the package version is the tag, which setuptools-scm reads.
|
||||
Add the release's section to [CHANGELOG.md](CHANGELOG.md), tag the commit and push the tag; CI checks the repository, builds the package, confirms the built version is the tag's, publishes it to the package registry and creates the release ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), [deploy/](deploy/README.md#publish-the-python-package)).
|
||||
|
||||
## Built with LLM assistance
|
||||
|
||||
Large language models (Anthropic's Claude, through Claude Code) write much of this code, its tests and its documentation.
|
||||
A person reviews every change, `pixi run check` gates it, and each assisted commit carries a `Co-Authored-By` trailer.
|
||||
|
||||
## Status and known limits
|
||||
|
||||
This is version 0.1.0.
|
||||
Of the 32 formats, 19 are `candidate` and 13 are `provisional`.
|
||||
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint; where those are described, they are marked as such.
|
||||
The repository includes a reference checker, test vectors and fixtures.
|
||||
|
||||
The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
|
||||
They do not replace publisher interchange formats.
|
||||
They do not fix what a converter reads wrongly; they record it and make it checkable.
|
||||
Not every addressable thing is a node with five hashes (the content values `text_hash`, `match_hash`, `align_key`, `own_hash` and `node_hash`, explained in [docs/concepts.md](docs/concepts.md)).
|
||||
Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](spec/04-identity.md), [spec/09](spec/09-figures-tables-equations.md)).
|
||||
|
||||
The Python package is an oracle, not a production reader or writer: it favors clarity over speed, and its VRF and ML-DSA-65 are written out in plain Python ([tools/](tools/README.md#about-the-oracle)).
|
||||
It needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
|
||||
Only `paperspec validate` and `paperspec load` work anywhere the package is installed; `check`, `build`, `stress` and `site` work on a checkout of this repository.
|
||||
|
||||
## Licenses
|
||||
|
||||
The specification and other prose are CC BY 4.0, so anyone may implement the formats.
|
||||
Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0.
|
||||
The code, the `paperspec` package in `src/` with its commands, is Apache-2.0.
|
||||
The code, the `paperspec` package in `src/` with its commands and its tests, is Apache-2.0, and its Python files open with an SPDX header that says so.
|
||||
This repository does not use the studio's Prosperity license, because the formats are meant to be implemented by anyone without asking.
|
||||
See [LICENSE.md](LICENSE.md).
|
||||
|
||||
## Credits
|
||||
|
||||
The reference implementation leaves the cryptography and the schema validation to established packages: [cryptography](https://cryptography.io) for Ed25519 and HKDF, and [jsonschema](https://python-jsonschema.dev) with [referencing](https://referencing.readthedocs.io) for JSON Schema.
|
||||
Canonical JSON follows [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785).
|
||||
The verifiable random function follows [RFC 9381](https://www.rfc-editor.org/rfc/rfc9381), and ML-DSA-65 follows [FIPS 204](https://csrc.nist.gov/pubs/fips/204/final); both are checked against those standards' own test vectors.
|
||||
The annotation layers build on the [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/).
|
||||
[pixi](https://pixi.sh), [ruff](https://docs.astral.sh/ruff/), [ty](https://docs.astral.sh/ty/), [pytest](https://pytest.org) and [markdownlint-cli2](https://github.com/DavidAnson/markdownlint-cli2) run the checks, and [Zensical](https://zensical.org) builds the package documentation.
|
||||
|
||||
## Contributing
|
||||
|
||||
Propose a change as a pull request; [CONTRIBUTING.md](CONTRIBUTING.md) says what a change needs, from the chapter to the vectors.
|
||||
`[CHECK: the studio's contributor license agreement does not exist yet; CONTRIBUTING.md invites pull requests as it did before]`
|
||||
Reference in new issue
Block a user