The work the other paper tools planned for paperspec (paperdesk PLAN M5.4, paperform PLAN 4.2): - paperspec correct and paperspec.revision apply proposed corrections to a package, folder or zip, and write it only once it validates. - doc.math.form is checked: label and style commands always, the form's own spelling when papermath is installed or forms= is passed. - paperspec.datasets and seven doc.dataset.* rules check a package's datasets. - fixtures/invalid gains suspect-witness and math-form-label (33 in all). - Canonical JSON reads numbers as doubles, so 1e20 round-trips. - The registries and the JSON-LD context ship in the wheel. - Public names for what paperdesk and paperform imported as private ones, with the old names kept as aliases. - The text and lossy-projection functions are about three times faster through the gate, tested equal to the old code on every code point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
238 lines
16 KiB
Markdown
238 lines
16 KiB
Markdown
# paperspec
|
|
|
|
**The data formats for storing a team's scientific papers once, correcting them once, addressing them down to the sentence, and keeping them.**
|
|
|
|
This page is for three readers.
|
|
A newcomer should read straight through; each term is explained where it first appears.
|
|
The tables below point an implementer, who will write a reader, writer, converter or checker, to the chapters, test vectors and fixtures that define exact behavior.
|
|
The sections on status and on differences give an expert in the neighboring standards the version, the maturity of each format and the comparison.
|
|
The specification and every file of the formats are published at <https://paperspec.scient.ing>, the documentation of the Python package is at <https://docs.scient.ing/paperspec/>, and the repository is at <https://git.scient.ing/papers/paperspec>.
|
|
|
|
A research team usually cares about a few dozen papers it understands completely, not the world's literature.
|
|
Those papers deserve better than a folder of PDFs.
|
|
Every sentence, table cell and equation should have a permanent address, and every correction should be recorded with its evidence.
|
|
Notes, claims and extracted numbers should point at exactly what they came from.
|
|
Anyone should be able to see how far each piece was checked, and by whom.
|
|
|
|
This repository defines the files that make that possible.
|
|
It holds no application.
|
|
Every format is documented here and readable without any program's code, so a reader, writer, converter, checker or store can each be a separate program.
|
|
Every file is JSON (a plain-text format for structured data) or JSON Lines (one JSON record per line).
|
|
A package is the unit these formats store a paper in: a folder or a zip file of such files holding the text as nodes (sections, sentences, cells, equations), page geometry, references, corrections, review receipts and the list of suspects ([spec/12](spec/12-package.md)).
|
|
|
|
```text
|
|
cornell1995second-saqk#k3m9q2a@4f1a0c[12:57]
|
|
└────── document ────┘└ node ┘└ pin ┘└span┘
|
|
```
|
|
|
|
A hash is a short fingerprint computed from text: the same text always gives the same fingerprint, and changing one character changes the hash.
|
|
The string above names code points 12 to 57 of one sentence, as it read when its hash began `4f1a0c`.
|
|
The node ID `k3m9q2a` encodes no page, position or kind.
|
|
It is minted once from the document key, the node's kind, its text hash and its occurrence, and then frozen.
|
|
The document key's last four characters, `saqk`, are the start of a hash of the paper's DOI, so two papers that a citation key alone would name alike never share a key ([spec/02 §2.4](spec/02-model.md#24-work-and-document-keys)).
|
|
|
|
Fixing a typo elsewhere in the paper leaves the ID alone, and converting the same input again mints the same IDs.
|
|
A teammate's download of the same PDF, stamped differently by the publisher, has the same page fingerprints.
|
|
If a different converter reads the paper, each sentence it reads identically keeps its ID. Every difference becomes a suspect, a place flagged for review.
|
|
Nothing changes until a correction with its evidence is accepted ([spec/04](spec/04-identity.md), [spec/05 §5.6](spec/05-hashes.md#56-alignment)).
|
|
The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and [vectors/addresses.jsonl](vectors/addresses.jsonl) holds valid and invalid examples.
|
|
|
|
## What is in the repository
|
|
|
|
| Folder | Holds |
|
|
| --- | --- |
|
|
| [spec/](spec/README.md) | The specification, in 24 chapters. Normative. |
|
|
| [schemas/](schemas/README.md) | JSON Schemas, which state what a valid file or record looks like, for every file and record. |
|
|
| [sql/](sql/README.md) | Tables for SQLite (a database kept in one file), Dolt and PostgreSQL, and the SQLite bundle with full-text search. |
|
|
| [context/](context/README.md) | The JSON-LD context (a file that gives JSON terms shared meanings) that makes annotation layers W3C Web Annotations. |
|
|
| [registry/](registry/README.md) | The list of formats, the rule IDs validators report, download-stamp rules, and the default rights each processing stage requires. |
|
|
| [vectors/](vectors/README.md) | Test vectors: inputs and the exact outputs every implementation must compute. |
|
|
| [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. |
|
|
| [tools/](tools/README.md) | How to run the commands that build and check everything above. |
|
|
| [src/paperspec/](src/paperspec/) | The reference implementation of every rule and those commands, installable as the Python package `paperspec`. |
|
|
| [tests/](tests/) | The pytest suite for the reference implementation, which takes its data from the vectors and fixtures. |
|
|
| [docs/](docs/README.md) | Guides to the formats for people getting to know the project, and the Python package's documentation in `docs/package/`. |
|
|
| [deploy/](deploy/README.md) | How the website at <https://paperspec.scient.ing> is built and served, and how the package is published. |
|
|
|
|
## Installing
|
|
|
|
The formats need no installation: they are files.
|
|
The Python package `paperspec` needs Python 3.14 and, for the `paperspec` command, the `tools` extra (`jsonschema`, `referencing` and `cryptography`).
|
|
|
|
### From Scienting's package registry
|
|
|
|
The registry does not mirror PyPI, so an installer needs two indexes: the registry for `paperspec`, and PyPI for the packages it depends on.
|
|
Tie `paperspec` to the registry wherever the installer allows it, so that a package of the same name on PyPI is never chosen instead.
|
|
`[CHECK: no version tag has been pushed, so the registry holds no release yet]`
|
|
|
|
With pixi, in `pixi.toml`:
|
|
|
|
```toml
|
|
[pypi-dependencies]
|
|
paperspec = { version = ">=0.1.1", extras = ["tools"], index = "https://git.scient.ing/api/packages/papers/pypi/simple" }
|
|
```
|
|
|
|
With uv, in `pyproject.toml`:
|
|
|
|
```toml
|
|
[[tool.uv.index]]
|
|
name = "scienting"
|
|
url = "https://git.scient.ing/api/packages/papers/pypi/simple"
|
|
explicit = true
|
|
|
|
[tool.uv.sources]
|
|
paperspec = { index = "scienting" }
|
|
```
|
|
|
|
With pip, which cannot tie a package to an index and takes the highest version from either:
|
|
|
|
```sh
|
|
pip install --index-url https://git.scient.ing/api/packages/papers/pypi/simple/ --extra-index-url https://pypi.org/simple/ "paperspec[tools]"
|
|
```
|
|
|
|
[deploy/](deploy/README.md#install-from-the-registry) says what a private registry needs.
|
|
|
|
### From a checkout
|
|
|
|
```sh
|
|
git clone https://git.scient.ing/papers/paperspec
|
|
pixi install
|
|
pixi run check
|
|
```
|
|
|
|
Without pixi, install the checkout as the package with `pip install -e ".[tools]" --group dev` on Python 3.14, then run `paperspec check`.
|
|
|
|
## Quick start
|
|
|
|
With [pixi](https://pixi.sh), one command sets up a pinned environment and runs every check:
|
|
|
|
```sh
|
|
pixi run check
|
|
```
|
|
|
|
It first checks the Python code: formatting and lint with ruff, types with ty, and the pytest suite.
|
|
Then it lints the Markdown.
|
|
Last it runs `paperspec check`, the repository's own checker, which you can also run alone with `pixi run verify`.
|
|
That checker validates every schema, record and package.
|
|
It confirms that every broken package is refused for the stated reason.
|
|
It recomputes every test vector, and it rebuilds everything and compares it byte for byte.
|
|
It also loads every package into SQLite and back, and checks the prose and the links.
|
|
The run ends with `All checks pass.`
|
|
|
|
To validate the example paper, a synthetic chemistry paper with 12 corrections, and load it into SQLite to query it:
|
|
|
|
```sh
|
|
pixi run validate fixtures/packages/hale2026harmonic-damb
|
|
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb
|
|
```
|
|
|
|
[docs/examples.md](docs/examples.md) has queries to start from.
|
|
To correct a package, try the proposal with `paperspec validate --correction fix.json PACKAGE`, then apply it with `paperspec correct --correction fix.json PACKAGE`, which writes the corrected package.
|
|
`paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)).
|
|
`pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)).
|
|
|
|
## Documentation
|
|
|
|
The documentation of the Python package and its command is at <https://docs.scient.ing/paperspec/>: how to install it, guides for validating a package, loading it into SQLite, using the library and checking the repository, the command line and the modules.
|
|
Its source is in `docs/package/`, and `pixi run docs-build` builds it.
|
|
The guides to the formats are at <https://paperspec.scient.ing/docs/>:
|
|
|
|
| If you want to | Read |
|
|
| --- | --- |
|
|
| Understand what the project is and how the parts fit | [docs/overview.md](docs/overview.md) |
|
|
| Learn the ideas one at a time, each with an example | [docs/concepts.md](docs/concepts.md) |
|
|
| Follow one paper from its first reading to a signed document | [docs/walkthrough.md](docs/walkthrough.md) |
|
|
| Run something in the next five minutes | [docs/examples.md](docs/examples.md) |
|
|
| Implement a reader or writer | [spec/](spec/README.md), then [vectors/](vectors/README.md) |
|
|
| Know why a rule is the way it is | [docs/rationale.md](docs/rationale.md) |
|
|
| Look up a term | [docs/glossary.md](docs/glossary.md) |
|
|
| Compare with neighboring standards | [docs/comparison.md](docs/comparison.md) |
|
|
|
|
## Format names
|
|
|
|
Every format is named `scienting-<name>/<major>`, such as `scienting-doc/1` or `scienting-anchor/1`.
|
|
The `scienting-` prefix is the formats' namespace, a separate matter from the repository's name.
|
|
A format is named after what it holds, never after a product or a repository, and products can be renamed without touching data.
|
|
Schemas use the identifier base `https://paperspec.scient.ing/schemas/`.
|
|
[registry/formats.json](registry/formats.json) lists every format with its media type, schema, status and chapter.
|
|
|
|
## How it differs from neighboring standards
|
|
|
|
- JATS is the XML format in which publishers and archives exchange journal content ([NISO Z39.96-2024](https://www.niso.org/publications/z3996-2024-jats)).
|
|
Keep using it for exchange.
|
|
Here, a JATS file is one kind of reading a package can be established from (`jats` is a witness kind, [spec/02 §2.3](spec/02-model.md)).
|
|
- TEI has a sentence element, `<s>` ([TEI P5](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html)), and recommends author-assigned `xml:id` values for durable pointers ([TEI P5 pointers](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)).
|
|
Here, node IDs are minted by a rule and, when a node is merged or deleted, retired with a recorded successor ([spec/04 §4.3](spec/04-identity.md)).
|
|
- W3C Web Annotation is a general model for annotating any resource, with text quote, text position and other selectors ([W3C Recommendation](https://www.w3.org/TR/annotation-model/)).
|
|
Use it for annotation in general.
|
|
Here, annotation layers are Web Annotations whose target selector is a custom `AnchorSelector`, optionally followed by a standard `TextQuoteSelector` for tools that do not know ours ([spec/18](spec/18-layers.md)).
|
|
- Docling is an open-source document conversion toolkit ([arXiv:2501.17887](https://arxiv.org/abs/2501.17887)).
|
|
Use a converter like it to read PDFs.
|
|
It identifies items by position in a list, such as `#/texts/12` ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
|
|
Here, IDs do not depend on position, and a second converter's reading is aligned with the first and compared.
|
|
|
|
## Development
|
|
|
|
The pixi tasks:
|
|
|
|
| Task | Does |
|
|
| --- | --- |
|
|
| `fmt`, `fmt-check` | Format the code with ruff, or check the formatting. |
|
|
| `lint`, `lint-fix` | Lint with ruff, or lint and apply the safe fixes. |
|
|
| `typecheck` | Type-check `src/` with ty. |
|
|
| `test`, `cov` | Run the pytest suite, or run it with branch coverage that fails under 90%. |
|
|
| `check` | Every check CI runs, then the repository's own checker (`verify`). |
|
|
| `hooks` | Install the git pre-commit hooks. |
|
|
| `docs-build`, `docs-serve` | Build the package documentation strictly into `build/docs`, or preview it. |
|
|
| `dist` | Build the sdist and wheel into `dist/` (the `release` environment). |
|
|
|
|
Work happens on the `dev` branch.
|
|
A release is a tag `vX.Y.Z` on the commit to release, and the package version is the tag, which setuptools-scm reads.
|
|
Add the release's section to [CHANGELOG.md](CHANGELOG.md), tag the commit and push the tag; CI checks the repository, builds the package, confirms the built version is the tag's, publishes it to the package registry and creates the release ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), [deploy/](deploy/README.md#publish-the-python-package)).
|
|
|
|
## Built with LLM assistance
|
|
|
|
Large language models (Anthropic's Claude, through Claude Code) write much of this code, its tests and its documentation.
|
|
A person reviews every change, `pixi run check` gates it, and each assisted commit carries a `Co-Authored-By` trailer.
|
|
|
|
## Status and known limits
|
|
|
|
This is version 0.1.0.
|
|
Of the 32 formats, 19 are `candidate` and 13 are `provisional`.
|
|
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint; where those are described, they are marked as such.
|
|
The repository includes a reference checker, test vectors and fixtures.
|
|
|
|
The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
|
|
They do not replace publisher interchange formats.
|
|
They do not fix what a converter reads wrongly; they record it and make it checkable.
|
|
Not every addressable thing is a node with five hashes (the content values `text_hash`, `match_hash`, `align_key`, `own_hash` and `node_hash`, explained in [docs/concepts.md](docs/concepts.md)).
|
|
Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](spec/04-identity.md), [spec/09](spec/09-figures-tables-equations.md)).
|
|
|
|
The Python package is an oracle, not a production reader or writer: it favors clarity over speed, and its VRF and ML-DSA-65 are written out in plain Python ([tools/](tools/README.md#about-the-oracle)).
|
|
It needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
|
|
Only `paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is installed; `check`, `build`, `stress` and `site` work on a checkout of this repository.
|
|
`paperspec correct` cannot rebuild what pinned tools derived from the text: it leaves out the lines of `derived/` files and the datasets that read corrected text, and says which, for the tool that made them to rebuild.
|
|
The validator checks the math form `scienting-math/1` in full only when papermath is installed; without it, it refuses only what the form never holds, such as a `\tag`.
|
|
It accepts more than one location record for a node on one page, which [spec/07 §7.4](spec/07-geometry.md#74-locationsjsonl) rules out.
|
|
|
|
## Licenses
|
|
|
|
The specification and other prose are CC BY 4.0, so anyone may implement the formats.
|
|
Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0.
|
|
The code, the `paperspec` package in `src/` with its commands and its tests, is Apache-2.0, and its Python files open with an SPDX header that says so.
|
|
This repository does not use Scienting's usual Prosperity license, because the formats are meant to be implemented by anyone without asking.
|
|
See [LICENSE.md](LICENSE.md).
|
|
|
|
## Credits
|
|
|
|
The reference implementation leaves the cryptography and the schema validation to established packages: [cryptography](https://cryptography.io) for Ed25519 and HKDF, and [jsonschema](https://python-jsonschema.dev) with [referencing](https://referencing.readthedocs.io) for JSON Schema.
|
|
Canonical JSON follows [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785).
|
|
The verifiable random function follows [RFC 9381](https://www.rfc-editor.org/rfc/rfc9381), and ML-DSA-65 follows [FIPS 204](https://csrc.nist.gov/pubs/fips/204/final); both are checked against those standards' own test vectors.
|
|
The annotation layers build on the [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/).
|
|
[pixi](https://pixi.sh), [ruff](https://docs.astral.sh/ruff/), [ty](https://docs.astral.sh/ty/), [pytest](https://pytest.org) and [markdownlint-cli2](https://github.com/DavidAnson/markdownlint-cli2) run the checks, and [Zensical](https://zensical.org) builds the package documentation.
|
|
|
|
## Contributing
|
|
|
|
Propose a change as a pull request; [CONTRIBUTING.md](CONTRIBUTING.md) says what a change needs, from the chapter to the vectors.
|
|
`[CHECK: Scienting's contributor license agreement does not exist yet; CONTRIBUTING.md invites pull requests as it did before]`
|