# PaperSpec **The data formats for storing a team's scientific papers once, correcting them once, addressing them down to the sentence, and keeping them.** This page is for three readers. A newcomer should read straight through; each term is explained where it first appears. The tables below point an implementer, who will write a reader, writer, converter or checker, to the chapters, test vectors and fixtures that define exact behavior. The sections on status and on differences give an expert in the neighboring standards the version, the maturity of each format and the comparison. The specification and every file of the formats are published at , the documentation of the Python package is at , and the repository is at . A research team usually cares about a few dozen papers it understands completely, not the world's literature. Those papers deserve better than a folder of PDFs. Every sentence, table cell and equation should have a permanent address, and every correction should be recorded with its evidence. Notes, claims and extracted numbers should point at exactly what they came from. Anyone should be able to see how far each piece was checked, and by whom. This repository defines the files that make that possible. It holds no application. Every format is documented here and readable without any program's code, so a reader, writer, converter, checker or store can each be a separate program. Every file is JSON (a plain-text format for structured data) or JSON Lines (one JSON record per line). A package is the unit these formats store a paper in: a folder or a zip file of such files holding the text as nodes (sections, sentences, cells, equations), page geometry, references, corrections, review receipts and the list of suspects ([spec/12](spec/12-package.md)). ```text cornell1995second-saqk#k3m9q2a@4f1a0c[12:57] └────── document ────┘└ node ┘└ pin ┘└span┘ ``` A hash is a short fingerprint computed from text: the same text always gives the same fingerprint, and changing one character changes the hash. The string above names code points 12 to 57 of one sentence, as it read when its hash began `4f1a0c`. The node ID `k3m9q2a` encodes no page, position or kind. It is minted once from the document key, the node's kind, its text hash and its occurrence, and then frozen. The document key's last four characters, `saqk`, are the start of a hash of the paper's DOI, so two papers that a citation key alone would name alike never share a key ([spec/02 §2.4](spec/02-model.md#24-work-and-document-keys)). Fixing a typo elsewhere in the paper leaves the ID alone, and converting the same input again mints the same IDs. A teammate's download of the same PDF, stamped differently by the publisher, has the same page fingerprints. If a different converter reads the paper, each sentence it reads identically keeps its ID. Every difference becomes a suspect, a place flagged for review. Nothing changes until a correction with its evidence is accepted ([spec/04](spec/04-identity.md), [spec/05 §5.6](spec/05-hashes.md#56-alignment)). The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and [vectors/addresses.jsonl](vectors/addresses.jsonl) holds valid and invalid examples. ## What is in the repository | Folder | Holds | | --- | --- | | [spec/](spec/README.md) | The specification, in 24 chapters. Normative. | | [schemas/](schemas/README.md) | JSON Schemas, which state what a valid file or record looks like, for every file and record. | | [sql/](sql/README.md) | Tables for SQLite (a database kept in one file), Dolt and PostgreSQL, and the SQLite bundle with full-text search. | | [context/](context/README.md) | The JSON-LD context (a file that gives JSON terms shared meanings) that makes annotation layers W3C Web Annotations. | | [registry/](registry/README.md) | The list of formats, the rule IDs validators report, download-stamp rules, and the default rights each processing stage requires. | | [vectors/](vectors/README.md) | Test vectors: inputs and the exact outputs every implementation must compute. | | [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. | | [tools/](tools/README.md) | How to run the commands that build and check everything above. | | [src/paperspec/](src/paperspec/) | The reference implementation of every rule and those commands, installable as the Python package `paperspec`. | | [tests/](tests/) | The pytest suite for the reference implementation, which takes its data from the vectors and fixtures. | | [docs/](docs/README.md) | Guides to the formats for people getting to know the project, and the Python package's documentation in `docs/package/`. | | [deploy/](deploy/README.md) | How the website at is built and served, and how the package is published. | ## Installing The formats need no installation: they are files. The Python package `paperspec` needs Python 3.14 and, for the `paperspec` command, the `tools` extra (`jsonschema`, `referencing` and `cryptography`). ### From Scienting's package registry The registry does not mirror PyPI, so an installer needs two indexes: the registry for `paperspec`, and PyPI for the packages it depends on. Tie `paperspec` to the registry wherever the installer allows it, so that a package of the same name on PyPI is never chosen instead. `[CHECK: no version tag has been pushed, so the registry holds no release yet]` With pixi, in `pixi.toml`: ```toml [pypi-dependencies] paperspec = { version = ">=0.1.1", extras = ["tools"], index = "https://git.scient.ing/api/packages/papers/pypi/simple" } ``` With uv, in `pyproject.toml`: ```toml [[tool.uv.index]] name = "scienting" url = "https://git.scient.ing/api/packages/papers/pypi/simple" explicit = true [tool.uv.sources] paperspec = { index = "scienting" } ``` With pip, which cannot tie a package to an index and takes the highest version from either: ```sh pip install --index-url https://git.scient.ing/api/packages/papers/pypi/simple/ --extra-index-url https://pypi.org/simple/ "paperspec[tools]" ``` [deploy/](deploy/README.md#install-from-the-registry) says what a private registry needs. ### From a checkout ```sh git clone https://git.scient.ing/papers/paperspec pixi install pixi run check ``` Without pixi, install the checkout as the package with `pip install -e ".[tools]" --group dev` on Python 3.14, then run `paperspec check`. ## Quick start With [pixi](https://pixi.sh), one command sets up a pinned environment and runs every check: ```sh pixi run check ``` It first checks the Python code: formatting and lint with ruff, types with ty, and the pytest suite. Then it lints the Markdown. Last it runs `paperspec check`, the repository's own checker, which you can also run alone with `pixi run verify`. That checker validates every schema, record and package. It confirms that every broken package is refused for the stated reason. It recomputes every test vector, and it rebuilds everything and compares it byte for byte. It also loads every package into SQLite and back, and checks the prose and the links. The run ends with `All checks pass.` To validate the example paper, a synthetic chemistry paper with 12 corrections, and load it into SQLite to query it: ```sh pixi run validate fixtures/packages/hale2026harmonic-damb pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb ``` [docs/examples.md](docs/examples.md) has queries to start from. To correct a package, try the proposal with `paperspec validate --correction fix.json PACKAGE`, then apply it with `paperspec correct --correction fix.json PACKAGE`, which writes the corrected package. `paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)). `pixi run site` builds the website into `_site/` ([deploy/](deploy/README.md)). ## Documentation The documentation of the Python package and its command is at : how to install it, guides for validating a package, loading it into SQLite, using the library and checking the repository, the command line and the modules. Its source is in `docs/package/`, and `pixi run docs-build` builds it. The guides to the formats are at : | If you want to | Read | | --- | --- | | Understand what the project is and how the parts fit | [docs/overview.md](docs/overview.md) | | Learn the ideas one at a time, each with an example | [docs/concepts.md](docs/concepts.md) | | Follow one paper from its first reading to a signed document | [docs/walkthrough.md](docs/walkthrough.md) | | Run something in the next five minutes | [docs/examples.md](docs/examples.md) | | Implement a reader or writer | [spec/](spec/README.md), then [vectors/](vectors/README.md) | | Know why a rule is the way it is | [docs/rationale.md](docs/rationale.md) | | Look up a term | [docs/glossary.md](docs/glossary.md) | | Compare with neighboring standards | [docs/comparison.md](docs/comparison.md) | ## Format names Every format is named `scienting-/`, such as `scienting-doc/1` or `scienting-anchor/1`. The `scienting-` prefix is the formats' namespace, a separate matter from the repository's name. A format is named after what it holds, never after a product or a repository, and products can be renamed without touching data. Schemas use the identifier base `https://paperspec.scient.ing/schemas/`. [registry/formats.json](registry/formats.json) lists every format with its media type, schema, status and chapter. ## How it differs from neighboring standards - JATS is the XML format in which publishers and archives exchange journal content ([NISO Z39.96-2024](https://www.niso.org/publications/z3996-2024-jats)). Keep using it for exchange. Here, a JATS file is one kind of reading a package can be established from (`jats` is a witness kind, [spec/02 §2.3](spec/02-model.md)). - TEI has a sentence element, `` ([TEI P5](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html)), and recommends author-assigned `xml:id` values for durable pointers ([TEI P5 pointers](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)). Here, node IDs are minted by a rule and, when a node is merged or deleted, retired with a recorded successor ([spec/04 §4.3](spec/04-identity.md)). - W3C Web Annotation is a general model for annotating any resource, with text quote, text position and other selectors ([W3C Recommendation](https://www.w3.org/TR/annotation-model/)). Use it for annotation in general. Here, annotation layers are Web Annotations whose target selector is a custom `AnchorSelector`, optionally followed by a standard `TextQuoteSelector` for tools that do not know ours ([spec/18](spec/18-layers.md)). - Docling is an open-source document conversion toolkit ([arXiv:2501.17887](https://arxiv.org/abs/2501.17887)). Use a converter like it to read PDFs. It identifies items by position in a list, such as `#/texts/12` ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)). Here, IDs do not depend on position, and a second converter's reading is aligned with the first and compared. ## Development The pixi tasks: | Task | Does | | --- | --- | | `fmt`, `fmt-check` | Format the code with ruff, or check the formatting. | | `lint`, `lint-fix` | Lint with ruff, or lint and apply the safe fixes. | | `typecheck` | Type-check `src/` with ty. | | `test`, `cov` | Run the pytest suite, or run it with branch coverage that fails under 90%. | | `check` | Every check CI runs, then the repository's own checker (`verify`). | | `hooks` | Install the git pre-commit hooks. | | `docs-build`, `docs-serve` | Build the package documentation strictly into `build/docs`, or preview it. | | `dist` | Build the sdist and wheel into `dist/` (the `release` environment). | Work happens on the `dev` branch. A release is a tag `vX.Y.Z` on the commit to release, and the package version is the tag, which setuptools-scm reads. Add the release's section to [CHANGELOG.md](CHANGELOG.md), tag the commit and push the tag; CI checks the repository, builds the package, confirms the built version is the tag's, publishes it to the package registry and creates the release ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), [deploy/](deploy/README.md#publish-the-python-package)). ## Built with LLM assistance Large language models (Anthropic's Claude, through Claude Code) write much of this code, its tests and its documentation. A person reviews every change, `pixi run check` gates it, and each assisted commit carries a `Co-Authored-By` trailer. ## Status and known limits This is version 0.1.0. Of the 32 formats, 19 are `candidate` and 13 are `provisional`. The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint; where those are described, they are marked as such. The repository includes a reference checker, test vectors and fixtures. The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers. They do not replace publisher interchange formats. They do not fix what a converter reads wrongly; they record it and make it checkable. Not every addressable thing is a node with five hashes (the content values `text_hash`, `match_hash`, `align_key`, `own_hash` and `node_hash`, explained in [docs/concepts.md](docs/concepts.md)). Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](spec/04-identity.md), [spec/09](spec/09-figures-tables-equations.md)). The Python package is an oracle, not a production reader or writer: it favors clarity over speed, and its VRF and ML-DSA-65 are written out in plain Python ([tools/](tools/README.md#about-the-oracle)). It needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other. Only `paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is installed; `check`, `build`, `stress` and `site` work on a checkout of this repository. `paperspec correct` cannot rebuild what pinned tools derived from the text: it leaves out the lines of `derived/` files and the datasets that read corrected text, and says which, for the tool that made them to rebuild. The validator checks the math form `scienting-math/1` in full only when PaperMath is installed; without it, it refuses only what the form never holds, such as a `\tag`. It accepts more than one location record for a node on one page, which [spec/07 §7.4](spec/07-geometry.md#74-locationsjsonl) rules out. ## Licenses The specification and other prose are CC BY 4.0, so anyone may implement the formats. Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0. The code, the `paperspec` package in `src/` with its commands and its tests, is Apache-2.0, and its Python files open with an SPDX header that says so. This repository does not use Scienting's usual Prosperity license, because the formats are meant to be implemented by anyone without asking. See [LICENSE.md](LICENSE.md). ## Credits The reference implementation leaves the cryptography and the schema validation to established packages: [cryptography](https://cryptography.io) for Ed25519 and HKDF, and [jsonschema](https://python-jsonschema.dev) with [referencing](https://referencing.readthedocs.io) for JSON Schema. Canonical JSON follows [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785). The verifiable random function follows [RFC 9381](https://www.rfc-editor.org/rfc/rfc9381), and ML-DSA-65 follows [FIPS 204](https://csrc.nist.gov/pubs/fips/204/final); both are checked against those standards' own test vectors. The annotation layers build on the [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/). [pixi](https://pixi.sh), [ruff](https://docs.astral.sh/ruff/), [ty](https://docs.astral.sh/ty/), [pytest](https://pytest.org) and [markdownlint-cli2](https://github.com/DavidAnson/markdownlint-cli2) run the checks, and [Zensical](https://zensical.org) builds the package documentation. ## Contributing Propose a change as a pull request; [CONTRIBUTING.md](CONTRIBUTING.md) says what a change needs, from the chapter to the vectors. `[CHECK: Scienting's contributor license agreement does not exist yet; CONTRIBUTING.md invites pull requests as it did before]`