paperspec/docs/overview.md

# The paperspec formats

This page serves three readers.
A student new to the problem will find each term defined where it first appears.
A developer who will write a reader, writer, converter or checker will find each topic pointing to its chapter.
A researcher or standards author will find the status, the limits and a comparison stated outright.

## The idea

Scientists who rely on a paper keep asking two questions.

1.
   Does the cited passage say what is claimed? A citation to a real paper can still be inaccurate, incomplete or beside the point, so a reader needs to see the exact sentence it rests on.
2.
   Where did this number come from, and how far was it checked? A force constant in a table, a value read off a plot, a parameter in a model: each should lead back to the cell or sentence it came from, with a record of who compared it with the page.

Most tools for scientific literature aggregate.
They index hundreds of millions of papers and answer questions over all of them, shallowly. paperspec does the opposite.

A team working on a focused question cares more about fifty papers it understands completely than about the world's literature.
Those fifty papers deserve to be converted once, corrected once, split down to the sentence, annotated by the people who know them, linked to checked claims, and kept.

A model can summarize a paper.
It cannot give you a signed statement that a named person checked a passage, IDs that never move, or a store its owner controls.
The formats supply those three, and they are worth most at the scale of one team.

paperspec is a set of file formats for scientific papers that have been converted, split into addressable pieces and checked.
The repository is at <https://git.scient.ing/infra/paperspec>.
It is version 0.1.0 and defines 32 formats; in [registry/formats.json](../registry/formats.json), 19 are `candidate` and 13 are `provisional`.

## Who it is for

| User | Example |
| --- | --- |
| One researcher | A thesis student, a science writer or fact-checker, a Wikipedia editor checking a citation, a scientist keeping a deep personal library. |
| An academic lab | A research group, a thesis committee, a journal club reading one annotated paper together. |
| A course | An instructor whose students learn to check whether a source supports a claim, with answers that are addresses and can be graded. |
| An industrial team | A computational chemistry or materials group that needs to show which passage of which paper each parameter came from. |

"Team" throughout means roughly five to twenty people with a few dozen to a few hundred papers that matter.
The design targets that scale: hundreds of documents (500 papers come to about a million and a half database rows).

## What the repository contains

The repository defines files and checks them.
It holds no application.

The formats are identified as `scienting-<name>/<major>`, such as `scienting-doc/1`.
That prefix is the formats' namespace.
It appears inside hashed data and test vectors, and it is separate from the repository's name.

| Folder | Holds |
| --- | --- |
| [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. |
| [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. |
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
| [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. |
| [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. |
| [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
| [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. |
| [docs/](README.md) | Guides, including this page. |

## Roles

The formats serve five roles.
A role is a job someone could implement.
The repository ships reference tools only for the checker and store roles.

| Role | Does |
| --- | --- |
| Reader | Opens a package, resolves addresses and anchors, and shows text, annotations and each passage's fidelity level. |
| Writer | Produces files in these formats: nodes, corrections, receipts, annotations, datasets. |
| Converter | Turns a witness (a PDF, JATS XML, the article format publishers use, a LaTeX source archive, an HTML page, an EPUB or a Typst project) into a package and raises doubtful readings as suspects (flagged disagreements awaiting review). |
| Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `paperspec check` does this for the repository. |
| Store | Keeps packages and records in a database a team can clone. `paperspec load` loads packages into SQLite. |

The [specification's reading paths](../spec/README.md) say which chapters each role needs.

## How a paper moves through the formats

1.
   Acquire.
   A witness arrives with a rights record that says how the team holds it and what that allows.
   A witness is one file of one version of a paper, such as the publisher's PDF, identified by the SHA-256 hash of its bytes.
2.
   Read and reconcile.
   A converter reads the witness with one or more backends, aligns the readings sentence by sentence and compares them.
   Each page gets a fingerprint with download stamps removed, and differently stamped downloads of one page are recognized as the same.
3.
   Segment and mint.
   Text is split into sentences.
   Every node (a section, paragraph, sentence, figure, panel, caption, table, row, cell, equation, footnote or reference) gets a permanent ID and five content hashes.
   A hash is a fixed-length fingerprint of content: change one character and it changes.
4.
   Review.
   Disagreements between readings become suspects with page crops.
   A person or a model answers each as a correction, and the gate refuses any correction that changes a digit without image evidence.
   In the example paper the first reading has a cell `3l0.0` where the page says `310.0`.
5.
   Package.
   The result is a package: a folder of canonical JSON files (JSON written in one fixed form, so equal content gives equal bytes) with a manifest whose hash pins the whole.
   Five files are required (`MANIFEST.json`, `document.json`, `nodes.jsonl`, `witnesses.json`, `provenance.json`).
   These are optional: `pages.jsonl`, `locations.jsonl`, `references.jsonl`, `corrections.jsonl`, `receipts.jsonl`, `ids.jsonl`, `suspects.jsonl`, `roster.json`, `assets/`, `datasets/` and `derived/` ([chapter 12](../spec/12-package.md)).
6.
   Use.
   A store loads the package into a database.
   A reader renders it with layers of annotations.
   Anchors (stored references to a sentence, with its hash and a quotation) let a claim, a dataset value or a draft's citation point at a passage.
   Datasets, drafts, views, projects, exercises, checklists and the transparency log are `provisional` formats.
7.
   Check and sign.
   People proofread what matters and sign receipts; second reviewers spot-check them.
   A passage is at level F3 (proofread) only while a valid receipt lists it at its current hash.
   Every level counts only at the current text.

[The walkthrough](walkthrough.md) follows one paper through every step.

## Principles

| Principle | Means |
| --- | --- |
| Quality over quantity | A small corpus processed to a stated and checkable fidelity beats a large one processed to an unknown one. |
| Do the work once | Conversion, correction, segmentation and digitization are recorded and kept. A fresh conversion of the same input mints the same IDs. A different converter's reading is aligned with the established text: sentences the alignment pairs keep their IDs, and every difference becomes a suspect. |
| Address everything | Every node has a permanent ID and five content hashes. Table columns and figure-text items also have IDs. They are not nodes and have no five hashes. IDs encode no page, ordinal or kind; each is minted once from the document key, kind, text hash and occurrence, then frozen. |
| Keep source and commentary apart | What a source says is stored as the source said it. Everything people or models add is stored beside it and points at it. |
| Count only what was done | A level says how far something was checked, and it falls back when the checked content changes. |
| Models propose, people sign | Models may draft, segment, gloss and digitize, and a model's annotation stays `proposed` until a person confirms it. The proofread level comes only from a receipt signed by a named person. |
| The team owns its data | Packages are folders of JSON files, the database form is plain SQL tables, and every format is documented in the specification. A team can clone, read and keep its data without anyone's software. |

## How it compares

paperspec adds to existing standards and replaces none of them. [The comparison page](comparison.md) has the full account; five points follow.

- JATS is the NISO standard by which [publishers and archives exchange journal content](https://www.niso.org/publications/z3996-2024-jats).
  Its [version 1.4 element list](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) shows no sentence element.
  A JATS file is an input here (witness kind `jats`).
- TEI has a [sentence-like element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html) and [markup that keeps both readings of a correction](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html), and recommends `xml:id` pointers ["for durability in the presence of editing"](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html). paperspec mints and retires IDs by rule and keeps corrections as a separate, evidence-gated record.
  TEI is built for encoding scholarly text in general, including correction markup within a text.
- GROBID [converts PDFs to TEI](https://grobid.readthedocs.io/en/latest/Introduction/) and [reports F1 scores](https://grobid.readthedocs.io/en/latest/Introduction/) for its automatic reading.
  Its output can be held as one reading, under witness kind `xml`.
- The W3C Web Annotation Data Model is [a Recommendation of 2017](https://www.w3.org/TR/annotation-model/).
  Layers are Web Annotations, with a custom `AnchorSelector` followed by a standard `TextQuoteSelector` fallback.
  The W3C model calls the `TextPositionSelector` ["very brittle"](https://www.w3.org/TR/annotation-model/) against changes to the resource, and the anchors here pin a hash instead.
  For annotation in general, use the W3C model.
- Docling's [document schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json) refers to items by position (`#/texts/12`) and lists no per-item hash field.
  Docling is a conversion toolkit that, [by its own abstract](https://arxiv.org/abs/2501.17887), uses AI models for layout analysis and table structure recognition. paperspec defines no converter.

The formats are at version 0.1.0, the repository names no outside adopter, and the reference tools belong to the same repository.
The formats do not replace publisher interchange formats.
They also do not fix what a converter reads wrongly; they record it and make it checkable.

## What it is not

It is not a search engine over all literature, a citation index, a recommendation feed or a chat-with-any-PDF tool.
Tools may import from Crossref, OpenAlex, PubMed Central or arXiv, and none should depend on them to work.

## Where next

- [Concepts](concepts.md) explains each idea with an example.
- [The specification](../spec/README.md) defines each format exactly.