infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a09d158e16158a6052e8a550f5f14c201effe1fb
paperspec/docs/comparison.md
T
alexmandClaude a09d158e16 feat: minted keys, VRF seeds, RFC 9162 proofs, history roots, tag URIs and a transparency log 
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
  Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
  where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
  openalex in same_as, else the folded title, year and family name. A
  collision lengthens the later key by two characters; a key is minted
  once and never recomputed. A document key adds .si, .v<seq> or .<type>
  for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
  okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
  every node and record ID was re-minted and the docs rewritten to match.
  The validator checks the document key against the work key and version
  (doc.key.work) and the hash part against the identifier (doc.key.suffix).
  vectors/keys.jsonl pins the cases paperstack's tests pin.

Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
  implemented in tools/oracle/vrf.py in plain Python and reproduced
  against the RFC's Example 16; the sample ranks by the VRF output. The
  earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
  verify_inclusion, consistency_proof, verify_consistency) with
  scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
  the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
  checkpoints with key hashes as transparency logs share them, the rule
  that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).

Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
  namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
  the checker refuses another version.

Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
  checkpoint records; a check that every ID and hash the prose quotes
  exists in the fixtures or vectors; three more broken packages
  (history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
  keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
  section, media-type registration and OCI distribution notes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-14 23:21:57 -04:00

482 lines
54 KiB
Markdown
Raw Blame History

# How paperspec compares with other schemas and standards
A newcomer, such as a student who knows what a scientific paper and a citation are and has not met hashes, JSON-LD or annotation standards, can read each section's first paragraph and stop.
An implementer, who will write a reader, writer, converter or checker, gets the chapter that defines each behavior and the places where the formats differ from a neighbor in ways that cause bugs.
An expert, such as a researcher, standards author or reviewer who knows the neighboring standards, gets stated limits, the status of every format, and where the other standard is the better choice.
The repository is at <https://git.scient.ing/infra/paperspec>.
This page was checked against the repository, at the version given under "Limits and status".
Every statement about another standard links to the page it was read on, and those pages were read on 2026-10-12, so versions and dates are as of that day and may have moved.
Where the checked sources say nothing, this page says "not seen", "not specified" or "not in scope", and each phrase keeps one meaning: "not in scope" means the standard's own stated purpose excludes the feature, "not seen" means it was not found in the pages read (which is not proof it is absent), and "not specified" means the page read does not define it.
A note on names.
The project is called paperspec.
Its formats are named `scienting-<name>/<major>`, for example `scienting-doc/1` and `scienting-anchor/1`.
That prefix is the formats' namespace, and the repository's name is a separate matter; formats are named after what they hold, never after a product or a repository.
The identifiers appear inside hashed data and test vectors, so they are not renamed.
## The problem, and how to choose
Suppose a research team keeps fifty papers it needs to understand completely.
A page number names a region of a PDF and cannot name a statement, and it changes between a preprint and the published version.
The team wants to point at one sentence or one table cell, such as `hale2026harmonic-damb#rr20tev@76e06b[111:130]` in the fixtures, and have that pointer keep working after someone fixes a misread digit elsewhere in the paper.
The pointer has four parts:
- The document key, `hale2026harmonic-damb`.
- After `#`, the node ID, `rr20tev`.
- After `@`, the first six digits of the node's hash, `76e06b`, so the pointer names the sentence as it then read.
- In brackets, the code-point range of the words inside the sentence, `[111:130]`.
It also wants to know who compared a passage with the page image, and how far.
That is the problem paperspec's formats solve: they store a paper as a package of files in which every sentence, table cell and equation has a permanent address and every correction is recorded with its evidence.
Each level of checking is tied to the exact text it was checked against.
Most of the standards below solve different problems, so this page compares by problem solved, not by rank.
The next table maps each job to the standard to start with and the section that compares it.
| If your job is | Start with | Compare in |
| --- | --- | --- |
| Publishers and archives exchanging journal articles | JATS (Journal Article Tag Suite) | [JATS and JATS4R](#jats-niso-z3996-and-jats4r) |
| Encoding scholarly text, or extracting structure from PDFs | TEI, GROBID, Docling, MinerU, Marker | [TEI and GROBID](#tei-and-grobid), [PDF converters](#docling-mineru-and-marker) |
| Annotating a resource on the web | W3C Web Annotation | [Web Annotation](#w3c-web-annotation-selectors-and-hypothesis-style-anchoring) |
| Mining a corpus of millions of papers | S2ORC | [S2ORC and CORD-19](#s2orc-and-cord-19) |
| Publishing one claim as a citable object | Nanopublications, micropublications | [Claims](#nanopublications-and-micropublications) |
| Packaging data or files with metadata or fixity | Frictionless, RO-Crate, BagIt | [Packaging](#packaging-frictionless-data-package-ro-crate-and-bagit) |
| Identifying and registering works and datasets | schema.org, Crossref, DataCite | [Metadata](#metadata-schemaorg-crossref-and-datacite) |
| Generic provenance, canonical JSON (JavaScript Object Notation), Merkle trees | PROV-O, RFC 8785, RFC 6962, RFC 9162 | [Building blocks](#building-blocks-prov-o-rfc-8785-rfc-6962-and-rfc-9162) |
| Content-addressed storage | IPFS CIDs, SWHIDs | [Content addressing](#content-addressing-ipfs-cids-and-swhids) |
| Linking into a PDF, text or web page | RFC 8118, Media Fragments, RFC 5147, Text Fragments | [Fragment identifiers](#fragment-identifiers-and-pandoc) |
| Exchanging text-mining annotations, bibliographies, books | BioC, CSL-JSON, EPUB 3 | [BioC, CSL-JSON and EPUB 3](#bioc-csl-json-and-epub-3) |
## What paperspec is, in the terms used below
A few terms recur.
A hash is a fixed-length fingerprint computed from content, so any change to the content changes the hash; paperspec uses SHA-256, written as 64 lowercase hexadecimal digits ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
Canonical JSON is a rule for writing JSON so that equal data always gives identical bytes, which is what makes a hash of JSON repeatable; paperspec uses the JSON Canonicalization Scheme of RFC 8785 ([spec/01 §1.1](../spec/01-conventions.md#11-canonical-json)).
A witness is one file of a paper, such as the publisher's PDF or a JATS XML file, identified by the SHA-256 of its bytes; a document is the text established from one or more witnesses ([spec/02 §2.1](../spec/02-model.md#21-four-objects)).
A package is a folder of canonical JSON files, or the same folder zipped as `.sct.zip`, with a manifest that lists every other file with its SHA-256 and size ([spec/12](../spec/12-package.md)).
The design choices that matter for comparison:
- **Node IDs:** A document is a tree of nodes (sections, paragraphs, sentences, cells, equations and so on, [spec/03](../spec/03-nodes.md)).
Each node has a 7-character ID, about 34.5 bits, valid as an HTML `id` and a URL fragment.
An ID is minted once from the document key, the node kind, the text hash and an occurrence count, then frozen: it is never recomputed, reassigned or reused.
It encodes no page, ordinal or kind.
A fresh conversion of the same input mints the same IDs ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
- **Five hashes per node:** `text_hash` covers the exact normalized text; `match_hash` is a strict projection that is the only value allowed to certify that two readings agree; `align_key` is a lossy projection that only proposes alignments; `own_hash` covers the node's own content; `node_hash` covers `{own, children}` as canonical JSON, so a change to one sentence changes the `node_hash` of each ancestor and nothing else ([spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)).
- **Anchors:** An anchor stores a node ID, hashes, an optional span in Unicode code points and a quotation with up to 32 code points of context on each side.
Resolving it gives one of five results: `current`, `changed-below`, `moved`, `reattached` or `stale`, with a drift measure for `changed-below` ([spec/06](../spec/06-anchors.md)).
- **Corrections:** Corrections are appended, never edited.
Each carries evidence, an author and a reason, and a gate refuses a digit change without image evidence ([spec/11](../spec/11-corrections.md)).
In `fixtures/packages/hale2026harmonic-damb`, correction 1 replaces `3l0.0` with `310.0` because "OCR (optical character recognition) read the digit 1 as the letter l", with a crop of the page as evidence.
- **Fidelity and receipts:** Levels F0 (converted) to F3 (proofread).
F3 comes only from a signed receipt that lists the node at its current `node_hash`; a correction drops the node back ([spec/08](../spec/08-fidelity.md)).
- **Scale:** The formats are built for a team of roughly five to 20 people with a few dozen to a few hundred papers.
A 30-page paper is about 3,000 nodes, so 500 documents are about 1.5 million database rows ([spec/13 §13.2](../spec/13-database.md#132-tables)).
## Summary table
Each row summarizes the pages read for that standard.
Cells keep the wording of the verified source; the paperspec row is checked against the repository.
Maturity is left to the sections below, because a single word hides too much.
| Standard | Sentence-level permanent IDs | Content hashes | Anchors or selectors | Correction history | Fidelity or signatures | Per-file rights | Numeric data anchors | Full-text structure |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **paperspec 0.1.0** | yes: 7-character IDs minted once, retired with a successor (spec/04) | yes: five SHA-256 values per node, plus keyed HMAC (hash-based message authentication code) forms (spec/05) | yes: ID, hash, quote, span, five outcomes with drift (spec/06) | yes: append-only, evidence-gated (spec/11) | yes: F0 to F3, signed receipts, spot-checks, error bounds (spec/08) | yes: per witness, with AI purpose and processor (spec/14) | yes: dataset values anchored to cells (spec/15; format `provisional`) | yes: sections to cells, equations, references (spec/03, 09) |
| [JATS standard page](https://www.niso.org/publications/z3996-2024-jats) (Z39.96-2024, v1.4) | not seen: no sentence element in the list shown | not seen | not in scope as an annotation model | not seen | not seen | partly: `permissions` and `license` elements exist | not seen | yes |
| [TEI P5](https://tei-c.org/guidelines/P5/) (v4.12.0) | partly: `<s>` and author-assigned `xml:id` | not seen | partly: `xml:id` pointers, XPointer schemes | partly: `<corr>`, `<sic>`, `<choice>` | not seen | not seen | not seen | yes |
| [GROBID TEI output](https://grobid.readthedocs.io/en/latest/Grobid-service/) | partly: optional sentence elements; IDs not described | not seen | partly: optional PDF coordinates | no | no; reports F1 figures | not in scope | not seen | partly: header, references, citation contexts, full text |
| [W3C Web Annotation](https://www.w3.org/TR/annotation-model/) | not in scope | not in scope | yes: TextQuote, TextPosition, Fragment, XPath, Range selectors, refinement, States | not in scope | not in scope | not in scope | not in scope | not in scope |
| [Hypothesis anchoring](https://web.hypothes.is/blog/fuzzy-anchoring/) (2013 post) | no | no | yes: Range, Position, Quote selectors, 4 reattach strategies | not in scope | not in scope | not in scope | not in scope | not in scope |
| [DoclingDocument](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json) (schema 1.10.0) | no: positional `self_ref` | no per-item hash seen; `binary_hash` is file-level | partly: `prov` with page, bbox, `charspan` | not seen | not seen | not seen | partly: table cells with spans and header flags | yes |
| [MinerU](https://github.com/opendatalab/MinerU) | no: locators embed page and block numbers | not seen | partly: locators to a character offset within a block | not seen | not seen | not seen | not seen | yes |
| [Marker](https://github.com/datalab-to/marker) | no: block IDs such as `/page/10/Page/366` | not seen | not specified | not seen | not seen | not seen | not seen | yes: tree of blocks |
| [S2ORC / doc2json](https://github.com/allenai/s2orc-doc2json) | no: paragraph level | partly: `pdf_hash` (file-level; algorithm not verified) | partly: character spans for citations, refs, equations | no | no | partly: dataset-level ODC-By 1.0 | no | partly: paragraphs with section labels |
| [CORD-19](https://arxiv.org/abs/2004.10706) | not seen | not seen | not seen | not seen | not seen | not seen | not seen | partly: "structured full text" |
| [Nanopublications](https://nanopub.net/guidelines/working_draft/) | not in scope | partly: trusty URI hash identifies the unit | not in scope | partly: integrity key shows a unit changed | not seen | not seen | not in scope | not in scope |
| [Micropublications](https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-28) | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope (claim and argument model) |
| [Frictionless Data Package](https://datapackage.org/standard/data-package/) (v2) | not in scope | partly: optional resource `hash` (MD5 default) | not in scope | not in scope | not in scope | partly: `licenses` property | no | not in scope |
| [RO-Crate](https://www.researchobject.org/ro-crate/) (1.2 Rec) | not in scope | not seen (`contentSize` recommended) | not in scope | not seen | not seen | partly: schema.org license terms | not in scope | not in scope |
| [BagIt](https://www.rfc-editor.org/rfc/rfc8493) (RFC 8493) | not in scope | yes: per-file manifests, SHA-256 or SHA-512 | not in scope | not in scope | no: "not designed to be secure against active attacks" | not in scope | not in scope | not in scope |
| [schema.org ScholarlyArticle](https://schema.org/ScholarlyArticle) | no | partly: `sha256` on MediaObject | not in scope | partly: `correction` at work level | not in scope | partly: `license` (URL) at work level | not in scope | partly: `articleBody`, `hasPart`, `citation` |
| [Crossref deposit schema](https://www.crossref.org/documentation/schema-library/) (5.5.0) | not in scope | not seen | not in scope | partly: Crossmark, work level | not in scope | not verified | not in scope | not in scope: metadata only |
| [DataCite](https://schema.datacite.org/) (4.7) | not in scope | not seen | not in scope | not seen | not in scope | not seen | not in scope | not in scope: metadata only |
| [PROV-O Recommendation](https://www.w3.org/TR/prov-o/) | not in scope | not in scope | not in scope | partly: `wasInvalidatedBy`, `wasDerivedFrom` | not in scope | not in scope | not in scope | not in scope |
| [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785) | not in scope | not in scope (canonical form only) | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope |
| [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962) / [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162) | not in scope | yes: Merkle Tree Hash definition | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope |
| [IPFS content addressing page](https://docs.ipfs.tech/concepts/content-addressing/) | no: addresses blocks and files | yes: identity is the hash | not in scope | no: new content gives a new CID | no | not in scope | not in scope | not in scope |
| [SWHID core identifiers](https://www.swhid.org/specification/v1.2/5.Core_identifiers/) (ISO/IEC 18670:2025) | not in scope | yes: SHA-1 based intrinsic identifiers | not in scope here | not in scope | not in scope | not in scope | not in scope | not in scope |
| [RFC 8118](https://www.rfc-editor.org/rfc/rfc8118) (PDF) | no | no | partly: page, destination, region, first-word search | no | no | no | no | no |
| [Media Fragments](https://www.w3.org/TR/media-frags/) | no | no | partly: time and region; nothing on text | no | no | no | no | no |
| [RFC 5147](https://www.rfc-editor.org/rfc/rfc5147) (text/plain) | no | partly: `md5=` and `length=` checks | yes: `char=` and `line=` | no | no | no | no | no |
| [Text Fragments](https://wicg.github.io/scroll-to-text-fragment/) (WICG draft) | no | no | partly: exact-text match with optional prefix and suffix | no | no | no | no | no |
| [Pandoc AST](https://hackage.haskell.org/package/pandoc-types) (pandoc-types 1.23.1) | not verified | not verified | not verified | not in scope | not in scope | not in scope | not in scope | yes: document AST (details not verified) |
| [BioC](https://pmc.ncbi.nlm.nih.gov/articles/PMC3889917) | not seen | not seen | yes: document-level offset and length per annotation | not in scope | not in scope | not in scope | not in scope | partly: passages and optional sentences |
| [CSL-JSON markup page](https://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html) | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope | not in scope: bibliographic items |
| [EPUB 3.3](https://www.w3.org/TR/epub-33/) | not seen | not seen | not seen (CFI not confirmed) | not seen | not seen | not seen | not seen | yes |
## JATS (NISO Z39.96) and JATS4R
*What it is.*
JATS, the Journal Tag Suite, is an XML vocabulary in which publishers and archives exchange journal articles.
It is ANSI/NISO Z39.96-2024, version 1.4, approved by ANSI on October 9, 2024 and published October 31, 2024, under continuous maintenance ([NISO page](https://www.niso.org/publications/z3996-2024-jats)).
Its tag sets are Journal Archiving and Interchange (most permissive), Journal Publishing (moderately prescriptive) and Article Authoring (most prescriptive), continuing NLM DTD work begun in 2002 ([jats.nlm.nih.gov](https://jats.nlm.nih.gov/)).
JATS4R is a NISO working group that publishes tagging recommendations and a validator ([JATS4R](https://jats4r.niso.org/)).
*What it does well.*
It models journal-article structure and metadata: sections, paragraphs, table wrappers and cells, reference lists, related articles, footnotes, display formulas, supplementary material, permissions and licenses ([1.4 tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/)).
Crossref's 5.4.0 deposit schema imports JATS ([Crossref](https://www.crossref.org/documentation/schema-library/metadata-deposit-schema-5-4-0/)), and a toolkit for S2ORC has a script for PMC JATS XML ([allenai/s2orc-doc2json](https://github.com/allenai/s2orc-doc2json)).
*How paperspec differs.*
JATS has `<permissions>` and `<license>` elements; where they may be placed was not verified, so no per-file rights comparison is made.
*Where JATS is the better choice.*
For publisher and archive exchange of journal content, and for the JATS4R recommendations on tagging consistency.
paperspec is not a publisher interchange format and does not replace JATS ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
*Interoperation.*
JATS is an input.
`jats` is a witness kind, and the fixture `okafor2025solvent-acth` was established from a JATS file and a PDF that agree ([fixtures/README.md](../fixtures/README.md)).
The document's `meta` is one CSL-JSON item, not JATS front matter ([spec/02 §2.2](../spec/02-model.md#22-documentjson)).
The specification defines no JATS exporter.
## TEI and GROBID
*What they are.*
The TEI P5 Guidelines are the Text Encoding Initiative's reference documentation, maintained by the TEI Consortium; the release table shows version 4.12.0, released 2026-07-28 ([TEI P5](https://tei-c.org/guidelines/P5/)).
GROBID is a Java machine-learning library that converts PDFs and similar documents to structured data, with full encoding in TEI for results ([GROBID introduction](https://grobid.readthedocs.io/en/latest/Introduction/)).
It is Apache-2.0 licensed and can return PDF coordinates ([repository](https://github.com/kermitt2/grobid)).
*What they do well.*
TEI has a sentence-like element `<s>` ([TEI element s](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html)).
Its `<corr>`, `<sic>` and `<choice>` markup preserves both readings of a corrected passage ([TEI element corr](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html)), and `<zone>` marks facsimile regions ([TEI element zone](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-zone.html)).
Pointers use `xml:id`, recommended "For durability in the presence of editing", plus XPointer schemes such as `xpath()`, `string-range()` and `match()` ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)).
GROBID extracts header metadata, references, citation contexts and full text with coordinates, and reports, in its own documentation's documentation, an F1 of about 0.87 for references on a PubMed Central set and 0.76 to 0.91 for citation contexts; it describes itself as production ready ([introduction](https://grobid.readthedocs.io/en/latest/Introduction/)).
Its `segmentSentences` option splits paragraphs into sentence elements, and `teiCoordinates=s` adds a bounding box to each ([service documentation](https://grobid.readthedocs.io/en/latest/Grobid-service/)).
*How paperspec differs.*
TEI's `xml:id` is author-assigned and durable by recommendation ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)); paperspec IDs are minted by a rule and retired by recorded successor.
GROBID's output is an automatic reading with the accuracy figures above, not a corrected, reviewed text ([introduction](https://grobid.readthedocs.io/en/latest/Introduction/)).
*Where they are the better choice.*
TEI for scholarly text encoding in general, and for apparatus-style correction markup inside a text.
GROBID as a PDF-to-structure converter for the reference and citation-context tasks it reports figures for.
*Interoperation.*
TEI from GROBID could be one reading of a PDF.
paperspec has no `tei` witness kind, so that reading would be recorded as `xml` ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
The mechanism for comparing that reading with another is sentence alignment, which pairs sentences, keeps the first reading's IDs for paired sentences, and turns every disagreement into a suspect for review ([spec/05 §5.6](../spec/05-hashes.md#56-alignment)).
No TEI or GROBID mapping is defined in the repository.
## W3C Web Annotation, selectors, and Hypothesis-style anchoring
*What it is.*
The Web Annotation Data Model is a W3C Recommendation of 23 February 2017, edited by Robert Sanderson, Paolo Ciccarese and Benjamin Young ([model](https://www.w3.org/TR/annotation-model/)); the Web Annotation Vocabulary has the same date ([vocabulary](https://www.w3.org/TR/annotation-vocab/)).
Hypothesis is an annotation tool whose 2013 blog post describes its anchoring; current client behavior was not verified ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
*What it does well.*
An annotation has a body, a target and a motivation, and the vocabulary lists motivations such as commenting, describing, highlighting, linking and questioning ([vocabulary](https://www.w3.org/TR/annotation-vocab/)).
Targets are narrowed by selectors ([model](https://www.w3.org/TR/annotation-model/)):
- `TextQuoteSelector`: `exact` required, `prefix` and `suffix` recommended.
- `TextPositionSelector`: `start` and `end`, which the model calls "very brittle with regards to changes to the resource".
- `FragmentSelector`, `XPathSelector` and `RangeSelector`, and refinement of one selector by another.
- States (`TimeState`, `HttpRequestState`), which pin a representation of the whole resource.
Hypothesis stores a range selector (XPaths plus offsets), a position selector and a quote selector with 32 characters of context, and reattaches in four steps: range, position, context-first fuzzy match, then selector-only fuzzy match ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
It identifies a PDF by an embedded fingerprint, which "persists across copies of the same document, but not across variants" ([Hypothesis help](https://web.hypothes.is/help/annotations-not-syncing-across-copies-of-a-pdf/)).
*How paperspec differs.*
Hypothesis's post lists its own limits: it assumes the system can tell two formats are the same document, mapping text to the DOM (the browser's tree of page elements) is slow on long documents, and mappings update only when Hypothesis itself changes the DOM ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
paperspec defines exact resolution outcomes and a hash pin where Hypothesis uses fuzzy matching.
*Where they are the better choice.*
The Web Annotation model and vocabulary for annotation in general, existing Web Annotation tools, and Hypothesis as a working annotation service on web pages and PDFs.
*Interoperation.*
A paperspec layer is a set of W3C Web Annotations whose target lists a custom `AnchorSelector` carrying a `scienting-anchor/1` record, optionally followed by a standard `TextQuoteSelector` "for tools that do not know ours" ([spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
The layers do not use `TextPositionSelector` or `FragmentSelector`.
The JSON-LD context (a file that maps the added terms to IRIs (internationalized web identifiers), so linked-data tools can read them) is at `https://paperspec.scient.ing/context/scienting-1.jsonld` ([context/README.md](../context/README.md)); whether that address resolves was not checked.
Whether a generic Web Annotation client displays an `AnchorSelector` was not verified.
Both Hypothesis and paperspec store a quote with 32 characters of prefix and suffix ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor)).
## Docling, MinerU and Marker
*What they are.*
Docling is an open-source document conversion toolkit, MIT-licensed (the paper is an arXiv preprint: [paper, arXiv:2501.17887](https://arxiv.org/abs/2501.17887)); `docling-core` defines its data types ([repository](https://github.com/docling-project/docling-core)), and DoclingDocument is "a unified document representation introduced with Docling v2" ([documentation](https://docling-project.github.io/docling/concepts/docling_document/)).
MinerU converts PDFs and Office documents to Markdown or JSON, states that it is not a RAG (retrieval-augmented generation) framework, and uses a custom license based on Apache 2.0 with additional conditions ([MinerU](https://github.com/opendatalab/MinerU)).
Marker converts documents to Markdown, JSON, chunks and HTML; its code is Apache 2.0 and its model weights carry a modified AI Pubs Open Rail-M license ([Marker](https://github.com/datalab-to/marker)).
*What they do well.*
Docling's schema (version 1.10.0 on `main`) covers texts, pictures, tables, groups, furniture and pages; items carry a `self_ref` such as `#/texts/12`, `label`, `prov` (page, bounding box, `charspan`) and text; tables have cells with spans and header flags ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
Docling uses AI models for layout analysis and table structure recognition and is reported integrated with LangChain, LlamaIndex and spaCy ([abstract](https://arxiv.org/abs/2501.17887)).
MinerU describes locators of the form `doc:{id}/tier:{tier}/page:{n}/block:{n}/char:{offset}` ([MinerU](https://github.com/opendatalab/MinerU)); Marker's JSON is a tree of blocks with an `id`, a `section_hierarchy` and `images` ([Marker](https://github.com/datalab-to/marker)).
*How paperspec differs.*
The Docling `TextItem` field list has no per-item hash and its `label` enum has no sentence value, so items are block-grained ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
Identifiers in all three are positional: `self_ref` indexes into lists, Marker's example IDs embed a page number, and MinerU's locators embed page and block numbers.
MinerU's guide says to preserve locators for citation.
It "does not explain what makes them stable or guarantee that they won't change across reparses" ([MinerU](https://github.com/opendatalab/MinerU)).
Docling's `charspan` is a span within an item on a page.
It is not a pinned, quote-verified anchor with resolution outcomes.
*Where they are the better choice.*
For converting PDFs and other formats, layout analysis, table structure recognition, integrations, and one-shot structured output.
paperspec does not read PDFs; extracting lines from a PDF is outside its text ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
*Interoperation.*
These are converters, which paperspec treats as backends.
A backend declaration records the tool's code and weight licenses, and a stage may run only if they pass the gate ([spec/14 §14.4](../spec/14-rights.md#144-backends-and-profiles)); `provenance.json` records the backends and cached readings used ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
Docling appears in the repository only as data: an illustrative declaration, version `illustrative`, in `fixtures/records/backend-docling.json` and `vectors/rights.jsonl`.
Their JSON output is not a named witness kind.
Reconciling their readings with others uses alignment and suspects ([spec/05 §5.6 and §5.7](../spec/05-hashes.md#56-alignment)).
## S2ORC and CORD-19
*What they are.*
S2ORC, "The Semantic Scholar Open Research Corpus", is a corpus of 81.1 million English-language academic papers with structured full text for 8.1 million open-access papers, resolved bibliographic references and inline mentions of citations, figures and tables linked to paper records (Lo, Wang, Neumann, Kinney and Weld, ACL 2020, pp. 4969 to 4983, [ACL Anthology](https://aclanthology.org/2020.acl-main.447/)).
CORD-19, "The COVID-19 Open Research Dataset", was submitted April 22, 2020 as an arXiv preprint ([arXiv:2004.10706](https://arxiv.org/abs/2004.10706)).
S2ORC is now maintained by the Semantic Scholar API product team; the original files were refactored into datasets on the APIs, which need a key, and the original 2019 and 2020 releases "are no longer available for download" ([repository](https://github.com/allenai/s2orc)).
*What they do well.*
Scale, and citation-mention linking across papers: the paper, an arXiv preprint, reports 380.5 million resolved citation links, 156.5 million of them tied to inline mentions ([arXiv:1911.02782](https://arxiv.org/pdf/1911.02782)).
The schema classes include `Paper`, with `pdf_hash`, `metadata`, `body_text`, `bib_entries` and `ref_entries`, and `Paragraph`, with `cite_spans`, `ref_spans` and `eq_spans` given as `start`, `end` and `text` ([s2orc.py](https://raw.githubusercontent.com/allenai/s2orc-doc2json/master/doc2json/s2orc.py)).
Sources are PDF via GROBID, PMC JATS XML and arXiv LaTeX ([doc2json](https://github.com/allenai/s2orc-doc2json)).
*How paperspec differs.*
Table cells are not nodes ([s2orc.py](https://raw.githubusercontent.com/allenai/s2orc-doc2json/master/doc2json/s2orc.py)).
The paper states limitations, among them GROBID-derived inline-citation errors, linking that depends on titles, and LaTeX-derived metadata "worse than those extracted from PDF" ([arXiv:1911.02782](https://arxiv.org/pdf/1911.02782)).
The release formats "have changed over time" ([doc2json](https://github.com/allenai/s2orc-doc2json)).
*Where they are the better choice.*
Corpus-scale text mining and citation-graph work across the literature, which paperspec explicitly does not target.
*Interoperation.*
The purposes and scales differ.
S2ORC-style JSON could be a source of text readings. paperspec has no S2ORC witness kind.
The `same_as` list takes `doi:`, `arxiv:`, `isbn:`, `pmid:`, `pmcid:` and `url:` among its prefixes and has no Semantic Scholar identifier ([spec/02 §2.1](../spec/02-model.md#21-four-objects)).
## Nanopublications and micropublications
*What they are.*
A nanopublication is "a small knowledge graph snippet with metadata" in RDF ([nanopub.net](https://nanopub.net/)).
The guidelines describe four graphs (head, assertion, provenance, publication info), an identifier for each nanopublication, and trusty URIs as the recommended integrity mechanism ([guidelines](https://nanopub.net/guidelines/working_draft/)).
That page carries no version number, date or status.
Trusty URIs put cryptographic hash values in URIs so that resources can be verified and kept unchanged (an arXiv preprint: [Kuhn and Dumontier, arXiv:1401.5775](https://arxiv.org/abs/1401.5775)).
Micropublications are a semantic model for claims, evidence, arguments and annotations in biomedical communication ([Clark, Ciccarese and Goble, J Biomed Semantics 2014, 5:28](https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-28)).
*What they do well.*
A claim-sized unit with machine-readable provenance and publication metadata, with an integrity check on the unit.
Micropublications add a claim, its supporting representations as a directed acyclic graph rooted at the claim, and challenges; the paper says they aim to present primary evidence while nanopublications "model only the indicated statement" ([paper](https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-28)).
*How paperspec differs.*
These models do not store the paper's text.
The guidelines read do not describe pinning a passage, and they do not mention signatures; a Java library's README shows RSA public-key signing; the signature structure was not verified ([nanopub-java](https://github.com/Nanopublication/nanopub-java)).
*Where they are the better choice.*
Publishing a claim as a citable, hash-verifiable, linked-data object, and representing support and challenge among claims.
paperspec does not define claim semantics; a statement true independently of one paper belongs outside this repository, and a statement about reading the paper belongs in a layer ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
*Interoperation.*
A layer annotation can say that a span states, uses or disputes a claim kept elsewhere ([spec/18 §18.4](../spec/18-layers.md#184-bodies)).
A nanopublication's provenance could point at a paperspec address, since an address is a URL fragment (`…#node`, [spec/06 §6.1](../spec/06-anchors.md#61-addresses)); that is an inference, and no such mapping is defined in either project.
## Packaging: Frictionless Data Package, RO-Crate and BagIt
These three describe or protect a bundle of files.
*Frictionless Data Package.*
It is "a simple container format for describing a coherent collection of data in a single package"; the descriptor `datapackage.json` requires `resources`, and version 2 is labeled current ([datapackage.org](https://datapackage.org/standard/data-package/); the v1 page shows 2007 to 2017, [specs.frictionlessdata.io](https://specs.frictionlessdata.io/data-package/)).
A resource's `hash` is MD5 by default, with other algorithms written as `algorithm:value` ([data resource](https://datapackage.org/standard/data-resource/)).
It describes tabular files and their schemas portably.
Not seen: hashing that covers descriptor and files in canonical form, per-value anchors to a source text, units and uncertainty.
It remains the right choice for the tabular container and its tooling, which paperspec adopts: a `scienting-dataset/1` is a Data Package plus units (strings a unit parser called Pint can read), uncertainty and an `origin`, with reported values anchored to the table cells or sentences that print them ([spec/15](../spec/15-datasets.md)).
The Data Package version paperspec uses is not stated in spec/15.
The dataset format is `provisional` ([registry/formats.json](../registry/formats.json)).
*RO-Crate.*
It is "a community effort to establish a lightweight approach to packaging research data with their metadata", using schema.org annotations in JSON-LD ([RO-Crate](https://www.researchobject.org/ro-crate/)).
Version 1.2 is a Recommendation published 2025-06-04, and the site calls 1.3 the newest ([1.2](https://www.researchobject.org/ro-crate/specification/1.2/index.html)).
It describes a folder's contents in `ro-crate-metadata.json`, and file entities should include `contentSize` ([data entities](https://www.researchobject.org/ro-crate/specification/1.2/data-entities.html)).
Use it for general research-object description and discovery with schema.org vocabularies.
A paperspec package folder could be described by an RO-Crate.
No such mapping exists in either project.
*BagIt.*
RFC 8493 defines a bag: `bagit.txt`, a `data/` payload of opaque octet streams, and at least one `manifest-<algorithm>.txt`; BagIt 1.0 tools must support SHA-256 and SHA-512 ([RFC 8493](https://www.rfc-editor.org/rfc/rfc8493)).
The RFC says it "is not designed to be secure against active attacks", and it has no signatures, content-level structure or payload semantics.
BagIt fits fixity and transfer of arbitrary payloads, archives and repositories.
paperspec's `MANIFEST.json` plays a similar fixity role, with a path, SHA-256 and size per file, and the hash of the manifest pins the package; the file layout differs ([spec/12 §12.3](../spec/12-package.md#123-the-manifest)).
A package could be a bag's payload, since payload files are opaque (an inference).
## Metadata: schema.org, Crossref and DataCite
*What they are.*
`ScholarlyArticle` is a schema.org type under Thing, CreativeWork and Article ([schema.org](https://schema.org/ScholarlyArticle)); release 30.1 is listed as 2026-09-16 ([releases](https://schema.org/docs/releases.html)).
Crossref's deposit schema is at 5.5.0, the version Crossref recommends ([schema library](https://www.crossref.org/documentation/schema-library/)).
DataCite Metadata Schema 4.7 was released 3 March 2026 and defines "core metadata properties chosen for an accurate and consistent identification of a resource" ([DataCite](https://schema.datacite.org/)).
*What they do well.*
Bibliographic and identification metadata.
schema.org's `CreativeWork` has `text`, `citation`, `hasPart`, `version`, `license` and a `correction` property ([CreativeWork](https://schema.org/CreativeWork)), and `MediaObject` has `sha256` ([MediaObject](https://schema.org/MediaObject)).
Crossref's Crossmark lets publishers show whether an item was corrected, retracted or updated, including to readers holding an older download ([Crossmark](https://www.crossref.org/documentation/crossmark/)).
*How paperspec differs.*
These are record-level.
A correction is flagged on the work, not on a sentence with evidence for each changed digit; the body has no node hashes; there are no fidelity levels or signed receipts; and rights are license-level information about the work, not per-file use grants with an AI purpose and processor.
None of the schema pages read describes body text, and the extent of Crossref's license and text-mining fields was not verified ([Crossref 5.4.0](https://www.crossref.org/documentation/schema-library/metadata-deposit-schema-5-4-0/)).
*Where they are the better choice.*
Identifying and describing works and datasets with DOIs (digital object identifiers), discovery, registering corrections and retractions at the work level, and structured data on web pages.
paperspec is not a citation index and imports from Crossref, OpenAlex, PubMed Central and arXiv and never depends on them ([docs/overview.md](../docs/overview.md#what-it-is-not)).
*Interoperation.*
The document's `meta` is a CSL-JSON item and `same_as` holds a DOI.
For open-access papers a producer sets `license` and a provisional rights basis from Crossref, and Crossref's license field is a named kind of evidence in a rights record ([spec/14 §14.1](../spec/14-rights.md#141-the-rights-record)).
No mapping from `meta` to Crossref, DataCite or schema.org is defined.
## Building blocks: PROV-O, RFC 8785, RFC 6962 and RFC 9162
*PROV-O.*
It is a W3C Recommendation of 30 April 2013 that expresses the PROV Data Model in OWL2, with the classes Entity, Activity and Agent and properties such as `wasGeneratedBy`, `wasDerivedFrom` and `wasInvalidatedBy` ([PROV-O Recommendation](https://www.w3.org/TR/prov-o/)).
It is a general vocabulary and defines no node hashes, anchor resolution, corrections or receipts.
paperspec does not use it: provenance is a file, `provenance.json`, defined in the repository ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
PROV-O is the better choice for linked-data provenance that other systems can query generically.
A mapping of stages and corrections to Activity and Entity is possible.
Nothing in either project defines one.
*RFC 8785 (JSON Canonicalization Scheme).*
Published June 2020 as an Informational RFC, it sorts object properties recursively in UTF-16 code unit order, keeps array order, writes numbers by ECMAScript rules, uses no whitespace, outputs UTF-8, and does not normalize Unicode strings ([RFC 8785](https://www.rfc-editor.org/rfc/rfc8785)).
It is a building block that defines no document model, and paperspec uses it: every hash of JSON is over RFC 8785 output.
paperspec adds rules on top: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits, and a document with a lone surrogate, `NaN` or an infinity has no canonical form and is invalid ([spec/01 §1.1 and §1.2](../spec/01-conventions.md#11-canonical-json)).
*RFC 6962 and RFC 9162 (Merkle trees).*
A Merkle tree hashes items pairwise up to one root, so one short value commits to all of them.
RFC 6962 (Certificate Transparency, June 2013, Experimental) is obsolete; RFC 9162 (Certificate Transparency Version 2.0, published December 9, 2021, Experimental) "obsoletes RFC 6962" ([RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162)).
Both give the same Merkle Tree Hash: a leaf is `HASH(0x00 || d)`, an interior node is `HASH(0x01 || left || right)`, and a list splits at the largest power of two smaller than n.
paperspec's receipts use this construction with SHA-256 and cite RFC 6962 ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
They differ in one rule: a receipt with no items is invalid in paperspec, while the RFC defines the hash of an empty list.
The citation is to the obsolete RFC; the tree it defines is the one RFC 9162 also defines.
paperspec's `node_hash` is not this construction (see "Limits and status" below).
## Content addressing: IPFS CIDs and SWHIDs
*IPFS CID.*
A content identifier is derived from the content's hash, and "any difference in the content will produce a different CID"; a CID holds the hash and the codec, with sha-256 as the default hash ([IPFS docs](https://docs.ipfs.tech/concepts/content-addressing/)).
It gives verifiable, location-independent identity to immutable data.
A CID covers the root block of a DAG (directed acyclic graph), "not a direct hash of the file's bytes", so chunk size, layout, codec, version and hash all change the CID for the same file.
Identity and content are the same thing, so an edit makes a new identifier; the page describes no ID that survives edits, no sentence granularity, corrections or fidelity.
Keep using CIDs for storage and distribution of immutable blobs.
paperspec's asset store names files by SHA-256, not CID ([spec/12 §12.6](../spec/12-package.md#126-assets)).
*Software Heritage SWHID.*
ISO/IEC 18670:2025 specifies the SoftWare Hash IDentifier, version 1.2, published April 2025 ([ISO catalog page](https://www.iso.org/standard/89985.html); [swhid.org](https://www.swhid.org/specification/v1.2/)).
Core identifiers have the form `swh:1:<tag>:<hash>` for contents, directories, revisions, releases and snapshots, each SHA-1 over a typed header and a serialized form, and "intrinsic": computed from the object's content and relevant metadata, not its location ([core identifiers](https://www.swhid.org/specification/v1.2/5.Core_identifiers/)).
The qualified-identifier clause was not read.
Keep using SWHIDs for persistent, content-derived identification and archiving of source code.
*The contrast.*
A CID or SWHID is a function of content at all times.
A paperspec node ID is derived from content once, at minting, and then frozen; the content hashes live separately in `node_hash`, so the pair of ID and hash tells whether the text has changed ([spec/04 §4.2 and §4.3](../spec/04-identity.md#42-minting-node-ids), [spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)).
paperspec's hashes are SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
The cost of the choice: because the ID is minted from the text hash, two independent conversions that read a sentence differently by one character mint different IDs for it.
## Fragment identifiers and Pandoc
A fragment identifier is the part of a URL after `#` that says where to look inside the resource.
- **PDF fragments (RFC 8118):** Informational, March 2017.
Parameters include `page`, `nameddest`, `structelem`, `zoom`, `viewrect`, `highlight` and `search`; none selects text by character offset, `search` "selects the first matching word in the document", and `highlight` takes rectangle coordinates ([RFC 8118](https://www.rfc-editor.org/rfc/rfc8118)).
Keep using it to open a PDF at a page, destination or region. paperspec's `locations.jsonl` stores per-node page boxes as `[x0, y0, x1, y1]` in PDF points against page fingerprints ([spec/07 §7.4](../spec/07-geometry.md#74-locationsjsonl)), so a tool could build an RFC 8118 link; the repository does not define that.
- **Media Fragments URI 1.0:** A W3C Recommendation of 25 September 2012 that defines time (`t=`) and spatial (`xywh=`) fragments for audio, video and images, and says nothing about text ([Media Fragments](https://www.w3.org/TR/media-frags/)).
It is a weak fit for text passages.
- **RFC 5147 (text/plain):** A Proposed Standard of April 2008 with `char=` and `line=` positions counted from zero and integrity checks `length=` and `md5=`, "to detect whether the resource has changed" ([RFC 5147](https://www.rfc-editor.org/rfc/rfc5147)).
It is the closest older precedent to a position-plus-integrity address; it uses MD5 only and applies to plain text, not to paper structure.
- **Text Fragments:** A WICG Draft Community Group Report, not a W3C standard; the version read is dated 13 December 2023 and may not be the latest.
The syntax is `#:~:text=[prefix-,]start[,end][,-suffix]`, it matches the first exact occurrence, and it is restricted to top-level navigations ([WICG draft report](https://wicg.github.io/scroll-to-text-fragment/)).
- **DOM and XPath approaches:** W3C `XPathSelector` and `RangeSelector` ([model](https://www.w3.org/TR/annotation-model/)), TEI's `xpath()` scheme ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)) and Hypothesis's first strategy, which "fails if the XPath expressions can no longer be applied" ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
Path addressing breaks when structure changes; paperspec's display paths are never stored or cited ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
- **Pandoc:** The pandoc-types library defines the `Pandoc` document type with modules for building, walking and JSON conversion; the latest on Hackage is 1.23.1, uploaded 2023-08-06 ([Hackage](https://hackage.haskell.org/package/pandoc-types)).
Its Block and Inline types and its treatment of identifiers were not verified. `latex` is a paperspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)); Pandoc is not mentioned in the repository.
On the web, a paperspec node ID is the URL fragment, so `…#k3m9q2a` lands on the sentence ([spec/06 §6.1](../spec/06-anchors.md#61-addresses)).
This is a plain HTML fragment, not a registered fragment scheme; the chapters define no fragment syntax for the package media type.
## BioC, CSL-JSON and EPUB 3
*BioC.*
BioC is a simple extensible XML format for sharing text documents and annotations between natural-language-processing and text-mining tools, with code to read and write it ([BioC paper](https://pmc.ncbi.nlm.nih.gov/articles/PMC3889917); [bioc.sourceforge.net](https://bioc.sourceforge.net/)).
Its model has collection, source, document, passage, optional sentence, annotation and relation, with `infon` key-value pairs, and an annotation's location is an offset and length relative to the whole document.
Its authors state that it "targets tool developers and not end users" and "focuses on text data rather than other media".
Keep using it to exchange text-mining annotations among biomedical natural-language-processing (NLP) tools.
Offsets would need conversion, since paperspec counts Unicode code points ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)) and the length unit in BioC was taken from a page summary and not confirmed.
No BioC mapping is defined.
*CSL-JSON.*
It is a JSON format for bibliographic items consumed by Citation Style Language (CSL) processors; items require `id` and `type`, `author` is an array of name objects, and `issued` uses `date-parts` or `raw` ([markup page of the CSL-JSON documentation](https://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html)).
Its schema is described as "not yet fully normative" ([CSL schema](https://github.com/citation-style-language/schema)).
Keep using it for bibliographic interchange with citation processors and reference managers.
paperspec embeds CSL-JSON as the document's `meta` and as the optional parsed `csl` of each reference ([spec/02 §2.2](../spec/02-model.md#22-documentjson), [spec/10](../spec/10-references.md)).
*EPUB 3.*
EPUB 3.3 is a W3C Recommendation, "a distribution and interchange format for digital publications and documents": a ZIP container with a package document listing a manifest and a spine, and XHTML or SVG content documents ([EPUB 3.3](https://www.w3.org/TR/epub-33/)).
It suits reflowable publications.
Keep using it as a reading and distribution format.
`epub` is a paperspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)), and the zip package rules (UTF-8 names, one top-level folder, no symlinks) are the package's own ([spec/12 §12.2](../spec/12-package.md#122-paths)).
## Limits and status
*Status.*
The specification is version 0.1.0.
`registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `tools/check.py`.
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; tools Apache-2.0 ([README.md](../README.md#licenses)).
*No adoption claim.*
The repository names no outside adopter and no independent implementation, and this page claims none.
*Scope of this comparison.*
The pages read cover no ORKG, DocBook, PubAnnotation, OpenAlex or PMC OA XML material, so those are not compared.
A statement that no single comparator combines paperspec's features describes the pages read and surveys no other systems.
*What paperspec does not do.*
- It is not a publisher interchange format; it ingests JATS and other formats as witnesses and does not replace them ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
- It is not a search engine, a citation index, a recommendation feed or a chat-with-a-PDF tool ([docs/overview.md](../docs/overview.md#what-it-is-not)).
Search is a per-bundle SQLite FTS5 index (the bundle format is `provisional`), which is the extent of its search ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)).
- It does not define claim semantics ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
- It does not model glyph-level PDF layout beyond per-node boxes; extracting lines from a PDF is outside its text ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
- It compares complex math by source only; comparing canonical MathML is a possible later addition ([spec/05 §5.3](../spec/05-hashes.md#53-the-strict-projection), [docs/rationale.md](../docs/rationale.md#questions-the-formats-leave-open)).
- It is built for hundreds of documents, not hundreds of millions, and it does not fix what a converter reads wrongly; it records the error and makes it checkable.
*Where the repository's guides are looser than the specification.*
The specification governs.
These are the places where the guides and the specification differ, and this page uses the corrected statements.
1. **Reconversion:** `README.md` says an address "survives reconversion with a better tool".
The specification says a fresh conversion of the same input mints the same IDs ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
With a different converter, sentences the alignment pairs keep their IDs and every difference becomes a suspect ([spec/05 §5.6](../spec/05-hashes.md#56-alignment); [docs/faq.md](../docs/faq.md)).
IDs survive a better converter only for paired sentences.
2. **"Merkle":** The guides and spec/05 §5.1 call the root `node_hash` a Merkle root.
A `node_hash` is a hash of the canonical JSON of `{"own": own_hash, "children": [...]}`.
Only receipts follow the RFC 6962 construction ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
The two constructions are not the same.
3. **"Opaque" IDs:** Calling IDs meaningless or random is loose.
An ID is minted by the rule given above, so it is neither meaningless nor random.
4. **Five hashes everywhere:** `docs/overview.md` lists the things that get "a permanent ID and five content hashes" without naming them.
Table columns and figure-text items have IDs, are not nodes, and carry no five hashes ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids), [spec/09 §9.1](../spec/09-figures-tables-equations.md#91-figures-and-panels)).
5. **Dataset digits:** spec/15 says values keep their printed digits, and in JSON they are strings.
The fixture descriptor `fixtures/packages/hale2026harmonic-damb/datasets/ds-hale-bond-params/datapackage.json` types the field `k_r` as `number`.
How Frictionless tools treat that was not verified, so take care when comparing with Frictionless Table Schema.
6. **Package tree:** `docs/concepts.md` omits `suspects.jsonl` from its package tree, and [spec/12 §12.1](../spec/12-package.md#121-files) lists it.
7. **RFC 6962 is obsolete:** spec/08 §8.5 cites it; RFC 9162 obsoletes it (see above).
8. **Status:** The guides describe the formats as one set.
Of the 32 formats, 13 are `provisional`, including `scienting-dataset/1`, which the datasets paragraphs in `docs/concepts.md` present without that qualifier.
*The distinguishing combination, and what it costs.*
paperspec puts in one package permanent sentence- and cell-level IDs, five hashes per node, anchors with explicit resolution outcomes and drift, append-only evidence-gated corrections, signed fidelity receipts with spot-check bounds, per-witness rights with AI-use axes, and value-to-cell anchors.
Individual pieces have precedents in the comparators: TEI correction markup, Web Annotation selectors, RFC 5147 integrity checks, RFC 6962 Merkle trees, BagIt manifests and Frictionless packaging.
No single comparator in the pages read combines them.
The price is weight: five hashes per node, a gate on corrections, signed records and a custom selector type.
The design target is a team's few dozen to few hundred papers; the weight is out of proportion for publisher exchange or corpus-scale mining.
Reference in new issue View Git Blame Copy Permalink