infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: rename to paperspec and add the website

Browse Source 
- Rename manuspec to paperspec everywhere outside pixi.toml: prose, the
  tools, provenance tool names and the fixture key derivations, so every
  fixture and vector is rebuilt.
- Move the namespace addresses under the site, mirroring the repository:
  schema $id https://paperspec.scient.ing/schemas/<file>, the JSON-LD
  context https://paperspec.scient.ing/context/scienting-1.jsonld, and
  term IRIs https://paperspec.scient.ing/vocab#<term>. Format
  identifiers (scienting-*) and URNs (urn:scienting:) are unchanged.
- tools/site.py (standard library only) builds _site/: every file of the
  repository at its own path, each Markdown file rendered as a page with
  sidebar navigation, heading anchors matching the forge's, folder
  listings, a vocabulary page for the context's IRIs, a heading search
  and a 404 page; light and dark themes.
- check.py builds the site and checks that every Markdown file has a
  page, every internal link and fragment resolves, no Markdown is left
  unrendered, and every $id, the context address and the vocabulary IRIs
  are files of the site.
- deploy/: a Caddyfile with the media types and CORS the data needs, and
  a guide; .gitea/workflows/site.yml builds on main and publishes with
  rsync when the SITE_* secrets are set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 21:27:21 -04:00
1 parent ea0a7efa78
commit 76d76353d9
213 files changed
+1233 -386
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/comparison.md
+53 -53
View File
@@ -1,16 +1,16 @@
# How manuspec compares with other schemas and standards
# How paperspec compares with other schemas and standards
A newcomer, such as a student who knows what a scientific paper and a citation are and has not met hashes, JSON-LD or annotation standards, can read each section's first paragraph and stop.
An implementer, who will write a reader, writer, converter or checker, gets the chapter that defines each behavior and the places where the formats differ from a neighbor in ways that cause bugs.
An expert, such as a researcher, standards author or reviewer who knows the neighboring standards, gets stated limits, the status of every format, and where the other standard is the better choice.
The repository is at <https://git.scient.ing/infra/manuspec>.
The repository is at <https://git.scient.ing/infra/paperspec>.
This page was checked against the repository, at the version given under "Limits and status".
Every statement about another standard links to the page it was read on, and those pages were read on 2026-10-12, so versions and dates are as of that day and may have moved.
Where the checked sources say nothing, this page says "not seen", "not specified" or "not in scope", and each phrase keeps one meaning: "not in scope" means the standard's own stated purpose excludes the feature, "not seen" means it was not found in the pages read (which is not proof it is absent), and "not specified" means the page read does not define it.
A note on names.
The project is called manuspec.
The project is called paperspec.
Its formats are named `scienting-<name>/<major>`, for example `scienting-doc/1` and `scienting-anchor/1`.
That prefix is the formats' namespace, and the repository's name is a separate matter; formats are named after what they hold, never after a product or a repository.
The identifiers appear inside hashed data and test vectors, so they are not renamed.
@@ -27,7 +27,7 @@ The pointer has four parts:
- After `@`, the first six digits of the node's hash, `76e06b`, so the pointer names the sentence as it then read.
- In brackets, the code-point range of the words inside the sentence, `[111:130]`.
It also wants to know who compared a passage with the page image, and how far.
That is the problem manuspec's formats solve: they store a paper as a package of files in which every sentence, table cell and equation has a permanent address and every correction is recorded with its evidence.
That is the problem paperspec's formats solve: they store a paper as a package of files in which every sentence, table cell and equation has a permanent address and every correction is recorded with its evidence.
Each level of checking is tied to the exact text it was checked against.
Most of the standards below solve different problems, so this page compares by problem solved, not by rank.
@@ -47,11 +47,11 @@ The next table maps each job to the standard to start with and the section that
| Linking into a PDF, text or web page | RFC 8118, Media Fragments, RFC 5147, Text Fragments | [Fragment identifiers](#fragment-identifiers-and-pandoc) |
| Exchanging text-mining annotations, bibliographies, books | BioC, CSL-JSON, EPUB 3 | [BioC, CSL-JSON and EPUB 3](#bioc-csl-json-and-epub-3) |
## What manuspec is, in the terms used below
## What paperspec is, in the terms used below
A few terms recur.
A hash is a fixed-length fingerprint computed from content, so any change to the content changes the hash; manuspec uses SHA-256, written as 64 lowercase hexadecimal digits ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
Canonical JSON is a rule for writing JSON so that equal data always gives identical bytes, which is what makes a hash of JSON repeatable; manuspec uses the JSON Canonicalization Scheme of RFC 8785 ([spec/01 §1.1](../spec/01-conventions.md#11-canonical-json)).
A hash is a fixed-length fingerprint computed from content, so any change to the content changes the hash; paperspec uses SHA-256, written as 64 lowercase hexadecimal digits ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
Canonical JSON is a rule for writing JSON so that equal data always gives identical bytes, which is what makes a hash of JSON repeatable; paperspec uses the JSON Canonicalization Scheme of RFC 8785 ([spec/01 §1.1](../spec/01-conventions.md#11-canonical-json)).
A witness is one file of a paper, such as the publisher's PDF or a JATS XML file, identified by the SHA-256 of its bytes; a document is the text established from one or more witnesses ([spec/02 §2.1](../spec/02-model.md#21-four-objects)).
A package is a folder of canonical JSON files, or the same folder zipped as `.sct.zip`, with a manifest that lists every other file with its SHA-256 and size ([spec/12](../spec/12-package.md)).
@@ -76,12 +76,12 @@ The design choices that matter for comparison:
## Summary table
Each row summarizes the pages read for that standard.
Cells keep the wording of the verified source; the manuspec row is checked against the repository.
Cells keep the wording of the verified source; the paperspec row is checked against the repository.
Maturity is left to the sections below, because a single word hides too much.
| Standard | Sentence-level permanent IDs | Content hashes | Anchors or selectors | Correction history | Fidelity or signatures | Per-file rights | Numeric data anchors | Full-text structure |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **manuspec 0.1.0** | yes: 7-character IDs minted once, retired with a successor (spec/04) | yes: five SHA-256 values per node, plus keyed HMAC (hash-based message authentication code) forms (spec/05) | yes: ID, hash, quote, span, five outcomes with drift (spec/06) | yes: append-only, evidence-gated (spec/11) | yes: F0 to F3, signed receipts, spot-checks, error bounds (spec/08) | yes: per witness, with AI purpose and processor (spec/14) | yes: dataset values anchored to cells (spec/15; format `provisional`) | yes: sections to cells, equations, references (spec/03, 09) |
| **paperspec 0.1.0** | yes: 7-character IDs minted once, retired with a successor (spec/04) | yes: five SHA-256 values per node, plus keyed HMAC (hash-based message authentication code) forms (spec/05) | yes: ID, hash, quote, span, five outcomes with drift (spec/06) | yes: append-only, evidence-gated (spec/11) | yes: F0 to F3, signed receipts, spot-checks, error bounds (spec/08) | yes: per witness, with AI purpose and processor (spec/14) | yes: dataset values anchored to cells (spec/15; format `provisional`) | yes: sections to cells, equations, references (spec/03, 09) |
| [JATS standard page](https://www.niso.org/publications/z3996-2024-jats) (Z39.96-2024, v1.4) | not seen: no sentence element in the list shown | not seen | not in scope as an annotation model | not seen | not seen | partly: `permissions` and `license` elements exist | not seen | yes |
| [TEI P5](https://tei-c.org/guidelines/P5/) (v4.12.0) | partly: `<s>` and author-assigned `xml:id` | not seen | partly: `xml:id` pointers, XPointer schemes | partly: `<corr>`, `<sic>`, `<choice>` | not seen | not seen | not seen | yes |
| [GROBID TEI output](https://grobid.readthedocs.io/en/latest/Grobid-service/) | partly: optional sentence elements; IDs not described | not seen | partly: optional PDF coordinates | no | no; reports F1 figures | not in scope | not seen | partly: header, references, citation contexts, full text |
@@ -126,12 +126,12 @@ JATS4R is a NISO working group that publishes tagging recommendations and a vali
It models journal-article structure and metadata: sections, paragraphs, table wrappers and cells, reference lists, related articles, footnotes, display formulas, supplementary material, permissions and licenses ([1.4 tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/)).
Crossref's 5.4.0 deposit schema imports JATS ([Crossref](https://www.crossref.org/documentation/schema-library/metadata-deposit-schema-5-4-0/)), and a toolkit for S2ORC has a script for PMC JATS XML ([allenai/s2orc-doc2json](https://github.com/allenai/s2orc-doc2json)).
*How manuspec differs.*
*How paperspec differs.*
JATS has `<permissions>` and `<license>` elements; where they may be placed was not verified, so no per-file rights comparison is made.
*Where JATS is the better choice.*
For publisher and archive exchange of journal content, and for the JATS4R recommendations on tagging consistency.
manuspec is not a publisher interchange format and does not replace JATS ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
paperspec is not a publisher interchange format and does not replace JATS ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
*Interoperation.*
JATS is an input.
@@ -153,8 +153,8 @@ Pointers use `xml:id`, recommended "For durability in the presence of editing",
GROBID extracts header metadata, references, citation contexts and full text with coordinates, and reports, in its own documentation's documentation, an F1 of about 0.87 for references on a PubMed Central set and 0.76 to 0.91 for citation contexts; it describes itself as production ready ([introduction](https://grobid.readthedocs.io/en/latest/Introduction/)).
Its `segmentSentences` option splits paragraphs into sentence elements, and `teiCoordinates=s` adds a bounding box to each ([service documentation](https://grobid.readthedocs.io/en/latest/Grobid-service/)).
*How manuspec differs.*
TEI's `xml:id` is author-assigned and durable by recommendation ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)); manuspec IDs are minted by a rule and retired by recorded successor.
*How paperspec differs.*
TEI's `xml:id` is author-assigned and durable by recommendation ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)); paperspec IDs are minted by a rule and retired by recorded successor.
GROBID's output is an automatic reading with the accuracy figures above, not a corrected, reviewed text ([introduction](https://grobid.readthedocs.io/en/latest/Introduction/)).
*Where they are the better choice.*
@@ -163,7 +163,7 @@ GROBID as a PDF-to-structure converter for the reference and citation-context ta
*Interoperation.*
TEI from GROBID could be one reading of a PDF.
manuspec has no `tei` witness kind, so that reading would be recorded as `xml` ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
paperspec has no `tei` witness kind, so that reading would be recorded as `xml` ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
The mechanism for comparing that reading with another is sentence alignment, which pairs sentences, keeps the first reading's IDs for paired sentences, and turns every disagreement into a suspect for review ([spec/05 §5.6](../spec/05-hashes.md#56-alignment)).
No TEI or GROBID mapping is defined in the repository.
@@ -185,19 +185,19 @@ Targets are narrowed by selectors ([model](https://www.w3.org/TR/annotation-mode
Hypothesis stores a range selector (XPaths plus offsets), a position selector and a quote selector with 32 characters of context, and reattaches in four steps: range, position, context-first fuzzy match, then selector-only fuzzy match ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
It identifies a PDF by an embedded fingerprint, which "persists across copies of the same document, but not across variants" ([Hypothesis help](https://web.hypothes.is/help/annotations-not-syncing-across-copies-of-a-pdf/)).
*How manuspec differs.*
*How paperspec differs.*
Hypothesis's post lists its own limits: it assumes the system can tell two formats are the same document, mapping text to the DOM (the browser's tree of page elements) is slow on long documents, and mappings update only when Hypothesis itself changes the DOM ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
manuspec defines exact resolution outcomes and a hash pin where Hypothesis uses fuzzy matching.
paperspec defines exact resolution outcomes and a hash pin where Hypothesis uses fuzzy matching.
*Where they are the better choice.*
The Web Annotation model and vocabulary for annotation in general, existing Web Annotation tools, and Hypothesis as a working annotation service on web pages and PDFs.
*Interoperation.*
A manuspec layer is a set of W3C Web Annotations whose target lists a custom `AnchorSelector` carrying a `scienting-anchor/1` record, optionally followed by a standard `TextQuoteSelector` "for tools that do not know ours" ([spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
A paperspec layer is a set of W3C Web Annotations whose target lists a custom `AnchorSelector` carrying a `scienting-anchor/1` record, optionally followed by a standard `TextQuoteSelector` "for tools that do not know ours" ([spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
The layers do not use `TextPositionSelector` or `FragmentSelector`.
The JSON-LD context (a file that maps the added terms to IRIs (internationalized web identifiers), so linked-data tools can read them) is at `https://scient.ing/ns/scienting/1.jsonld` ([context/README.md](../context/README.md)); whether that address resolves was not checked.
The JSON-LD context (a file that maps the added terms to IRIs (internationalized web identifiers), so linked-data tools can read them) is at `https://paperspec.scient.ing/context/scienting-1.jsonld` ([context/README.md](../context/README.md)); whether that address resolves was not checked.
Whether a generic Web Annotation client displays an `AnchorSelector` was not verified.
Both Hypothesis and manuspec store a quote with 32 characters of prefix and suffix ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor)).
Both Hypothesis and paperspec store a quote with 32 characters of prefix and suffix ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor)).
## Docling, MinerU and Marker
@@ -211,7 +211,7 @@ Docling's schema (version 1.10.0 on `main`) covers texts, pictures, tables, grou
Docling uses AI models for layout analysis and table structure recognition and is reported integrated with LangChain, LlamaIndex and spaCy ([abstract](https://arxiv.org/abs/2501.17887)).
MinerU describes locators of the form `doc:{id}/tier:{tier}/page:{n}/block:{n}/char:{offset}` ([MinerU](https://github.com/opendatalab/MinerU)); Marker's JSON is a tree of blocks with an `id`, a `section_hierarchy` and `images` ([Marker](https://github.com/datalab-to/marker)).
*How manuspec differs.*
*How paperspec differs.*
The Docling `TextItem` field list has no per-item hash and its `label` enum has no sentence value, so items are block-grained ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
Identifiers in all three are positional: `self_ref` indexes into lists, Marker's example IDs embed a page number, and MinerU's locators embed page and block numbers.
MinerU's guide says to preserve locators for citation.
@@ -221,10 +221,10 @@ It is not a pinned, quote-verified anchor with resolution outcomes.
*Where they are the better choice.*
For converting PDFs and other formats, layout analysis, table structure recognition, integrations, and one-shot structured output.
manuspec does not read PDFs; extracting lines from a PDF is outside its text ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
paperspec does not read PDFs; extracting lines from a PDF is outside its text ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
*Interoperation.*
These are converters, which manuspec treats as backends.
These are converters, which paperspec treats as backends.
A backend declaration records the tool's code and weight licenses, and a stage may run only if they pass the gate ([spec/14 §14.4](../spec/14-rights.md#144-backends-and-profiles)); `provenance.json` records the backends and cached readings used ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
Docling appears in the repository only as data: an illustrative declaration, version `illustrative`, in `fixtures/records/backend-docling.json` and `vectors/rights.jsonl`.
Their JSON output is not a named witness kind.
@@ -242,17 +242,17 @@ Scale, and citation-mention linking across papers: the paper, an arXiv preprint,
The schema classes include `Paper`, with `pdf_hash`, `metadata`, `body_text`, `bib_entries` and `ref_entries`, and `Paragraph`, with `cite_spans`, `ref_spans` and `eq_spans` given as `start`, `end` and `text` ([s2orc.py](https://raw.githubusercontent.com/allenai/s2orc-doc2json/master/doc2json/s2orc.py)).
Sources are PDF via GROBID, PMC JATS XML and arXiv LaTeX ([doc2json](https://github.com/allenai/s2orc-doc2json)).
*How manuspec differs.*
*How paperspec differs.*
Table cells are not nodes ([s2orc.py](https://raw.githubusercontent.com/allenai/s2orc-doc2json/master/doc2json/s2orc.py)).
The paper states limitations, among them GROBID-derived inline-citation errors, linking that depends on titles, and LaTeX-derived metadata "worse than those extracted from PDF" ([arXiv:1911.02782](https://arxiv.org/pdf/1911.02782)).
The release formats "have changed over time" ([doc2json](https://github.com/allenai/s2orc-doc2json)).
*Where they are the better choice.*
Corpus-scale text mining and citation-graph work across the literature, which manuspec explicitly does not target.
Corpus-scale text mining and citation-graph work across the literature, which paperspec explicitly does not target.
*Interoperation.*
The purposes and scales differ.
S2ORC-style JSON could be a source of text readings. manuspec has no S2ORC witness kind.
S2ORC-style JSON could be a source of text readings. paperspec has no S2ORC witness kind.
The `same_as` list takes `doi:`, `arxiv:`, `isbn:`, `pmid:`, `pmcid:` and `url:` among its prefixes and has no Semantic Scholar identifier ([spec/02 §2.1](../spec/02-model.md#21-four-objects)).
## Nanopublications and micropublications
@@ -268,17 +268,17 @@ Micropublications are a semantic model for claims, evidence, arguments and annot
A claim-sized unit with machine-readable provenance and publication metadata, with an integrity check on the unit.
Micropublications add a claim, its supporting representations as a directed acyclic graph rooted at the claim, and challenges; the paper says they aim to present primary evidence while nanopublications "model only the indicated statement" ([paper](https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-28)).
*How manuspec differs.*
*How paperspec differs.*
These models do not store the paper's text.
The guidelines read do not describe pinning a passage, and they do not mention signatures; a Java library's README shows RSA public-key signing; the signature structure was not verified ([nanopub-java](https://github.com/Nanopublication/nanopub-java)).
*Where they are the better choice.*
Publishing a claim as a citable, hash-verifiable, linked-data object, and representing support and challenge among claims.
manuspec does not define claim semantics; a statement true independently of one paper belongs outside this repository, and a statement about reading the paper belongs in a layer ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
paperspec does not define claim semantics; a statement true independently of one paper belongs outside this repository, and a statement about reading the paper belongs in a layer ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
*Interoperation.*
A layer annotation can say that a span states, uses or disputes a claim kept elsewhere ([spec/18 §18.4](../spec/18-layers.md#184-bodies)).
A nanopublication's provenance could point at a manuspec address, since an address is a URL fragment (`…#node`, [spec/06 §6.1](../spec/06-anchors.md#61-addresses)); that is an inference, and no such mapping is defined in either project.
A nanopublication's provenance could point at a paperspec address, since an address is a URL fragment (`…#node`, [spec/06 §6.1](../spec/06-anchors.md#61-addresses)); that is an inference, and no such mapping is defined in either project.
## Packaging: Frictionless Data Package, RO-Crate and BagIt
@@ -289,8 +289,8 @@ It is "a simple container format for describing a coherent collection of data in
A resource's `hash` is MD5 by default, with other algorithms written as `algorithm:value` ([data resource](https://datapackage.org/standard/data-resource/)).
It describes tabular files and their schemas portably.
Not seen: hashing that covers descriptor and files in canonical form, per-value anchors to a source text, units and uncertainty.
It remains the right choice for the tabular container and its tooling, which manuspec adopts: a `scienting-dataset/1` is a Data Package plus units (strings a unit parser called Pint can read), uncertainty and an `origin`, with reported values anchored to the table cells or sentences that print them ([spec/15](../spec/15-datasets.md)).
The Data Package version manuspec uses is not stated in spec/15.
It remains the right choice for the tabular container and its tooling, which paperspec adopts: a `scienting-dataset/1` is a Data Package plus units (strings a unit parser called Pint can read), uncertainty and an `origin`, with reported values anchored to the table cells or sentences that print them ([spec/15](../spec/15-datasets.md)).
The Data Package version paperspec uses is not stated in spec/15.
The dataset format is `provisional` ([registry/formats.json](../registry/formats.json)).
*RO-Crate.*
@@ -298,14 +298,14 @@ It is "a community effort to establish a lightweight approach to packaging resea
Version 1.2 is a Recommendation published 2025-06-04, and the site calls 1.3 the newest ([1.2](https://www.researchobject.org/ro-crate/specification/1.2/index.html)).
It describes a folder's contents in `ro-crate-metadata.json`, and file entities should include `contentSize` ([data entities](https://www.researchobject.org/ro-crate/specification/1.2/data-entities.html)).
Use it for general research-object description and discovery with schema.org vocabularies.
A manuspec package folder could be described by an RO-Crate.
A paperspec package folder could be described by an RO-Crate.
No such mapping exists in either project.
*BagIt.*
RFC 8493 defines a bag: `bagit.txt`, a `data/` payload of opaque octet streams, and at least one `manifest-<algorithm>.txt`; BagIt 1.0 tools must support SHA-256 and SHA-512 ([RFC 8493](https://www.rfc-editor.org/rfc/rfc8493)).
The RFC says it "is not designed to be secure against active attacks", and it has no signatures, content-level structure or payload semantics.
BagIt fits fixity and transfer of arbitrary payloads, archives and repositories.
manuspec's `MANIFEST.json` plays a similar fixity role, with a path, SHA-256 and size per file, and the hash of the manifest pins the package; the file layout differs ([spec/12 §12.3](../spec/12-package.md#123-the-manifest)).
paperspec's `MANIFEST.json` plays a similar fixity role, with a path, SHA-256 and size per file, and the hash of the manifest pins the package; the file layout differs ([spec/12 §12.3](../spec/12-package.md#123-the-manifest)).
A package could be a bag's payload, since payload files are opaque (an inference).
## Metadata: schema.org, Crossref and DataCite
@@ -320,14 +320,14 @@ Bibliographic and identification metadata.
schema.org's `CreativeWork` has `text`, `citation`, `hasPart`, `version`, `license` and a `correction` property ([CreativeWork](https://schema.org/CreativeWork)), and `MediaObject` has `sha256` ([MediaObject](https://schema.org/MediaObject)).
Crossref's Crossmark lets publishers show whether an item was corrected, retracted or updated, including to readers holding an older download ([Crossmark](https://www.crossref.org/documentation/crossmark/)).
*How manuspec differs.*
*How paperspec differs.*
These are record-level.
A correction is flagged on the work, not on a sentence with evidence for each changed digit; the body has no node hashes; there are no fidelity levels or signed receipts; and rights are license-level information about the work, not per-file use grants with an AI purpose and processor.
None of the schema pages read describes body text, and the extent of Crossref's license and text-mining fields was not verified ([Crossref 5.4.0](https://www.crossref.org/documentation/schema-library/metadata-deposit-schema-5-4-0/)).
*Where they are the better choice.*
Identifying and describing works and datasets with DOIs (digital object identifiers), discovery, registering corrections and retractions at the work level, and structured data on web pages.
manuspec is not a citation index and imports from Crossref, OpenAlex, PubMed Central and arXiv and never depends on them ([docs/overview.md](../docs/overview.md#what-it-is-not)).
paperspec is not a citation index and imports from Crossref, OpenAlex, PubMed Central and arXiv and never depends on them ([docs/overview.md](../docs/overview.md#what-it-is-not)).
*Interoperation.*
The document's `meta` is a CSL-JSON item and `same_as` holds a DOI.
@@ -339,24 +339,24 @@ No mapping from `meta` to Crossref, DataCite or schema.org is defined.
*PROV-O.*
It is a W3C Recommendation of 30 April 2013 that expresses the PROV Data Model in OWL2, with the classes Entity, Activity and Agent and properties such as `wasGeneratedBy`, `wasDerivedFrom` and `wasInvalidatedBy` ([PROV-O Recommendation](https://www.w3.org/TR/prov-o/)).
It is a general vocabulary and defines no node hashes, anchor resolution, corrections or receipts.
manuspec does not use it: provenance is a file, `provenance.json`, defined in the repository ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
paperspec does not use it: provenance is a file, `provenance.json`, defined in the repository ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
PROV-O is the better choice for linked-data provenance that other systems can query generically.
A mapping of stages and corrections to Activity and Entity is possible.
Nothing in either project defines one.
*RFC 8785 (JSON Canonicalization Scheme).*
Published June 2020 as an Informational RFC, it sorts object properties recursively in UTF-16 code unit order, keeps array order, writes numbers by ECMAScript rules, uses no whitespace, outputs UTF-8, and does not normalize Unicode strings ([RFC 8785](https://www.rfc-editor.org/rfc/rfc8785)).
It is a building block that defines no document model, and manuspec uses it: every hash of JSON is over RFC 8785 output.
manuspec adds rules on top: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits, and a document with a lone surrogate, `NaN` or an infinity has no canonical form and is invalid ([spec/01 §1.1 and §1.2](../spec/01-conventions.md#11-canonical-json)).
It is a building block that defines no document model, and paperspec uses it: every hash of JSON is over RFC 8785 output.
paperspec adds rules on top: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits, and a document with a lone surrogate, `NaN` or an infinity has no canonical form and is invalid ([spec/01 §1.1 and §1.2](../spec/01-conventions.md#11-canonical-json)).
*RFC 6962 and RFC 9162 (Merkle trees).*
A Merkle tree hashes items pairwise up to one root, so one short value commits to all of them.
RFC 6962 (Certificate Transparency, June 2013, Experimental) is obsolete; RFC 9162 (Certificate Transparency Version 2.0, published December 9, 2021, Experimental) "obsoletes RFC 6962" ([RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162)).
Both give the same Merkle Tree Hash: a leaf is `HASH(0x00 || d)`, an interior node is `HASH(0x01 || left || right)`, and a list splits at the largest power of two smaller than n.
manuspec's receipts use this construction with SHA-256 and cite RFC 6962 ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
They differ in one rule: a receipt with no items is invalid in manuspec, while the RFC defines the hash of an empty list.
paperspec's receipts use this construction with SHA-256 and cite RFC 6962 ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
They differ in one rule: a receipt with no items is invalid in paperspec, while the RFC defines the hash of an empty list.
The citation is to the obsolete RFC; the tree it defines is the one RFC 9162 also defines.
manuspec's `node_hash` is not this construction (see "Limits and status" below).
paperspec's `node_hash` is not this construction (see "Limits and status" below).
## Content addressing: IPFS CIDs and SWHIDs
@@ -366,7 +366,7 @@ It gives verifiable, location-independent identity to immutable data.
A CID covers the root block of a DAG (directed acyclic graph), "not a direct hash of the file's bytes", so chunk size, layout, codec, version and hash all change the CID for the same file.
Identity and content are the same thing, so an edit makes a new identifier; the page describes no ID that survives edits, no sentence granularity, corrections or fidelity.
Keep using CIDs for storage and distribution of immutable blobs.
manuspec's asset store names files by SHA-256, not CID ([spec/12 §12.6](../spec/12-package.md#126-assets)).
paperspec's asset store names files by SHA-256, not CID ([spec/12 §12.6](../spec/12-package.md#126-assets)).
*Software Heritage SWHID.*
ISO/IEC 18670:2025 specifies the SoftWare Hash IDentifier, version 1.2, published April 2025 ([ISO catalog page](https://www.iso.org/standard/89985.html); [swhid.org](https://www.swhid.org/specification/v1.2/)).
@@ -376,8 +376,8 @@ Keep using SWHIDs for persistent, content-derived identification and archiving o
*The contrast.*
A CID or SWHID is a function of content at all times.
A manuspec node ID is derived from content once, at minting, and then frozen; the content hashes live separately in `node_hash`, so the pair of ID and hash tells whether the text has changed ([spec/04 §4.2 and §4.3](../spec/04-identity.md#42-minting-node-ids), [spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)).
manuspec's hashes are SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
A paperspec node ID is derived from content once, at minting, and then frozen; the content hashes live separately in `node_hash`, so the pair of ID and hash tells whether the text has changed ([spec/04 §4.2 and §4.3](../spec/04-identity.md#42-minting-node-ids), [spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)).
paperspec's hashes are SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
The cost of the choice: because the ID is minted from the text hash, two independent conversions that read a sentence differently by one character mint different IDs for it.
## Fragment identifiers and Pandoc
@@ -386,7 +386,7 @@ A fragment identifier is the part of a URL after `#` that says where to look ins
- **PDF fragments (RFC 8118):** Informational, March 2017.
Parameters include `page`, `nameddest`, `structelem`, `zoom`, `viewrect`, `highlight` and `search`; none selects text by character offset, `search` "selects the first matching word in the document", and `highlight` takes rectangle coordinates ([RFC 8118](https://www.rfc-editor.org/rfc/rfc8118)).
Keep using it to open a PDF at a page, destination or region. manuspec's `locations.jsonl` stores per-node page boxes as `[x0, y0, x1, y1]` in PDF points against page fingerprints ([spec/07 §7.4](../spec/07-geometry.md#74-locationsjsonl)), so a tool could build an RFC 8118 link; the repository does not define that.
Keep using it to open a PDF at a page, destination or region. paperspec's `locations.jsonl` stores per-node page boxes as `[x0, y0, x1, y1]` in PDF points against page fingerprints ([spec/07 §7.4](../spec/07-geometry.md#74-locationsjsonl)), so a tool could build an RFC 8118 link; the repository does not define that.
- **Media Fragments URI 1.0:** A W3C Recommendation of 25 September 2012 that defines time (`t=`) and spatial (`xywh=`) fragments for audio, video and images, and says nothing about text ([Media Fragments](https://www.w3.org/TR/media-frags/)).
It is a weak fit for text passages.
- **RFC 5147 (text/plain):** A Proposed Standard of April 2008 with `char=` and `line=` positions counted from zero and integrity checks `length=` and `md5=`, "to detect whether the resource has changed" ([RFC 5147](https://www.rfc-editor.org/rfc/rfc5147)).
@@ -394,11 +394,11 @@ A fragment identifier is the part of a URL after `#` that says where to look ins
- **Text Fragments:** A WICG Draft Community Group Report, not a W3C standard; the version read is dated 13 December 2023 and may not be the latest.
The syntax is `#:~:text=[prefix-,]start[,end][,-suffix]`, it matches the first exact occurrence, and it is restricted to top-level navigations ([WICG draft report](https://wicg.github.io/scroll-to-text-fragment/)).
- **DOM and XPath approaches:** W3C `XPathSelector` and `RangeSelector` ([model](https://www.w3.org/TR/annotation-model/)), TEI's `xpath()` scheme ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)) and Hypothesis's first strategy, which "fails if the XPath expressions can no longer be applied" ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
Path addressing breaks when structure changes; manuspec's display paths are never stored or cited ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
Path addressing breaks when structure changes; paperspec's display paths are never stored or cited ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
- **Pandoc:** The pandoc-types library defines the `Pandoc` document type with modules for building, walking and JSON conversion; the latest on Hackage is 1.23.1, uploaded 2023-08-06 ([Hackage](https://hackage.haskell.org/package/pandoc-types)).
Its Block and Inline types and its treatment of identifiers were not verified. `latex` is a manuspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)); Pandoc is not mentioned in the repository.
Its Block and Inline types and its treatment of identifiers were not verified. `latex` is a paperspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)); Pandoc is not mentioned in the repository.
On the web, a manuspec node ID is the URL fragment, so `…#k3m9q2a` lands on the sentence ([spec/06 §6.1](../spec/06-anchors.md#61-addresses)).
On the web, a paperspec node ID is the URL fragment, so `…#k3m9q2a` lands on the sentence ([spec/06 §6.1](../spec/06-anchors.md#61-addresses)).
This is a plain HTML fragment, not a registered fragment scheme; the chapters define no fragment syntax for the package media type.
## BioC, CSL-JSON and EPUB 3
@@ -408,20 +408,20 @@ BioC is a simple extensible XML format for sharing text documents and annotation
Its model has collection, source, document, passage, optional sentence, annotation and relation, with `infon` key-value pairs, and an annotation's location is an offset and length relative to the whole document.
Its authors state that it "targets tool developers and not end users" and "focuses on text data rather than other media".
Keep using it to exchange text-mining annotations among biomedical natural-language-processing (NLP) tools.
Offsets would need conversion, since manuspec counts Unicode code points ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)) and the length unit in BioC was taken from a page summary and not confirmed.
Offsets would need conversion, since paperspec counts Unicode code points ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)) and the length unit in BioC was taken from a page summary and not confirmed.
No BioC mapping is defined.
*CSL-JSON.*
It is a JSON format for bibliographic items consumed by Citation Style Language (CSL) processors; items require `id` and `type`, `author` is an array of name objects, and `issued` uses `date-parts` or `raw` ([markup page of the CSL-JSON documentation](https://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html)).
Its schema is described as "not yet fully normative" ([CSL schema](https://github.com/citation-style-language/schema)).
Keep using it for bibliographic interchange with citation processors and reference managers.
manuspec embeds CSL-JSON as the document's `meta` and as the optional parsed `csl` of each reference ([spec/02 §2.2](../spec/02-model.md#22-documentjson), [spec/10](../spec/10-references.md)).
paperspec embeds CSL-JSON as the document's `meta` and as the optional parsed `csl` of each reference ([spec/02 §2.2](../spec/02-model.md#22-documentjson), [spec/10](../spec/10-references.md)).
*EPUB 3.*
EPUB 3.3 is a W3C Recommendation, "a distribution and interchange format for digital publications and documents": a ZIP container with a package document listing a manifest and a spine, and XHTML or SVG content documents ([EPUB 3.3](https://www.w3.org/TR/epub-33/)).
It suits reflowable publications.
Keep using it as a reading and distribution format.
`epub` is a manuspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)), and the zip package rules (UTF-8 names, one top-level folder, no symlinks) are the package's own ([spec/12 §12.2](../spec/12-package.md#122-paths)).
`epub` is a paperspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)), and the zip package rules (UTF-8 names, one top-level folder, no symlinks) are the package's own ([spec/12 §12.2](../spec/12-package.md#122-paths)).
## Limits and status
@@ -437,9 +437,9 @@ The repository names no outside adopter and no independent implementation, and t
*Scope of this comparison.*
The pages read cover no ORKG, DocBook, PubAnnotation, OpenAlex or PMC OA XML material, so those are not compared.
A statement that no single comparator combines manuspec's features describes the pages read and surveys no other systems.
A statement that no single comparator combines paperspec's features describes the pages read and surveys no other systems.
*What manuspec does not do.*
*What paperspec does not do.*
- It is not a publisher interchange format; it ingests JATS and other formats as witnesses and does not replace them ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
- It is not a search engine, a citation index, a recommendation feed or a chat-with-a-PDF tool ([docs/overview.md](../docs/overview.md#what-it-is-not)).
@@ -474,7 +474,7 @@ These are the places where the guides and the specification differ, and this pag
Of the 29 formats, 11 are `provisional`, including `scienting-dataset/1`, which the datasets paragraphs in `docs/concepts.md` present without that qualifier.
*The distinguishing combination, and what it costs.*
manuspec puts in one package permanent sentence- and cell-level IDs, five hashes per node, anchors with explicit resolution outcomes and drift, append-only evidence-gated corrections, signed fidelity receipts with spot-check bounds, per-witness rights with AI-use axes, and value-to-cell anchors.
paperspec puts in one package permanent sentence- and cell-level IDs, five hashes per node, anchors with explicit resolution outcomes and drift, append-only evidence-gated corrections, signed fidelity receipts with spot-check bounds, per-witness rights with AI-use axes, and value-to-cell anchors.
Individual pieces have precedents in the comparators: TEI correction markup, Web Annotation selectors, RFC 5147 integrity checks, RFC 6962 Merkle trees, BagIt manifests and Frictionless packaging.
No single comparator in the pages read combines them.
The price is weight: five hashes per node, a gate on corrections, signed records and a custom selector type.