docs: the tools' names in TitleCase

Prose names are PaperSpec, PaperDesk, PaperClaim and so on; commands, packages and paths stay lowercase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014zNf4shD4zaCKLucZWPYL9
This commit is contained in:
alexmandClaude committed 2026-10-21 09:49:58 -04:00
1 parent 105896a18b
commit 03b4722023
24 files changed
+111 -111

No files matched your search

+14 -14
View File
@@ -38,7 +38,7 @@ Packaging, commands and lazy imports only; the formats stay at 0.1.0, and no com
### Unreleased
Two changes asked for by the first producer that compares a converted reading with a second witness (paperforms with a JATS file).
Two changes asked for by the first producer that compares a converted reading with a second witness (PaperForms with a JATS file).
Both are minor changes in the sense of [spec/00](spec/00-introduction.md): no valid file becomes invalid, and no hash, ID or key changes; the next release numbers them.
- Alignment has a substitution step ([spec/05 §5.6](spec/05-hashes.md#56-alignment), step 3): in each stretch between pairs, an unpaired sentence of the first reading is paired with the first unpaired sentence of the second whose similarity (2*L* / (|*a*| + |*b*|) over code points of the lossy projections, *L* their longest common subsequence) is at least 0.8, both of prose length 20 or more.
@@ -62,7 +62,7 @@ They are minor changes too: new operations, enumeration values and optional prop
- The rules `correction.emend`, `correction.relink`, `correction.resolve` and `doc.correction.resolution` are registered, the `corrections` table's CHECK takes the three operations ([sql/doc.sql](sql/doc.sql)), `corrections.State` carries the reference records, and `paperspec stress` walks the three operations with their invariants.
`corrections.jsonl` gains thirteen cases and `projections.jsonl` and `projection-groups.jsonl` two each; no existing case changes.
Four more, asked for by papermath, which defines the math form `scienting-math/1`, and by paperclaim, the claim store whose claims and symbols layers point at.
Four more, asked for by PaperMath, which defines the math form `scienting-math/1`, and by PaperClaim, the claim store whose claims and symbols layers point at.
The first two are minor changes: an optional property, a derived file and its schema, with no hash, ID or key of an existing file changed.
The last two make some files that were valid invalid, which [spec/00](spec/00-introduction.md) calls breaking; they are made now because no release of `scienting-doc/1` or `scienting-layer/1` has been tagged, and a reference worth changing is changed before one is ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), item 4).
@@ -71,14 +71,14 @@ The last two make some files that were valid invalid, which [spec/00](spec/00-in
`form` enters no content value ([spec/05 §5.2](spec/05-hashes.md#52-own-content)): own content leaves it out of math runs (`paperspec.hashing`), an equation's own content never held more than its notation and source, and no projection reads it, so adding `form` to a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does.
Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first.
`hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator now does in part (see the last section).
- `derived/math.jsonl` ([spec/12 §12.7](spec/12-package.md#127-derived-files), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `math.schema.json`): one line per formula, `{node, run, tool, form, tree, symbols, complete, issues, label?}`, papermath's reading without the canonical TeX, which is the source; `run` is null for an equation, as in `derived/mathml.jsonl`, and `label` is omitted when the source has no `\tag`.
- `derived/math.jsonl` ([spec/12 §12.7](spec/12-package.md#127-derived-files), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `math.schema.json`): one line per formula, `{node, run, tool, form, tree, symbols, complete, issues, label?}`, PaperMath's reading without the canonical TeX, which is the source; `run` is null for an equation, as in `derived/mathml.jsonl`, and `label` is omitted when the source has no `\tag`.
A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 49 kinds and six codes of version 1.
The validator now checks derived files: each line against its schema, that it names a formula of the package (`doc.derived.node`), that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`), and that a checks-only package carries none (`doc.policy.withheld`), since a tree or MathML reads the withheld text.
The main example gains `derived/math.jsonl`, papermath 0.1.0's reading of its equation and of two math runs, copied by hand because the oracle does not implement the form; `paperspec check` validates fourteen variants of it, and no fixture's hashes change.
The main example gains `derived/math.jsonl`, PaperMath 0.1.0's reading of its equation and of two math runs, copied by hand because the oracle does not implement the form; `paperspec check` validates fourteen variants of it, and no fixture's hashes change.
There are now 43 schemas, which the 0.1.0 entry above, held to the repository by `paperspec check`, states too.
- Claim references ([spec/18 §18.4](spec/18-layers.md#184-bodies), [spec/02 §2.4](spec/02-model.md#24-work-and-document-keys), [spec/20](spec/20-projects.md), `claimRef` and `claimPin` in `common.schema.json`): the `source` of a `claim-link` or `symbol` body, a `claim:` identifier in `same_as`, and a project's claim step are paperclaim references, `claim:<tag>` or `claim:<host>/<tag>` with the tag eight Crockford base32 capitals, and a `pin` is the reference, `@` and six lowercase hex digits of the claim's statement hash, as paperclaim writes them; a project names its claim namespace as `claims.paperclaim`.
A reference by slug, such as `amber/k-bond`, which paperclaim may rename, or under any other prefix, was valid before and is not now.
The fixtures' slug reference and `illustrative-pin` become `claim:272XTX7C` and `claim:272XTX7C@02f47b`, the claim `amber/k-bond` of paperclaim's example store, which changes the claim link's and the symbol's record IDs, the layer receipt and `project.json`.
- Claim references ([spec/18 §18.4](spec/18-layers.md#184-bodies), [spec/02 §2.4](spec/02-model.md#24-work-and-document-keys), [spec/20](spec/20-projects.md), `claimRef` and `claimPin` in `common.schema.json`): the `source` of a `claim-link` or `symbol` body, a `claim:` identifier in `same_as`, and a project's claim step are PaperClaim references, `claim:<tag>` or `claim:<host>/<tag>` with the tag eight Crockford base32 capitals, and a `pin` is the reference, `@` and six lowercase hex digits of the claim's statement hash, as PaperClaim writes them; a project names its claim namespace as `claims.paperclaim`.
A reference by slug, such as `amber/k-bond`, which PaperClaim may rename, or under any other prefix, was valid before and is not now.
The fixtures' slug reference and `illustrative-pin` become `claim:272XTX7C` and `claim:272XTX7C@02f47b`, the claim `amber/k-bond` of PaperClaim's example store, which changes the claim link's and the symbol's record IDs, the layer receipt and `project.json`.
The main example's two targeted receipts take the scope `targeted:paperclaim`, which changes their IDs and signatures, Ben's spot-check of the second (its seed and its sample of ten, still without errors), the QA report, the inclusion proof, the transparency log's checkpoint and tiles, and the signature, spot-check, VRF and checkpoint vectors built from them; no node's level or the report's counts change.
- Symbol keys ([spec/18 §18.4](spec/18-layers.md#184-bodies), `annotation.schema.json`, the `key` term of the JSON-LD context, `sct:symbolKey`): a `symbol` body names its symbol by `key`, its `scienting-math/1` key, besides the span its target selects, so a symbol is found across papers and matched to its claim by key; a `symbol` body without `key` was valid before and is not now.
The example layer gains a proposed symbol annotation on `$K_r$`, with the key `K_{r}`.
@@ -88,7 +88,7 @@ The last two make some files that were valid invalid, which [spec/00](spec/00-in
The third was the stress harness's own: a truncation could cut a file at its full length, leaving the package unchanged and valid, so it now always shortens the file.
So was a fourth, with 1000 rounds: the random walk of corrections could delete the body section, leaving nothing to correct and too few corrections accepted, so a delete now leaves at least eight nodes.
One more, asked for by paperdesk, which lets a reader follow a citation to the passage it rests on.
One more, asked for by PaperDesk, which lets a reader follow a citation to the passage it rests on.
It is a minor change in the sense of [spec/00](spec/00-introduction.md): a new body kind, optional properties and two rules, with no valid file made invalid, no computed value changed and no existing vector case or package changed.
- Traced citations ([spec/18 §18.5](spec/18-layers.md#185-traced-citations), `annotation.schema.json`, the `note` and `proposer` terms of the JSON-LD context): a `trace` body follows one citation of a sentence of a citing paper to a passage of the cited paper.
@@ -103,7 +103,7 @@ It is a minor change in the sense of [spec/00](spec/00-introduction.md): a new b
`paperspec check` runs both layers through the record checks and the store.
The example project holds the new layer, which changes `project.json`; no other fixture changes.
One more, asked for by paperchem and paperbio, which tie molecules and sequences to the words of a paper.
One more, asked for by PaperChem and PaperBio, which tie molecules and sequences to the words of a paper.
It is a minor change in the sense of [spec/00](spec/00-introduction.md): two new body kinds, optional properties and seven rules, with no valid file made invalid and no computed value changed; the example layers, the traces and every receipt are as they were.
- Chemical structures ([spec/18 §18.6](spec/18-layers.md#186-chemical-structures), `annotation.schema.json`): a `structure` body ties a SMILES or SMARTS to a span, with a `role` (`names`, `depicts`, `parametrizes` or `uses`), an `identity` that finds the same structure however it was written (the InChIKey of a SMILES, else `smiles:` or `smarts:` and twelve hex digits of the SHA-256 of the value), the paper's own `written` notation, a `label`, outside `accessions` and `terms`: the bonds, angles, torsions and impropers a paper gives constants for.
@@ -122,7 +122,7 @@ It is a minor change in the sense of [spec/00](spec/00-introduction.md): two new
The draft gains a last sentence, which the biology layer is about, so the draft package, its draft IDs and the one traces vector that anchors its paragraph change; the traces layer's anchors, its receipt and every other vector are as they were.
- The prose check's UK spelling pattern no longer takes `organism` for a form of `organise`.
One more, asked for by paperdata, which finds data by its kind and by the molecules and sequences it is about.
One more, asked for by PaperData, which finds data by its kind and by the molecules and sequences it is about.
It is a minor change in the sense of [spec/00](spec/00-introduction.md): one new body kind, optional properties and four rules, with no valid file made invalid and no computed value changed; the other layers, the traces and every receipt are as they were.
- Data ([spec/18 §18.8](spec/18-layers.md#188-data), `annotation.schema.json`): a `data` body says what kind of data a span presents, with a `data_type` from a fixed list of fifteen, a `role` (`reports` or `uses`), the `dataset` that holds the values, the `quantities` its fields measure (a field, a name and the dataset's unit), the `subjects` it is about (the identities of `structure` and `sequence` bodies), a `method`, a `label` and outside `accessions`, so that a term of an ontology can say more than the list does.
@@ -154,16 +154,16 @@ Tooling and code quality only; no computed value, vector, fixture, schema or SQL
- Python files open with an `SPDX-License-Identifier: Apache-2.0` header, which ruff and a test check.
- The checker and the site builder skip `work/`, caches and build output.
One more, the work the other paper tools planned for paperspec (paperdesk's plan, item M5.4, and paperforms's, §4.2): a command that applies corrections, two checks the validator had left to producers, and public names for what the tools had been importing as private ones.
One more, the work the other paper tools planned for PaperSpec (PaperDesk's plan, item M5.4, and PaperForms's, §4.2): a command that applies corrections, two checks the validator had left to producers, and public names for what the tools had been importing as private ones.
It is a minor change in the sense of [spec/00](spec/00-introduction.md): new rules, functions and a command, with no fixture package made invalid and no computed value changed.
- `paperspec correct` and `paperspec.revision` apply proposed corrections to a package, a folder or a zip, through the gate ([spec/11](spec/11-corrections.md)), and write the package they make, in place or to `--out`, only once it validates.
A changed leaf becomes F1 when a crop of it is the evidence and F0 otherwise ([spec/08 §8.1](spec/08-fidelity.md#81-levels-of-fidelity)), and an `emend`, `relink` or `resolve` keeps its level.
A retired node's locations move to its successor, suspects at removed nodes and those named by `--answered` are dropped, derived lines and datasets that read the corrected text are left out and reported, and provenance gains a `review` stage.
With no corrections it writes every example package and every valid package of paperforms's bench back byte for byte (`test_revision.py`, `test_cli_correct.py`).
With no corrections it writes every example package and every valid package of PaperForms's bench back byte for byte (`test_revision.py`, `test_cli_correct.py`).
- `doc.math.form` is checked ([spec/03 §3.7](spec/03-nodes.md#37-math-forms)).
The validator refuses a source in the form that holds a label or style command the form never holds, such as `\tag`, `\label`, `\nonumber`, `\displaystyle` or `\dfrac`.
When papermath is installed, or a caller passes `forms=`, it also refuses a source that is not the form's own spelling or that the form cannot read (`test_math_form.py`).
When PaperMath is installed, or a caller passes `forms=`, it also refuses a source that is not the form's own spelling or that the form cannot read (`test_math_form.py`).
- The validator checks a package's datasets ([spec/15](spec/15-datasets.md), `paperspec.datasets`): the descriptor against its schema, the folder named by the ID, a unique primary key, each value anchor resolving as current to the text the CSV holds, and every numeric value of a reported resource anchored.
These are the rules `doc.dataset.schema`, `.id`, `.file`, `.key`, `.anchor`, `.value` and `.unanchored` (`test_datasets.py`).
- `fixtures/invalid/` gains `suspect-witness` and `math-form-label`, so there are 33 broken packages, which the 0.1.0 entry above states too (`paperspec check`).
@@ -172,4 +172,4 @@ It is a minor change in the sense of [spec/00](spec/00-introduction.md): new rul
- Public names, with the old private names kept as aliases: `package.parse_canonical`, `parse_json`, `parse_jsonl`, `check_zip_entries` and `read_zip_files`; `tree.flatten` and `establish_with_side`, which keeps a producer's private keys; `hashing.strict_runs`; and `corrections.OPS` and `EMEND_KINDS` (`test_public_api.py`).
Also `anchors.follow` and `resolve_address`, `claims.parse_ref`, `parse_pin`, `format_ref` and `format_pin`, `layers.annotation_id`, `anchors_of` and the lists of body kinds, categories, verdicts and statuses, and `db.load_layer`, `read_state`, `delete` and `upgrade` (`test_anchors.py`, `test_claims.py`, `test_layers.py`, `test_db.py`).
- `text.collapse_ws`, `trim`, `remove_ws` and `digits` and `hashing.lossy_from_strict` are rewritten for speed, so a correction goes through the gate about three times faster, and are tested equal to the old code on every code point (`test_text.py`, `test_hashing.py`).
- The private aliases kept for the siblings are gone, now that paperdesk and paperforms use the public names: `package._canonical_bytes`, `package._zip_entries`, `anchors._follow` and `traces._anchors` (`tests/test_public_api.py::test_parse_canonical_reads_doubles_and_refuses_what_is_not_canonical`, `::test_check_zip_entries_returns_the_folder_without_reading`, `tests/test_anchors.py::TestFollow::test_follow_has_no_alias_under_its_old_private_name`).
- The private aliases kept for the siblings are gone, now that PaperDesk and PaperForms use the public names: `package._canonical_bytes`, `package._zip_entries`, `anchors._follow` and `traces._anchors` (`tests/test_public_api.py::test_parse_canonical_reads_doubles_and_refuses_what_is_not_canonical`, `::test_check_zip_entries_returns_the_folder_without_reading`, `tests/test_anchors.py::TestFollow::test_follow_has_no_alias_under_its_old_private_name`).
+2 -2
View File
@@ -1,4 +1,4 @@
# paperspec
# PaperSpec
**The data formats for storing a team's scientific papers once, correcting them once, addressing them down to the sentence, and keeping them.**
@@ -212,7 +212,7 @@ The Python package is an oracle, not a production reader or writer: it favors cl
It needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
Only `paperspec validate`, `paperspec correct` and `paperspec load` work anywhere the package is installed; `check`, `build`, `stress` and `site` work on a checkout of this repository.
`paperspec correct` cannot rebuild what pinned tools derived from the text: it leaves out the lines of `derived/` files and the datasets that read corrected text, and says which, for the tool that made them to rebuild.
The validator checks the math form `scienting-math/1` in full only when papermath is installed; without it, it refuses only what the form never holds, such as a `\tag`.
The validator checks the math form `scienting-math/1` in full only when PaperMath is installed; without it, it refuses only what the form never holds, such as a `\tag`.
It accepts more than one location record for a node on one page, which [spec/07 §7.4](spec/07-geometry.md#74-locationsjsonl) rules out.
## Licenses
+1 -1
View File
@@ -29,7 +29,7 @@ Every layer and every annotation lists both contexts, Web Annotation first:
```
The `scienting-` prefix is the formats' namespace, as in `scienting-layer/1` and `scienting-anchor/1`.
The repository's name, paperspec, is a separate matter.
The repository's name, PaperSpec, is a separate matter.
## An example from the fixtures
+1 -1
View File
@@ -1,6 +1,6 @@
# Deploying the website and the Python package
This folder holds what it takes to serve the paperspec website at <https://paperspec.scient.ing>.
This folder holds what it takes to serve the PaperSpec website at <https://paperspec.scient.ing>.
The last sections say how the Python package `paperspec` is published to the Gitea package registry, and how other repositories install it from there.
The site is static: `paperspec site` builds it into `_site/`, and any web server can serve that folder.
+4 -4
View File
@@ -1,18 +1,18 @@
# Guides
These guides explain paperspec to someone meeting it for the first time.
These guides explain PaperSpec to someone meeting it for the first time.
They serve three readers: a newcomer who has not seen the problem before, an implementer who will write a reader, writer, converter or checker, and an expert who knows the neighboring standards.
The [specification](../spec/README.md) says exactly what each format is.
The guides say what each format is for, with examples drawn from the fixtures.
| Guide | For |
| --- | --- |
| [Overview](overview.md) | What paperspec is, who it serves, and how its parts fit together. |
| [Overview](overview.md) | What PaperSpec is, who it serves, and how its parts fit together. |
| [Concepts](concepts.md) | The ideas behind the formats, one at a time, each with a real example. |
| [Walkthrough](walkthrough.md) | One paper, from a flawed first reading to a corrected, reviewed and signed document. |
| [Examples](examples.md) | Things to run: checking hashes by hand, reading a package with a few lines of Python, querying a database. |
| [Rationale](rationale.md) | Why the formats look the way they do, and the questions they leave open. |
| [Comparison](comparison.md) | How paperspec compares with JATS, TEI, W3C Web Annotation and other schemas. |
| [Comparison](comparison.md) | How PaperSpec compares with JATS, TEI, W3C Web Annotation and other schemas. |
| [*FAQ*](faq.md) | Short answers to the questions people ask first. |
| [Glossary](glossary.md) | Every term, in a sentence. |
@@ -27,7 +27,7 @@ Next, run the section "Test your implementation against a vector" in the example
Check your work against the [vectors](../vectors/README.md) and the [fixtures](../fixtures/README.md).
Chapter 22 of the specification defines conformance.
**Expert.** Read the overview for scope and for what paperspec is not, then the comparison for how it differs from existing standards.
**Expert.** Read the overview for scope and for what PaperSpec is not, then the comparison for how it differs from existing standards.
After that, read the rationale for the reason behind each rule and the questions it leaves open.
It ends with a list of limits.
Section 0.4 of the specification and [registry/formats.json](../registry/formats.json) give the status of each format: the version is 0.1.0, and each format is `candidate` or `provisional`.
+51 -51
View File
@@ -1,4 +1,4 @@
# How paperspec compares with other schemas and standards
# How PaperSpec compares with other schemas and standards
A newcomer, such as a student who knows what a scientific paper and a citation are and has not met hashes, JSON-LD or annotation standards, can read each section's first paragraph and stop.
An implementer, who will write a reader, writer, converter or checker, gets the chapter that defines each behavior and the places where the formats differ from a neighbor in ways that cause bugs.
@@ -10,7 +10,7 @@ Every statement about another standard links to the page it was read on, and tho
Where the checked sources say nothing, this page says "not seen", "not specified" or "not in scope", and each phrase keeps one meaning: "not in scope" means the standard's own stated purpose excludes the feature, "not seen" means it was not found in the pages read (which is not proof it is absent), and "not specified" means the page read does not define it.
A note on names.
The project is called paperspec.
The project is called PaperSpec.
Its formats are named `scienting-<name>/<major>`, for example `scienting-doc/1` and `scienting-anchor/1`.
That prefix is the formats' namespace, and the repository's name is a separate matter; formats are named after what they hold, never after a product or a repository.
The identifiers appear inside hashed data and test vectors, so they are not renamed.
@@ -27,7 +27,7 @@ The pointer has four parts:
- After `@`, the first six digits of the node's hash, `76e06b`, so the pointer names the sentence as it then read.
- In brackets, the code-point range of the words inside the sentence, `[111:130]`.
It also wants to know who compared a passage with the page image, and how far.
That is the problem paperspec's formats solve: they store a paper as a package of files in which every sentence, table cell and equation has a permanent address and every correction is recorded with its evidence.
That is the problem PaperSpec's formats solve: they store a paper as a package of files in which every sentence, table cell and equation has a permanent address and every correction is recorded with its evidence.
Each level of checking is tied to the exact text it was checked against.
Most of the standards below solve different problems, so this page compares by problem solved, not by rank.
@@ -47,11 +47,11 @@ The next table maps each job to the standard to start with and the section that
| Linking into a PDF, text or web page | RFC 8118, Media Fragments, RFC 5147, Text Fragments | [Fragment identifiers](#fragment-identifiers-and-pandoc) |
| Exchanging text-mining annotations, bibliographies, books | BioC, CSL-JSON, EPUB 3 | [BioC, CSL-JSON and EPUB 3](#bioc-csl-json-and-epub-3) |
## What paperspec is, in the terms used below
## What PaperSpec is, in the terms used below
A few terms recur.
A hash is a fixed-length fingerprint computed from content, so any change to the content changes the hash; paperspec uses SHA-256, written as 64 lowercase hexadecimal digits ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
Canonical JSON is a rule for writing JSON so that equal data always gives identical bytes, which is what makes a hash of JSON repeatable; paperspec uses the JSON Canonicalization Scheme of RFC 8785 ([spec/01 §1.1](../spec/01-conventions.md#11-canonical-json)).
A hash is a fixed-length fingerprint computed from content, so any change to the content changes the hash; PaperSpec uses SHA-256, written as 64 lowercase hexadecimal digits ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
Canonical JSON is a rule for writing JSON so that equal data always gives identical bytes, which is what makes a hash of JSON repeatable; PaperSpec uses the JSON Canonicalization Scheme of RFC 8785 ([spec/01 §1.1](../spec/01-conventions.md#11-canonical-json)).
A witness is one file of a paper, such as the publisher's PDF or a JATS XML file, identified by the SHA-256 of its bytes; a document is the text established from one or more witnesses ([spec/02 §2.1](../spec/02-model.md#21-four-objects)).
A package is a folder of canonical JSON files, or the same folder zipped as `.sct.zip`, with a manifest that lists every other file with its SHA-256 and size ([spec/12](../spec/12-package.md)).
@@ -76,12 +76,12 @@ The design choices that matter for comparison:
## Summary table
Each row summarizes the pages read for that standard.
Cells keep the wording of the verified source; the paperspec row is checked against the repository.
Cells keep the wording of the verified source; the PaperSpec row is checked against the repository.
Maturity is left to the sections below, because a single word hides too much.
| Standard | Sentence-level permanent IDs | Content hashes | Anchors or selectors | Correction history | Fidelity or signatures | Per-file rights | Numeric data anchors | Full-text structure |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **paperspec 0.1.0** | yes: 7-character IDs minted once, retired with a successor (spec/04) | yes: five SHA-256 values per node, plus keyed HMAC (hash-based message authentication code) forms (spec/05) | yes: ID, hash, quote, span, five outcomes with drift (spec/06) | yes: append-only, evidence-gated (spec/11) | yes: F0 to F3, signed receipts, spot-checks, error bounds (spec/08) | yes: per witness, with AI purpose and processor (spec/14) | yes: dataset values anchored to cells (spec/15; format `provisional`) | yes: sections to cells, equations, references (spec/03, 09) |
| **PaperSpec 0.1.0** | yes: 7-character IDs minted once, retired with a successor (spec/04) | yes: five SHA-256 values per node, plus keyed HMAC (hash-based message authentication code) forms (spec/05) | yes: ID, hash, quote, span, five outcomes with drift (spec/06) | yes: append-only, evidence-gated (spec/11) | yes: F0 to F3, signed receipts, spot-checks, error bounds (spec/08) | yes: per witness, with AI purpose and processor (spec/14) | yes: dataset values anchored to cells (spec/15; format `provisional`) | yes: sections to cells, equations, references (spec/03, 09) |
| [JATS standard page](https://www.niso.org/publications/z3996-2024-jats) (Z39.96-2024, v1.4) | not seen: no sentence element in the list shown | not seen | not in scope as an annotation model | not seen | not seen | partly: `permissions` and `license` elements exist | not seen | yes |
| [TEI P5](https://tei-c.org/guidelines/P5/) (v4.12.0) | partly: `<s>` and author-assigned `xml:id` | not seen | partly: `xml:id` pointers, XPointer schemes | partly: `<corr>`, `<sic>`, `<choice>` | not seen | not seen | not seen | yes |
| [GROBID TEI output](https://grobid.readthedocs.io/en/latest/Grobid-service/) | partly: optional sentence elements; IDs not described | not seen | partly: optional PDF coordinates | no | no; reports F1 figures | not in scope | not seen | partly: header, references, citation contexts, full text |
@@ -126,12 +126,12 @@ JATS4R is a NISO working group that publishes tagging recommendations and a vali
It models journal-article structure and metadata: sections, paragraphs, table wrappers and cells, reference lists, related articles, footnotes, display formulas, supplementary material, permissions and licenses ([1.4 tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/)).
Crossref's 5.4.0 deposit schema imports JATS ([Crossref](https://www.crossref.org/documentation/schema-library/metadata-deposit-schema-5-4-0/)), and a toolkit for S2ORC has a script for PMC JATS XML ([allenai/s2orc-doc2json](https://github.com/allenai/s2orc-doc2json)).
*How paperspec differs.*
*How PaperSpec differs.*
JATS has `<permissions>` and `<license>` elements; where they may be placed was not verified, so no per-file rights comparison is made.
*Where JATS is the better choice.*
For publisher and archive exchange of journal content, and for the JATS4R recommendations on tagging consistency.
paperspec is not a publisher interchange format and does not replace JATS ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
PaperSpec is not a publisher interchange format and does not replace JATS ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
*Interoperation.*
JATS is an input.
@@ -153,8 +153,8 @@ Pointers use `xml:id`, recommended "For durability in the presence of editing",
GROBID extracts header metadata, references, citation contexts and full text with coordinates, and reports, in its own documentation's documentation, an F1 of about 0.87 for references on a PubMed Central set and 0.76 to 0.91 for citation contexts; it describes itself as production ready ([introduction](https://grobid.readthedocs.io/en/latest/Introduction/)).
Its `segmentSentences` option splits paragraphs into sentence elements, and `teiCoordinates=s` adds a bounding box to each ([service documentation](https://grobid.readthedocs.io/en/latest/Grobid-service/)).
*How paperspec differs.*
TEI's `xml:id` is author-assigned and durable by recommendation ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)); paperspec IDs are minted by a rule and retired by recorded successor.
*How PaperSpec differs.*
TEI's `xml:id` is author-assigned and durable by recommendation ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)); PaperSpec IDs are minted by a rule and retired by recorded successor.
GROBID's output is an automatic reading with the accuracy figures above, not a corrected, reviewed text ([introduction](https://grobid.readthedocs.io/en/latest/Introduction/)).
*Where they are the better choice.*
@@ -163,7 +163,7 @@ GROBID as a PDF-to-structure converter for the reference and citation-context ta
*Interoperation.*
TEI from GROBID could be one reading of a PDF.
paperspec has no `tei` witness kind, so that reading would be recorded as `xml` ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
PaperSpec has no `tei` witness kind, so that reading would be recorded as `xml` ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
The mechanism for comparing that reading with another is sentence alignment, which pairs sentences, keeps the first reading's IDs for paired sentences, and turns every disagreement into a suspect for review ([spec/05 §5.6](../spec/05-hashes.md#56-alignment)).
No TEI or GROBID mapping is defined in the repository.
@@ -185,19 +185,19 @@ Targets are narrowed by selectors ([model](https://www.w3.org/TR/annotation-mode
Hypothesis stores a range selector (XPaths plus offsets), a position selector and a quote selector with 32 characters of context, and reattaches in four steps: range, position, context-first fuzzy match, then selector-only fuzzy match ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
It identifies a PDF by an embedded fingerprint, which "persists across copies of the same document, but not across variants" ([Hypothesis help](https://web.hypothes.is/help/annotations-not-syncing-across-copies-of-a-pdf/)).
*How paperspec differs.*
*How PaperSpec differs.*
Hypothesis's post lists its own limits: it assumes the system can tell two formats are the same document, mapping text to the DOM (the browser's tree of page elements) is slow on long documents, and mappings update only when Hypothesis itself changes the DOM ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
paperspec defines exact resolution outcomes and a hash pin where Hypothesis uses fuzzy matching.
PaperSpec defines exact resolution outcomes and a hash pin where Hypothesis uses fuzzy matching.
*Where they are the better choice.*
The Web Annotation model and vocabulary for annotation in general, existing Web Annotation tools, and Hypothesis as a working annotation service on web pages and PDFs.
*Interoperation.*
A paperspec layer is a set of W3C Web Annotations whose target lists a custom `AnchorSelector` carrying a `scienting-anchor/1` record, optionally followed by a standard `TextQuoteSelector` "for tools that do not know ours" ([spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
A PaperSpec layer is a set of W3C Web Annotations whose target lists a custom `AnchorSelector` carrying a `scienting-anchor/1` record, optionally followed by a standard `TextQuoteSelector` "for tools that do not know ours" ([spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
The layers do not use `TextPositionSelector` or `FragmentSelector`.
The JSON-LD context (a file that maps the added terms to IRIs (internationalized web identifiers), so linked-data tools can read them) is at `https://paperspec.scient.ing/context/scienting-1.jsonld` ([context/README.md](../context/README.md)); whether that address resolves was not checked.
Whether a generic Web Annotation client displays an `AnchorSelector` was not verified.
Both Hypothesis and paperspec store a quote with 32 characters of prefix and suffix ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor)).
Both Hypothesis and PaperSpec store a quote with 32 characters of prefix and suffix ([spec/06 §6.2](../spec/06-anchors.md#62-the-anchor)).
## Docling, MinerU and Marker
@@ -211,7 +211,7 @@ Docling's schema (version 1.10.0 on `main`) covers texts, pictures, tables, grou
Docling uses AI models for layout analysis and table structure recognition and is reported integrated with LangChain, LlamaIndex and spaCy ([abstract](https://arxiv.org/abs/2501.17887)).
MinerU describes locators of the form `doc:{id}/tier:{tier}/page:{n}/block:{n}/char:{offset}` ([MinerU](https://github.com/opendatalab/MinerU)); Marker's JSON is a tree of blocks with an `id`, a `section_hierarchy` and `images` ([Marker](https://github.com/datalab-to/marker)).
*How paperspec differs.*
*How PaperSpec differs.*
The Docling `TextItem` field list has no per-item hash and its `label` enum has no sentence value, so items are block-grained ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
Identifiers in all three are positional: `self_ref` indexes into lists, Marker's example IDs embed a page number, and MinerU's locators embed page and block numbers.
MinerU's guide says to preserve locators for citation.
@@ -221,10 +221,10 @@ It is not a pinned, quote-verified anchor with resolution outcomes.
*Where they are the better choice.*
For converting PDFs and other formats, layout analysis, table structure recognition, integrations, and one-shot structured output.
paperspec does not read PDFs; extracting lines from a PDF is outside its text ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
PaperSpec does not read PDFs; extracting lines from a PDF is outside its text ([spec/07 §7.1](../spec/07-geometry.md#71-page-fingerprints)).
*Interoperation.*
These are converters, which paperspec treats as backends.
These are converters, which PaperSpec treats as backends.
A backend declaration records the tool's code and weight licenses, and a stage may run only if they pass the gate ([spec/14 §14.4](../spec/14-rights.md#144-backends-and-profiles)); `provenance.json` records the backends and cached readings used ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
Docling appears in the repository only as data: an illustrative declaration, version `illustrative`, in `fixtures/records/backend-docling.json` and `vectors/rights.jsonl`.
Their JSON output is not a named witness kind.
@@ -242,17 +242,17 @@ Scale, and citation-mention linking across papers: the paper, an arXiv preprint,
The schema classes include `Paper`, with `pdf_hash`, `metadata`, `body_text`, `bib_entries` and `ref_entries`, and `Paragraph`, with `cite_spans`, `ref_spans` and `eq_spans` given as `start`, `end` and `text` ([s2orc.py](https://raw.githubusercontent.com/allenai/s2orc-doc2json/master/doc2json/s2orc.py)).
Sources are PDF via GROBID, PMC JATS XML and arXiv LaTeX ([doc2json](https://github.com/allenai/s2orc-doc2json)).
*How paperspec differs.*
*How PaperSpec differs.*
Table cells are not nodes ([s2orc.py](https://raw.githubusercontent.com/allenai/s2orc-doc2json/master/doc2json/s2orc.py)).
The paper states limitations, among them GROBID-derived inline-citation errors, linking that depends on titles, and LaTeX-derived metadata "worse than those extracted from PDF" ([arXiv:1911.02782](https://arxiv.org/pdf/1911.02782)).
The release formats "have changed over time" ([doc2json](https://github.com/allenai/s2orc-doc2json)).
*Where they are the better choice.*
Corpus-scale text mining and citation-graph work across the literature, which paperspec explicitly does not target.
Corpus-scale text mining and citation-graph work across the literature, which PaperSpec explicitly does not target.
*Interoperation.*
The purposes and scales differ.
S2ORC-style JSON could be a source of text readings. paperspec has no S2ORC witness kind.
S2ORC-style JSON could be a source of text readings. PaperSpec has no S2ORC witness kind.
The `same_as` list takes `doi:`, `arxiv:`, `isbn:`, `pmid:`, `pmcid:` and `url:` among its prefixes and has no Semantic Scholar identifier ([spec/02 §2.1](../spec/02-model.md#21-four-objects)).
## Nanopublications and micropublications
@@ -268,17 +268,17 @@ Micropublications are a semantic model for claims, evidence, arguments and annot
A claim-sized unit with machine-readable provenance and publication metadata, with an integrity check on the unit.
Micropublications add a claim, its supporting representations as a directed acyclic graph rooted at the claim, and challenges; the paper says they aim to present primary evidence while nanopublications "model only the indicated statement" ([paper](https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-28)).
*How paperspec differs.*
*How PaperSpec differs.*
These models do not store the paper's text.
The guidelines read do not describe pinning a passage, and they do not mention signatures; a Java library's README shows RSA public-key signing; the signature structure was not verified ([nanopub-java](https://github.com/Nanopublication/nanopub-java)).
*Where they are the better choice.*
Publishing a claim as a citable, hash-verifiable, linked-data object, and representing support and challenge among claims.
paperspec does not define claim semantics; a statement true independently of one paper belongs outside this repository, and a statement about reading the paper belongs in a layer ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
PaperSpec does not define claim semantics; a statement true independently of one paper belongs outside this repository, and a statement about reading the paper belongs in a layer ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
*Interoperation.*
A layer annotation can say that a span states, uses or disputes a claim kept elsewhere ([spec/18 §18.4](../spec/18-layers.md#184-bodies)).
A nanopublication's provenance could point at a paperspec address, since an address is a URL fragment (`…#node`, [spec/06 §6.1](../spec/06-anchors.md#61-addresses)); that is an inference, and no such mapping is defined in either project.
A nanopublication's provenance could point at a PaperSpec address, since an address is a URL fragment (`…#node`, [spec/06 §6.1](../spec/06-anchors.md#61-addresses)); that is an inference, and no such mapping is defined in either project.
## Packaging: Frictionless Data Package, RO-Crate and BagIt
@@ -289,8 +289,8 @@ It is "a simple container format for describing a coherent collection of data in
A resource's `hash` is MD5 by default, with other algorithms written as `algorithm:value` ([data resource](https://datapackage.org/standard/data-resource/)).
It describes tabular files and their schemas portably.
Not seen: hashing that covers descriptor and files in canonical form, per-value anchors to a source text, units and uncertainty.
It remains the right choice for the tabular container and its tooling, which paperspec adopts: a `scienting-dataset/1` is a Data Package plus units (strings a unit parser called Pint can read), uncertainty and an `origin`, with reported values anchored to the table cells or sentences that print them ([spec/15](../spec/15-datasets.md)).
The Data Package version paperspec uses is not stated in spec/15.
It remains the right choice for the tabular container and its tooling, which PaperSpec adopts: a `scienting-dataset/1` is a Data Package plus units (strings a unit parser called Pint can read), uncertainty and an `origin`, with reported values anchored to the table cells or sentences that print them ([spec/15](../spec/15-datasets.md)).
The Data Package version PaperSpec uses is not stated in spec/15.
The dataset format is `provisional` ([registry/formats.json](../registry/formats.json)).
*RO-Crate.*
@@ -298,14 +298,14 @@ It is "a community effort to establish a lightweight approach to packaging resea
Version 1.2 is a Recommendation published 2025-06-04, and the site calls 1.3 the newest ([1.2](https://www.researchobject.org/ro-crate/specification/1.2/index.html)).
It describes a folder's contents in `ro-crate-metadata.json`, and file entities should include `contentSize` ([data entities](https://www.researchobject.org/ro-crate/specification/1.2/data-entities.html)).
Use it for general research-object description and discovery with schema.org vocabularies.
A paperspec package folder could be described by an RO-Crate.
A PaperSpec package folder could be described by an RO-Crate.
No such mapping exists in either project.
*BagIt.*
RFC 8493 defines a bag: `bagit.txt`, a `data/` payload of opaque octet streams, and at least one `manifest-<algorithm>.txt`; BagIt 1.0 tools must support SHA-256 and SHA-512 ([RFC 8493](https://www.rfc-editor.org/rfc/rfc8493)).
The RFC says it "is not designed to be secure against active attacks", and it has no signatures, content-level structure or payload semantics.
BagIt fits fixity and transfer of arbitrary payloads, archives and repositories.
paperspec's `MANIFEST.json` plays a similar fixity role, with a path, SHA-256 and size per file, and the hash of the manifest pins the package; the file layout differs ([spec/12 §12.3](../spec/12-package.md#123-the-manifest)).
PaperSpec's `MANIFEST.json` plays a similar fixity role, with a path, SHA-256 and size per file, and the hash of the manifest pins the package; the file layout differs ([spec/12 §12.3](../spec/12-package.md#123-the-manifest)).
A package could be a bag's payload, since payload files are opaque (an inference).
## Metadata: schema.org, Crossref and DataCite
@@ -320,14 +320,14 @@ Bibliographic and identification metadata.
schema.org's `CreativeWork` has `text`, `citation`, `hasPart`, `version`, `license` and a `correction` property ([CreativeWork](https://schema.org/CreativeWork)), and `MediaObject` has `sha256` ([MediaObject](https://schema.org/MediaObject)).
Crossref's Crossmark lets publishers show whether an item was corrected, retracted or updated, including to readers holding an older download ([Crossmark](https://www.crossref.org/documentation/crossmark/)).
*How paperspec differs.*
*How PaperSpec differs.*
These are record-level.
A correction is flagged on the work, not on a sentence with evidence for each changed digit; the body has no node hashes; there are no fidelity levels or signed receipts; and rights are license-level information about the work, not per-file use grants with an AI purpose and processor.
None of the schema pages read describes body text, and the extent of Crossref's license and text-mining fields was not verified ([Crossref 5.4.0](https://www.crossref.org/documentation/schema-library/metadata-deposit-schema-5-4-0/)).
*Where they are the better choice.*
Identifying and describing works and datasets with DOIs (digital object identifiers), discovery, registering corrections and retractions at the work level, and structured data on web pages.
paperspec is not a citation index and imports from Crossref, OpenAlex, PubMed Central and arXiv and never depends on them ([docs/overview.md](../docs/overview.md#what-it-is-not)).
PaperSpec is not a citation index and imports from Crossref, OpenAlex, PubMed Central and arXiv and never depends on them ([docs/overview.md](../docs/overview.md#what-it-is-not)).
*Interoperation.*
The document's `meta` is a CSL-JSON item and `same_as` holds a DOI.
@@ -339,24 +339,24 @@ No mapping from `meta` to Crossref, DataCite or schema.org is defined.
*PROV-O.*
It is a W3C Recommendation of 30 April 2013 that expresses the PROV Data Model in OWL2, with the classes Entity, Activity and Agent and properties such as `wasGeneratedBy`, `wasDerivedFrom` and `wasInvalidatedBy` ([PROV-O Recommendation](https://www.w3.org/TR/prov-o/)).
It is a general vocabulary and defines no node hashes, anchor resolution, corrections or receipts.
paperspec does not use it: provenance is a file, `provenance.json`, defined in the repository ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
PaperSpec does not use it: provenance is a file, `provenance.json`, defined in the repository ([spec/12 §12.4](../spec/12-package.md#124-provenance)).
PROV-O is the better choice for linked-data provenance that other systems can query generically.
A mapping of stages and corrections to Activity and Entity is possible.
Nothing in either project defines one.
*RFC 8785 (JSON Canonicalization Scheme).*
Published June 2020 as an Informational RFC, it sorts object properties recursively in UTF-16 code unit order, keeps array order, writes numbers by ECMAScript rules, uses no whitespace, outputs UTF-8, and does not normalize Unicode strings ([RFC 8785](https://www.rfc-editor.org/rfc/rfc8785)).
It is a building block that defines no document model, and paperspec uses it: every hash of JSON is over RFC 8785 output.
paperspec adds rules on top: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits, and a document with a lone surrogate, `NaN` or an infinity has no canonical form and is invalid ([spec/01 §1.1 and §1.2](../spec/01-conventions.md#11-canonical-json)).
It is a building block that defines no document model, and PaperSpec uses it: every hash of JSON is over RFC 8785 output.
PaperSpec adds rules on top: a decimal that carries meaning is a string, so `310.0` and `1.5260` keep their printed digits, and a document with a lone surrogate, `NaN` or an infinity has no canonical form and is invalid ([spec/01 §1.1 and §1.2](../spec/01-conventions.md#11-canonical-json)).
*RFC 6962 and RFC 9162 (Merkle trees).*
A Merkle tree hashes items pairwise up to one root, so one short value commits to all of them.
RFC 6962 (Certificate Transparency, June 2013, Experimental) is obsolete; RFC 9162 (Certificate Transparency Version 2.0, published December 9, 2021, Experimental) "obsoletes RFC 6962" ([RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162)).
Both give the same Merkle Tree Hash: a leaf is `HASH(0x00 || d)`, an interior node is `HASH(0x01 || left || right)`, and a list splits at the largest power of two smaller than n.
paperspec's receipts use this construction with SHA-256 and cite RFC 6962 ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
They differ in one rule: a receipt with no items is invalid in paperspec, while the RFC defines the hash of an empty list.
PaperSpec's receipts use this construction with SHA-256 and cite RFC 6962 ([spec/08 §8.5](../spec/08-fidelity.md#85-merkle-roots)).
They differ in one rule: a receipt with no items is invalid in PaperSpec, while the RFC defines the hash of an empty list.
The citation is to the obsolete RFC; the tree it defines is the one RFC 9162 also defines.
paperspec's `node_hash` is not this construction (see "Limits and status" below).
PaperSpec's `node_hash` is not this construction (see "Limits and status" below).
## Content addressing: IPFS CIDs and SWHIDs
@@ -366,7 +366,7 @@ It gives verifiable, location-independent identity to immutable data.
A CID covers the root block of a DAG (directed acyclic graph), "not a direct hash of the file's bytes", so chunk size, layout, codec, version and hash all change the CID for the same file.
Identity and content are the same thing, so an edit makes a new identifier; the page describes no ID that survives edits, no sentence granularity, corrections or fidelity.
Keep using CIDs for storage and distribution of immutable blobs.
paperspec's asset store names files by SHA-256, not CID ([spec/12 §12.6](../spec/12-package.md#126-assets)).
PaperSpec's asset store names files by SHA-256, not CID ([spec/12 §12.6](../spec/12-package.md#126-assets)).
*Software Heritage SWHID.*
ISO/IEC 18670:2025 specifies the SoftWare Hash IDentifier, version 1.2, published April 2025 ([ISO catalog page](https://www.iso.org/standard/89985.html); [swhid.org](https://www.swhid.org/specification/v1.2/)).
@@ -376,8 +376,8 @@ Keep using SWHIDs for persistent, content-derived identification and archiving o
*The contrast.*
A CID or SWHID is a function of content at all times.
A paperspec node ID is derived from content once, at minting, and then frozen; the content hashes live separately in `node_hash`, so the pair of ID and hash tells whether the text has changed ([spec/04 §4.2 and §4.3](../spec/04-identity.md#42-minting-node-ids), [spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)).
paperspec's hashes are SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
A PaperSpec node ID is derived from content once, at minting, and then frozen; the content hashes live separately in `node_hash`, so the pair of ID and hash tells whether the text has changed ([spec/04 §4.2 and §4.3](../spec/04-identity.md#42-minting-node-ids), [spec/05 §5.1](../spec/05-hashes.md#51-the-five-values)).
PaperSpec's hashes are SHA-256 ([spec/01 §1.3](../spec/01-conventions.md#13-hashes)).
The cost of the choice: because the ID is minted from the text hash, two independent conversions that read a sentence differently by one character mint different IDs for it.
## Fragment identifiers and Pandoc
@@ -386,7 +386,7 @@ A fragment identifier is the part of a URL after `#` that says where to look ins
- **PDF fragments (RFC 8118):** Informational, March 2017.
Parameters include `page`, `nameddest`, `structelem`, `zoom`, `viewrect`, `highlight` and `search`; none selects text by character offset, `search` "selects the first matching word in the document", and `highlight` takes rectangle coordinates ([RFC 8118](https://www.rfc-editor.org/rfc/rfc8118)).
Keep using it to open a PDF at a page, destination or region. paperspec's `locations.jsonl` stores per-node page boxes as `[x0, y0, x1, y1]` in PDF points against page fingerprints ([spec/07 §7.4](../spec/07-geometry.md#74-locationsjsonl)), so a tool could build an RFC 8118 link; the repository does not define that.
Keep using it to open a PDF at a page, destination or region. PaperSpec's `locations.jsonl` stores per-node page boxes as `[x0, y0, x1, y1]` in PDF points against page fingerprints ([spec/07 §7.4](../spec/07-geometry.md#74-locationsjsonl)), so a tool could build an RFC 8118 link; the repository does not define that.
- **Media Fragments URI 1.0:** A W3C Recommendation of 25 September 2012 that defines time (`t=`) and spatial (`xywh=`) fragments for audio, video and images, and says nothing about text ([Media Fragments](https://www.w3.org/TR/media-frags/)).
It is a weak fit for text passages.
- **RFC 5147 (text/plain):** A Proposed Standard of April 2008 with `char=` and `line=` positions counted from zero and integrity checks `length=` and `md5=`, "to detect whether the resource has changed" ([RFC 5147](https://www.rfc-editor.org/rfc/rfc5147)).
@@ -394,11 +394,11 @@ A fragment identifier is the part of a URL after `#` that says where to look ins
- **Text Fragments:** A WICG Draft Community Group Report, not a W3C standard; the version read is dated 13 December 2023 and may not be the latest.
The syntax is `#:~:text=[prefix-,]start[,end][,-suffix]`, it matches the first exact occurrence, and it is restricted to top-level navigations ([WICG draft report](https://wicg.github.io/scroll-to-text-fragment/)).
- **DOM and XPath approaches:** W3C `XPathSelector` and `RangeSelector` ([model](https://www.w3.org/TR/annotation-model/)), TEI's `xpath()` scheme ([TEI pointers chapter](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)) and Hypothesis's first strategy, which "fails if the XPath expressions can no longer be applied" ([post](https://web.hypothes.is/blog/fuzzy-anchoring/)).
Path addressing breaks when structure changes; paperspec's display paths are never stored or cited ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
Path addressing breaks when structure changes; PaperSpec's display paths are never stored or cited ([spec/04 §4.3](../spec/04-identity.md#43-permanence-retirement-and-aliases)).
- **Pandoc:** The pandoc-types library defines the `Pandoc` document type with modules for building, walking and JSON conversion; the latest on Hackage is 1.23.1, uploaded 2023-08-06 ([Hackage](https://hackage.haskell.org/package/pandoc-types)).
Its Block and Inline types and its treatment of identifiers were not verified. `latex` is a paperspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)); Pandoc is not mentioned in the repository.
Its Block and Inline types and its treatment of identifiers were not verified. `latex` is a PaperSpec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)); Pandoc is not mentioned in the repository.
On the web, a paperspec node ID is the URL fragment, so `…#k3m9q2a` lands on the sentence ([spec/06 §6.1](../spec/06-anchors.md#61-addresses)).
On the web, a PaperSpec node ID is the URL fragment, so `…#k3m9q2a` lands on the sentence ([spec/06 §6.1](../spec/06-anchors.md#61-addresses)).
This is a plain HTML fragment, not a registered fragment scheme; the chapters define no fragment syntax for the package media type.
## BioC, CSL-JSON and EPUB 3
@@ -408,20 +408,20 @@ BioC is a simple extensible XML format for sharing text documents and annotation
Its model has collection, source, document, passage, optional sentence, annotation and relation, with `infon` key-value pairs, and an annotation's location is an offset and length relative to the whole document.
Its authors state that it "targets tool developers and not end users" and "focuses on text data rather than other media".
Keep using it to exchange text-mining annotations among biomedical natural-language-processing (NLP) tools.
Offsets would need conversion, since paperspec counts Unicode code points ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)) and the length unit in BioC was taken from a page summary and not confirmed.
Offsets would need conversion, since PaperSpec counts Unicode code points ([spec/01 §1.5](../spec/01-conventions.md#15-character-offsets)) and the length unit in BioC was taken from a page summary and not confirmed.
No BioC mapping is defined.
*CSL-JSON.*
It is a JSON format for bibliographic items consumed by Citation Style Language (CSL) processors; items require `id` and `type`, `author` is an array of name objects, and `issued` uses `date-parts` or `raw` ([markup page of the CSL-JSON documentation](https://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html)).
Its schema is described as "not yet fully normative" ([CSL schema](https://github.com/citation-style-language/schema)).
Keep using it for bibliographic interchange with citation processors and reference managers.
paperspec embeds CSL-JSON as the document's `meta` and as the optional parsed `csl` of each reference ([spec/02 §2.2](../spec/02-model.md#22-documentjson), [spec/10](../spec/10-references.md)).
PaperSpec embeds CSL-JSON as the document's `meta` and as the optional parsed `csl` of each reference ([spec/02 §2.2](../spec/02-model.md#22-documentjson), [spec/10](../spec/10-references.md)).
*EPUB 3.*
EPUB 3.3 is a W3C Recommendation, "a distribution and interchange format for digital publications and documents": a ZIP container with a package document listing a manifest and a spine, and XHTML or SVG content documents ([EPUB 3.3](https://www.w3.org/TR/epub-33/)).
It suits reflowable publications.
Keep using it as a reading and distribution format.
`epub` is a paperspec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)), and the zip package rules (UTF-8 names, one top-level folder, no symlinks) are the package's own ([spec/12 §12.2](../spec/12-package.md#122-paths)).
`epub` is a PaperSpec witness kind ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)), and the zip package rules (UTF-8 names, one top-level folder, no symlinks) are the package's own ([spec/12 §12.2](../spec/12-package.md#122-paths)).
## Limits and status
@@ -437,9 +437,9 @@ The repository names no outside adopter and no independent implementation, and t
*Scope of this comparison.*
The pages read cover no ORKG, DocBook, PubAnnotation, OpenAlex or PMC OA XML material, so those are not compared.
A statement that no single comparator combines paperspec's features describes the pages read and surveys no other systems.
A statement that no single comparator combines PaperSpec's features describes the pages read and surveys no other systems.
*What paperspec does not do.*
*What PaperSpec does not do.*
- It is not a publisher interchange format; it ingests JATS and other formats as witnesses and does not replace them ([spec/02 §2.3](../spec/02-model.md#23-witnessesjson)).
- It is not a search engine, a citation index, a recommendation feed or a chat-with-a-PDF tool ([docs/overview.md](../docs/overview.md#what-it-is-not)).
@@ -474,7 +474,7 @@ These are the places where the guides and the specification differ, and this pag
Of the 32 formats, 13 are `provisional`, including `scienting-dataset/1`, which the datasets paragraphs in `docs/concepts.md` present without that qualifier.
*The distinguishing combination, and what it costs.*
paperspec puts in one package permanent sentence- and cell-level IDs, five hashes per node, anchors with explicit resolution outcomes and drift, append-only evidence-gated corrections, signed fidelity receipts with spot-check bounds, per-witness rights with AI-use axes, and value-to-cell anchors.
PaperSpec puts in one package permanent sentence- and cell-level IDs, five hashes per node, anchors with explicit resolution outcomes and drift, append-only evidence-gated corrections, signed fidelity receipts with spot-check bounds, per-witness rights with AI-use axes, and value-to-cell anchors.
Individual pieces have precedents in the comparators: TEI correction markup, Web Annotation selectors, RFC 5147 integrity checks, RFC 6962 Merkle trees, BagIt manifests and Frictionless packaging.
No single comparator in the pages read combines them.
The price is weight: five hashes per node, a gate on corrections, signed records and a custom selector type.
+2 -2
View File
@@ -16,7 +16,7 @@ It defines 32 formats, 19 of them `candidate` and 13 `provisional`, as [registry
The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint formats; the sections below say so where they describe them.
Format identifiers such as `scienting-anchor/1` and `scienting-correction/1` begin with `scienting-`, which is the formats' namespace.
The repository's name, paperspec, is a separate matter, and the formats are named after what they hold.
The repository's name, PaperSpec, is a separate matter, and the formats are named after what they hold.
The repository is at <https://git.scient.ing/papers/paperspec>.
## Hashes and canonical JSON
@@ -103,7 +103,7 @@ Page breaks, running heads and columns are not nodes; they are geometry ([chapte
See [chapter 3](../spec/03-nodes.md).
Close relatives: [JATS 1.4](https://www.niso.org/publications/z3996-2024-jats) is an XML format in which publishers and archives exchange journal content, and TEI is a set of guidelines for encoding scholarly text.
Each is the better choice for its own job, and the paperspec formats do not replace them; a JATS file can be one witness of a document.
Each is the better choice for its own job, and the PaperSpec formats do not replace them; a JATS file can be one witness of a document.
The element list in the [JATS 1.4 archiving tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) shows sections, paragraphs, table wrappers and cells and has no sentence element, while [TEI has an `<s>` element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html).
Here the sentence is the unit that gets an ID and hashes.
See [the comparison page](comparison.md).
+3 -3
View File
@@ -1,6 +1,6 @@
# FAQ
This page answers questions about paperspec, a set of file formats that give every sentence and table cell of a scientific paper a permanent address.
This page answers questions about PaperSpec, a set of file formats that give every sentence and table cell of a scientific paper a permanent address.
Newcomers, implementers of a reader, writer, converter or checker, and experts who know JATS (Journal Article Tag Suite), TEI (Text Encoding Initiative) or W3C Web Annotations can all use it.
Each answer gives the plain statement first, then the precise one, then the chapter that defines it.
The repository is <https://git.scient.ing/papers/paperspec>, and the formats are at version 0.1.0.
@@ -88,13 +88,13 @@ Whether a team may share notes on a paper this way is a question for its own cou
They overlap in subject, not in job.
JATS, the [Journal Article Tag Suite](https://www.niso.org/publications/z3996-2024-jats) (ANSI/NISO Z39.96-2024), is an XML format for publishers and archives to exchange journal content. paperspec does not replace it.
JATS, the [Journal Article Tag Suite](https://www.niso.org/publications/z3996-2024-jats) (ANSI/NISO Z39.96-2024), is an XML format for publishers and archives to exchange journal content. PaperSpec does not replace it.
A JATS file is one kind of witness (a source file), an input from which a document is established.
None of the 312 elements in the [JATS 1.4 Archiving tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) marks a sentence; its smallest text unit is the paragraph, `<p>`.
TEI, the Text Encoding Initiative, has a sentence-like [`<s>` element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html) and keeps both readings of a corrected passage with [`<corr>` and `<sic>`](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html).
The encoder assigns its `xml:id`, and the guidelines [recommend it for durability](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html).
paperspec mints and retires IDs by rule, keeps corrections as separate records with evidence, and hashes each node.
PaperSpec mints and retires IDs by rule, keeps corrections as separate records with evidence, and hashes each node.
Keep JATS for publisher and archive exchange and TEI for general scholarly text encoding.
A TEI file is a witness of kind `xml` ([spec/02 §2.3](../spec/02-model.md)).
+4 -4
View File
@@ -1,6 +1,6 @@
# Glossary
A scientific paper changes after it is published: errata appear, readings get corrected, annotations pile up. paperspec is a set of file formats for scientific papers that keeps each sentence findable through those changes (repository: <https://git.scient.ing/papers/paperspec>).
A scientific paper changes after it is published: errata appear, readings get corrected, annotations pile up. PaperSpec is a set of file formats for scientific papers that keeps each sentence findable through those changes (repository: <https://git.scient.ing/papers/paperspec>).
This glossary defines its terms for readers from newcomer to expert, implementers included.
Each entry gives the plain meaning, then the precise one, then the chapter that defines it.
@@ -83,7 +83,7 @@ History root
JATS
: Outside term.
Journal Article Tag Suite, ANSI/NISO Z39.96-2024, an XML format in which publishers and archives exchange journal content ([the NISO page](https://www.niso.org/publications/z3996-2024-jats)).
In paperspec, `jats` is a witness kind: the fixture `okafor2025solvent-acth` was established from a JATS file and a PDF that agree.
In PaperSpec, `jats` is a witness kind: the fixture `okafor2025solvent-acth` was established from a JATS file and a PDF that agree.
The formats ingest JATS and do not replace it. ([2.3](../spec/02-model.md#23-witnessesjson))
Keyed hash
@@ -106,7 +106,7 @@ A complex-notation key is `notation:source` for a piece of math too complex for
Math form
: A standard spelling of formulas, so that one formula is written alike in every paper and claim.
`scienting-math/1` is canonical TeX, with a tree read from it and a key for each symbol; papermath implements it.
`scienting-math/1` is canonical TeX, with a tree read from it and a key for each symbol; PaperMath implements it.
An equation or math run in the form says so with `form`, which enters no hash, and its tree lives in `derived/math.jsonl`. ([3.7](../spec/03-nodes.md#37-math-forms), [9.4](../spec/09-figures-tables-equations.md#94-the-math-form))
ML-DSA-65
@@ -191,7 +191,7 @@ Suspect
Symbol key
: The name of a symbol in a formula, its canonical TeX with `\mathrm` spelled `\text`: `K_{r}` for `K_r`.
A paper's symbols and a paperclaim claim's have one key, so a `symbol` annotation names its symbol by it. ([9.4](../spec/09-figures-tables-equations.md#94-the-math-form), [18.4](../spec/18-layers.md#184-bodies))
A paper's symbols and a PaperClaim claim's have one key, so a `symbol` annotation names its symbol by it. ([9.4](../spec/09-figures-tables-equations.md#94-the-math-form), [18.4](../spec/18-layers.md#184-bodies))
Targeted F3
: Proofreading the nodes a team's claims, quotations and datasets rest on, rather than every node.
+6 -6
View File
@@ -1,4 +1,4 @@
# The paperspec formats
# The PaperSpec formats
This page serves three readers.
A student new to the problem will find each term defined where it first appears.
@@ -15,7 +15,7 @@ Scientists who rely on a paper keep asking two questions.
Where did this number come from, and how far was it checked? A force constant in a table, a value read off a plot, a parameter in a model: each should lead back to the cell or sentence it came from, with a record of who compared it with the page.
Most tools for scientific literature aggregate.
They index hundreds of millions of papers and answer questions over all of them, shallowly. paperspec does the opposite.
They index hundreds of millions of papers and answer questions over all of them, shallowly. PaperSpec does the opposite.
A team working on a focused question cares more about fifty papers it understands completely than about the world's literature.
Those fifty papers deserve to be converted once, corrected once, split down to the sentence, annotated by the people who know them, linked to checked claims, and kept.
@@ -24,7 +24,7 @@ A model can summarize a paper.
It cannot give you a signed statement that a named person checked a passage, IDs that never move, or a store its owner controls.
The formats supply those three, and they are worth most at the scale of one team.
paperspec is a set of file formats for scientific papers that have been converted, split into addressable pieces and checked.
PaperSpec is a set of file formats for scientific papers that have been converted, split into addressable pieces and checked.
The repository is at <https://git.scient.ing/papers/paperspec>.
It is version 0.1.0 and defines 32 formats; in [registry/formats.json](../registry/formats.json), 19 are `candidate` and 13 are `provisional`.
@@ -131,12 +131,12 @@ The [specification's reading paths](../spec/README.md) say which chapters each r
## How it compares
paperspec adds to existing standards and replaces none of them. [The comparison page](comparison.md) has the full account; five points follow.
PaperSpec adds to existing standards and replaces none of them. [The comparison page](comparison.md) has the full account; five points follow.
- JATS is the NISO standard by which [publishers and archives exchange journal content](https://www.niso.org/publications/z3996-2024-jats).
Its [version 1.4 element list](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) shows no sentence element.
A JATS file is an input here (witness kind `jats`).
- TEI has a [sentence-like element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html) and [markup that keeps both readings of a correction](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html), and recommends `xml:id` pointers ["for durability in the presence of editing"](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html). paperspec mints and retires IDs by rule and keeps corrections as a separate, evidence-gated record.
- TEI has a [sentence-like element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html) and [markup that keeps both readings of a correction](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html), and recommends `xml:id` pointers ["for durability in the presence of editing"](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html). PaperSpec mints and retires IDs by rule and keeps corrections as a separate, evidence-gated record.
TEI is built for encoding scholarly text in general, including correction markup within a text.
- GROBID [converts PDFs to TEI](https://grobid.readthedocs.io/en/latest/Introduction/) and [reports F1 scores](https://grobid.readthedocs.io/en/latest/Introduction/) for its automatic reading.
Its output can be held as one reading, under witness kind `xml`.
@@ -145,7 +145,7 @@ paperspec adds to existing standards and replaces none of them. [The comparison
The W3C model calls the `TextPositionSelector` ["very brittle"](https://www.w3.org/TR/annotation-model/) against changes to the resource, and the anchors here pin a hash instead.
For annotation in general, use the W3C model.
- Docling's [document schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json) refers to items by position (`#/texts/12`) and lists no per-item hash field.
Docling is a conversion toolkit that, [by its own abstract](https://arxiv.org/abs/2501.17887), uses AI models for layout analysis and table structure recognition. paperspec defines no converter.
Docling is a conversion toolkit that, [by its own abstract](https://arxiv.org/abs/2501.17887), uses AI models for layout analysis and table structure recognition. PaperSpec defines no converter.
The formats are at version 0.1.0, the repository names no outside adopter, and the reference tools belong to the same repository.
The formats do not replace publisher interchange formats.
+1 -1
View File
@@ -1,6 +1,6 @@
# License
paperspec is not under Scienting's usual license.
PaperSpec is not under Scienting's usual license.
Because the formats are meant to be implemented by anyone, in any product, without asking, each path of the repository keeps the license that `LICENSE.md` gives it.
| Paths | License | SPDX |
+1 -1
View File
@@ -1,6 +1,6 @@
# Check and rebuild the repository
Four commands work on a checkout of the paperspec repository, not on an installed package: `check`, `build`, `stress` and `site`.
Four commands work on a checkout of the PaperSpec repository, not on an installed package: `check`, `build`, `stress` and `site`.
The checkout is the one an editable install points at, or else the one around the current folder.
## Check everything
+2 -2
View File
@@ -1,6 +1,6 @@
# paperspec
# PaperSpec
paperspec is the reference implementation of the scienting formats, the files in which a team stores its scientific papers once, corrects them once and addresses them down to the sentence.
PaperSpec is the reference implementation of the scienting formats, the files in which a team stores its scientific papers once, corrects them once and addresses them down to the sentence.
This site documents the Python package `paperspec` and its command.
The formats themselves, with the specification, schemas, test vectors and fixtures, are published at <https://paperspec.scient.ing>.
+2 -2
View File
@@ -60,7 +60,7 @@ A rule that is broken raises `tree.Refusal`, whose `rule` attribute names the ru
| --- | --- | --- |
| `rights` | Rights records, policies and stage checks. | `effective`, `combine`, `check`, `backend_allowed` |
| `layers` | What every annotation shares: its record ID, its anchors and the lists its fields take. | `annotation_id`, `anchors_of`, `BODY_KINDS`, `CATEGORIES`, `VERDICTS`, `STATUSES` |
| `claims` | References to paperclaim claims and their pins. | `parse_ref`, `parse_pin`, `format_ref`, `format_pin` |
| `claims` | References to PaperClaim claims and their pins. | `parse_ref`, `parse_pin`, `format_ref`, `format_pin` |
| `traces` | Traced citations. | `resolve`, `problems`, `disputed` |
| `structures` | Chemical structures tied to words. | `scan`, `problems` |
| `sequences` | Biological sequences tied to words. | `residues`, `problems` |
@@ -92,7 +92,7 @@ A proposal the gate refuses raises `revision.Refused`, with the proposal's `inde
## Math forms
`package.validate` reports `doc.math.form` for a source with `form` that is not the form's own spelling ([spec/03 §3.7](https://paperspec.scient.ing/spec/03-nodes.html#37-math-forms)).
Without help it knows only what `scienting-math/1` never holds, a `\tag`, `\label`, `\nonumber`, `\notag`, style command, `\dfrac` or `\tfrac`; with papermath installed it spells every source again and requires it back unchanged.
Without help it knows only what `scienting-math/1` never holds, a `\tag`, `\label`, `\nonumber`, `\notag`, style command, `\dfrac` or `\tfrac`; with PaperMath installed it spells every source again and requires it back unchanged.
Pass `forms={}` to check only the first, or a mapping from a form to a function that spells a source in it.
## The command
+1 -1
View File
@@ -10,7 +10,7 @@ Experts find the cost of each choice and, where a neighboring standard chose dif
The formats are at version 0.1.0.
Of the 29 in `registry/formats.json`, 19 are `candidate` and 13 are `provisional`, and the provisional ones are named where they appear.
The repository is paperspec, at <https://git.scient.ing/papers/paperspec>.
The repository is PaperSpec, at <https://git.scient.ing/papers/paperspec>.
The `scienting-` prefix in format names such as `scienting-anchor/1` is the formats' namespace, separate from the repository's name.
## Identity
+2 -2
View File
@@ -7,7 +7,7 @@ Its problems are modeled on the failures that the specification's rules address.
Three readers can use the guide, and each gets something different.
A newcomer knows what a paper and a citation are, not hashes or JSON, so every term is defined where it first appears.
An implementer who will write a reader, writer, converter or checker gets the chapter and test vector for each step.
An expert gets the status: [paperspec](https://git.scient.ing/papers/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (19 candidate, 13 provisional).
An expert gets the status: [PaperSpec](https://git.scient.ing/papers/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (19 candidate, 13 provisional).
The `scienting-` prefix on format names, as in `scienting-doc/1`, is the formats' namespace.
It has nothing to do with the repository's name.
@@ -132,7 +132,7 @@ The table shows how each anchor resolves at revision 12 ([vectors/resolve.jsonl]
A reader shows each note by its result.
A checker of exact words accepts only `current`, `moved` and `reattached`.
The [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/) is the general W3C Recommendation for annotating web resources. paperspec's annotation layers build on it, with a custom `AnchorSelector` followed by a standard `TextQuoteSelector` as a fallback.
The [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/) is the general W3C Recommendation for annotating web resources. PaperSpec's annotation layers build on it, with a custom `AnchorSelector` followed by a standard `TextQuoteSelector` as a fallback.
It has no permanent IDs for sentences inside the source, no hashes of the target text, and no results such as `moved` or drift.
See [comparison.md](comparison.md).
+2 -2
View File
@@ -1,7 +1,7 @@
# Fixtures
A reader, writer, converter or checker of stored papers needs files to test against, and a newcomer needs examples of what a stored paper looks like.
This folder holds that example data of paperspec (<https://git.scient.ing/papers/paperspec>), version 0.1.0.
This folder holds that example data of PaperSpec (<https://git.scient.ing/papers/paperspec>), version 0.1.0.
All text is synthetic, written for these files and dedicated CC0; license fields inside the fixtures describe fictional works.
Three readers use these files.
@@ -47,7 +47,7 @@ Each record is a single file, or a folder where a format needs several files.
| `backend-docling.json`, `backend-examplevendor.json` | The code and weight licenses of a conversion tool and the conditions attached to them. |
| `profile-acme.json`, `profile-smithlab.json` | What those conditions ask about a team: its purpose, its funding and whom it competes with. |
| `roster.json` | The keys that may sign, the role of each key's holder in the team, and what each key may sign. Ana is the owner, so she may revoke anyone's receipt. |
| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). Its claim link and its symbol name the claim `amber/k-bond` of paperclaim's example store as paperclaim references it, `claim:272XTX7C`, pinned at `claim:272XTX7C@02f47b`; the symbol names `K_r` by its key, `K_{r}`. |
| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). Its claim link and its symbol name the claim `amber/k-bond` of PaperClaim's example store as PaperClaim references it, `claim:272XTX7C`, pinned at `claim:272XTX7C@02f47b`; the symbol names `K_r` by its key, `K_{r}`. |
| `traces/` | A layer of traced citations on the Typst draft, which cites the main example (`layer.json`, `annotations.jsonl`, `receipt.json`): a trace a model proposed from the draft's citation of "a two-step fit" to the abstract of the main example and a person confirmed, with its receipt; a second candidate, the introduction's sentence, that a person rejected; a person's trace of the same sentence to Table 1; and a person's reply that disputes the first ([spec/18 §18.5](../spec/18-layers.md#185-traced-citations)). `vectors/traces.jsonl` resolves them after corrections. |
| `chemistry/` | A layer of chemical structures on the main example (`layer.json`, `annotations.jsonl`, `receipt.json`): ethane, named in the methods and confirmed; water, named by a model and still `proposed`; the CT–CT pattern of Table 1, a SMARTS with the bond constants of both parameter sets as terms; and water again as a SMILES that writes its hydrogens, with the OW–HW bond constants. The two writings of water share an InChIKey ([spec/18 §18.6](../spec/18-layers.md#186-chemical-structures)). `vectors/structures.jsonl` pins the rules. |
| `biology/` | A layer of sequences on the Typst draft (`layer.json`, `annotations.jsonl`, `receipt.json`): the villin headpiece subdomain HP35, named by the words that name it, shown by the sequence the sentence prints, with its three helices as regions, and its N68H variant as a variant a model proposed ([spec/18 §18.7](../spec/18-layers.md#187-sequences)). The residues are numbered 42 to 76, as the headpiece is. `vectors/sequences.jsonl` pins the rules. |
+2 -2
View File
@@ -1,6 +1,6 @@
# Schemas
This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the paperspec formats (paperspec is the specification this repository holds) define: 43 schema files.
This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the PaperSpec formats (PaperSpec is the specification this repository holds) define: 43 schema files.
A JSON Schema is a file, itself written in JSON, that states what a valid JSON file looks like.
Newcomers can read it as a form: it lists the fields a record must have and what kind of value each field takes.
Implementers will find the loading recipe and the traps under "Validating" and "Writing schemas".
@@ -38,7 +38,7 @@ The rest belong to `candidate` formats or to no format of their own.
The bundle has no JSON Schema: its registry entry points to `sql/bundle-sqlite.sql`.
Of the 43 schemas, 12 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `math`, `annotation` and `value-anchor`.
A line of `derived/math.jsonl` carries `form`, the math form `scienting-math/1` it reads the formula in; that names a spelling of TeX, which papermath implements ([spec/09 §9.4](../spec/09-figures-tables-equations.md#94-the-math-form)), not a file format, so the registry does not list it.
A line of `derived/math.jsonl` carries `form`, the math form `scienting-math/1` it reads the formula in; that names a spelling of TeX, which PaperMath implements ([spec/09 §9.4](../spec/09-figures-tables-equations.md#94-the-math-form)), not a file format, so the registry does not list it.
Of the schemas that do have one, 27 require a `format` property; `spot-check.schema.json` describes the input a VRF proof is made over rather than a file.
The others (`page`, `draft-ids`, `view` and `log-entry`) describe a line of a file, a Vega-Lite specification (a JSON description of a chart) or an item of a tree, and the record itself does not need to carry the identifier.
+1 -1
View File
@@ -41,7 +41,7 @@ An implementation states, for each format, which versions it reads and writes.
Every format has an identifier of the form `scienting-<name>/<major>`, such as `scienting-doc/1`.
The identifier tells a reader which rules apply to a file.
The prefix `scienting-` is the namespace of the formats.
The namespace is not the name of this repository, paperspec, and formats are named after what they hold.
The namespace is not the name of this repository, PaperSpec, and formats are named after what they hold.
A file or record that can travel on its own carries its identifier in a `format` property.
The anchor in `fixtures/records/anchor-leaf.json` begins `{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1", ...`.
+1 -1
View File
@@ -150,5 +150,5 @@ A consumer MAY rely on these properties of a source that carries `form`:
A producer writes `form` only on a source it has spelled in that form.
A correction that changes such a source writes the new source in the form too, since the gate does not implement the form and cannot check it.
A validator that implements the form reports a source with `form` that is not its own canonical spelling as `doc.math.form`.
One that does not can still report a `scienting-math/1` source that holds `\tag`, `\label`, `\nonumber`, `\notag`, a style command such as `\displaystyle`, `\dfrac` or `\tfrac`, which the form never holds ([9.4](09-figures-tables-equations.md#94-the-math-form)); the reference validator does that, and checks every source with papermath when it is installed.
One that does not can still report a `scienting-math/1` source that holds `\tag`, `\label`, `\nonumber`, `\notag`, a style command such as `\displaystyle`, `\dfrac` or `\tfrac`, which the form never holds ([9.4](09-figures-tables-equations.md#94-the-math-form)); the reference validator does that, and checks every source with PaperMath when it is installed.
`form` enters no content value, so adding it to a source already in the form changes no hash, ID or anchor, while respelling a source in the form changes them as any edit of the source does ([5.2](05-hashes.md#52-own-content)).
+3 -3
View File
@@ -41,7 +41,7 @@ MathML is derived by a pinned tool and recorded, with the tool's name and versio
`scienting-math/1` is the standard form of a formula in a paper: one canonical TeX spelling, a tree read from it, a key for each symbol, and the issues a reader should look at.
A converter writes a package's equations and math runs in it, a reader renders and searches it, and a claim store reads a paper's equations and its own claims through it, so the two can be matched.
papermath is its reference implementation, and what papermath computes is what the form says; this section states what the specification relies on.
PaperMath is its reference implementation, and what PaperMath computes is what the form says; this section states what the specification relies on.
The identifier names a spelling of formulas, not a file format, so [registry/formats.json](../registry/formats.json) does not list it; its readings travel in a package's `derived/math.jsonl`.
The form reads TeX only.
@@ -94,7 +94,7 @@ A later minor version of the form may add a kind, and a consumer that meets a ki
**A symbol** is a base, which is a letter, a Greek letter, or a font or accent command on a group (`\mathbf{F}`, `\hat{H}`), with any subscript, any superscript that names rather than raises (`R^{*}`, `A^{\dagger}`, `E^{\text{LJ}}`), and primes.
A text group alone is a symbol too, unless it holds several words, is a condition in `cases`, or is the set under a big operator.
A symbol's key is its canonical TeX with `\mathrm` spelled `\text`, `\ast` spelled `*`, `\prime` spelled `'`, and no space inside text: `K_r` and `K_{r}` have the key `K_{r}`, and `r_{\mathrm{eq}}` has the key `r_{\text{eq}}`.
The key is the one a paperclaim claim store gives a symbol, so a symbol in a paper and the same symbol in a claim have one key, and a `symbol` annotation names its symbol by it ([18.4](18-layers.md#184-bodies)).
The key is the one a PaperClaim claim store gives a symbol, so a symbol in a paper and the same symbol in a claim have one key, and a `symbol` annotation names its symbol by it ([18.4](18-layers.md#184-bodies)).
**Issues** never stop the reading; each is `{code, message, at}`, with `at` the canonical TeX it is about.
@@ -113,4 +113,4 @@ A later minor version may add a code.
`run` is the index of the math run among the node's runs, or null for an equation or equation line, as in `derived/mathml.jsonl`; `symbols` lists the keys in the order the formula first writes them; `label` is the label the tool lifted from a `\tag`, omitted when the source has none, as a canonical source never does.
The schema is `math.schema.json`.
A validator checks what it can without implementing the form: that each line names a formula of the package (`doc.derived.node`), and that the formula is TeX, `complete` is true exactly when the tree has no `opaque` node, and `symbols` holds each `sym` key of the tree once (`doc.derived.math`).
`fixtures/packages/hale2026harmonic-damb/derived/math.jsonl` holds papermath's reading of the example's equation, of a math run that is one symbol, and of the run `_2` of "H₂O", which the tree cannot read alone.
`fixtures/packages/hale2026harmonic-damb/derived/math.jsonl` holds PaperMath's reading of the example's equation, of a math run that is one symbol, and of the run `_2` of "H₂O", which the tree cannot read alone.
+2 -2
View File
@@ -62,7 +62,7 @@ Replies are annotations that target annotations, so discussions are threads anch
A textual body has a `value`, and optionally a `format` (`text/plain` or `text/markdown`) and a `language`.
The `source` of a `claim-link` or `symbol` body is a claim in a paperclaim claim store, written as paperclaim writes a reference: `claim:` and the claim's tag, eight Crockford base32 capitals (the digits and the letters without I, L, O and U), as in `claim:272XTX7C`, or `claim:<host>/<tag>` for a claim in another store, `<host>` being a domain name.
The `source` of a `claim-link` or `symbol` body is a claim in a PaperClaim claim store, written as PaperClaim writes a reference: `claim:` and the claim's tag, eight Crockford base32 capitals (the digits and the letters without I, L, O and U), as in `claim:272XTX7C`, or `claim:<host>/<tag>` for a claim in another store, `<host>` being a domain name.
A tag never changes, while a claim's slug, such as `amber/k-bond`, may be renamed, so a slug is never a reference.
The optional `pin` fixes the claim as the annotator read it: the same reference, `@`, and the first six lowercase hex digits of the claim's statement hash, as in `claim:272XTX7C@02f47b`.
`common.schema.json` defines both, as `claimRef` and `claimPin`.
@@ -90,7 +90,7 @@ The store's `annotations` table keeps the category beside the body's kind ([13.4
Nothing a model writes is ever stored as source text.
A model may propose a `description`, and like any annotation it stays `proposed` until a person confirms it; an accessible export uses confirmed descriptions only, and a figure with none has no description.
Other text a model writes about a span, such as a summary, is a `generated` annotation, which names its `generator`.
`fixtures/records/layer/` holds a layer with one annotation of each common kind, each comment, caveat and question with its category, a claim link and a symbol that reference a paperclaim claim and pin it, and the receipt that confirms some of them.
`fixtures/records/layer/` holds a layer with one annotation of each common kind, each comment, caveat and question with its category, a claim link and a symbol that reference a PaperClaim claim and pin it, and the receipt that confirms some of them.
## 18.5 Traced citations
+2 -2
View File
@@ -1,6 +1,6 @@
# The paperspec specification
# The PaperSpec specification
This specification states exactly what each paperspec file holds and how every value in it is computed.
This specification states exactly what each PaperSpec file holds and how every value in it is computed.
The schemas, SQL, JSON-LD context, registries, vectors and fixtures are normative too, and each chapter says which of them it relies on.
A chapter and a vector that disagree signal a bug, so please report it in the repository, <https://git.scient.ing/papers/paperspec>.
+1 -1
View File
@@ -76,7 +76,7 @@ A node's five values depend only on its own subtree, so after a correction the o
`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
paperstack, paperforms, paperdesk and paperclaim import it, and paperchem, paperbio and paperdata test their annotation bodies against it.
PaperStack, PaperForms, PaperDesk and PaperClaim import it, and PaperChem, PaperBio and PaperData test their annotation bodies against it.
The library surface they use is `keys`, `text`, `jcs`, `hashing`, `tree`, `anchors`, `package`, `revision`, `corrections`, `db`, `schema`, `fingerprint`, `rights`, `layers`, `claims`, `traces`, `structures`, `sequences` and `data`, with the schemas, SQL, vectors, registries and JSON-LD context as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository, where `paperspec.resources.path("fixtures")` finds them in a checkout or an editable install.
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.