0 Fork 0

Files

T

feat: minted keys, VRF seeds, RFC 9162 proofs, history roots, tag URIs and a transparency log

Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
  Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
  where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
  openalex in same_as, else the folded title, year and family name. A
  collision lengthens the later key by two characters; a key is minted
  once and never recomputed. A document key adds .si, .v<seq> or .<type>
  for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
  okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
  every node and record ID was re-minted and the docs rewritten to match.
  The validator checks the document key against the work key and version
  (doc.key.work) and the hash part against the identifier (doc.key.suffix).
  vectors/keys.jsonl pins the cases paperstack's tests pin.

Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
  implemented in tools/oracle/vrf.py in plain Python and reproduced
  against the RFC's Example 16; the sample ranks by the VRF output. The
  earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
  verify_inclusion, consistency_proof, verify_consistency) with
  scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
  the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
  checkpoints with key hashes as transparency logs share them, the rule
  that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).

Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
  namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
  the checker refuses another version.

Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
  checkpoint records; a check that every ID and hash the prose quotes
  exists in the fixtures or vectors; three more broken packages
  (history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
  keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
  section, media-type registration and OCI distribution notes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-10-14 23:21:57 -04:00

9.8 KiB

Raw Blame History

FAQ

This page answers questions about paperspec, a set of file formats that give every sentence and table cell of a scientific paper a permanent address. Newcomers, implementers of a reader, writer, converter or checker, and experts who know JATS (Journal Article Tag Suite), TEI (Text Encoding Initiative) or W3C Web Annotations can all use it. Each answer gives the plain statement first, then the precise one, then the chapter that defines it. The repository is https://git.scient.ing/infra/paperspec, and the formats are at version 0.1.0.

Can I read these files without special software?

Yes. Every file is JSON or JSON Lines (one JSON value per line), and spec/ specifies every format. The examples read a package with a few lines of Python.

Why not keep using PDFs and page numbers?

A page number names a region, not a statement, and it changes between a preprint and the version of record. An address names a sentence or a cell by its ID, such as hale2026harmonic-damb#v3063qv. It can add a pin (the first six hex digits of the node's hash) and character offsets, which lets a reader check the address against the exact text (spec/06 §6.1).

An ID encodes no page, position or kind. The system mints it once from the document key, kind, text hash and occurrence, then freezes it (spec/04 §4.2).

Why does a key end in four odd characters?

hale2026harmonic-damb is the paper's key: the citation key a reader would use, then four characters of a hash of its DOI. Citation keys alone collide, between a preprint and its published version or two papers by one author in one year; the hash settles that, and any tool that knows the DOI can recompute it. A key is minted once and never recomputed, so a corrected title does not change a key someone has cited (spec/02 §2.4).

Why are numbers stored as strings?

Databases and JSON libraries rewrite numbers: 310.0 becomes 310 and 1.5260 becomes 1.526, and the printed digits are part of what a paper says. Canonical JSON, the byte-exact form used for hashing, writes numbers as ECMAScript writes a double, so it would erase the same difference. A decimal that carries meaning is a string. JSON numbers are for counts, indexes, offsets, sizes and geometry (spec/01 §1.2).

What happens when a better converter comes along?

The system aligns the new reading with the established document, sentence by sentence. A sentence the alignment pairs keeps its ID even where the readings disagree, which catches a lost superscript instead of accepting it. Every disagreement, and every sentence found in only one reading, becomes a suspect, a doubtful reading raised for review. Nothing changes until a correction passes the gate, the rules that refuse, for example, a digit change with no crop of the page image as evidence.

Converting the same input again with the same converter mints the same IDs (spec/05 §5.6, spec/11 §11.4, spec/04 §4.2).

Do I need receipts, rights records and spot-checks for a personal library?

You need no receipts and no spot-checks. A receipt is a signed record that a named person compared nodes with the page image, and a spot-check is a second reviewer's sample of that work. One person can leave both out.

Each source file still carries a rights record, though the record can say little: the license as printed, a basis of unknown, and no uses, each of which then counts as unknown (spec/14 §14.1). The fields are defined so that a personal library loads into a team's store later without conversion (spec/00 §0.4).

Is a model allowed to correct a document?

A model may propose corrections and answer suspects, under the same gate as a person, and its corrections record the model and the run as author. Only a named person can sign a receipt, and a model's annotation stays proposed until a person confirms it (spec/11 §11.2, spec/18 §18.3).

Why are annotations W3C Web Annotations?

Tools that follow the W3C Web Annotation model can read the standard parts of these annotations. A layer, the file that groups annotations, follows the W3C Web Annotation Data Model, a W3C Recommendation of 2017. Each target holds an AnchorSelector carrying a scienting-anchor/1 record, followed by a standard TextQuoteSelector for tools that do not know the anchor selector. (The scienting- prefix is the formats' namespace, separate from the repository's name.) The formats use no TextPositionSelector or FragmentSelector.

The anchor adds what the standard selectors lack: the node's ID, its hashes, and a defined outcome (current, changed-below, moved, reattached or stale) when the text has changed. The Web Annotation model alone is the better-established choice where an annotation needs none of that. Whether a generic Web Annotation client displays an AnchorSelector is untested. See the comparison.

What is the difference between a claim and an annotation?

A statement true independently of the paper, such as "K_r is the bond force constant of the AMBER energy", is a claim. A claim has to be checked against evidence, so it belongs in a store of checked claims outside these formats, which define no claim semantics.

A statement about reading the paper, such as "the authors pass quickly over the scaling here", is an annotation in a layer, attributed and unchecked (spec/18 §18.1).

Yes, without sharing the paper. A checks-only package withholds all text and keeps IDs, labels, pages, boxes and keyed hashes. A public layer swaps each quotation for keyed hashes, under a key derived from the paper's page fingerprints, so a reader checks and shows it only over their own copy. The hash must be keyed because anyone can recover a plain hash of a short text such as 310.0 by hashing guesses (spec/12 §12.5, spec/05 §5.8, spec/06 §6.3).

Whether a team may share notes on a paper this way is a question for its own counsel; nothing here is legal advice.

How does this differ from JATS or TEI?

They overlap in subject, not in job.

JATS, the Journal Article Tag Suite (ANSI/NISO Z39.96-2024), is an XML format for publishers and archives to exchange journal content. paperspec does not replace it. A JATS file is one kind of witness (a source file), an input from which a document is established. None of the 312 elements in the JATS 1.4 Archiving tag library marks a sentence; its smallest text unit is the paragraph, <p>.

TEI, the Text Encoding Initiative, has a sentence-like <s> element and keeps both readings of a corrected passage with <corr> and <sic>. The encoder assigns its xml:id, and the guidelines recommend it for durability. paperspec mints and retires IDs by rule, keeps corrections as separate records with evidence, and hashes each node.

Keep JATS for publisher and archive exchange and TEI for general scholarly text encoding. A TEI file is a witness of kind xml (spec/02 §2.3). The full comparison is docs/comparison.md

What is its status and maturity?

Version 0.1.0 has 32 formats: 19 are candidate and 13 are provisional, as registry/formats.json lists. A candidate format is complete, with vectors and fixtures, and ready to implement. A provisional one is complete and expected to change as implementations gain experience. The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint.

The repository holds a reference checker (tools/check.py), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages (spec/00 §0.4, spec/22).

What does it not do?

It is built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers. It does not replace publisher interchange formats; it reads JATS as a witness. It does not fix what a converter reads wrongly. It records the reading and raises disagreements as suspects, and a correction is checkable. It is not a search engine, a citation index or a recommendation feed (docs/overview.md).

How do I add a node kind or a run kind?

A node kind is a type of document part, such as sentence; a run kind is a type of inline content within a sentence, such as math. Propose either with the vector or fixture that shows it. A new kind is a minor version (spec/00 §0.3). See CONTRIBUTING.md.

What are the files in `tools/` for?

build.py builds the fixtures and vectors. check.py recomputes and checks every one, and load.py loads packages into SQLite. The oracle/ modules compute every expected value, so that none is typed by hand. They are not a library; implementations follow the specification (tools/README.md).

9.8 KiB Raw Blame History

FAQ