infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ea0a7efa78f6c118a946d6fff89d875ce3610c3f
paperspec/docs/faq.md
T
alexmandClaude ea0a7efa78 feat: zipped packages, target set, checklists, audit fixes and a stress test 
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-14 20:07:22 -04:00

126 lines
9.3 KiB
Markdown
Raw Blame History

# FAQ
This page answers questions about manuspec, a set of file formats that give every sentence and table cell of a scientific paper a permanent address.
Newcomers, implementers of a reader, writer, converter or checker, and experts who know JATS (Journal Article Tag Suite), TEI (Text Encoding Initiative) or W3C Web Annotations can all use it.
Each answer gives the plain statement first, then the precise one, then the chapter that defines it.
The repository is <https://git.scient.ing/infra/manuspec>, and the formats are at version 0.1.0.
## Can I read these files without special software?
Yes.
Every file is JSON or JSON Lines (one JSON value per line), and [spec/](../spec/README.md) specifies every format. [The examples](examples.md) read a package with a few lines of Python.
## Why not keep using PDFs and page numbers?
A page number names a region, not a statement, and it changes between a preprint and the version of record.
An address names a sentence or a cell by its ID, such as `hale2026harmonic#j06tfhn`.
It can add a pin (the first six hex digits of the node's hash) and character offsets, which lets a reader check the address against the exact text ([spec/06 §6.1](../spec/06-anchors.md#61-addresses)).
An ID encodes no page, position or kind.
The system mints it once from the document key, kind, text hash and occurrence, then freezes it ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
## Why are numbers stored as strings?
Databases and JSON libraries rewrite numbers: `310.0` becomes `310` and `1.5260` becomes `1.526`, and the printed digits are part of what a paper says.
Canonical JSON, the byte-exact form used for hashing, writes numbers as ECMAScript writes a double, so it would erase the same difference.
A decimal that carries meaning is a string.
JSON numbers are for counts, indexes, offsets, sizes and geometry ([spec/01 §1.2](../spec/01-conventions.md#12-numbers)).
## What happens when a better converter comes along?
The system aligns the new reading with the established document, sentence by sentence.
A sentence the alignment pairs keeps its ID even where the readings disagree, which catches a lost superscript instead of accepting it.
Every disagreement, and every sentence found in only one reading, becomes a suspect, a doubtful reading raised for review.
Nothing changes until a correction passes the gate, the rules that refuse, for example, a digit change with no crop of the page image as evidence.
Converting the same input again with the same converter mints the same IDs ([spec/05 §5.6](../spec/05-hashes.md#56-alignment), [spec/11 §11.4](../spec/11-corrections.md#114-the-gate), [spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
## Do I need receipts, rights records and spot-checks for a personal library?
You need no receipts and no spot-checks.
A receipt is a signed record that a named person compared nodes with the page image, and a spot-check is a second reviewer's sample of that work.
One person can leave both out.
Each source file still carries a rights record, though the record can say little: the license as printed, a `basis` of `unknown`, and no uses, each of which then counts as unknown ([spec/14 §14.1](../spec/14-rights.md)).
The fields are defined so that a personal library loads into a team's store later without conversion ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
## Is a model allowed to correct a document?
A model may propose corrections and answer suspects, under the same gate as a person, and its corrections record the model and the run as author.
Only a named person can sign a receipt, and a model's annotation stays `proposed` until a person confirms it ([spec/11 §11.2](../spec/11-corrections.md#112-every-correction-records), [spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
## Why are annotations W3C Web Annotations?
Tools that follow the W3C Web Annotation model can read the standard parts of these annotations.
A layer, the file that groups annotations, follows the [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/), a W3C Recommendation of 2017.
Each target holds an `AnchorSelector` carrying a `scienting-anchor/1` record, followed by a standard `TextQuoteSelector` for tools that do not know the anchor selector.
(The `scienting-` prefix is the formats' namespace, separate from the repository's name.)
The formats use no `TextPositionSelector` or `FragmentSelector`.
The anchor adds what the standard selectors lack: the node's ID, its hashes, and a defined outcome (`current`, `changed-below`, `moved`, `reattached` or `stale`) when the text has changed.
The Web Annotation model alone is the better-established choice where an annotation needs none of that.
Whether a generic Web Annotation client displays an `AnchorSelector` is untested.
See [the comparison](comparison.md).
## What is the difference between a claim and an annotation?
A statement true independently of the paper, such as "K_r is the bond force constant of the AMBER energy", is a claim.
A claim has to be checked against evidence, so it belongs in a store of checked claims outside these formats, which define no claim semantics.
A statement about reading the paper, such as "the authors pass quickly over the scaling here", is an annotation in a layer, attributed and unchecked ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
## Can a team share notes on a paywalled paper?
Yes, without sharing the paper.
A checks-only package withholds all text and keeps IDs, labels, pages, boxes and keyed hashes.
A public layer swaps each quotation for keyed hashes, under a key derived from the paper's page fingerprints, so a reader checks and shows it only over their own copy.
The hash must be keyed because anyone can recover a plain hash of a short text such as `310.0` by hashing guesses ([spec/12 §12.5](../spec/12-package.md#125-text-policies), [spec/05 §5.8](../spec/05-hashes.md#58-keyed-hashes), [spec/06 §6.3](../spec/06-anchors.md#63-keyed-anchors)).
Whether a team may share notes on a paper this way is a question for its own counsel; nothing here is legal advice.
## How does this differ from JATS or TEI?
They overlap in subject, not in job.
JATS, the [Journal Article Tag Suite](https://www.niso.org/publications/z3996-2024-jats) (ANSI/NISO Z39.96-2024), is an XML format for publishers and archives to exchange journal content. manuspec does not replace it.
A JATS file is one kind of witness (a source file), an input from which a document is established.
None of the 312 elements in the [JATS 1.4 Archiving tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) marks a sentence; its smallest text unit is the paragraph, `<p>`.
TEI, the Text Encoding Initiative, has a sentence-like [`<s>` element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html) and keeps both readings of a corrected passage with [`<corr>` and `<sic>`](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html).
The encoder assigns its `xml:id`, and the guidelines [recommend it for durability](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html).
manuspec mints and retires IDs by rule, keeps corrections as separate records with evidence, and hashes each node.
Keep JATS for publisher and archive exchange and TEI for general scholarly text encoding.
A TEI file is a witness of kind `xml` ([spec/02 §2.3](../spec/02-model.md)).
The full comparison is [docs/comparison.md](comparison.md)
## What is its status and maturity?
Version 0.1.0 has 29 formats: 18 are `candidate` and 11 are `provisional`, as [registry/formats.json](../registry/formats.json) lists.
A `candidate` format is complete, with vectors and fixtures, and ready to implement.
A `provisional` one is complete and expected to change as implementations gain experience.
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade and checklist.
The repository holds a reference checker (`tools/check.py`), 21 vector files, five valid packages, one of them also as a zip, and 28 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
## What does it not do?
It is built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
It does not replace publisher interchange formats; it reads JATS as a witness.
It does not fix what a converter reads wrongly.
It records the reading and raises disagreements as suspects, and a correction is checkable.
It is not a search engine, a citation index or a recommendation feed ([docs/overview.md](overview.md)).
## How do I add a node kind or a run kind?
A node kind is a type of document part, such as `sentence`; a run kind is a type of inline content within a sentence, such as `math`.
Propose either with the vector or fixture that shows it.
A new kind is a minor version ([spec/00 §0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
See [CONTRIBUTING.md](../CONTRIBUTING.md).
## What are the files in `tools/` for?
`build.py` builds the fixtures and vectors. `check.py` recomputes and checks every one, and `load.py` loads packages into SQLite.
The `oracle/` modules compute every expected value, so that none is typed by hand.
They are not a library; implementations follow the specification ([tools/README.md](../tools/README.md)).
Reference in new issue View Git Blame Copy Permalink