infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ea0a7efa78f6c118a946d6fff89d875ce3610c3f
paperspec/README.md
T
alexmandClaude ea0a7efa78 feat: zipped packages, target set, checklists, audit fixes and a stress test 
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-14 20:07:22 -04:00

133 lines
9.1 KiB
Markdown
Raw Blame History

# manuspec
**The data formats for storing a team's scientific papers once, correcting them once, addressing them down to the sentence, and keeping them.**
This page is for three readers.
A newcomer should read straight through; each term is explained where it first appears.
The tables below point an implementer, who will write a reader, writer, converter or checker, to the chapters, test vectors and fixtures that define exact behavior.
The sections on status and on differences give an expert in the neighboring standards the version, the maturity of each format and the comparison.
The repository is at <https://git.scient.ing/infra/manuspec>.
A research team usually cares about a few dozen papers it understands completely, not the world's literature.
Those papers deserve better than a folder of PDFs.
Every sentence, table cell and equation should have a permanent address, and every correction should be recorded with its evidence.
Notes, claims and extracted numbers should point at exactly what they came from.
Anyone should be able to see how far each piece was checked, and by whom.
This repository defines the files that make that possible.
It holds no application.
Every format is documented here and readable without any program's code, so a reader, writer, converter, checker or store can each be a separate program.
Every file is JSON (a plain-text format for structured data) or JSON Lines (one JSON record per line).
A package is the unit these formats store a paper in: a folder or a zip file of such files holding the text as nodes (sections, sentences, cells, equations), page geometry, references, corrections, review receipts and the list of suspects ([spec/12](spec/12-package.md)).
```text
cornell1995second#k3m9q2a@4f1a0c[12:57]
└──── document ──┘└ node ┘└ pin ┘└span┘
```
A hash is a short fingerprint computed from text: the same text always gives the same fingerprint, and changing one character changes the hash.
The string above names code points 12 to 57 of one sentence, as it read when its hash began `4f1a0c`.
The node ID `k3m9q2a` encodes no page, position or kind.
It is minted once from the document key, the node's kind, its text hash and its occurrence, and then frozen.
Fixing a typo elsewhere in the paper leaves the ID alone, and converting the same input again mints the same IDs.
A teammate's download of the same PDF, stamped differently by the publisher, has the same page fingerprints.
If a different converter reads the paper, each sentence it reads identically keeps its ID. Every difference becomes a suspect, a place flagged for review.
Nothing changes until a correction with its evidence is accepted ([spec/04](spec/04-identity.md), [spec/05 §5.6](spec/05-hashes.md#56-alignment)).
The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and [vectors/addresses.jsonl](vectors/addresses.jsonl) holds valid and invalid examples.
## Start here
| If you want to | Read |
| --- | --- |
| Understand what the project is and how the parts fit | [docs/overview.md](docs/overview.md) |
| Learn the ideas one at a time, each with an example | [docs/concepts.md](docs/concepts.md) |
| Follow one paper from its first reading to a signed document | [docs/walkthrough.md](docs/walkthrough.md) |
| Run something in the next five minutes | [docs/examples.md](docs/examples.md) |
| Implement a reader or writer | [spec/](spec/README.md), then [vectors/](vectors/README.md) |
| Know why a rule is the way it is | [docs/rationale.md](docs/rationale.md) |
| Look up a term | [docs/glossary.md](docs/glossary.md) |
| Compare with neighboring standards | [docs/comparison.md](docs/comparison.md) |
## What is in the repository
| Folder | Holds |
| --- | --- |
| [spec/](spec/README.md) | The specification, in 23 chapters. Normative. |
| [schemas/](schemas/README.md) | JSON Schemas, which state what a valid file or record looks like, for every file and record. |
| [sql/](sql/README.md) | Tables for SQLite (a database kept in one file), Dolt and PostgreSQL, and the SQLite bundle with full-text search. |
| [context/](context/README.md) | The JSON-LD context (a file that gives JSON terms shared meanings) that makes annotation layers W3C Web Annotations. |
| [registry/](registry/README.md) | The list of formats, the rule IDs validators report, download-stamp rules, and the default rights each processing stage requires. |
| [vectors/](vectors/README.md) | Test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. |
| [tools/](tools/README.md) | The programs that build and check everything above. |
| [docs/](docs/README.md) | Guides for people getting to know the project. |
## Try it
With [pixi](https://pixi.sh), one command sets up a pinned environment and checks everything:
```sh
pixi run check
```
The checker validates every schema, record and package.
It confirms that every broken package is refused for the stated reason.
It recomputes every test vector, and it rebuilds everything and compares it byte for byte.
Last, it loads every package into SQLite and back, and checks the prose and the links.
Pixi lints the Markdown first.
The run ends with `All checks pass.`
`pixi task list` shows the other tasks: `quick`, `build`, `lint`, `load` and `validate`.
The tools need Python 3.14 and two packages, `jsonschema` and `cryptography`, which pixi installs along with markdownlint for the Markdown.
Without pixi, install those two with pip and run `python3 tools/check.py`.
To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from.
```sh
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic
```
## Format names
Every format is named `scienting-<name>/<major>`, such as `scienting-doc/1` or `scienting-anchor/1`.
The `scienting-` prefix is the formats' namespace, a separate matter from the repository's name.
A format is named after what it holds, never after a product or a repository, and products can be renamed without touching data.
Schemas use the identifier base `https://scient.ing/ns/schemas/`.
[registry/formats.json](registry/formats.json) lists every format with its media type, schema, status and chapter.
## Status and limits
This is version 0.1.0.
Of the 29 formats, 18 are `candidate` and 11 are `provisional`.
The `provisional` formats are project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade and checklist; where those are described, they are marked as such.
The repository includes a reference checker, test vectors and fixtures.
The formats are built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
They do not replace publisher interchange formats.
They do not fix what a converter reads wrongly; they record it and make it checkable.
Not every addressable thing is a node with five hashes (the content values `text_hash`, `match_hash`, `align_key`, `own_hash` and `node_hash`, explained in [docs/concepts.md](docs/concepts.md)).
Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](spec/04-identity.md), [spec/09](spec/09-figures-tables-equations.md)).
## How it differs from neighboring standards
- JATS is the XML format in which publishers and archives exchange journal content ([NISO Z39.96-2024](https://www.niso.org/publications/z3996-2024-jats)).
Keep using it for exchange.
Here, a JATS file is one kind of reading a package can be established from (`jats` is a witness kind, [spec/02 §2.3](spec/02-model.md)).
- TEI has a sentence element, `<s>` ([TEI P5](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html)), and recommends author-assigned `xml:id` values for durable pointers ([TEI P5 pointers](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html)).
Here, node IDs are minted by a rule and, when a node is merged or deleted, retired with a recorded successor ([spec/04 §4.3](spec/04-identity.md)).
- W3C Web Annotation is a general model for annotating any resource, with text quote, text position and other selectors ([W3C Recommendation](https://www.w3.org/TR/annotation-model/)).
Use it for annotation in general.
Here, annotation layers are Web Annotations whose target selector is a custom `AnchorSelector`, optionally followed by a standard `TextQuoteSelector` for tools that do not know ours ([spec/18](spec/18-layers.md)).
- Docling is an open-source document conversion toolkit ([arXiv:2501.17887](https://arxiv.org/abs/2501.17887)).
Use a converter like it to read PDFs.
It identifies items by position in a list, such as `#/texts/12` ([schema](https://raw.githubusercontent.com/docling-project/docling-core/main/docs/DoclingDocument.json)).
Here, IDs do not depend on position, and a second converter's reading is aligned with the first and compared.
## Licenses
The specification and other prose are CC BY 4.0, so anyone may implement the formats.
Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0.
The tools are Apache-2.0.
See [LICENSE.md](LICENSE.md).
Reference in new issue View Git Blame Copy Permalink