infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: validator, bundle builder and release checks for 0.1.0

Browse Source 
- Add tools/validate.py: validates packages, and with --correction runs
  proposed corrections through the gate against a package without
  writing anything (spec/11 §11.4). The gate now refuses a proposal that
  fails the correction schema, rule correction.schema.
- tools/load.py validates every package before loading, and with
  --bundle --built-from builds a scienting-bundle/1 with the FTS5 search
  index and bundle_meta (spec/13 §13.5). Search rows are defined per
  unit (sentence, cell, equation outside a sentence, figure-text) with
  the parent kind as context.
- Roster keys carry a role, and a key with the role owner may revoke any
  receipt (spec/08 §8.7, spec/20); roster schema, store table and the
  fixture roster updated. The checker signs a trial revocation to prove
  the rule both ways.
- A qa: receipt raises the nodes it sampled except those in its errors
  (spec/08 §8.2); the example paper's fidelity counts become 27 F3,
  13 F2, 4 F1 and 1 F0.
- The destination chooses the keyed scheme, not the text policy:
  hmac-team between the team's machines, hkdf-pages outside the team
  (spec/05 §5.8, spec/12 §12.5).
- A TextQuoteSelector must repeat the anchor's plain quote and is absent
  for hashed or keyed anchors (spec/18 §18.3).
- Add fixtures/records/suspect.json, a scienting-suspect/1 record.
- Oracle: hashes are kept incrementally and checked after every
  correction against a full recomputation; load and write refuse unsafe
  manifest paths; Ed25519, HKDF and schema validation come from the
  cryptography and jsonschema packages; canonical JSON uses the built-in
  encoder with an exact RFC 8785 fallback.
- The tools need Python 3.14 with cryptography and jsonschema; the py310
  CI environment is dropped and the workflow runs the default only.
- Resolve every [CHECK] note in the docs, READMEs and context; add
  CONTRIBUTING.md; every tool answers --help and refuses unknown options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 18:58:00 -04:00
1 parent b357af84d8
commit 8c8b51cc3f
89 files changed
+1141 -513
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/README.md
+1 -1
View File
@@ -12,7 +12,7 @@ The guides say what each format is for, with examples drawn from the fixtures.
| [Walkthrough](walkthrough.md) | One paper, from a flawed first reading to a corrected, reviewed and signed document. |
| [Examples](examples.md) | Things to run: checking hashes by hand, reading a package with a few lines of Python, querying a database. |
| [Rationale](rationale.md) | Why the formats look the way they do, and the questions they leave open. |
| [Comparison](comparison.md) | How manuspec compares with JATS, TEI, W3C Web Annotation and other schemas. [CHECK: docs/comparison.md is not in the repository yet; confirm the page exists and covers these standards] |
| [Comparison](comparison.md) | How manuspec compares with JATS, TEI, W3C Web Annotation and other schemas. |
| [*FAQ*](faq.md) | Short answers to the questions people ask first. |
| [Glossary](glossary.md) | Every term, in a sentence. |
docs/concepts.md
+4 -4
View File
@@ -260,7 +260,7 @@ Every leaf has a level that says how far its text was checked.
A level counts only at the node's current hash, so a correction drops a proofread sentence back until someone signs for it again.
The pipeline records F0, F1 or F2; F3 comes only from a valid receipt.
The example reports 28 leaves at F3, 12 at F2, 4 at F1 and 1 at F0, which is its 45 leaves.
The example reports 27 leaves at F3, 13 at F2, 4 at F1 and 1 at F0, which is its 45 leaves.
See [chapter 8](../spec/08-fidelity.md).
## Receipts and spot-checks
@@ -307,9 +307,9 @@ A keyed hash (an HMAC) needs a secret key to compute, so guessing does not work
Values are plain SHA-256 inside the team.
Exported to another of the team's machines, they are HMACs under the team's key.
In a public layer, the key is derived from the paper's page fingerprints (see the previous section), so only someone holding the paper can check them. [fixtures/packages/hale2026harmonic.checks-only](../fixtures/packages/hale2026harmonic.checks-only) is the example paper exported this way, under the team key.
[CHECK: spec/05 §5.8 lists the page-derived key for "any export on a work whose text is withheld", but spec/12 §12.5, fixtures/README.md and the checks-only fixture's document.json use the team key (hmac-team); confirm which the spec intends before stage 03]
In a public layer, or in an export outside the team, the key is derived from the paper's page fingerprints (see the previous section), so only someone holding the paper can check them.
Where the values go chooses the key, not whether the text is withheld.
[fixtures/packages/hale2026harmonic.checks-only](../fixtures/packages/hale2026harmonic.checks-only) is the example paper exported to another of the team's machines, so it uses the team key.
A checks-only package also has no corrections, suspects or datasets.
Pinned by [vectors/keyed.jsonl](../vectors/keyed.jsonl).
docs/examples.md
+3 -4
View File
@@ -9,7 +9,7 @@ The formats are version 0.1.0, and each is `candidate` or `provisional` in `regi
Run every command from the root of the repository, <https://git.scient.ing/infra/manuspec>.
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
Every command also works with plain Python 3.10 or later and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
Every command also works without pixi, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
## Run the checks
@@ -154,7 +154,7 @@ okafor2025open: 15 nodes
smithlab2027charges: 15 nodes
```
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only [CHECK: confirm that no run on Dolt or PostgreSQL is claimed anywhere].
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
```python
@@ -210,8 +210,7 @@ The first query reads the level the converter recorded, not the final one.
A receipt is a signed record that a reviewer checked certain nodes against the page.
Receipts by `ana` raise `e4fkseg` and `r1gacps` to F3; `eajq93t` stays at F0 because the only receipt that covered it, by `cara`, was revoked.
To compute the final level, check each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
The `scope` column of the third query shows the value `targeted:lode`, which is data in the fixture, copied here exactly as the query prints it [CHECK: house rule 2 bans naming programs that use the formats; this value is real fixture data in fixtures/src/hale2026harmonic.json and the package receipts, so either keep the output exact or change the fixture].
The `scope` column of the third query shows the value `targeted:lode`, which is data in the fixture, copied here exactly as the query prints it.
## Test your implementation against a vector
docs/faq.md
+17 -17
View File
@@ -5,12 +5,12 @@ Newcomers, implementers of a reader, writer, converter or checker, and experts w
Each answer gives the plain statement first, then the precise one, then the chapter that defines it.
The repository is <https://git.scient.ing/infra/manuspec>, and the formats are at version 0.1.0.
### Can I read these files without special software?
## Can I read these files without special software?
Yes.
Every file is JSON or JSON Lines (one JSON value per line), and [spec/](../spec/README.md) specifies every format. [The examples](examples.md) read a package with a few lines of Python.
### Why not keep using PDFs and page numbers?
## Why not keep using PDFs and page numbers?
A page number names a region, not a statement, and it changes between a preprint and the version of record.
An address names a sentence or a cell by its ID, such as `hale2026harmonic#j06tfhn`.
@@ -19,14 +19,14 @@ It can add a pin (the first six hex digits of the node's hash) and character off
An ID encodes no page, position or kind.
The system mints it once from the document key, kind, text hash and occurrence, then freezes it ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
### Why are numbers stored as strings?
## Why are numbers stored as strings?
Databases and JSON libraries rewrite numbers: `310.0` becomes `310` and `1.5260` becomes `1.526`, and the printed digits are part of what a paper says.
Canonical JSON, the byte-exact form used for hashing, writes numbers as ECMAScript writes a double, so it would erase the same difference.
A decimal that carries meaning is a string.
JSON numbers are for counts, indexes, offsets, sizes and geometry ([spec/01 §1.2](../spec/01-conventions.md#12-numbers)).
### What happens when a better converter comes along?
## What happens when a better converter comes along?
The system aligns the new reading with the established document, sentence by sentence.
A sentence the alignment pairs keeps its ID even where the readings disagree, which catches a lost superscript instead of accepting it.
@@ -35,7 +35,7 @@ Nothing changes until a correction passes the gate, the rules that refuse, for e
Converting the same input again with the same converter mints the same IDs ([spec/05 §5.6](../spec/05-hashes.md#56-alignment), [spec/11 §11.4](../spec/11-corrections.md#114-the-gate), [spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
### Do I need receipts, rights records and spot-checks for a personal library?
## Do I need receipts, rights records and spot-checks for a personal library?
You need no receipts and no spot-checks.
A receipt is a signed record that a named person compared nodes with the page image, and a spot-check is a second reviewer's sample of that work.
@@ -44,12 +44,12 @@ One person can leave both out.
Each source file still carries a rights record, though the record can say little: the license as printed, a `basis` of `unknown`, and no uses, each of which then counts as unknown ([spec/14 §14.1](../spec/14-rights.md)).
The fields are defined so that a personal library loads into a team's store later without conversion ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
### Is a model allowed to correct a document?
## Is a model allowed to correct a document?
A model may propose corrections and answer suspects, under the same gate as a person, and its corrections record the model and the run as author.
Only a named person can sign a receipt, and a model's annotation stays `proposed` until a person confirms it ([spec/11 §11.2](../spec/11-corrections.md#112-every-correction-records), [spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
### Why are annotations W3C Web Annotations?
## Why are annotations W3C Web Annotations?
Tools that follow the W3C Web Annotation model can read the standard parts of these annotations.
A layer, the file that groups annotations, follows the [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/), a W3C Recommendation of 2017.
@@ -59,17 +59,17 @@ The formats use no `TextPositionSelector` or `FragmentSelector`.
The anchor adds what the standard selectors lack: the node's ID, its hashes, and a defined outcome (`current`, `changed-below`, `moved`, `reattached` or `stale`) when the text has changed.
The Web Annotation model alone is the better-established choice where an annotation needs none of that.
Whether a generic Web Annotation client displays an `AnchorSelector` is untested [CHECK: comparison.md says this was not verified].
Whether a generic Web Annotation client displays an `AnchorSelector` is untested.
See [the comparison](comparison.md).
### What is the difference between a claim and an annotation?
## What is the difference between a claim and an annotation?
A statement true independently of the paper, such as "K_r is the bond force constant of the AMBER energy", is a claim.
A claim has to be checked against evidence, so it belongs in a store of checked claims outside these formats, which define no claim semantics.
A statement about reading the paper, such as "the authors pass quickly over the scaling here", is an annotation in a layer, attributed and unchecked ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
### Can a team share notes on a paywalled paper?
## Can a team share notes on a paywalled paper?
Yes, without sharing the paper.
A checks-only package withholds all text and keeps IDs, labels, pages, boxes and keyed hashes.
@@ -78,13 +78,13 @@ The hash must be keyed because anyone can recover a plain hash of a short text s
Whether a team may share notes on a paper this way is a question for its own counsel; nothing here is legal advice.
### How does this differ from JATS or TEI?
## How does this differ from JATS or TEI?
They overlap in subject, not in job.
JATS, the [Journal Article Tag Suite](https://www.niso.org/publications/z3996-2024-jats) (ANSI/NISO Z39.96-2024), is an XML format for publishers and archives to exchange journal content. manuspec does not replace it.
A JATS file is one kind of witness (a source file), an input from which a document is established.
The JATS 1.4 tag library lists no sentence-level element [CHECK: comparison.md found none on the element list page it read; confirm against the full tag library].
None of the 312 elements in the [JATS 1.4 Archiving tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) marks a sentence; its smallest text unit is the paragraph, `<p>`.
TEI, the Text Encoding Initiative, has a sentence-like [`<s>` element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html) and keeps both readings of a corrected passage with [`<corr>` and `<sic>`](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html).
The encoder assigns its `xml:id`, and the guidelines [recommend it for durability](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html).
@@ -92,9 +92,9 @@ manuspec mints and retires IDs by rule, keeps corrections as separate records wi
Keep JATS for publisher and archive exchange and TEI for general scholarly text encoding.
A TEI file is a witness of kind `xml` ([spec/02 §2.3](../spec/02-model.md)).
The full comparison is [docs/comparison.md](comparison.md) [CHECK: that page is written elsewhere and does not yet exist in the repository; confirm the path].
The full comparison is [docs/comparison.md](comparison.md)
### What is its status and maturity?
## What is its status and maturity?
Version 0.1.0 has 27 formats: 20 are `candidate` and 7 are `provisional`, as [registry/formats.json](../registry/formats.json) lists.
A `candidate` format is complete, with vectors and fixtures, and ready to implement.
@@ -103,7 +103,7 @@ The provisional formats are dataset, digitization, view, project, bundle, snapsh
The repository holds a reference checker (`tools/check.py`), 20 vector files, and 22 example packages, of which 17 are deliberately broken ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
### What does it not do?
## What does it not do?
It is built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
It does not replace publisher interchange formats; it reads JATS as a witness.
@@ -111,14 +111,14 @@ It does not fix what a converter reads wrongly.
It records the reading and raises disagreements as suspects, and a correction is checkable.
It is not a search engine, a citation index or a recommendation feed ([docs/overview.md](overview.md)).
### How do I add a node kind or a run kind?
## How do I add a node kind or a run kind?
A node kind is a type of document part, such as `sentence`; a run kind is a type of inline content within a sentence, such as `math`.
Propose either with the vector or fixture that shows it.
A new kind is a minor version ([spec/00 §0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
See [CONTRIBUTING.md](../CONTRIBUTING.md).
### What are the files in `tools/` for?
## What are the files in `tools/` for?
`build.py` builds the fixtures and vectors. `check.py` recomputes and checks every one, and `load.py` loads packages into SQLite.
The `oracle/` modules compute every expected value, so that none is typed by hand.
docs/glossary.md
+6 -2
View File
@@ -62,7 +62,8 @@ Layer
: A named set of annotations with one purpose, audience, visibility and license. ([18.2](../spec/18-layers.md#182-the-layer))
Leaf
: A node that holds text itself: a sentence, cell, reference, author line, keyword line, equation, equation line, code or algorithm [CHECK: spec/03 §3.6 says an `equation-line` is never a text leaf, while spec/06 §6.4 lists it as a leaf anchor kind; confirm the intended list]. ([3.6](../spec/03-nodes.md#36-the-plain-projection))
: A node that holds text itself: a sentence, cell, reference, author line, keyword line, equation, equation line, code or algorithm.
A leaf anchor points at one whole. ([6.4](../spec/06-anchors.md#64-four-kinds-of-anchor))
Lossy projection
: A node's text with scripts flattened, citation marks, whitespace and dashes removed, and case folded; used for proposing alignments. ([5.5](../spec/05-hashes.md#55-the-lossy-projection))
@@ -112,7 +113,7 @@ Rights record
: A witness's license, the basis on which the team holds it, and the uses that basis allows. ([14.1](../spec/14-rights.md#141-the-rights-record))
Roster
: The list of keys that may sign, with their owners and purposes. ([8.7](../spec/08-fidelity.md#87-revocations-and-the-roster))
: The list of keys that may sign, with the person who holds each, that person's role in the team, and what the key may sign. ([8.7](../spec/08-fidelity.md#87-revocations-and-the-roster))
Selector
: Outside term.
@@ -136,6 +137,9 @@ Targeted F3
: Proofreading the nodes a team's claims, quotations and datasets rest on, rather than every node.
The target set also includes every display equation and, under the project's digit rule (`all`, `anchored-sections` or `none`), sentences with a digit in a text or math run (an inline stretch of plain text, or of TeX or Typst math, within the sentence; [3.5](../spec/03-nodes.md#35-inline-runs)). ([8.10](../spec/08-fidelity.md#810-targeted-f3))
Text leaf
: A leaf whose text a container's projections are made of: any leaf except an equation line, or an equation inside a sentence, whose text is already part of its equation or its sentence. ([3.6](../spec/03-nodes.md#36-the-plain-projection))
Text policy
: Whether a package carries its text (`full`) or withholds it (`checks-only`). ([12.5](../spec/12-package.md#125-text-policies))
docs/rationale.md
+2 -2
View File
@@ -99,7 +99,7 @@ Only the three text values, `text_hash`, `match_hash` and `align_key`, normalize
### Hashes that leave the team are keyed
A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages [CHECK: the overlapping-anchors claim is the author's; spec/05 §5.8 gives only the guessing argument].
A plain hash of "310.0" is recovered by hashing guesses, and overlapping anchors can rebuild passages.
Inside a team, plain hashes are fine.
An export to another machine of the team uses an HMAC (hash-based message authentication code), a hash that needs a secret key, under the team key.
@@ -124,7 +124,7 @@ Deciding whether a merged paragraph still holds "at least half" of what was anch
### Long quotations become a hash and a length
Quotations over 200 code points are replaced by a hash and a length, which keeps anchors small and limits how much text an anchor carries [CHECK: reason for the cap is the author's; spec/06 §6.2 gives the number but not the reason].
Quotations over 200 code points are replaced by a hash and a length, which keeps anchors small and limits how much text an anchor carries.
A hashed window can still be found again in the node.
### Prefix and suffix pick between repeated quotes
docs/walkthrough.md
+8 -5
View File
@@ -134,7 +134,7 @@ A checker of exact words accepts only `current`, `moved` and `reattached`.
The [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/) is the general W3C Recommendation for annotating web resources. manuspec's annotation layers build on it, with a custom `AnchorSelector` followed by a standard `TextQuoteSelector` as a fallback.
It has no permanent IDs for sentences inside the source, no hashes of the target text, and no results such as `moved` or drift.
See [comparison.md](comparison.md) [CHECK: docs/comparison.md is absent from the repository snapshot; confirm the link target].
See [comparison.md](comparison.md).
## 6. Review and signatures
@@ -154,8 +154,10 @@ His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes
Those sentences fall back to their earlier levels, which is why `eajq93t` is still F0.
- `ana` signs `gena76nkb6cnc` for the figure's three caption sentences. `ben`'s spot-check `tejzs01eqn2ym` finds a missing period in `nmm4md0`, which suspends the receipt until she reviews the sitting again and signs a new one.
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic/document.json), counts the result over 45 leaves: 28 at F3, 12 at F2, 4 at F1 and 1 at F0.
The 28 are the 23 nodes of `eqgbbpwb6725v`, the 2 of `zaxky8c1y12tc`, and the 3 caption sentences, which still count because Ben's own valid receipt lists them at F3 [CHECK: confirm this is intended; tools/oracle/receipts.py computes it this way].
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
The 27 are the 23 nodes of `eqgbbpwb6725v`, the 2 of `zaxky8c1y12tc`, and 2 of the 3 caption sentences.
Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right.
The third, `nmm4md0`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
## 7. Data out of the table
@@ -168,14 +170,15 @@ The reference checker, [`tools/check.py`](../tools/check.py), confirms each CSV
## 8. Sharing without the text
The team wants to share its notes with another machine and keep the paper to itself.
The team wants to move its notes to another of its own machines without copying the paper there.
[`hale2026harmonic.checks-only`](../fixtures/packages/hale2026harmonic.checks-only/) is the same document with every leaf's text withheld and every hash replaced by an HMAC under the team's key ([spec/12 §12.5](../spec/12-package.md), [spec/05 §5.8](../spec/05-hashes.md)).
An HMAC is a hash mixed with a secret key.
Anyone can recover a plain hash of a short text such as "310.0" by hashing guesses; a keyed hash resists that.
Corrections, receipts, the roster and the dataset are left out too.
IDs, labels, pages and boxes survive, so annotations still land in the right place.
Only someone holding the key and the text can check the values.
[CHECK: the author wrote "a collaborator's machine"; the fixture uses the team-key scheme (`hmac-team`), which spec/05 §5.8 assigns to exports between the team's own machines].
The team key suits an export between the team's own machines.
An export to a collaborator outside the team would use a key derived from the pages of the paper instead, so that only someone holding the paper can check it.
## 9. Recompute, print and query the package