feat: validator, bundle builder and release checks for 0.1.0
- Add tools/validate.py: validates packages, and with --correction runs proposed corrections through the gate against a package without writing anything (spec/11 §11.4). The gate now refuses a proposal that fails the correction schema, rule correction.schema. - tools/load.py validates every package before loading, and with --bundle --built-from builds a scienting-bundle/1 with the FTS5 search index and bundle_meta (spec/13 §13.5). Search rows are defined per unit (sentence, cell, equation outside a sentence, figure-text) with the parent kind as context. - Roster keys carry a role, and a key with the role owner may revoke any receipt (spec/08 §8.7, spec/20); roster schema, store table and the fixture roster updated. The checker signs a trial revocation to prove the rule both ways. - A qa: receipt raises the nodes it sampled except those in its errors (spec/08 §8.2); the example paper's fidelity counts become 27 F3, 13 F2, 4 F1 and 1 F0. - The destination chooses the keyed scheme, not the text policy: hmac-team between the team's machines, hkdf-pages outside the team (spec/05 §5.8, spec/12 §12.5). - A TextQuoteSelector must repeat the anchor's plain quote and is absent for hashed or keyed anchors (spec/18 §18.3). - Add fixtures/records/suspect.json, a scienting-suspect/1 record. - Oracle: hashes are kept incrementally and checked after every correction against a full recomputation; load and write refuse unsafe manifest paths; Ed25519, HKDF and schema validation come from the cryptography and jsonschema packages; canonical JSON uses the built-in encoder with an exact RFC 8785 fallback. - The tools need Python 3.14 with cryptography and jsonschema; the py310 CI environment is dropped and the workflow runs the default only. - Resolve every [CHECK] note in the docs, READMEs and context; add CONTRIBUTING.md; every tool answers --help and refuses unknown options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
89 files changed
+1141
-513
No files matched your search
+17
-17
@@ -5,12 +5,12 @@ Newcomers, implementers of a reader, writer, converter or checker, and experts w
|
||||
Each answer gives the plain statement first, then the precise one, then the chapter that defines it.
|
||||
The repository is <https://git.scient.ing/infra/manuspec>, and the formats are at version 0.1.0.
|
||||
|
||||
### Can I read these files without special software?
|
||||
## Can I read these files without special software?
|
||||
|
||||
Yes.
|
||||
Every file is JSON or JSON Lines (one JSON value per line), and [spec/](../spec/README.md) specifies every format. [The examples](examples.md) read a package with a few lines of Python.
|
||||
|
||||
### Why not keep using PDFs and page numbers?
|
||||
## Why not keep using PDFs and page numbers?
|
||||
|
||||
A page number names a region, not a statement, and it changes between a preprint and the version of record.
|
||||
An address names a sentence or a cell by its ID, such as `hale2026harmonic#j06tfhn`.
|
||||
@@ -19,14 +19,14 @@ It can add a pin (the first six hex digits of the node's hash) and character off
|
||||
An ID encodes no page, position or kind.
|
||||
The system mints it once from the document key, kind, text hash and occurrence, then freezes it ([spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
|
||||
|
||||
### Why are numbers stored as strings?
|
||||
## Why are numbers stored as strings?
|
||||
|
||||
Databases and JSON libraries rewrite numbers: `310.0` becomes `310` and `1.5260` becomes `1.526`, and the printed digits are part of what a paper says.
|
||||
Canonical JSON, the byte-exact form used for hashing, writes numbers as ECMAScript writes a double, so it would erase the same difference.
|
||||
A decimal that carries meaning is a string.
|
||||
JSON numbers are for counts, indexes, offsets, sizes and geometry ([spec/01 §1.2](../spec/01-conventions.md#12-numbers)).
|
||||
|
||||
### What happens when a better converter comes along?
|
||||
## What happens when a better converter comes along?
|
||||
|
||||
The system aligns the new reading with the established document, sentence by sentence.
|
||||
A sentence the alignment pairs keeps its ID even where the readings disagree, which catches a lost superscript instead of accepting it.
|
||||
@@ -35,7 +35,7 @@ Nothing changes until a correction passes the gate, the rules that refuse, for e
|
||||
|
||||
Converting the same input again with the same converter mints the same IDs ([spec/05 §5.6](../spec/05-hashes.md#56-alignment), [spec/11 §11.4](../spec/11-corrections.md#114-the-gate), [spec/04 §4.2](../spec/04-identity.md#42-minting-node-ids)).
|
||||
|
||||
### Do I need receipts, rights records and spot-checks for a personal library?
|
||||
## Do I need receipts, rights records and spot-checks for a personal library?
|
||||
|
||||
You need no receipts and no spot-checks.
|
||||
A receipt is a signed record that a named person compared nodes with the page image, and a spot-check is a second reviewer's sample of that work.
|
||||
@@ -44,12 +44,12 @@ One person can leave both out.
|
||||
Each source file still carries a rights record, though the record can say little: the license as printed, a `basis` of `unknown`, and no uses, each of which then counts as unknown ([spec/14 §14.1](../spec/14-rights.md)).
|
||||
The fields are defined so that a personal library loads into a team's store later without conversion ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
|
||||
|
||||
### Is a model allowed to correct a document?
|
||||
## Is a model allowed to correct a document?
|
||||
|
||||
A model may propose corrections and answer suspects, under the same gate as a person, and its corrections record the model and the run as author.
|
||||
Only a named person can sign a receipt, and a model's annotation stays `proposed` until a person confirms it ([spec/11 §11.2](../spec/11-corrections.md#112-every-correction-records), [spec/18 §18.3](../spec/18-layers.md#183-the-annotation)).
|
||||
|
||||
### Why are annotations W3C Web Annotations?
|
||||
## Why are annotations W3C Web Annotations?
|
||||
|
||||
Tools that follow the W3C Web Annotation model can read the standard parts of these annotations.
|
||||
A layer, the file that groups annotations, follows the [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/), a W3C Recommendation of 2017.
|
||||
@@ -59,17 +59,17 @@ The formats use no `TextPositionSelector` or `FragmentSelector`.
|
||||
|
||||
The anchor adds what the standard selectors lack: the node's ID, its hashes, and a defined outcome (`current`, `changed-below`, `moved`, `reattached` or `stale`) when the text has changed.
|
||||
The Web Annotation model alone is the better-established choice where an annotation needs none of that.
|
||||
Whether a generic Web Annotation client displays an `AnchorSelector` is untested [CHECK: comparison.md says this was not verified].
|
||||
Whether a generic Web Annotation client displays an `AnchorSelector` is untested.
|
||||
See [the comparison](comparison.md).
|
||||
|
||||
### What is the difference between a claim and an annotation?
|
||||
## What is the difference between a claim and an annotation?
|
||||
|
||||
A statement true independently of the paper, such as "K_r is the bond force constant of the AMBER energy", is a claim.
|
||||
A claim has to be checked against evidence, so it belongs in a store of checked claims outside these formats, which define no claim semantics.
|
||||
|
||||
A statement about reading the paper, such as "the authors pass quickly over the scaling here", is an annotation in a layer, attributed and unchecked ([spec/18 §18.1](../spec/18-layers.md#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)).
|
||||
|
||||
### Can a team share notes on a paywalled paper?
|
||||
## Can a team share notes on a paywalled paper?
|
||||
|
||||
Yes, without sharing the paper.
|
||||
A checks-only package withholds all text and keeps IDs, labels, pages, boxes and keyed hashes.
|
||||
@@ -78,13 +78,13 @@ The hash must be keyed because anyone can recover a plain hash of a short text s
|
||||
|
||||
Whether a team may share notes on a paper this way is a question for its own counsel; nothing here is legal advice.
|
||||
|
||||
### How does this differ from JATS or TEI?
|
||||
## How does this differ from JATS or TEI?
|
||||
|
||||
They overlap in subject, not in job.
|
||||
|
||||
JATS, the [Journal Article Tag Suite](https://www.niso.org/publications/z3996-2024-jats) (ANSI/NISO Z39.96-2024), is an XML format for publishers and archives to exchange journal content. manuspec does not replace it.
|
||||
A JATS file is one kind of witness (a source file), an input from which a document is established.
|
||||
The JATS 1.4 tag library lists no sentence-level element [CHECK: comparison.md found none on the element list page it read; confirm against the full tag library].
|
||||
None of the 312 elements in the [JATS 1.4 Archiving tag library](https://jats.nlm.nih.gov/archiving/tag-library/1.4/) marks a sentence; its smallest text unit is the paragraph, `<p>`.
|
||||
|
||||
TEI, the Text Encoding Initiative, has a sentence-like [`<s>` element](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-s.html) and keeps both readings of a corrected passage with [`<corr>` and `<sic>`](https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-corr.html).
|
||||
The encoder assigns its `xml:id`, and the guidelines [recommend it for durability](https://tei-c.org/release/doc/tei-p5-doc/en/html/SA.html).
|
||||
@@ -92,9 +92,9 @@ manuspec mints and retires IDs by rule, keeps corrections as separate records wi
|
||||
|
||||
Keep JATS for publisher and archive exchange and TEI for general scholarly text encoding.
|
||||
A TEI file is a witness of kind `xml` ([spec/02 §2.3](../spec/02-model.md)).
|
||||
The full comparison is [docs/comparison.md](comparison.md) [CHECK: that page is written elsewhere and does not yet exist in the repository; confirm the path].
|
||||
The full comparison is [docs/comparison.md](comparison.md)
|
||||
|
||||
### What is its status and maturity?
|
||||
## What is its status and maturity?
|
||||
|
||||
Version 0.1.0 has 27 formats: 20 are `candidate` and 7 are `provisional`, as [registry/formats.json](../registry/formats.json) lists.
|
||||
A `candidate` format is complete, with vectors and fixtures, and ready to implement.
|
||||
@@ -103,7 +103,7 @@ The provisional formats are dataset, digitization, view, project, bundle, snapsh
|
||||
|
||||
The repository holds a reference checker (`tools/check.py`), 20 vector files, and 22 example packages, of which 17 are deliberately broken ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
|
||||
|
||||
### What does it not do?
|
||||
## What does it not do?
|
||||
|
||||
It is built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
|
||||
It does not replace publisher interchange formats; it reads JATS as a witness.
|
||||
@@ -111,14 +111,14 @@ It does not fix what a converter reads wrongly.
|
||||
It records the reading and raises disagreements as suspects, and a correction is checkable.
|
||||
It is not a search engine, a citation index or a recommendation feed ([docs/overview.md](overview.md)).
|
||||
|
||||
### How do I add a node kind or a run kind?
|
||||
## How do I add a node kind or a run kind?
|
||||
|
||||
A node kind is a type of document part, such as `sentence`; a run kind is a type of inline content within a sentence, such as `math`.
|
||||
Propose either with the vector or fixture that shows it.
|
||||
A new kind is a minor version ([spec/00 §0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
|
||||
See [CONTRIBUTING.md](../CONTRIBUTING.md).
|
||||
|
||||
### What are the files in `tools/` for?
|
||||
## What are the files in `tools/` for?
|
||||
|
||||
`build.py` builds the fixtures and vectors. `check.py` recomputes and checks every one, and `load.py` loads packages into SQLite.
|
||||
The `oracle/` modules compute every expected value, so that none is typed by hand.
|
||||
|
||||
Reference in new issue
Block a user