0 Fork 0

Files

T

feat: validator, bundle builder and release checks for 0.1.0

- Add tools/validate.py: validates packages, and with --correction runs
  proposed corrections through the gate against a package without
  writing anything (spec/11 §11.4). The gate now refuses a proposal that
  fails the correction schema, rule correction.schema.
- tools/load.py validates every package before loading, and with
  --bundle --built-from builds a scienting-bundle/1 with the FTS5 search
  index and bundle_meta (spec/13 §13.5). Search rows are defined per
  unit (sentence, cell, equation outside a sentence, figure-text) with
  the parent kind as context.
- Roster keys carry a role, and a key with the role owner may revoke any
  receipt (spec/08 §8.7, spec/20); roster schema, store table and the
  fixture roster updated. The checker signs a trial revocation to prove
  the rule both ways.
- A qa: receipt raises the nodes it sampled except those in its errors
  (spec/08 §8.2); the example paper's fidelity counts become 27 F3,
  13 F2, 4 F1 and 1 F0.
- The destination chooses the keyed scheme, not the text policy:
  hmac-team between the team's machines, hkdf-pages outside the team
  (spec/05 §5.8, spec/12 §12.5).
- A TextQuoteSelector must repeat the anchor's plain quote and is absent
  for hashed or keyed anchors (spec/18 §18.3).
- Add fixtures/records/suspect.json, a scienting-suspect/1 record.
- Oracle: hashes are kept incrementally and checked after every
  correction against a full recomputation; load and write refuse unsafe
  manifest paths; Ed25519, HKDF and schema validation come from the
  cryptography and jsonschema packages; canonical JSON uses the built-in
  encoder with an exact RFC 8785 fallback.
- The tools need Python 3.14 with cryptography and jsonschema; the py310
  CI environment is dropped and the workflow runs the default only.
- Resolve every [CHECK] note in the docs, READMEs and context; add
  CONTRIBUTING.md; every tool answers --help and refuses unknown options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-10-14 18:58:00 -04:00

8.6 KiB

Raw Blame History

Fixtures

A reader, writer, converter or checker of stored papers needs files to test against, and a newcomer needs examples of what a stored paper looks like. This folder holds that example data of manuspec (https://git.scient.ing/infra/manuspec), version 0.1.0. All text is synthetic, written for these files and dedicated CC0; license fields inside the fixtures describe fictional works.

Three readers use these files. A newcomer learns what a stored paper looks like and what can go wrong with one. An implementer, who writes a reader, writer, converter or checker, tests it against the files. An expert reviewing the work finds which cases the tests pin and what the published keys do and do not prove.

Each format here has the status candidate or provisional in registry/formats.json.

A hash is a fixed-length fingerprint of data: any change to the data changes it. The hash used here is SHA-256, written in hexadecimal. A package is one document stored as a folder of files, with a manifest listing every file and its hash. A node is an addressable part of a document, such as a section, a sentence or a table cell. A witness is a source file a reading came from, such as a PDF. A receipt is a signed statement that a named person reviewed listed nodes.

Packages

Each package under packages/ is a valid document. A conforming reader accepts each one and recomputes every hash it contains.

Package	What it is for
`packages/hale2026harmonic/`	The main example, followed step by step in the walkthrough: a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, and derived MathML.
`packages/hale2026harmonic.checks-only/`	The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` (spec/05, spec/12 section 12.5). It shows how a team can share its checks on a paywalled paper without sharing the paper.
`packages/okafor2025open/`	An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness.
`packages/smithlab2027charges/`	A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#r1gacps`, a sentence of the main example.
`packages/tiny2026example/`	A small document, the starting point of the correction vectors (`vectors/corrections.jsonl`) and of every broken package below.

Records

records/ holds a valid example of every other format that is a JSON file of its own. Three formats have their example elsewhere: the stage requirements (scienting-stages/1) and the stamp rules (scienting-stamps/1) are in registry/, and a bundle (scienting-bundle/1) is a SQLite file that tools/load.py --bundle builds from the packages. Each record is a single file, or a folder where a format needs several files.

Files	What they are for
`anchor-span.json`, `anchor-leaf.json`, `anchor-cell.json`, `anchor-paragraph.json`	An anchor is an address pinned to a hash, with a quotation and context, so it can be found again after the text changes. These four anchors point at a span of a sentence, a sentence, a table cell and a paragraph of the main example.
`anchor-keyed.json`	An anchor that leaves the team, as in a public layer: its table context is stored as keyed hashes, so the labels cannot be read from it (spec/06 section 6.3).
`rights-open.json`, `rights-subscription.json`, `rights-collective.json`	A witness's license and what uses it allows, under an open license, a subscription and a collective license.
`policy.json`	A signed team decision that settles a use the rights record leaves unknown.
`backend-docling.json`, `backend-examplevendor.json`	The code and weight licenses of a conversion tool and the conditions attached to them.
`profile-acme.json`, `profile-smithlab.json`	What those conditions ask about a team: its purpose, its funding and whom it competes with.
`roster.json`	The keys that may sign, the role of each key's holder in the team, and what each key may sign. Ana is the owner, so she may revoke anyone's receipt.
`layer/`	A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`).
`submissions/`, `grades/`	A student's answers to exercises in the layer, and the grader's result for each.
`suspect.json`	A doubtful reading raised for review: the converter's reading of an exponent in the main example, before the first correction answered it.
`qa-report.json`	Reviewer error bounds for a delivery that covers two of the main example's receipts.
`project.json`, `digitization.json`, `digitization-confirmation.json`, `datasets/`, `view.vl.json`, `snapshot.json`, `draft-ids.jsonl`	A team project; values read off the main example's figure by hand, the receipt that confirms them, and the dataset they yield; a Vega-Lite chart pinned to that dataset; what the draft cites, pinned; and the sentence IDs of the draft's Typst project. These formats are `provisional`.

Broken packages

Each folder in invalid/ holds a package/ that breaks exactly one rule, built from packages/tiny2026example/ by one edit, and an expect.json that names the rule. A conforming implementation refuses every one of them and should report that rule. registry/rules.json lists every rule ID with its meaning, and a validator reports each failure as path:line: rule: message (spec/22 section 22.2).

Case	What is wrong	Rule
`alias-dangling/`	An alias whose node does not exist.	`doc.id.successor`
`bad-id/`	An ID that uses a letter Crockford base32 lacks (i, l, o or u).	`package.schema`
`checks-only-plain/`	A checks-only package that still carries text and plain hashes.	`doc.policy.withheld`
`cite-not-reference/`	A cite run names a node that is not a reference.	`doc.run.cite`
`empty-optional-file/`	An optional JSONL file with no records.	`package.file.empty`
`hash-mismatch/`	A node_hash that does not recompute.	`doc.hash.node-hash`
`heading-not-first/`	A heading must be a section's first child.	`doc.tree.child`
`location-unknown-page/`	A location on a page no witness has.	`doc.location.page`
`manifest-hash/`	A file changed after the manifest was written.	`package.manifest.hash`
`not-canonical/`	A line that is valid JSON but not canonical JSON (the fixed form of RFC 8785, which gives equal data identical bytes).	`package.file.canonical`
`receipt-root/`	A receipt whose Merkle root (one hash that summarizes a list of hashes) does not recompute.	`doc.receipt.root`
`receipt-signature/`	A receipt changed after it was signed.	`doc.receipt.signature`
`root-mismatch/`	document.json's root is not the root node's node_hash.	`doc.hash.root`
`style-sup-sub/`	A run cannot be both a subscript and a superscript.	`doc.run.style`
`text-edited/`	Text changed without its hashes.	`doc.hash.text-hash`
`unknown-kind/`	Kinds outside the list are refused.	`package.schema`
`unlisted-file/`	A file the manifest does not list.	`package.manifest.unlisted`

Keys

keys/ holds the Ed25519 secret keys (ana, ben, cara, and ben-spot, whose signature picks the nodes of a spot-check) that signed the fixtures' receipts, and the team secret of the checks-only package. Ed25519 is a signature scheme: the secret key makes a signature, and the public key in a roster verifies it. These are published test keys. Anyone can reproduce the signatures, so a signature made with these keys proves nothing about who made it. Use them for nothing else.

Sources

src/ holds the hand-written descriptions of the three main documents, as first read; see src/README.md. Everything else here is generated by python3 tools/build.py, so change the sources or the builder, never the generated files.

8.6 KiB Raw Blame History