alexmandClaude
8c8b51cc3f
- Add tools/validate.py: validates packages, and with --correction runs proposed corrections through the gate against a package without writing anything (spec/11 §11.4). The gate now refuses a proposal that fails the correction schema, rule correction.schema. - tools/load.py validates every package before loading, and with --bundle --built-from builds a scienting-bundle/1 with the FTS5 search index and bundle_meta (spec/13 §13.5). Search rows are defined per unit (sentence, cell, equation outside a sentence, figure-text) with the parent kind as context. - Roster keys carry a role, and a key with the role owner may revoke any receipt (spec/08 §8.7, spec/20); roster schema, store table and the fixture roster updated. The checker signs a trial revocation to prove the rule both ways. - A qa: receipt raises the nodes it sampled except those in its errors (spec/08 §8.2); the example paper's fidelity counts become 27 F3, 13 F2, 4 F1 and 1 F0. - The destination chooses the keyed scheme, not the text policy: hmac-team between the team's machines, hkdf-pages outside the team (spec/05 §5.8, spec/12 §12.5). - A TextQuoteSelector must repeat the anchor's plain quote and is absent for hashed or keyed anchors (spec/18 §18.3). - Add fixtures/records/suspect.json, a scienting-suspect/1 record. - Oracle: hashes are kept incrementally and checked after every correction against a full recomputation; load and write refuse unsafe manifest paths; Ed25519, HKDF and schema validation come from the cryptography and jsonschema packages; canonical JSON uses the built-in encoder with an exact RFC 8785 fallback. - The tools need Python 3.14 with cryptography and jsonschema; the py310 CI environment is dropped and the workflow runs the default only. - Resolve every [CHECK] note in the docs, READMEs and context; add CONTRIBUTING.md; every tool answers --help and refuses unknown options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Registry
The registry holds lists that tools read at run time. Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program. This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.
| File | Holds |
|---|---|
formats.json |
Every format: its identifier, media type (null for four formats that have none), schema, status, chapter and a one-line description. |
rules.json |
Every validation rule ID, with what it means and the chapter that defines it. |
stamps/scienting-stamps-1.json |
The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them. |
stages/scienting-stages-default.json |
The rights each processing stage needs, as a default a team may replace. |
Format status
Each entry in formats.json has a status.
A candidate format is complete, with vectors and fixtures, and ready to implement.
It changes only with a recorded reason and a new or changed vector.
A provisional format is complete and expected to change as implementations gain experience, so it may change more freely.
At version 0.1.0, 20 of the 27 formats are candidate and seven are provisional.
The third defined value, reserved, is unused.
The values are defined in Chapter 0, section 0.4, and the stability rules in Chapter 22, section 22.5.
Rule IDs
A validator checks a package and reports each failure as path:line: rule: message, for example:
nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute
A node_hash is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document.
The line above is the failure that fixtures/invalid/hash-mismatch/ is built to cause: its sixth node record carries a node_hash of all zeros.
A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs.
A tool can match on the rule field, whose meaning is looked up in rules.json, and treat the message as text for people.
Chapter 22, section 22.4 lists the ID prefixes.
Each directory in fixtures/invalid/ has an expect.json that names the rule it breaks.
The reference checker (tools/check.py) fails if that rule is not in rules.json.
Adding stamp rules
Many publishers add a line to each PDF they serve, naming the institution and the time of download.
This page calls that line a stamp.
Two downloads of one paper then differ, so a hash of the file cannot match them.
A page fingerprint is a hash of a page's size, its text and its images.
It is computed after removing every line that a stamp rule matches (Chapter 7, section 7.1).
The vectors in vectors/fingerprints.jsonl show this: acs-download-1 and acs-download-2 differ in the institution and the time on their stamp lines and have the same fingerprint.
Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes. Record a rule in a file of your own, marked as learned:
{"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
"rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}
Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed (Chapter 1, section 1.4).
A rule has an id, a pattern, and a where that is any or margin.
A margin rule matches only lines whose box lies wholly within margin points of one page edge (36 when the rule gives none).
Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case.
The allowed syntax is literals, ., *, +, ?, ^, $, classes and escaped punctuation.
A page record in pages.jsonl lists, in stamps_removed, the IDs of the rules that removed lines.
A rule that is too broad hides a real difference between two pages.
A stamp that no rule recognizes splits one edition into two (Chapter 7, section 7.2).
Replacing the stage defaults
A stage is one step of processing, such as conversion or indexing. Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them. The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.
A team copies scienting-stages-default.json, adjusts it with its counsel and points its project's settings at the copy.
The file itself calls its contents "proposed defaults from the design, for counsel to set".
In a project, the settings.stages field holds the id of the stage file in use (Chapter 20), so a team's copy needs an id of its own.
The project format is provisional.
Chapter 14 explains how stages are checked.