alexmandClaude
a09d158e16
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
openalex in same_as, else the folded title, year and family name. A
collision lengthens the later key by two characters; a key is minted
once and never recomputed. A document key adds .si, .v<seq> or .<type>
for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
every node and record ID was re-minted and the docs rewritten to match.
The validator checks the document key against the work key and version
(doc.key.work) and the hash part against the identifier (doc.key.suffix).
vectors/keys.jsonl pins the cases paperstack's tests pin.
Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
implemented in tools/oracle/vrf.py in plain Python and reproduced
against the RFC's Example 16; the sample ranks by the VRF output. The
earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
verify_inclusion, consistency_proof, verify_consistency) with
scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
checkpoints with key hashes as transparency logs share them, the rule
that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).
Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
the checker refuses another version.
Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
checkpoint records; a check that every ID and hash the prose quotes
exists in the fixtures or vectors; three more broken packages
(history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
section, media-type registration and OCI distribution notes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5.8 KiB
Registry
The registry holds lists that tools read at run time. Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program. This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.
| File | Holds |
|---|---|
formats.json |
Every format: its identifier, media type (null for the six that have none), schema, status, chapter and a one-line description. |
rules.json |
Every validation rule ID, with what it means and the chapter that defines it. |
stamps/scienting-stamps-1.json |
The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them. |
stages/scienting-stages-default.json |
The rights each processing stage needs, as a default a team may replace. |
Format status
Each entry in formats.json has a status.
A candidate format is complete, with vectors and fixtures, and ready to implement.
It changes only with a recorded reason and a new or changed vector.
A provisional format is complete and expected to change as implementations gain experience, so it may change more freely.
At version 0.1.0, 19 of the 32 formats are candidate and 13 are provisional.
The third defined value, reserved, is unused.
The values are defined in Chapter 0, section 0.4, and the stability rules in Chapter 22, section 22.5.
Registering the media types
The application/vnd.scienting.* media types are vendor-tree types, which IANA registers on request without a standards document (RFC 6838 §3.2).
Register each type the formats serve over the web before the first release that other tools read: the zipped package, the anchor, the receipt and the layer first.
Until then the types are unregistered and work as they are; a registration changes no byte of any file.
The JSON-LD context is application/ld+json and a checkpoint is text/plain, both already registered.
Rule IDs
A validator checks a package and reports each failure as path:line: rule: message, for example:
nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute
A node_hash is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document.
The line above is the failure that fixtures/invalid/hash-mismatch/ is built to cause: its sixth node record carries a node_hash of all zeros.
A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs.
A tool can match on the rule field, whose meaning is looked up in rules.json, and treat the message as text for people.
Chapter 22, section 22.4 lists the ID prefixes.
Each directory in fixtures/invalid/ has an expect.json that names the rule it breaks.
The reference checker (tools/check.py) fails if that rule is not in rules.json.
Adding stamp rules
Many publishers add a line to each PDF they serve, naming the institution and the time of download.
This page calls that line a stamp.
Two downloads of one paper then differ, so a hash of the file cannot match them.
A page fingerprint is a hash of a page's size, its text and its images.
It is computed after removing every line that a stamp rule matches (Chapter 7, section 7.1).
The vectors in vectors/fingerprints.jsonl show this: acs-download-1 and acs-download-2 differ in the institution and the time on their stamp lines and have the same fingerprint.
Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes. Record a rule in a file of your own, marked as learned:
{"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
"rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}
Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed (Chapter 1, section 1.4).
A rule has an id, a pattern, and a where that is any or margin.
A margin rule matches only lines whose box lies wholly within margin points of one page edge (36 when the rule gives none).
Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case.
The allowed syntax is literals, ., *, +, ?, ^, $, classes and escaped punctuation.
A page record in pages.jsonl lists, in stamps_removed, the IDs of the rules that removed lines.
A rule that is too broad hides a real difference between two pages.
A stamp that no rule recognizes splits one edition into two (Chapter 7, section 7.2).
Replacing the stage defaults
A stage is one step of processing, such as conversion or indexing. Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them. The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.
A team copies scienting-stages-default.json, adjusts it with its counsel and points its project's settings at the copy.
The file itself calls its contents "proposed defaults from the design, for counsel to set".
In a project, the settings.stages field holds the id of the stage file in use (Chapter 20), so a team's copy needs an id of its own.
The project format is provisional.
Chapter 14 explains how stages are checked.