paperspec/registry

Registry

The registry holds lists that tools read at run time. Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program. This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.

File	Holds
formats.json	Every format: its identifier, media type (null for the six that have none), schema, status, chapter and a one-line description.
rules.json	Every validation rule ID, with what it means and the chapter that defines it.
stamps/scienting-stamps-1.json	The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them.
stages/scienting-stages-default.json	The rights each processing stage needs, as a default a team may replace.

Format status

Each entry in formats.json has a status. A candidate format is complete, with vectors and fixtures, and ready to implement. It changes only with a recorded reason and a new or changed vector. A provisional format is complete and expected to change as implementations gain experience, so it may change more freely. At version 0.1.0, 19 of the 32 formats are candidate and 13 are provisional. The third defined value, reserved, is unused. The values are defined in Chapter 0, section 0.4, and the stability rules in Chapter 22, section 22.5.

Registering the media types

The application/vnd.scienting.* media types are vendor-tree types, which IANA registers on request without a standards document (RFC 6838 §3.2). Register each type the formats serve over the web before the first release that other tools read: the zipped package, the anchor, the receipt and the layer first. Until then the types are unregistered and work as they are; a registration changes no byte of any file. The JSON-LD context is application/ld+json and a checkpoint is text/plain, both already registered.

Rule IDs

A validator checks a package and reports each failure as path:line: rule: message, for example:

nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute

A node_hash is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document. The line above is the failure that fixtures/invalid/hash-mismatch/ is built to cause: its sixth node record carries a node_hash of all zeros.

A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs. A tool can match on the rule field, whose meaning is looked up in rules.json, and treat the message as text for people. Chapter 22, section 22.4 lists the ID prefixes. Each directory in fixtures/invalid/ has an expect.json that names the rule it breaks. <<<<<<< HEAD The reference checker (paperspec check) fails if that rule is not in rules.json. ||||||| parent of 4fa326d (First implementation (#1))

The reference checker (tools/check.py) fails if that rule is not in rules.json.

4fa326d (First implementation (#1))

Adding stamp rules

Many publishers add a line to each PDF they serve, naming the institution and the time of download. This page calls that line a stamp. Two downloads of one paper then differ, so a hash of the file cannot match them. A page fingerprint is a hash of a page's size, its text and its images. It is computed after removing every line that a stamp rule matches (Chapter 7, section 7.1). The vectors in vectors/fingerprints.jsonl show this: acs-download-1 and acs-download-2 differ in the institution and the time on their stamp lines and have the same fingerprint.

Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes. Record a rule in a file of your own, marked as learned:

{"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
 "rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}

Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed (Chapter 1, section 1.4).

A rule has an id, a pattern, and a where that is any or margin. A margin rule matches only lines whose box lies wholly within margin points of one page edge (36 when the rule gives none). Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case. The allowed syntax is literals, ., *, +, ?, ^, $, classes and escaped punctuation.

A page record in pages.jsonl lists, in stamps_removed, the IDs of the rules that removed lines. A rule that is too broad hides a real difference between two pages. A stamp that no rule recognizes splits one edition into two (Chapter 7, section 7.2).

Replacing the stage defaults

A stage is one step of processing, such as conversion or indexing. Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them. The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.

A team copies scienting-stages-default.json, adjusts it with its counsel and points its project's settings at the copy. The file itself calls its contents "proposed defaults from the design, for counsel to set". In a project, the settings.stages field holds the id of the stage file in use (Chapter 20), so a team's copy needs an id of its own. The project format is provisional. Chapter 14 explains how stages are checked.