0 Fork 0

Files

T

feat!: one paperspec command with subcommands, help and examples

The six console scripts (paperspec-check, -build, -validate, -load,
-stress, -site) become subcommands of a single `paperspec` command:
paperspec validate, load, check, build, stress and site. `python -m
paperspec COMMAND` runs the same thing.

- paperspec/cli/main.py builds the whole parser, apart from the commands,
  so `paperspec --help` and `paperspec COMMAND --help` work without a
  checkout or the tools extra; a command's module is imported only when
  it runs.
- `paperspec` with no command prints the overview: what each command does,
  examples, and which work anywhere (validate, load) and which need a
  checkout (check, build, stress, site). Each command's help has a
  description with advice, and worked examples; a misspelled command gets
  a suggestion (argparse suggest_on_error); `--version` prints the version
  and the pinned Unicode version.
- Each command module exposes run(args) and returns the exit status.
  Arguments that parse but do not fit together raise UsageError, which
  prints the command's usage and exits 2: load now also refuses
  --built-from or --store without --bundle.
- The failure messages say what to do: install paperspec[tools], clone the
  repository, rerun stress with the printed seed, or run paperspec build
  when the rebuild check differs. check and stress say how many failures
  they did not list.
- check runs the rebuild as `python -m paperspec build`.
- pyproject installs one script, `paperspec`; the pixi tasks and the docs
  use the subcommands.

pixi run check passes (run here as `python -m paperspec check` with src/ on
the path); the rebuild is byte-identical. pixi.lock still needs
`pixi lock`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-15 09:28:30 -04:00

5.8 KiB

Raw Blame History

Registry

The registry holds lists that tools read at run time. Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program. This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.

File	Holds
`formats.json`	Every format: its identifier, media type (null for the six that have none), schema, status, chapter and a one-line description.
`rules.json`	Every validation rule ID, with what it means and the chapter that defines it.
`stamps/scienting-stamps-1.json`	The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them.
`stages/scienting-stages-default.json`	The rights each processing stage needs, as a default a team may replace.

Format status

Each entry in formats.json has a status. A candidate format is complete, with vectors and fixtures, and ready to implement. It changes only with a recorded reason and a new or changed vector. A provisional format is complete and expected to change as implementations gain experience, so it may change more freely. At version 0.1.0, 19 of the 32 formats are candidate and 13 are provisional. The third defined value, reserved, is unused. The values are defined in Chapter 0, section 0.4, and the stability rules in Chapter 22, section 22.5.

Registering the media types

The application/vnd.scienting.* media types are vendor-tree types, which IANA registers on request without a standards document (RFC 6838 §3.2). Register each type the formats serve over the web before the first release that other tools read: the zipped package, the anchor, the receipt and the layer first. Until then the types are unregistered and work as they are; a registration changes no byte of any file. The JSON-LD context is application/ld+json and a checkpoint is text/plain, both already registered.

Rule IDs

A validator checks a package and reports each failure as path:line: rule: message, for example:

nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute

A node_hash is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document. The line above is the failure that fixtures/invalid/hash-mismatch/ is built to cause: its sixth node record carries a node_hash of all zeros.

A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs. A tool can match on the rule field, whose meaning is looked up in rules.json, and treat the message as text for people. Chapter 22, section 22.4 lists the ID prefixes. Each directory in fixtures/invalid/ has an expect.json that names the rule it breaks. The reference checker (paperspec check) fails if that rule is not in rules.json.

Adding stamp rules

Many publishers add a line to each PDF they serve, naming the institution and the time of download. This page calls that line a stamp. Two downloads of one paper then differ, so a hash of the file cannot match them. A page fingerprint is a hash of a page's size, its text and its images. It is computed after removing every line that a stamp rule matches (Chapter 7, section 7.1). The vectors in vectors/fingerprints.jsonl show this: acs-download-1 and acs-download-2 differ in the institution and the time on their stamp lines and have the same fingerprint.

Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes. Record a rule in a file of your own, marked as learned:

{"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
 "rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}

Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed (Chapter 1, section 1.4).

A rule has an id, a pattern, and a where that is any or margin. A margin rule matches only lines whose box lies wholly within margin points of one page edge (36 when the rule gives none). Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case. The allowed syntax is literals, ., *, +, ?, ^, $, classes and escaped punctuation.

A page record in pages.jsonl lists, in stamps_removed, the IDs of the rules that removed lines. A rule that is too broad hides a real difference between two pages. A stamp that no rule recognizes splits one edition into two (Chapter 7, section 7.2).

Replacing the stage defaults

A stage is one step of processing, such as conversion or indexing. Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them. The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.

A team copies scienting-stages-default.json, adjusts it with its counsel and points its project's settings at the copy. The file itself calls its contents "proposed defaults from the design, for counsel to set". In a project, the settings.stages field holds the id of the stage file in use (Chapter 20), so a team's copy needs an id of its own. The project format is provisional. Chapter 14 explains how stages are checked.

5.8 KiB Raw Blame History