feat: rewrite docs

registry/README.md

@@ -1,36 +1,79 @@
 # Registry
 
-Lists that tools read at run time.
+The registry holds lists that tools read at run time.
+Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program.
+This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.
 
 | File | Holds |
 | --- | --- |
-| `formats.json` | Every format: its identifier, media type, schema, status and chapter. |
-| `rules.json` | Every validation rule ID, with what it means. |
-| `stamps/scienting-stamps-1.json` | The published rules that recognize publishers' download stamps, so page fingerprints ignore them. |
+| `formats.json` | Every format: its identifier, media type (null for four formats that have none), schema, status, chapter and a one-line description. |
+| `rules.json` | Every validation rule ID, with what it means and the chapter that defines it. |
+| `stamps/scienting-stamps-1.json` | The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them. |
 | `stages/scienting-stages-default.json` | The rights each processing stage needs, as a default a team may replace. |
 
+## Format status
+
+Each entry in `formats.json` has a `status`.
+A `candidate` format is complete, with vectors and fixtures, and ready to implement.
+It changes only with a recorded reason and a new or changed vector.
+A `provisional` format is complete and expected to change as implementations gain experience, so it may change more freely.
+At version 0.1.0, 20 of the 27 formats are `candidate` and seven are `provisional`.
+The third defined value, `reserved`, is unused.
+The values are defined in [Chapter 0, section 0.4](../spec/00-introduction.md#04-status), and the stability rules in [Chapter 22, section 22.5](../spec/22-conformance.md#225-stability).
+
 ## Rule IDs
 
-Validators report failures as `path:line: rule: message`, for example:
+A validator checks a package and reports each failure as `path:line: rule: message`, for example:
 
 ```text
-nodes.jsonl:7: doc.hash.node-hash: node_hash does not recompute
+nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute
 ```
 
+A `node_hash` is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document.
+The line above is the failure that `fixtures/invalid/hash-mismatch/` is built to cause: its sixth node record carries a `node_hash` of all zeros.
+
 A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs.
+A tool can match on the `rule` field, whose meaning is looked up in `rules.json`, and treat the message as text for people.
+[Chapter 22, section 22.4](../spec/22-conformance.md#224-rules) lists the ID prefixes.
+Each directory in `fixtures/invalid/` has an `expect.json` that names the rule it breaks.
+The reference checker (`tools/check.py`) fails if that rule is not in `rules.json`.
 
 ## Adding stamp rules
 
-When two downloads of one paper differ only in a stamp line that no published rule recognizes, record a rule in a file of your own, marked as learned:
+Many publishers add a line to each PDF they serve, naming the institution and the time of download.
+This page calls that line a stamp.
+Two downloads of one paper then differ, so a hash of the file cannot match them.
+A page fingerprint is a hash of a page's size, its text and its images.
+It is computed after removing every line that a stamp rule matches ([Chapter 7, section 7.1](../spec/07-geometry.md#71-page-fingerprints)).
+The vectors in `vectors/fingerprints.jsonl` show this: `acs-download-1` and `acs-download-2` differ in the institution and the time on their stamp lines and have the same fingerprint.
+
+Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes.
+Record a rule in a file of your own, marked as learned:
 
 ```json
 {"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
  "rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}
 ```
 
-Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with ASCII letters compared without case.
+Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed ([Chapter 1, section 1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
+
+A rule has an `id`, a `pattern`, and a `where` that is `any` or `margin`.
+A `margin` rule matches only lines whose box lies wholly within `margin` points of one page edge (36 when the rule gives none).
+Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case.
+The allowed syntax is literals, `.`, `*`, `+`, `?`, `^`, `$`, classes and escaped punctuation.
+
+A page record in `pages.jsonl` lists, in `stamps_removed`, the IDs of the rules that removed lines.
+A rule that is too broad hides a real difference between two pages.
+A stamp that no rule recognizes splits one edition into two ([Chapter 7, section 7.2](../spec/07-geometry.md#72-stamp-rules)).
 
 ## Replacing the stage defaults
 
-The stage requirements are configuration, not code: licenses define text and data mining differently, so a team copies `scienting-stages-default.json`, adjusts it with its counsel, and points its project's settings at the copy.
+A stage is one step of processing, such as conversion or indexing.
+Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them.
+The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.
+
+A team copies `scienting-stages-default.json`, adjusts it with its counsel and points its project's settings at the copy.
+The file itself calls its contents "proposed defaults from the design, for counsel to set".
+In a project, the `settings.stages` field names the stage file in use ([Chapter 20](../spec/20-projects.md)) [CHECK: schema says the value is a key; confirm it is the stage file's `id`, so a copy needs its own `id`].
+The project format is `provisional`.
 [Chapter 14](../spec/14-rights.md) explains how stages are checked.