feat: rewrite docs

vectors/README.md

@@ -1,8 +1,104 @@
 # Test vectors
 
 Inputs and the exact outputs every implementation must compute.
-Each file holds one case per line, in canonical JSON, with an `id`, its inputs, and its expected outputs, usually under `expect`.
-Run them in your own test suite; [docs/examples.md](../docs/examples.md#test-your-implementation-against-a-vector) shows the shape of a harness.
+
+This folder is for three readers.
+New to the problem? The first section says what a test vector is and why one exists.
+Writing a reader, writer, converter or checker? Read the procedure for running a file, then use the table to find the vectors and chapters for the rule you are implementing.
+Judging the specification? The last section says what the vectors establish and what they leave open.
+
+## What a test vector is
+
+Two programs can read a rule written in prose differently, and then the values they compute no longer match.
+These formats store hashes of text, so a mismatch is visible and costly.
+A hash is a short fixed-length fingerprint computed from data: the same data always gives the same hash, and in practice any change gives a different one.
+These formats use SHA-256, written as 64 lowercase hexadecimal digits ([1.3](../spec/01-conventions.md#13-hashes)).
+The same sentence gets two hashes when one program collapses runs of spaces before hashing and another does not, and the second program sees a change that never happened.
+
+A test vector removes the doubt.
+It is one case: an input and the exact output that the specification requires for it.
+An implementation passes a case when its own output for that input is identical to the expected one.
+The vectors are normative, like the chapters: where a chapter and a vector disagree, the disagreement is a bug ([spec/README.md](../spec/README.md)).
+
+One case from `normalize.jsonl`, which pins the text normalization of [1.4](../spec/01-conventions.md#14-text-normalization-for-hashing):
+
+```json
+{"id":"collapse","input":"a \t\n b","output":"a b","sha256":"c8687a08aa5d6ed2044328fa6a697ab8e96dc34291e8c2034ae8c38e6fcc6d65"}
+```
+
+The input is the letter a, a space, a tab, a newline, a space and the letter b.
+The rule collapses every run of whitespace to one space, so the output is `a b`, and `sha256` is the SHA-256 of that output.
+
+JSON is a text format for structured data, and a file of one JSON value per line is called JSON Lines.
+Each file holds one case per line, in canonical JSON, with an `id`, its inputs, and its expected outputs.
+Canonical JSON is the one agreed way of writing a JSON value, the JSON Canonicalization Scheme of [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785), a numbered Internet standard (RFC means Request for Comments), plus the repository's own rules ([1.1](../spec/01-conventions.md#11-canonical-json)): no whitespace between tokens, object members in a fixed order, and numbers written one way.
+The same data is always the same bytes when every file is written this way, so it hashes the same.
+
+A format's identifier, for example `scienting-id/1`, appears inside several cases.
+The `scienting-` prefix is the formats' namespace; it has no connection to the name of this repository.
+
+## How to run a vector file
+
+Run the vectors in your own test suite, against your own implementation.
+[docs/examples.md](../docs/examples.md#test-your-implementation-against-a-vector) shows the shape of a harness for `projections.jsonl`.
+The procedure is the same for every file:
+
+1.
+   Read the file as UTF-8 (the standard Unicode text encoding) and split it at U+000A only.
+   `normalize.jsonl` contains a raw U+2028 (line separator) inside one case, so a reader that splits at every Unicode line break, such as Python's `str.splitlines()`, cuts that case in two.
+   Python's `for line in open(path, encoding="utf-8")` splits correctly.
+2.
+   Parse each line as JSON.
+3.
+   Find the inputs and the expected outputs of the case.
+   The table below says which fields are which, and the chapter it links defines the rule.
+   Only five files, `alignment`, `corrections`, `grading`, `resolve` and `rights`, put the expected outputs under `expect`.
+   In the other fifteen the expected outputs are fields beside the inputs, such as `output`, `sha256`, `canonical` or `parsed`.
+4.
+   Compute each output from the inputs with your implementation.
+5.
+   Compare each computed output with the expected one.
+   Strings must be identical.
+   Objects and arrays are compared as values, which is how the reference checker compares them.
+6.
+   Report every difference with the case's `id`.
+
+Some files change the procedure:
+
+- `canonical-json.jsonl` gives JSON text in `input_json`.
+  Parse its numbers as IEEE 754 doubles (the 64-bit floating-point numbers most languages use), as a JavaScript engine does.
+  The `numbers` case contains `123456789012345678901`, which a parser that keeps exact integers reads differently; the expected output is `123456789012345680000`.
+- `projection-groups.jsonl` has no inputs of its own.
+  Each of its cases lists ids from `projections.jsonl` and names a field in which those cases must be equal (`same`) or unequal (`differ`).
+  Compute the field for each listed case and compare across the group.
+- `fingerprints.jsonl` needs the download-stamp rules in [registry/stamps/scienting-stamps-1.json](../registry/stamps/scienting-stamps-1.json), which each case names in `rules`.
+- `resolve.jsonl` and `grading.jsonl` name a package by a path from the repository root.
+  Both files use `fixtures/packages/hale2026harmonic`.
+
+Several files also mix more than one kind of case, told apart by their fields.
+`ids.jsonl` has minting cases and letter-first cases, `keyed.jsonl` and `signatures.jsonl` have a case for each derivation or record, `spotcheck.jsonl` has sample-size cases and one full sample, and `rights.jsonl` has stage checks (a `stage` field), backend checks (a `backend` field) and one combining case (neither).
+
+The reference checker in [tools/](../tools/README.md) runs all twenty files, together with the rest of the repository's checks:
+
+```sh
+python3 tools/check.py --quick
+```
+
+It needs Python 3.10 or later and has no option to run one file.
+It tests the reference implementation in `tools/oracle/` against the vectors.
+It does not test yours.
+
+## What each file tests
+
+The table below uses a few terms that the chapters define:
+
+- A code point is one Unicode character; UTF-16 is an encoding that writes some characters as two units, so a JavaScript string offset can differ from a code-point offset ([1.5](../spec/01-conventions.md#15-character-offsets)).
+- An HMAC is a hash computed under a secret key, so that someone without the key cannot recompute it ([5.8](../spec/05-hashes.md#58-keyed-hashes)).
+- A Merkle root is one hash that commits to a list of items by hashing them in a binary tree ([8.5](../spec/08-fidelity.md#85-merkle-roots)).
+- A page fingerprint is a hash of a page's text and images with download stamps removed, so two differently stamped downloads of one paper share it ([7.1](../spec/07-geometry.md#71-page-fingerprints)).
+- The strict and lossy projections are two canonical forms of a sentence's text, one to certify that two readings agree and one to propose that they might ([5.3](../spec/05-hashes.md#53-the-strict-projection)).
+
+[docs/glossary.md](../docs/glossary.md) defines the rest.
 
 | File | Tests | Chapter |
 | --- | --- | --- |
@@ -27,4 +123,42 @@ Run them in your own test suite; [docs/examples.md](../docs/examples.md#test-you
 | `rights.jsonl` | Stage checks with policies and expiry, the combining rule, and backend conditions. | [14](../spec/14-rights.md) |
 | `grading.jsonl` | Exercises, submissions and the grades they earn. | [19.4](../spec/19-exercises.md#194-grading) |
 
-Vectors that refer to a package (`resolve.jsonl` and `grading.jsonl`) name it by path; resolve against the files in that package, not against anything else.
+Resolve those paths against the files in that package only.
+
+## What the vectors prove, and what they do not
+
+The specification, and the vectors with it, are at version 0.1.0 ([0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
+The specification has 27 formats: 20 are `candidate` and seven are `provisional` in [registry/formats.json](../registry/formats.json).
+A candidate format changes only with a recorded reason and a new or changed vector ([22.5](../spec/22-conformance.md#225-stability)).
+
+A passing run shows one thing: for these inputs, your implementation computes the same values as the reference implementation in `tools/oracle/`.
+That code computed the expected outputs, and `pixi run build` regenerates them ([tools/](../tools/README.md)).
+The vectors hold 203 cases in all, some of them small: `hashes.jsonl` has one document, `alignment.jsonl` one pair of readings, `offsets.jsonl` and `signatures.jsonl` two cases each.
+
+The vectors do not show the following.
+
+- Conformance: the vectors are one part of the suite in [chapter 22](../spec/22-conformance.md#221-the-suite).
+  A conforming implementation must also accept every valid package in `fixtures/packages/`, and refuse every package in `fixtures/invalid/`, naming the rule it breaks if it can.
+  No vector tests package validation.
+- Independent agreement: the only program in this repository that runs the vectors is the reference checker, and it uses the same oracle that produced them.
+  A rule that the oracle and the chapter both misstate would pass.
+  A separately written implementation that passes is evidence the repository cannot supply itself, and a disagreement between a chapter and a vector is a bug to report.
+- Rules the vectors skip:
+  - `canonical-json.jsonl` has five cases, none for the repository's rule that a lone surrogate, `NaN` or an infinity has no canonical form.
+  - `merkle.jsonl` has roots for 1, 2, 3, 4, 5 and 7 items.
+    A receipt with no items is invalid ([8.5](../spec/08-fidelity.md#85-merkle-roots)), so no case has an empty list, although the RFCs it follows define a hash for an empty list ([RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162)).
+  - Seven formats are `provisional`, and no case carries their identifiers.
+    Their chapters define values that no vector pins, such as a dataset's content hash ([15](../spec/15-datasets.md)) and the hash that confirms a digitization ([16](../spec/16-digitization.md)).
+    They are expected to change as implementations gain experience ([0.4](../spec/00-introduction.md#04-status)).
+- Reading a PDF: how text lines are extracted from a PDF is outside the specification; `fingerprints.jsonl` supplies the lines, so it tests the fingerprint function only ([7.1](../spec/07-geometry.md#71-page-fingerprints)).
+- Faithful conversion: the vectors check what is computed from text, not whether a converter read the paper correctly.
+  The formats record such an error and make it checkable through corrections and receipts ([chapter 8](../spec/08-fidelity.md), [chapter 11](../spec/11-corrections.md)); the vectors do not detect it.
+- Security: `signatures.jsonl` contains the RFC 8032 test vector and one receipt, so it checks that your Ed25519 and record hashing give the stated bytes.
+  The secrets in `keyed.jsonl` and `signatures.jsonl` are test values published in this repository; do not use them for anything real.
+
+A vector that rests on a building block defined elsewhere pins this specification's use of it.
+Canonical JSON is RFC 8785 with added rules.
+Receipts use a Merkle tree that follows RFC 6962 ([8.5](../spec/08-fidelity.md#85-merkle-roots)); RFC 9162 replaces RFC 6962 and defines the same tree.
+The `node_hash` of a node is a different construction: a hash of the canonical JSON of `{own, children}`.
+Here `own` is the node's own hash and `children` holds the `node_hash` of each child ([5.1](../spec/05-hashes.md#51-the-five-values)), and `hashes.jsonl` pins it.
+The comparison page covers how these formats differ from JATS (Journal Article Tag Suite), TEI (Text Encoding Initiative), W3C (World Wide Web Consortium) Web Annotation and the other neighbors [CHECK: docs/comparison.md does not exist in the reference repository; link it once it does].