First implementation (#1)

Reviewed-on: #1

vectors/README.md

@@ -0,0 +1,169 @@
+# Test vectors
+
+Inputs and the exact outputs every implementation must compute.
+
+This folder is for three readers.
+New to the problem? The first section says what a test vector is and why one exists.
+Writing a reader, writer, converter or checker? Read the procedure for running a file, then use the table to find the vectors and chapters for the rule you are implementing.
+Judging the specification? The last section says what the vectors establish and what they leave open.
+
+## What a test vector is
+
+Two programs can read a rule written in prose differently, and then the values they compute no longer match.
+These formats store hashes of text, so a mismatch is visible and costly.
+A hash is a short fixed-length fingerprint computed from data: the same data always gives the same hash, and in practice any change gives a different one.
+These formats use SHA-256, written as 64 lowercase hexadecimal digits ([1.3](../spec/01-conventions.md#13-hashes)).
+The same sentence gets two hashes when one program collapses runs of spaces before hashing and another does not, and the second program sees a change that never happened.
+
+A test vector removes the doubt.
+It is one case: an input and the exact output that the specification requires for it.
+An implementation passes a case when its own output for that input is identical to the expected one.
+The vectors are normative, like the chapters: where a chapter and a vector disagree, the disagreement is a bug ([spec/README.md](../spec/README.md)).
+
+One case from `normalize.jsonl`, which pins the text normalization of [1.4](../spec/01-conventions.md#14-text-normalization-for-hashing):
+
+```json
+{"id":"collapse","input":"a \t\n b","output":"a b","sha256":"c8687a08aa5d6ed2044328fa6a697ab8e96dc34291e8c2034ae8c38e6fcc6d65"}
+```
+
+The input is the letter a, a space, a tab, a newline, a space and the letter b.
+The rule collapses every run of whitespace to one space, so the output is `a b`, and `sha256` is the SHA-256 of that output.
+
+JSON is a text format for structured data, and a file of one JSON value per line is called JSON Lines.
+Each file holds one case per line, in canonical JSON, with an `id`, its inputs, and its expected outputs.
+Canonical JSON is the one agreed way of writing a JSON value, the JSON Canonicalization Scheme of [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785), a numbered Internet standard (RFC means Request for Comments), plus the repository's own rules ([1.1](../spec/01-conventions.md#11-canonical-json)): no whitespace between tokens, object members in a fixed order, and numbers written one way.
+The same data is always the same bytes when every file is written this way, so it hashes the same.
+
+A format's identifier, for example `scienting-id/1`, appears inside several cases.
+The `scienting-` prefix is the formats' namespace; it has no connection to the name of this repository.
+
+## How to run a vector file
+
+Run the vectors in your own test suite, against your own implementation.
+[docs/examples.md](../docs/examples.md#test-your-implementation-against-a-vector) shows the shape of a harness for `projections.jsonl`.
+The procedure is the same for every file:
+
+1.
+   Read the file as UTF-8 (the standard Unicode text encoding) and split it at U+000A only.
+   `normalize.jsonl` contains a raw U+2028 (line separator) inside one case, so a reader that splits at every Unicode line break, such as Python's `str.splitlines()`, cuts that case in two.
+   Python's `for line in open(path, encoding="utf-8")` splits correctly.
+2.
+   Parse each line as JSON.
+3.
+   Find the inputs and the expected outputs of the case.
+   The table below says which fields are which, and the chapter it links defines the rule.
+   Only six files, `alignment`, `corrections`, `grading`, `resolve`, `rights` and `targets`, put the expected outputs under `expect`.
+   In the other nineteen the expected outputs are fields beside the inputs, such as `output`, `sha256`, `canonical` or `parsed`.
+4.
+   Compute each output from the inputs with your implementation.
+5.
+   Compare each computed output with the expected one.
+   Strings must be identical.
+   Objects and arrays are compared as values, which is how the reference checker compares them.
+6.
+   Report every difference with the case's `id`.
+
+Some files change the procedure:
+
+- `canonical-json.jsonl` gives JSON text in `input_json`.
+  Parse its numbers as IEEE 754 doubles (the 64-bit floating-point numbers most languages use), as a JavaScript engine does.
+  The `numbers` case contains `123456789012345678901`, which a parser that keeps exact integers reads differently; the expected output is `123456789012345680000`.
+- `projection-groups.jsonl` has no inputs of its own.
+  Each of its cases lists ids from `projections.jsonl` and names a field in which those cases must be equal (`same`) or unequal (`differ`).
+  Compute the field for each listed case and compare across the group.
+- `fingerprints.jsonl` needs the download-stamp rules in [registry/stamps/scienting-stamps-1.json](../registry/stamps/scienting-stamps-1.json), which each case names in `rules`.
+- `resolve.jsonl`, `grading.jsonl` and `targets.jsonl` name a package by a path from the repository root.
+  All three use `fixtures/packages/hale2026harmonic`.
+
+Several files also mix more than one kind of case, told apart by their fields.
+`ids.jsonl` has minting cases and letter-first cases, `keyed.jsonl` and `signatures.jsonl` have a case for each derivation or record, `spotcheck.jsonl` has sample-size cases and one full sample, and `rights.jsonl` has stage checks (a `stage` field), backend checks (a `backend` field) and one combining case (neither).
+
+The reference checker in [tools/](../tools/README.md) runs all twenty-five files, together with the rest of the repository's checks:
+
+```sh
+python3 tools/check.py --quick
+```
+
+It needs Python 3.14 and two packages, `jsonschema` and `cryptography`, and has no option to run one file.
+It tests the reference implementation in `tools/oracle/` against the vectors.
+It does not test yours.
+
+## What each file tests
+
+The table below uses a few terms that the chapters define:
+
+- A code point is one Unicode character; UTF-16 is an encoding that writes some characters as two units, so a JavaScript string offset can differ from a code-point offset ([1.5](../spec/01-conventions.md#15-character-offsets)).
+- An HMAC is a hash computed under a secret key, so that someone without the key cannot recompute it ([5.8](../spec/05-hashes.md#58-keyed-hashes)).
+- A Merkle root is one hash that commits to a list of items by hashing them in a binary tree ([8.5](../spec/08-fidelity.md#85-merkle-roots)).
+- A page fingerprint is a hash of a page's text and images with download stamps removed, so two differently stamped downloads of one paper share it ([7.1](../spec/07-geometry.md#71-page-fingerprints)).
+- The strict and lossy projections are two canonical forms of a sentence's text, one to certify that two readings agree and one to propose that they might ([5.3](../spec/05-hashes.md#53-the-strict-projection)).
+
+[docs/glossary.md](../docs/glossary.md) defines the rest.
+
+| File | Tests | Chapter |
+| --- | --- | --- |
+| `canonical-json.jsonl` | `input_json`, parsed with every number as an IEEE 754 double, and its canonical `output`. | [1.1](../spec/01-conventions.md#11-canonical-json) |
+| `normalize.jsonl` | Normalization for hashing, and the SHA-256 of the result. | [1.4](../spec/01-conventions.md#14-text-normalization-for-hashing) |
+| `ids.jsonl` | Minting payloads and node IDs, and letter-first encodings of digests. | [4](../spec/04-identity.md) |
+| `notation.jsonl` | Each math source's canonical linear form, or null when it is complex. | [5.4](../spec/05-hashes.md#54-simple-notation) |
+| `projections.jsonl` | One-sentence runs: plain, normalized, strict and lossy projections, complex keys and the three text values. | [5.3](../spec/05-hashes.md#53-the-strict-projection) |
+| `projection-groups.jsonl` | Which cases must share, or must differ in, a value: H₂O in six spellings, cm⁻¹, 10⁻³, K_r, ligatures, the micro sign, "103" against 10³. | [5.3](../spec/05-hashes.md#53-the-strict-projection) |
+| `offsets.jsonl` | Code-point spans and their UTF-16 equivalents. | [1.5](../spec/01-conventions.md#15-character-offsets) |
+| `hashes.jsonl` | Every node of a small document, its own content and its five values. | [5.1](../spec/05-hashes.md#51-the-five-values) |
+| `addresses.jsonl` | How addresses parse; null for invalid ones. | [6.1](../spec/06-anchors.md#61-addresses) |
+| `keyed.jsonl` | The team key and its ID, an HMAC, page hashes, a page-derived key, and what someone without the paper cannot check. | [5.8](../spec/05-hashes.md#58-keyed-hashes) |
+| `fingerprints.jsonl` | Pages as text lines, the stamp rules that fired, and the fingerprint; two downloads that must agree, and a change that must not. | [7.1](../spec/07-geometry.md#71-page-fingerprints) |
+| `merkle.jsonl` | Receipt items and their Merkle roots. | [8.5](../spec/08-fidelity.md#85-merkle-roots) |
+| `signatures.jsonl` | The RFC 8032 test vector; a receipt's record hash, signing payload and signature; `did:key` identifiers for an Ed25519 and an ML-DSA-65 key; and ML-DSA-65 (FIPS 204): NIST's key-generation case and a deterministic signature. | [8.4](../spec/08-fidelity.md#84-record-hashes-and-signatures) |
+| `spotcheck.jsonl` | Seed payloads, seeds, sample sizes and samples. | [8.8](../spec/08-fidelity.md#88-spot-checks) |
+| `bounds.jsonl` | Reviewer error bounds as reported. | [8.9](../spec/08-fidelity.md#89-reviewer-error-bounds) |
+| `alignment.jsonl` | Two readings of a document, their pairs and their suspects. | [5.6](../spec/05-hashes.md#56-alignment) |
+| `corrections.jsonl` | Each operation and refusal on a small document: the state before, the correction, and the record and state after, or the rule broken. | [11](../spec/11-corrections.md) |
+| `resolve.jsonl` | Anchors made on the example paper's first reading, and how each resolves after its corrections. | [6.5](../spec/06-anchors.md#65-resolution) |
+| `rights.jsonl` | Stage checks with policies and expiry, the combining rule, and backend conditions. | [14](../spec/14-rights.md) |
+| `grading.jsonl` | Exercises, submissions and the grades they earn. | [19.4](../spec/19-exercises.md#194-grading) |
+| `targets.jsonl` | The target set of targeted F3 for the example paper under each digit rule, given the anchored nodes. | [8.10](../spec/08-fidelity.md#810-targeted-f3) |
+| `keys.jsonl` | Work keys: Scholar parts, identities, hash suffixes and document keys, with accented names, missing years and a collision. | [2.4](../spec/02-model.md#24-work-and-document-keys) |
+| `proofs.jsonl` | RFC 9162 inclusion paths for every item of small trees, consistency proofs for every prefix, the empty root, and the example paper's history. | [8.5](../spec/08-fidelity.md#85-merkle-roots), [11.5](../spec/11-corrections.md#115-the-history-root) |
+| `vrf.jsonl` | The VRF of spot-check seeds: RFC 9381's Example 16, two more inputs for its key, the example paper's seed, and proofs that must not verify (`beta` null). Bytes are hex. | [8.8](../spec/08-fidelity.md#88-spot-checks) |
+| `checkpoint.jsonl` | A transparency-log checkpoint: the entries, their root, the signed note and its signers. | [23](../spec/23-log.md) |
+
+Resolve those paths against the files in that package only.
+
+## What the vectors prove, and what they do not
+
+The specification, and the vectors with it, are at version 0.1.0 ([0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
+The specification has 32 formats: 19 are `candidate` and 13 are `provisional` in [registry/formats.json](../registry/formats.json).
+A candidate format changes only with a recorded reason and a new or changed vector ([22.5](../spec/22-conformance.md#225-stability)).
+
+A passing run shows one thing: for these inputs, your implementation computes the same values as the reference implementation in `tools/oracle/`.
+That code computed the expected outputs, and `pixi run build` regenerates them ([tools/](../tools/README.md)).
+The vectors hold 311 cases in all, some of them small: `hashes.jsonl` has one document, `alignment.jsonl` one pair of readings, `offsets.jsonl` two cases.
+
+The vectors do not show the following.
+
+- Conformance: the vectors are one part of the suite in [chapter 22](../spec/22-conformance.md#221-the-suite).
+  A conforming implementation must also accept every valid package in `fixtures/packages/`, and refuse every package in `fixtures/invalid/`, naming the rule it breaks if it can.
+  No vector tests package validation.
+- Independent agreement: the only program in this repository that runs the vectors is the reference checker, and it uses the same oracle that produced them.
+  A rule that the oracle and the chapter both misstate would pass.
+  A separately written implementation that passes is evidence the repository cannot supply itself, and a disagreement between a chapter and a vector is a bug to report.
+- Rules the vectors skip:
+  - `canonical-json.jsonl` has five cases, none for the repository's rule that a lone surrogate, `NaN` or an infinity has no canonical form.
+  - `merkle.jsonl` has roots for 1, 2, 3, 4, 5 and 7 items.
+    A receipt with no items is invalid ([8.5](../spec/08-fidelity.md#85-merkle-roots)), so no case has an empty list, although the RFCs it follows define a hash for an empty list ([RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162)).
+  - Thirteen formats are `provisional`, and only `grading.jsonl` carries any of their identifiers (exercise, submission and grade).
+    The chapters of the others define values that no vector pins, such as a dataset's content hash ([15](../spec/15-datasets.md)), the hash that confirms a digitization ([16](../spec/16-digitization.md)) and a checklist's content hash ([8.11](../spec/08-fidelity.md#811-checklists)).
+    They are expected to change as implementations gain experience ([0.4](../spec/00-introduction.md#04-status)).
+- Reading a PDF: how text lines are extracted from a PDF is outside the specification; `fingerprints.jsonl` supplies the lines, so it tests the fingerprint function only ([7.1](../spec/07-geometry.md#71-page-fingerprints)).
+- Faithful conversion: the vectors check what is computed from text, not whether a converter read the paper correctly.
+  The formats record such an error and make it checkable through corrections and receipts ([chapter 8](../spec/08-fidelity.md), [chapter 11](../spec/11-corrections.md)); the vectors do not detect it.
+- Security: `signatures.jsonl` contains the RFC 8032 test vector and one receipt, so it checks that your Ed25519 and record hashing give the stated bytes.
+  The secrets in `keyed.jsonl` and `signatures.jsonl` are test values published in this repository; do not use them for anything real.
+
+A vector that rests on a building block defined elsewhere pins this specification's use of it.
+Canonical JSON is RFC 8785 with added rules.
+Receipts use a Merkle tree that follows RFC 6962 ([8.5](../spec/08-fidelity.md#85-merkle-roots)); RFC 9162 replaces RFC 6962 and defines the same tree.
+The `node_hash` of a node is a different construction: a hash of the canonical JSON of `{own, children}`.
+Here `own` is the node's own hash and `children` holds the `node_hash` of each child ([5.1](../spec/05-hashes.md#51-the-five-values)), and `hashes.jsonl` pins it.
+The [comparison page](../docs/comparison.md) covers how these formats differ from JATS (Journal Article Tag Suite), TEI (Text Encoding Initiative), W3C (World Wide Web Consortium) Web Annotation and the other neighbors.