infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3a5f28d010383a3b3aaf3a265ebfd168c0e20002
paperspec/vectors/README.md
T
alexmandClaude 607062056f feat!: pin Python 3.14 and Unicode 16.0, and make the tools commands 
Python and Unicode
- requires-python is >=3.14,<3.15: 3.14 is the only Python with the
  Unicode 16.0 data the specification pins (3.13 has 15.1, 3.15 has 17.0).
- `import paperspec` raises ImportError unless unicodedata reports
  paperspec.UNICODE_VERSION (16.0.0), so a source checkout or an install
  that bypasses requires-python cannot compute values on other data.

Commands
- tools/{check,build,validate,load,stress,site}.py move with history to
  src/paperspec/cli/ and install as paperspec-check, -build, -validate,
  -load, -stress and -site; each also runs as `python -m paperspec.cli.X`.
- paperspec.cli.require() exits with "install paperspec[tools]" when
  cryptography or jsonschema is missing; paperspec.cli.checkout() finds
  the repository the check, build, stress and site commands work on (the
  editable install's own checkout, else the one around the current
  folder) and exits with a message outside one. tools/__init__.py is gone.
- validate and load read the schemas and SQL through the new
  paperspec.resources.path(), which finds them in a wheel and in an
  editable install alike, so they work anywhere the package is installed.
- check runs the rebuild as `python -m paperspec.cli.build`.
- build still writes the provenance tool label "paperspec tools/build.py":
  it is part of every hashed and signed fixture.
- pixi tasks and the check workflow run the commands; tools/README.md
  documents them, and the other docs name the commands instead of paths.

License
- LICENSE.md gives `src/` Apache-2.0 alongside `tools/`. Since the
  previous commit moved the code to src/, no row had covered it.

pixi run check passes (run as `python -m paperspec.cli.check` with src/ on
the path, since this environment cannot install the entry points); the
byte-for-byte rebuild matches, so no fixture or vector changed.
pixi.lock still needs `pixi lock`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 09:17:27 -04:00

170 lines
15 KiB
Markdown
Raw Blame History

# Test vectors
Inputs and the exact outputs every implementation must compute.
This folder is for three readers.
New to the problem? The first section says what a test vector is and why one exists.
Writing a reader, writer, converter or checker? Read the procedure for running a file, then use the table to find the vectors and chapters for the rule you are implementing.
Judging the specification? The last section says what the vectors establish and what they leave open.
## What a test vector is
Two programs can read a rule written in prose differently, and then the values they compute no longer match.
These formats store hashes of text, so a mismatch is visible and costly.
A hash is a short fixed-length fingerprint computed from data: the same data always gives the same hash, and in practice any change gives a different one.
These formats use SHA-256, written as 64 lowercase hexadecimal digits ([1.3](../spec/01-conventions.md#13-hashes)).
The same sentence gets two hashes when one program collapses runs of spaces before hashing and another does not, and the second program sees a change that never happened.
A test vector removes the doubt.
It is one case: an input and the exact output that the specification requires for it.
An implementation passes a case when its own output for that input is identical to the expected one.
The vectors are normative, like the chapters: where a chapter and a vector disagree, the disagreement is a bug ([spec/README.md](../spec/README.md)).
One case from `normalize.jsonl`, which pins the text normalization of [1.4](../spec/01-conventions.md#14-text-normalization-for-hashing):
```json
{"id":"collapse","input":"a \t\n b","output":"a b","sha256":"c8687a08aa5d6ed2044328fa6a697ab8e96dc34291e8c2034ae8c38e6fcc6d65"}
```
The input is the letter a, a space, a tab, a newline, a space and the letter b.
The rule collapses every run of whitespace to one space, so the output is `a b`, and `sha256` is the SHA-256 of that output.
JSON is a text format for structured data, and a file of one JSON value per line is called JSON Lines.
Each file holds one case per line, in canonical JSON, with an `id`, its inputs, and its expected outputs.
Canonical JSON is the one agreed way of writing a JSON value, the JSON Canonicalization Scheme of [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785), a numbered Internet standard (RFC means Request for Comments), plus the repository's own rules ([1.1](../spec/01-conventions.md#11-canonical-json)): no whitespace between tokens, object members in a fixed order, and numbers written one way.
The same data is always the same bytes when every file is written this way, so it hashes the same.
A format's identifier, for example `scienting-id/1`, appears inside several cases.
The `scienting-` prefix is the formats' namespace; it has no connection to the name of this repository.
## How to run a vector file
Run the vectors in your own test suite, against your own implementation.
[docs/examples.md](../docs/examples.md#test-your-implementation-against-a-vector) shows the shape of a harness for `projections.jsonl`.
The procedure is the same for every file:
1.
Read the file as UTF-8 (the standard Unicode text encoding) and split it at U+000A only.
`normalize.jsonl` contains a raw U+2028 (line separator) inside one case, so a reader that splits at every Unicode line break, such as Python's `str.splitlines()`, cuts that case in two.
Python's `for line in open(path, encoding="utf-8")` splits correctly.
2.
Parse each line as JSON.
3.
Find the inputs and the expected outputs of the case.
The table below says which fields are which, and the chapter it links defines the rule.
Only six files, `alignment`, `corrections`, `grading`, `resolve`, `rights` and `targets`, put the expected outputs under `expect`.
In the other nineteen the expected outputs are fields beside the inputs, such as `output`, `sha256`, `canonical` or `parsed`.
4.
Compute each output from the inputs with your implementation.
5.
Compare each computed output with the expected one.
Strings must be identical.
Objects and arrays are compared as values, which is how the reference checker compares them.
6.
Report every difference with the case's `id`.
Some files change the procedure:
- `canonical-json.jsonl` gives JSON text in `input_json`.
Parse its numbers as IEEE 754 doubles (the 64-bit floating-point numbers most languages use), as a JavaScript engine does.
The `numbers` case contains `123456789012345678901`, which a parser that keeps exact integers reads differently; the expected output is `123456789012345680000`.
- `projection-groups.jsonl` has no inputs of its own.
Each of its cases lists ids from `projections.jsonl` and names a field in which those cases must be equal (`same`) or unequal (`differ`).
Compute the field for each listed case and compare across the group.
- `fingerprints.jsonl` needs the download-stamp rules in [registry/stamps/scienting-stamps-1.json](../registry/stamps/scienting-stamps-1.json), which each case names in `rules`.
- `resolve.jsonl`, `grading.jsonl` and `targets.jsonl` name a package by a path from the repository root.
All three use `fixtures/packages/hale2026harmonic`.
Several files also mix more than one kind of case, told apart by their fields.
`ids.jsonl` has minting cases and letter-first cases, `keyed.jsonl` and `signatures.jsonl` have a case for each derivation or record, `spotcheck.jsonl` has sample-size cases and one full sample, and `rights.jsonl` has stage checks (a `stage` field), backend checks (a `backend` field) and one combining case (neither).
The reference checker in [tools/](../tools/README.md) runs all twenty-five files, together with the rest of the repository's checks:
```sh
paperspec-check --quick
```
It needs Python 3.14 and two packages, `jsonschema` and `cryptography`, and has no option to run one file.
It tests the reference implementation in `src/paperspec/` against the vectors.
It does not test yours.
## What each file tests
The table below uses a few terms that the chapters define:
- A code point is one Unicode character; UTF-16 is an encoding that writes some characters as two units, so a JavaScript string offset can differ from a code-point offset ([1.5](../spec/01-conventions.md#15-character-offsets)).
- An HMAC is a hash computed under a secret key, so that someone without the key cannot recompute it ([5.8](../spec/05-hashes.md#58-keyed-hashes)).
- A Merkle root is one hash that commits to a list of items by hashing them in a binary tree ([8.5](../spec/08-fidelity.md#85-merkle-roots)).
- A page fingerprint is a hash of a page's text and images with download stamps removed, so two differently stamped downloads of one paper share it ([7.1](../spec/07-geometry.md#71-page-fingerprints)).
- The strict and lossy projections are two canonical forms of a sentence's text, one to certify that two readings agree and one to propose that they might ([5.3](../spec/05-hashes.md#53-the-strict-projection)).
[docs/glossary.md](../docs/glossary.md) defines the rest.
| File | Tests | Chapter |
| --- | --- | --- |
| `canonical-json.jsonl` | `input_json`, parsed with every number as an IEEE 754 double, and its canonical `output`. | [1.1](../spec/01-conventions.md#11-canonical-json) |
| `normalize.jsonl` | Normalization for hashing, and the SHA-256 of the result. | [1.4](../spec/01-conventions.md#14-text-normalization-for-hashing) |
| `ids.jsonl` | Minting payloads and node IDs, and letter-first encodings of digests. | [4](../spec/04-identity.md) |
| `notation.jsonl` | Each math source's canonical linear form, or null when it is complex. | [5.4](../spec/05-hashes.md#54-simple-notation) |
| `projections.jsonl` | One-sentence runs: plain, normalized, strict and lossy projections, complex keys and the three text values. | [5.3](../spec/05-hashes.md#53-the-strict-projection) |
| `projection-groups.jsonl` | Which cases must share, or must differ in, a value: H₂O in six spellings, cm⁻¹, 10⁻³, K_r, ligatures, the micro sign, "103" against 10³. | [5.3](../spec/05-hashes.md#53-the-strict-projection) |
| `offsets.jsonl` | Code-point spans and their UTF-16 equivalents. | [1.5](../spec/01-conventions.md#15-character-offsets) |
| `hashes.jsonl` | Every node of a small document, its own content and its five values. | [5.1](../spec/05-hashes.md#51-the-five-values) |
| `addresses.jsonl` | How addresses parse; null for invalid ones. | [6.1](../spec/06-anchors.md#61-addresses) |
| `keyed.jsonl` | The team key and its ID, an HMAC, page hashes, a page-derived key, and what someone without the paper cannot check. | [5.8](../spec/05-hashes.md#58-keyed-hashes) |
| `fingerprints.jsonl` | Pages as text lines, the stamp rules that fired, and the fingerprint; two downloads that must agree, and a change that must not. | [7.1](../spec/07-geometry.md#71-page-fingerprints) |
| `merkle.jsonl` | Receipt items and their Merkle roots. | [8.5](../spec/08-fidelity.md#85-merkle-roots) |
| `signatures.jsonl` | The RFC 8032 test vector; a receipt's record hash, signing payload and signature; `did:key` identifiers for an Ed25519 and an ML-DSA-65 key; and ML-DSA-65 (FIPS 204): NIST's key-generation case and a deterministic signature. | [8.4](../spec/08-fidelity.md#84-record-hashes-and-signatures) |
| `spotcheck.jsonl` | Seed payloads, seeds, sample sizes and samples. | [8.8](../spec/08-fidelity.md#88-spot-checks) |
| `bounds.jsonl` | Reviewer error bounds as reported. | [8.9](../spec/08-fidelity.md#89-reviewer-error-bounds) |
| `alignment.jsonl` | Two readings of a document, their pairs and their suspects. | [5.6](../spec/05-hashes.md#56-alignment) |
| `corrections.jsonl` | Each operation and refusal on a small document: the state before, the correction, and the record and state after, or the rule broken. | [11](../spec/11-corrections.md) |
| `resolve.jsonl` | Anchors made on the example paper's first reading, and how each resolves after its corrections. | [6.5](../spec/06-anchors.md#65-resolution) |
| `rights.jsonl` | Stage checks with policies and expiry, the combining rule, and backend conditions. | [14](../spec/14-rights.md) |
| `grading.jsonl` | Exercises, submissions and the grades they earn. | [19.4](../spec/19-exercises.md#194-grading) |
| `targets.jsonl` | The target set of targeted F3 for the example paper under each digit rule, given the anchored nodes. | [8.10](../spec/08-fidelity.md#810-targeted-f3) |
| `keys.jsonl` | Work keys: Scholar parts, identities, hash suffixes and document keys, with accented names, missing years and a collision. | [2.4](../spec/02-model.md#24-work-and-document-keys) |
| `proofs.jsonl` | RFC 9162 inclusion paths for every item of small trees, consistency proofs for every prefix, the empty root, and the example paper's history. | [8.5](../spec/08-fidelity.md#85-merkle-roots), [11.5](../spec/11-corrections.md#115-the-history-root) |
| `vrf.jsonl` | The VRF of spot-check seeds: RFC 9381's Example 16, two more inputs for its key, the example paper's seed, and proofs that must not verify (`beta` null). Bytes are hex. | [8.8](../spec/08-fidelity.md#88-spot-checks) |
| `checkpoint.jsonl` | A transparency-log checkpoint: the entries, their root, the signed note and its signers. | [23](../spec/23-log.md) |
Resolve those paths against the files in that package only.
## What the vectors prove, and what they do not
The specification, and the vectors with it, are at version 0.1.0 ([0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
The specification has 32 formats: 19 are `candidate` and 13 are `provisional` in [registry/formats.json](../registry/formats.json).
A candidate format changes only with a recorded reason and a new or changed vector ([22.5](../spec/22-conformance.md#225-stability)).
A passing run shows one thing: for these inputs, your implementation computes the same values as the reference implementation in `src/paperspec/`.
That code computed the expected outputs, and `pixi run build` regenerates them ([tools/](../tools/README.md)).
The vectors hold 311 cases in all, some of them small: `hashes.jsonl` has one document, `alignment.jsonl` one pair of readings, `offsets.jsonl` two cases.
The vectors do not show the following.
- Conformance: the vectors are one part of the suite in [chapter 22](../spec/22-conformance.md#221-the-suite).
A conforming implementation must also accept every valid package in `fixtures/packages/`, and refuse every package in `fixtures/invalid/`, naming the rule it breaks if it can.
No vector tests package validation.
- Independent agreement: the only program in this repository that runs the vectors is the reference checker, and it uses the same oracle that produced them.
A rule that the oracle and the chapter both misstate would pass.
A separately written implementation that passes is evidence the repository cannot supply itself, and a disagreement between a chapter and a vector is a bug to report.
- Rules the vectors skip:
- `canonical-json.jsonl` has five cases, none for the repository's rule that a lone surrogate, `NaN` or an infinity has no canonical form.
- `merkle.jsonl` has roots for 1, 2, 3, 4, 5 and 7 items.
A receipt with no items is invalid ([8.5](../spec/08-fidelity.md#85-merkle-roots)), so no case has an empty list, although the RFCs it follows define a hash for an empty list ([RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162)).
- Thirteen formats are `provisional`, and only `grading.jsonl` carries any of their identifiers (exercise, submission and grade).
The chapters of the others define values that no vector pins, such as a dataset's content hash ([15](../spec/15-datasets.md)), the hash that confirms a digitization ([16](../spec/16-digitization.md)) and a checklist's content hash ([8.11](../spec/08-fidelity.md#811-checklists)).
They are expected to change as implementations gain experience ([0.4](../spec/00-introduction.md#04-status)).
- Reading a PDF: how text lines are extracted from a PDF is outside the specification; `fingerprints.jsonl` supplies the lines, so it tests the fingerprint function only ([7.1](../spec/07-geometry.md#71-page-fingerprints)).
- Faithful conversion: the vectors check what is computed from text, not whether a converter read the paper correctly.
The formats record such an error and make it checkable through corrections and receipts ([chapter 8](../spec/08-fidelity.md), [chapter 11](../spec/11-corrections.md)); the vectors do not detect it.
- Security: `signatures.jsonl` contains the RFC 8032 test vector and one receipt, so it checks that your Ed25519 and record hashing give the stated bytes.
The secrets in `keyed.jsonl` and `signatures.jsonl` are test values published in this repository; do not use them for anything real.
A vector that rests on a building block defined elsewhere pins this specification's use of it.
Canonical JSON is RFC 8785 with added rules.
Receipts use a Merkle tree that follows RFC 6962 ([8.5](../spec/08-fidelity.md#85-merkle-roots)); RFC 9162 replaces RFC 6962 and defines the same tree.
The `node_hash` of a node is a different construction: a hash of the canonical JSON of `{own, children}`.
Here `own` is the node's own hash and `children` holds the `node_hash` of each child ([5.1](../spec/05-hashes.md#51-the-five-values)), and `hashes.jsonl` pins it.
The [comparison page](../docs/comparison.md) covers how these formats differ from JATS (Journal Article Tag Suite), TEI (Text Encoding Initiative), W3C (World Wide Web Consortium) Web Annotation and the other neighbors.
Reference in new issue View Git Blame Copy Permalink