alexmandClaude
8c8b51cc3f
- Add tools/validate.py: validates packages, and with --correction runs proposed corrections through the gate against a package without writing anything (spec/11 §11.4). The gate now refuses a proposal that fails the correction schema, rule correction.schema. - tools/load.py validates every package before loading, and with --bundle --built-from builds a scienting-bundle/1 with the FTS5 search index and bundle_meta (spec/13 §13.5). Search rows are defined per unit (sentence, cell, equation outside a sentence, figure-text) with the parent kind as context. - Roster keys carry a role, and a key with the role owner may revoke any receipt (spec/08 §8.7, spec/20); roster schema, store table and the fixture roster updated. The checker signs a trial revocation to prove the rule both ways. - A qa: receipt raises the nodes it sampled except those in its errors (spec/08 §8.2); the example paper's fidelity counts become 27 F3, 13 F2, 4 F1 and 1 F0. - The destination chooses the keyed scheme, not the text policy: hmac-team between the team's machines, hkdf-pages outside the team (spec/05 §5.8, spec/12 §12.5). - A TextQuoteSelector must repeat the anchor's plain quote and is absent for hashed or keyed anchors (spec/18 §18.3). - Add fixtures/records/suspect.json, a scienting-suspect/1 record. - Oracle: hashes are kept incrementally and checked after every correction against a full recomputation; load and write refuse unsafe manifest paths; Ed25519, HKDF and schema validation come from the cryptography and jsonschema packages; canonical JSON uses the built-in encoder with an exact RFC 8785 fallback. - The tools need Python 3.14 with cryptography and jsonschema; the py310 CI environment is dropped and the workflow runs the default only. - Resolve every [CHECK] note in the docs, READMEs and context; add CONTRIBUTING.md; every tool answers --help and refuses unknown options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
250 lines
10 KiB
Markdown
250 lines
10 KiB
Markdown
# Examples
|
||
|
||
Things to run, each with the output it gives.
|
||
Three readers can use this page.
|
||
Each section opens with what it shows, in plain terms, for readers new to the problem, and every example uses the fixtures in this repository.
|
||
The last sections point implementers (anyone writing a reader, writer, converter or checker) to the test vectors, which pin the exact behavior (the vector files are listed in [vectors/README.md](../vectors/README.md)).
|
||
This page makes no comparison with the neighboring standards.
|
||
The formats are version 0.1.0, and each is `candidate` or `provisional` in `registry/formats.json`.
|
||
|
||
Run every command from the root of the repository, <https://git.scient.ing/infra/manuspec>.
|
||
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
|
||
Every command also works without pixi, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
|
||
|
||
## Run the checks
|
||
|
||
This shows how to confirm that the repository's specification, fixtures and test vectors agree with each other.
|
||
|
||
```sh
|
||
pixi run check
|
||
```
|
||
|
||
`pixi run check` lints the Markdown, then runs `python tools/check.py`, the repository's reference checker.
|
||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||
`pixi run quick`, or `python3 tools/check.py --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
|
||
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `tools/check.py` does not run it.
|
||
|
||
## Recompute a hash by hand
|
||
|
||
This shows that a hash in a package is not magic: you can reproduce it with a standard tool.
|
||
A hash is a fixed-length fingerprint of some text; change one character and the fingerprint changes.
|
||
A sentence's `text_hash` is the SHA-256 of its text after two steps.
|
||
The text is converted to Unicode NFC (a standard form that writes each accented or composed character one way), and each run of whitespace is collapsed to one space, as [chapter 1](../spec/01-conventions.md#14-text-normalization-for-hashing) defines.
|
||
For a sentence with math or citations, the text is its plain projection ([chapter 5](../spec/05-hashes.md#51-the-five-values)).
|
||
For a sentence of plain text, the whole computation is a one-liner:
|
||
|
||
```sh
|
||
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
|
||
```
|
||
|
||
```text
|
||
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
|
||
```
|
||
|
||
The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic`:
|
||
|
||
```sh
|
||
grep -o '"id":"j06tfhn".*' fixtures/packages/hale2026harmonic/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
|
||
```
|
||
|
||
```text
|
||
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
|
||
```
|
||
|
||
## Check a package's manifest
|
||
|
||
This shows how a reader detects that a package's files have changed.
|
||
A package is a folder of JSON (JavaScript Object Notation, a plain-text format for structured data) files describing one paper.
|
||
`MANIFEST.json` lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package ([chapter 12](../spec/12-package.md#123-the-manifest)).
|
||
|
||
```sh
|
||
cd fixtures/packages/okafor2025open
|
||
sha256sum nodes.jsonl
|
||
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
|
||
```
|
||
|
||
```text
|
||
0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94 nodes.jsonl
|
||
"path":"nodes.jsonl","sha256":"0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94"
|
||
```
|
||
|
||
## Read a package with nothing but a JSON parser
|
||
|
||
This shows that a package needs no special software to read.
|
||
JSON Lines is a text format with one JSON object per line, here one node (a section, sentence, figure, table or equation) per line of `nodes.jsonl`.
|
||
Save this script as `outline.py`.
|
||
It prints a document's sections, figures, tables and equations.
|
||
|
||
```python
|
||
import json, sys
|
||
|
||
pkg = sys.argv[1]
|
||
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
|
||
by_id = {n["id"]: n for n in nodes}
|
||
children = {}
|
||
for n in nodes:
|
||
children.setdefault(n["parent"], []).append(n)
|
||
|
||
|
||
def text(n):
|
||
"""The plain projection of a node with runs."""
|
||
out = []
|
||
for r in n["content"]:
|
||
if r["kind"] == "math":
|
||
out.append("$" + r["source"] + "$")
|
||
elif r["kind"] == "equation":
|
||
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
|
||
else:
|
||
out.append(r["text"])
|
||
return "".join(out)
|
||
|
||
|
||
for n in nodes:
|
||
if n["kind"] == "section":
|
||
heading = children[n["id"]][0]
|
||
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
|
||
elif n["kind"] in ("figure", "table", "equation"):
|
||
print(n["id"], " ", n["kind"], n.get("label", ""))
|
||
```
|
||
|
||
```sh
|
||
python3 outline.py fixtures/packages/hale2026harmonic
|
||
```
|
||
|
||
```text
|
||
bzqvbhd 1 Introduction
|
||
qn08tj9 equation (1)
|
||
f0tj6ca 2 Methods
|
||
ajfvq4s figure Figure 1
|
||
gm5z4rx table Table 1
|
||
```
|
||
|
||
## Follow an address
|
||
|
||
This shows how a citation points inside a sentence.
|
||
An address names a node, and optionally a span of its text ([chapter 6](../spec/06-anchors.md#61-addresses)).
|
||
`hale2026harmonic#f929eq9[29:44]` names code points 29 to 44 of sentence `f929eq9`.
|
||
With `text()` from the script above:
|
||
|
||
```python
|
||
print(text(by_id["f929eq9"])[29:44])
|
||
```
|
||
|
||
```text
|
||
first test case
|
||
```
|
||
|
||
Offsets count Unicode code points, one per character; in Python, string indexes already do.
|
||
JavaScript strings index UTF-16 units, so convert first.
|
||
For text with characters outside the Basic Multilingual Plane the two counts differ, and [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both.
|
||
|
||
## Load packages into SQLite
|
||
|
||
This shows that the same packages can live in a database and be queried with SQL.
|
||
|
||
```sh
|
||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic fixtures/packages/okafor2025open fixtures/packages/smithlab2027charges
|
||
```
|
||
|
||
Without pixi, run `python3 tools/load.py` with the same arguments.
|
||
|
||
```text
|
||
hale2026harmonic: 75 nodes
|
||
okafor2025open: 15 nodes
|
||
smithlab2027charges: 15 nodes
|
||
```
|
||
|
||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
|
||
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
|
||
|
||
```python
|
||
import json, sqlite3
|
||
|
||
db = sqlite3.connect("lab.db")
|
||
|
||
# Sentences the converter left below F2. Receipts can raise them; see below.
|
||
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic' "
|
||
"AND kind = 'sentence' AND fidelity IN ('F0', 'F1') ORDER BY seq"):
|
||
print(row)
|
||
|
||
# Every sentence that cites reference 2: the citation graph is sentence-level.
|
||
for row in db.execute("SELECT c.sentence FROM citations c JOIN refs r "
|
||
"ON r.doc_key = c.doc_key AND r.node = c.ref_node "
|
||
"WHERE c.doc_key = 'hale2026harmonic' AND r.label = '2'"):
|
||
print(row)
|
||
|
||
# Who signed what.
|
||
for row in db.execute("SELECT r.signer, r.scope, COUNT(*) FROM receipts r "
|
||
"JOIN receipt_nodes rn ON rn.receipt = r.id GROUP BY r.id ORDER BY r.made_at"):
|
||
print(row)
|
||
|
||
# Table 1's body, row by row.
|
||
rows = db.execute("SELECT r.id FROM nodes r JOIN nodes b ON b.doc_key = r.doc_key AND b.id = r.parent "
|
||
"WHERE r.doc_key = 'hale2026harmonic' AND r.kind = 'row' AND b.kind = 'table-body' "
|
||
"ORDER BY r.seq").fetchall()
|
||
for (row,) in rows:
|
||
cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic' AND parent = ? ORDER BY ord",
|
||
(row,)).fetchall()
|
||
print(" | ".join("".join(r.get("text", "") for r in json.loads(c)) for (c,) in cells))
|
||
```
|
||
|
||
```text
|
||
('e4fkseg', 'F1')
|
||
('r1gacps', 'F0')
|
||
('eajq93t', 'F0')
|
||
('f929eq9',)
|
||
('ana', 'targeted:lode', 3)
|
||
('ana', 'targeted:lode', 23)
|
||
('ana', 'section:bzqvbhd', 2)
|
||
('cara', 'page:2', 3)
|
||
('ana', 'targeted:captions', 3)
|
||
('ben', 'qa:eqgbbpwb6725v', 10)
|
||
('ben', 'qa:gena76nkb6cnc', 3)
|
||
Bond parameters
|
||
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
|
||
CT–HC | 340.0 | 1.090 | 338.7 | 1.0900
|
||
OW–HW | 553.0 | 0.9572 | 548.1 | 0.9570
|
||
```
|
||
|
||
The first query reads the level the converter recorded, not the final one.
|
||
A receipt is a signed record that a reviewer checked certain nodes against the page.
|
||
Receipts by `ana` raise `e4fkseg` and `r1gacps` to F3; `eajq93t` stays at F0 because the only receipt that covered it, by `cara`, was revoked.
|
||
To compute the final level, check each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
|
||
The `scope` column of the third query shows the value `targeted:lode`, which is data in the fixture, copied here exactly as the query prints it.
|
||
|
||
## Test your implementation against a vector
|
||
|
||
A test vector is a recorded input together with the output every correct implementation must produce.
|
||
Every vector file has one case per line, with its inputs and expected outputs.
|
||
A harness for the projections looks like this, with your functions in place of the comments:
|
||
|
||
```python
|
||
import json
|
||
|
||
for line in open("vectors/projections.jsonl", encoding="utf-8"):
|
||
case = json.loads(line)
|
||
runs = case["runs"]
|
||
# assert my_plain(runs) == case["plain"]
|
||
# assert my_strict(runs) == case["strict"]
|
||
# assert my_lossy(runs) == case["lossy"]
|
||
# assert my_sha256(case["strict"]) == case["match_hash"]
|
||
print(case["id"], case["strict"])
|
||
```
|
||
|
||
[vectors/README.md](../vectors/README.md) lists every file, what it tests and the chapter that defines it.
|
||
|
||
## Break something on purpose
|
||
|
||
This shows what the checks catch.
|
||
Change one digit in a fixture and run the checks:
|
||
|
||
```sh
|
||
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025open/nodes.jsonl
|
||
pixi run quick
|
||
```
|
||
|
||
The checker reports three failures: the SHA-256 of `nodes.jsonl` no longer matches the manifest, unloading the package from the database no longer gives the same bytes, and the package hash differs after unloading.
|
||
Restore the file with `git checkout fixtures/packages/okafor2025open/nodes.jsonl`.
|
||
The broken packages in [fixtures/invalid/](../fixtures/invalid/) are this kind of change, each with the rule it breaks.
|
||
`git checkout` restores a tracked file, so run this only in a clone you can reset.
|