222 lines
7.3 KiB
Markdown
222 lines
7.3 KiB
Markdown
# Examples
|
||
|
||
Things to run, each with the output it gives.
|
||
Run them from the repository's root.
|
||
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first; every command also works with plain Python 3.10 or later and standard command-line tools.
|
||
|
||
## Run the checks
|
||
|
||
```sh
|
||
pixi run check
|
||
```
|
||
|
||
`pixi run check` lints the Markdown and then runs `python tools/check.py`, which you can also run directly.
|
||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||
`pixi run quick`, or `--quick`, skips the slowest part: rebuilding everything and comparing it byte for byte.
|
||
|
||
## Recompute a hash by hand
|
||
|
||
A sentence's `text_hash` is SHA-256 of its text, after Unicode NFC and with whitespace collapsed.
|
||
For a sentence of plain text, that is a one-liner:
|
||
|
||
```sh
|
||
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
|
||
```
|
||
|
||
```text
|
||
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
|
||
```
|
||
|
||
The same value is in the package:
|
||
|
||
```sh
|
||
grep -o '"id":"j06tfhn".*' fixtures/packages/hale2026harmonic/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
|
||
```
|
||
|
||
```text
|
||
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
|
||
```
|
||
|
||
## Check a package's manifest
|
||
|
||
Every file of a package is listed in `MANIFEST.json` with its SHA-256, and the manifest's own hash pins the whole package.
|
||
|
||
```sh
|
||
cd fixtures/packages/okafor2025open
|
||
sha256sum nodes.jsonl
|
||
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
|
||
```
|
||
|
||
```text
|
||
0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94 nodes.jsonl
|
||
"path":"nodes.jsonl","sha256":"0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94"
|
||
```
|
||
|
||
## Read a package with nothing but a JSON parser
|
||
|
||
Packages are plain JSON Lines.
|
||
Save this script as `outline.py`; it prints a document's sections, figures, tables and equations.
|
||
|
||
```python
|
||
import json, sys
|
||
|
||
pkg = sys.argv[1]
|
||
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
|
||
by_id = {n["id"]: n for n in nodes}
|
||
children = {}
|
||
for n in nodes:
|
||
children.setdefault(n["parent"], []).append(n)
|
||
|
||
|
||
def text(n):
|
||
"""The plain projection of a node with runs."""
|
||
out = []
|
||
for r in n["content"]:
|
||
if r["kind"] == "math":
|
||
out.append("$" + r["source"] + "$")
|
||
elif r["kind"] == "equation":
|
||
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
|
||
else:
|
||
out.append(r["text"])
|
||
return "".join(out)
|
||
|
||
|
||
for n in nodes:
|
||
if n["kind"] == "section":
|
||
heading = children[n["id"]][0]
|
||
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
|
||
elif n["kind"] in ("figure", "table", "equation"):
|
||
print(n["id"], " ", n["kind"], n.get("label", ""))
|
||
```
|
||
|
||
```sh
|
||
python3 outline.py fixtures/packages/hale2026harmonic
|
||
```
|
||
|
||
```text
|
||
bzqvbhd 1 Introduction
|
||
qn08tj9 equation (1)
|
||
f0tj6ca 2 Methods
|
||
ajfvq4s figure Figure 1
|
||
gm5z4rx table Table 1
|
||
```
|
||
|
||
## Follow an address
|
||
|
||
`hale2026harmonic#f929eq9[29:44]` names code points 29 to 44 of sentence `f929eq9`.
|
||
With `text()` from the script above:
|
||
|
||
```python
|
||
print(text(by_id["f929eq9"])[29:44])
|
||
```
|
||
|
||
```text
|
||
first test case
|
||
```
|
||
|
||
Offsets count Unicode code points; in Python, string indexes already do.
|
||
In JavaScript, convert to UTF-16 units first; [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both for text with characters outside the Basic Multilingual Plane.
|
||
|
||
## Load packages into SQLite
|
||
|
||
```sh
|
||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic fixtures/packages/okafor2025open fixtures/packages/smithlab2027charges
|
||
```
|
||
|
||
Without pixi, run `python3 tools/load.py` with the same arguments.
|
||
|
||
```text
|
||
hale2026harmonic: 75 nodes
|
||
okafor2025open: 15 nodes
|
||
smithlab2027charges: 15 nodes
|
||
```
|
||
|
||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), and the same statements run on Dolt and PostgreSQL.
|
||
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
|
||
|
||
```python
|
||
import json, sqlite3
|
||
|
||
db = sqlite3.connect("lab.db")
|
||
|
||
# Sentences the converter left below F2. Receipts can raise them; see below.
|
||
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic' "
|
||
"AND kind = 'sentence' AND fidelity IN ('F0', 'F1') ORDER BY seq"):
|
||
print(row)
|
||
|
||
# Every sentence that cites reference 2: the citation graph is sentence-level.
|
||
for row in db.execute("SELECT c.sentence FROM citations c JOIN refs r "
|
||
"ON r.doc_key = c.doc_key AND r.node = c.ref_node "
|
||
"WHERE c.doc_key = 'hale2026harmonic' AND r.label = '2'"):
|
||
print(row)
|
||
|
||
# Who signed what.
|
||
for row in db.execute("SELECT r.signer, r.scope, COUNT(*) FROM receipts r "
|
||
"JOIN receipt_nodes rn ON rn.receipt = r.id GROUP BY r.id ORDER BY r.made_at"):
|
||
print(row)
|
||
|
||
# Table 1's body, row by row.
|
||
rows = db.execute("SELECT r.id FROM nodes r JOIN nodes b ON b.doc_key = r.doc_key AND b.id = r.parent "
|
||
"WHERE r.doc_key = 'hale2026harmonic' AND r.kind = 'row' AND b.kind = 'table-body' "
|
||
"ORDER BY r.seq").fetchall()
|
||
for (row,) in rows:
|
||
cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic' AND parent = ? ORDER BY ord",
|
||
(row,)).fetchall()
|
||
print(" | ".join("".join(r.get("text", "") for r in json.loads(c)) for (c,) in cells))
|
||
```
|
||
|
||
```text
|
||
('e4fkseg', 'F1')
|
||
('r1gacps', 'F0')
|
||
('eajq93t', 'F0')
|
||
('f929eq9',)
|
||
('ana', 'targeted:lode', 3)
|
||
('ana', 'targeted:lode', 23)
|
||
('ana', 'section:bzqvbhd', 2)
|
||
('cara', 'page:2', 3)
|
||
('ana', 'targeted:captions', 3)
|
||
('ben', 'qa:eqgbbpwb6725v', 10)
|
||
('ben', 'qa:gena76nkb6cnc', 3)
|
||
Bond parameters
|
||
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
|
||
CT–HC | 340.0 | 1.090 | 338.7 | 1.0900
|
||
OW–HW | 553.0 | 0.9572 | 548.1 | 0.9570
|
||
```
|
||
|
||
The first query reads the level the converter recorded, not the final one.
|
||
Receipts by `ana` raise `e4fkseg` and `r1gacps` to F3; `eajq93t` stays at F0 because the only receipt that covered it was revoked.
|
||
Computing the final level means checking each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
|
||
|
||
## Test your implementation against a vector
|
||
|
||
Every vector file has one case per line, with its inputs and expected outputs.
|
||
A harness for the projections looks like this, with your functions in place of the comments:
|
||
|
||
```python
|
||
import json
|
||
|
||
for line in open("vectors/projections.jsonl", encoding="utf-8"):
|
||
case = json.loads(line)
|
||
runs = case["runs"]
|
||
# assert my_plain(runs) == case["plain"]
|
||
# assert my_strict(runs) == case["strict"]
|
||
# assert my_lossy(runs) == case["lossy"]
|
||
# assert my_sha256(case["strict"]) == case["match_hash"]
|
||
print(case["id"], case["strict"])
|
||
```
|
||
|
||
[vectors/README.md](../vectors/README.md) lists every file and what it tests.
|
||
|
||
## Break something on purpose
|
||
|
||
Change one digit in a fixture and run the checks:
|
||
|
||
```sh
|
||
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025open/nodes.jsonl
|
||
pixi run quick
|
||
```
|
||
|
||
The checker reports the package's manifest hash no longer matches, and that unloading it from the database no longer gives the same bytes.
|
||
Restore the file with `git checkout fixtures/packages/okafor2025open/nodes.jsonl`.
|
||
The broken packages in [fixtures/invalid/](../fixtures/invalid/) are this kind of change, each with the rule it breaks.
|