feat: documents and session tokens
This commit is contained in:
39 files changed
+1523
-199
No files matched your search
@@ -0,0 +1,221 @@
|
||||
# Examples
|
||||
|
||||
Things to run, each with the output it gives.
|
||||
Run them from the repository's root.
|
||||
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first; every command also works with plain Python 3.10 or later and standard command-line tools.
|
||||
|
||||
## Run the checks
|
||||
|
||||
```sh
|
||||
pixi run check
|
||||
```
|
||||
|
||||
`pixi run check` lints the Markdown and then runs `python tools/check.py`, which you can also run directly.
|
||||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||||
`pixi run quick`, or `--quick`, skips the slowest part: rebuilding everything and comparing it byte for byte.
|
||||
|
||||
## Recompute a hash by hand
|
||||
|
||||
A sentence's `text_hash` is SHA-256 of its text, after Unicode NFC and with whitespace collapsed.
|
||||
For a sentence of plain text, that is a one-liner:
|
||||
|
||||
```sh
|
||||
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
|
||||
```
|
||||
|
||||
```text
|
||||
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
|
||||
```
|
||||
|
||||
The same value is in the package:
|
||||
|
||||
```sh
|
||||
grep -o '"id":"j06tfhn".*' fixtures/packages/hale2026harmonic/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
|
||||
```
|
||||
|
||||
```text
|
||||
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
|
||||
```
|
||||
|
||||
## Check a package's manifest
|
||||
|
||||
Every file of a package is listed in `MANIFEST.json` with its SHA-256, and the manifest's own hash pins the whole package.
|
||||
|
||||
```sh
|
||||
cd fixtures/packages/okafor2025open
|
||||
sha256sum nodes.jsonl
|
||||
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
|
||||
```
|
||||
|
||||
```text
|
||||
0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94 nodes.jsonl
|
||||
"path":"nodes.jsonl","sha256":"0fccd6492121fb91ebd1d32487e23ca5b30ad92627e7d3a0801d77549e1a2f94"
|
||||
```
|
||||
|
||||
## Read a package with nothing but a JSON parser
|
||||
|
||||
Packages are plain JSON Lines.
|
||||
Save this script as `outline.py`; it prints a document's sections, figures, tables and equations.
|
||||
|
||||
```python
|
||||
import json, sys
|
||||
|
||||
pkg = sys.argv[1]
|
||||
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
|
||||
by_id = {n["id"]: n for n in nodes}
|
||||
children = {}
|
||||
for n in nodes:
|
||||
children.setdefault(n["parent"], []).append(n)
|
||||
|
||||
|
||||
def text(n):
|
||||
"""The plain projection of a node with runs."""
|
||||
out = []
|
||||
for r in n["content"]:
|
||||
if r["kind"] == "math":
|
||||
out.append("$" + r["source"] + "$")
|
||||
elif r["kind"] == "equation":
|
||||
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
|
||||
else:
|
||||
out.append(r["text"])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
for n in nodes:
|
||||
if n["kind"] == "section":
|
||||
heading = children[n["id"]][0]
|
||||
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
|
||||
elif n["kind"] in ("figure", "table", "equation"):
|
||||
print(n["id"], " ", n["kind"], n.get("label", ""))
|
||||
```
|
||||
|
||||
```sh
|
||||
python3 outline.py fixtures/packages/hale2026harmonic
|
||||
```
|
||||
|
||||
```text
|
||||
bzqvbhd 1 Introduction
|
||||
qn08tj9 equation (1)
|
||||
f0tj6ca 2 Methods
|
||||
ajfvq4s figure Figure 1
|
||||
gm5z4rx table Table 1
|
||||
```
|
||||
|
||||
## Follow an address
|
||||
|
||||
`hale2026harmonic#f929eq9[29:44]` names code points 29 to 44 of sentence `f929eq9`.
|
||||
With `text()` from the script above:
|
||||
|
||||
```python
|
||||
print(text(by_id["f929eq9"])[29:44])
|
||||
```
|
||||
|
||||
```text
|
||||
first test case
|
||||
```
|
||||
|
||||
Offsets count Unicode code points; in Python, string indexes already do.
|
||||
In JavaScript, convert to UTF-16 units first; [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both for text with characters outside the Basic Multilingual Plane.
|
||||
|
||||
## Load packages into SQLite
|
||||
|
||||
```sh
|
||||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic fixtures/packages/okafor2025open fixtures/packages/smithlab2027charges
|
||||
```
|
||||
|
||||
Without pixi, run `python3 tools/load.py` with the same arguments.
|
||||
|
||||
```text
|
||||
hale2026harmonic: 75 nodes
|
||||
okafor2025open: 15 nodes
|
||||
smithlab2027charges: 15 nodes
|
||||
```
|
||||
|
||||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), and the same statements run on Dolt and PostgreSQL.
|
||||
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
|
||||
|
||||
```python
|
||||
import json, sqlite3
|
||||
|
||||
db = sqlite3.connect("lab.db")
|
||||
|
||||
# Sentences the converter left below F2. Receipts can raise them; see below.
|
||||
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic' "
|
||||
"AND kind = 'sentence' AND fidelity IN ('F0', 'F1') ORDER BY seq"):
|
||||
print(row)
|
||||
|
||||
# Every sentence that cites reference 2: the citation graph is sentence-level.
|
||||
for row in db.execute("SELECT c.sentence FROM citations c JOIN refs r "
|
||||
"ON r.doc_key = c.doc_key AND r.node = c.ref_node "
|
||||
"WHERE c.doc_key = 'hale2026harmonic' AND r.label = '2'"):
|
||||
print(row)
|
||||
|
||||
# Who signed what.
|
||||
for row in db.execute("SELECT r.signer, r.scope, COUNT(*) FROM receipts r "
|
||||
"JOIN receipt_nodes rn ON rn.receipt = r.id GROUP BY r.id ORDER BY r.made_at"):
|
||||
print(row)
|
||||
|
||||
# Table 1's body, row by row.
|
||||
rows = db.execute("SELECT r.id FROM nodes r JOIN nodes b ON b.doc_key = r.doc_key AND b.id = r.parent "
|
||||
"WHERE r.doc_key = 'hale2026harmonic' AND r.kind = 'row' AND b.kind = 'table-body' "
|
||||
"ORDER BY r.seq").fetchall()
|
||||
for (row,) in rows:
|
||||
cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic' AND parent = ? ORDER BY ord",
|
||||
(row,)).fetchall()
|
||||
print(" | ".join("".join(r.get("text", "") for r in json.loads(c)) for (c,) in cells))
|
||||
```
|
||||
|
||||
```text
|
||||
('e4fkseg', 'F1')
|
||||
('r1gacps', 'F0')
|
||||
('eajq93t', 'F0')
|
||||
('f929eq9',)
|
||||
('ana', 'targeted:lode', 3)
|
||||
('ana', 'targeted:lode', 23)
|
||||
('ana', 'section:bzqvbhd', 2)
|
||||
('cara', 'page:2', 3)
|
||||
('ana', 'targeted:captions', 3)
|
||||
('ben', 'qa:eqgbbpwb6725v', 10)
|
||||
('ben', 'qa:gena76nkb6cnc', 3)
|
||||
Bond parameters
|
||||
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
|
||||
CT–HC | 340.0 | 1.090 | 338.7 | 1.0900
|
||||
OW–HW | 553.0 | 0.9572 | 548.1 | 0.9570
|
||||
```
|
||||
|
||||
The first query reads the level the converter recorded, not the final one.
|
||||
Receipts by `ana` raise `e4fkseg` and `r1gacps` to F3; `eajq93t` stays at F0 because the only receipt that covered it was revoked.
|
||||
Computing the final level means checking each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
|
||||
|
||||
## Test your implementation against a vector
|
||||
|
||||
Every vector file has one case per line, with its inputs and expected outputs.
|
||||
A harness for the projections looks like this, with your functions in place of the comments:
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
for line in open("vectors/projections.jsonl", encoding="utf-8"):
|
||||
case = json.loads(line)
|
||||
runs = case["runs"]
|
||||
# assert my_plain(runs) == case["plain"]
|
||||
# assert my_strict(runs) == case["strict"]
|
||||
# assert my_lossy(runs) == case["lossy"]
|
||||
# assert my_sha256(case["strict"]) == case["match_hash"]
|
||||
print(case["id"], case["strict"])
|
||||
```
|
||||
|
||||
[vectors/README.md](../vectors/README.md) lists every file and what it tests.
|
||||
|
||||
## Break something on purpose
|
||||
|
||||
Change one digit in a fixture and run the checks:
|
||||
|
||||
```sh
|
||||
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025open/nodes.jsonl
|
||||
pixi run quick
|
||||
```
|
||||
|
||||
The checker reports the package's manifest hash no longer matches, and that unloading it from the database no longer gives the same bytes.
|
||||
Restore the file with `git checkout fixtures/packages/okafor2025open/nodes.jsonl`.
|
||||
The broken packages in [fixtures/invalid/](../fixtures/invalid/) are this kind of change, each with the rule it breaks.
|
||||
Reference in new issue
Block a user