400 lines
16 KiB
Markdown
400 lines
16 KiB
Markdown
# Examples
|
||
|
||
Things to run, each with the output it gives.
|
||
Three readers can use this page.
|
||
Each section opens with what it shows, in plain terms, for readers new to the problem, and every example uses the fixtures in this repository.
|
||
The last sections point implementers (anyone writing a reader, writer, converter or checker) to the test vectors, which pin the exact behavior (the vector files are listed in [vectors/README.md](../vectors/README.md)).
|
||
This page makes no comparison with the neighboring standards.
|
||
The formats are version 0.1.0, and each is `candidate` or `provisional` in `registry/formats.json`.
|
||
|
||
Run every command from the root of the repository, <https://git.scient.ing/infra/paperspec>.
|
||
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
|
||
<<<<<<< HEAD
|
||
Every command also works without pixi once the checkout is installed with `pip install -e ".[tools]"` on Python 3.14, given standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
|
||
|
||
## Run the checks
|
||
|
||
This shows how to confirm that the repository's specification, fixtures and test vectors agree with each other.
|
||
|
||
```sh
|
||
pixi run check
|
||
```
|
||
|
||
`pixi run check` lints the Markdown, then runs `paperspec check`, the repository's reference checker.
|
||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||
`pixi run quick`, or `paperspec check --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
|
||
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `paperspec check` does not run it.
|
||
|
||
## Recompute a hash by hand
|
||
|
||
This shows that a hash in a package is not magic: you can reproduce it with a standard tool.
|
||
A hash is a fixed-length fingerprint of some text; change one character and the fingerprint changes.
|
||
A sentence's `text_hash` is the SHA-256 of its text after two steps.
|
||
The text is converted to Unicode NFC (a standard form that writes each accented or composed character one way), and each run of whitespace is collapsed to one space, as [chapter 1](../spec/01-conventions.md#14-text-normalization-for-hashing) defines.
|
||
For a sentence with math or citations, the text is its plain projection ([chapter 5](../spec/05-hashes.md#51-the-five-values)).
|
||
For a sentence of plain text, the whole computation is a one-liner:
|
||
|
||
```sh
|
||
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
|
||
```
|
||
|
||
```text
|
||
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
|
||
```
|
||
|
||
The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic-damb`:
|
||
|
||
```sh
|
||
grep -o '"id":"v3063qv".*' fixtures/packages/hale2026harmonic-damb/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
|
||
```
|
||
|
||
```text
|
||
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
|
||
```
|
||
|
||
## Check a package's manifest
|
||
|
||
This shows how a reader detects that a package's files have changed.
|
||
A package is a folder of JSON (JavaScript Object Notation, a plain-text format for structured data) files describing one paper.
|
||
`MANIFEST.json` lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package ([chapter 12](../spec/12-package.md#123-the-manifest)).
|
||
|
||
```sh
|
||
cd fixtures/packages/okafor2025solvent-acth
|
||
sha256sum nodes.jsonl
|
||
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
|
||
```
|
||
|
||
```text
|
||
09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3 nodes.jsonl
|
||
"path":"nodes.jsonl","sha256":"09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3"
|
||
```
|
||
|
||
## Read a package with nothing but a JSON parser
|
||
|
||
This shows that a package needs no special software to read.
|
||
JSON Lines is a text format with one JSON object per line, here one node (a section, sentence, figure, table or equation) per line of `nodes.jsonl`.
|
||
Save this script as `outline.py`.
|
||
It prints a document's sections, figures, tables and equations.
|
||
|
||
```python
|
||
import json, sys
|
||
|
||
pkg = sys.argv[1]
|
||
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
|
||
by_id = {n["id"]: n for n in nodes}
|
||
children = {}
|
||
for n in nodes:
|
||
children.setdefault(n["parent"], []).append(n)
|
||
|
||
|
||
def text(n):
|
||
"""The plain projection of a node with runs."""
|
||
out = []
|
||
for r in n["content"]:
|
||
if r["kind"] == "math":
|
||
out.append("$" + r["source"] + "$")
|
||
elif r["kind"] == "equation":
|
||
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
|
||
else:
|
||
out.append(r["text"])
|
||
return "".join(out)
|
||
|
||
|
||
for n in nodes:
|
||
if n["kind"] == "section":
|
||
heading = children[n["id"]][0]
|
||
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
|
||
elif n["kind"] in ("figure", "table", "equation"):
|
||
print(n["id"], " ", n["kind"], n.get("label", ""))
|
||
```
|
||
|
||
```sh
|
||
python3 outline.py fixtures/packages/hale2026harmonic-damb
|
||
```
|
||
|
||
```text
|
||
p36k8t6 1 Introduction
|
||
tkxsmts equation (1)
|
||
d400rz3 2 Methods
|
||
v7cx6qs figure Figure 1
|
||
hj4s7pb table Table 1
|
||
```
|
||
|
||
## Follow an address
|
||
|
||
This shows how a citation points inside a sentence.
|
||
An address names a node, and optionally a span of its text ([chapter 6](../spec/06-anchors.md#61-addresses)).
|
||
`hale2026harmonic-damb#hca6dc5[29:44]` names code points 29 to 44 of sentence `hca6dc5`.
|
||
With `text()` from the script above:
|
||
|
||
```python
|
||
print(text(by_id["hca6dc5"])[29:44])
|
||
```
|
||
|
||
```text
|
||
first test case
|
||
```
|
||
|
||
Offsets count Unicode code points, one per character; in Python, string indexes already do.
|
||
JavaScript strings index UTF-16 units, so convert first.
|
||
For text with characters outside the Basic Multilingual Plane the two counts differ, and [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both.
|
||
|
||
## Load packages into SQLite
|
||
|
||
This shows that the same packages can live in a database and be queried with SQL.
|
||
|
||
```sh
|
||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
|
||
```
|
||
|
||
Without pixi, run `paperspec load` with the same arguments.
|
||
|
||
```text
|
||
hale2026harmonic-damb: 75 nodes
|
||
okafor2025solvent-acth: 15 nodes
|
||
writer2027fixed-ynr7.v3: 15 nodes
|
||
```
|
||
|
||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
|
||
||||||| parent of 4fa326d (First implementation (#1))
|
||
=======
|
||
Every command also works without pixi, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
|
||
|
||
## Run the checks
|
||
|
||
This shows how to confirm that the repository's specification, fixtures and test vectors agree with each other.
|
||
|
||
```sh
|
||
pixi run check
|
||
```
|
||
|
||
`pixi run check` lints the Markdown, then runs `python tools/check.py`, the repository's reference checker.
|
||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||
`pixi run quick`, or `python3 tools/check.py --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
|
||
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `tools/check.py` does not run it.
|
||
|
||
## Recompute a hash by hand
|
||
|
||
This shows that a hash in a package is not magic: you can reproduce it with a standard tool.
|
||
A hash is a fixed-length fingerprint of some text; change one character and the fingerprint changes.
|
||
A sentence's `text_hash` is the SHA-256 of its text after two steps.
|
||
The text is converted to Unicode NFC (a standard form that writes each accented or composed character one way), and each run of whitespace is collapsed to one space, as [chapter 1](../spec/01-conventions.md#14-text-normalization-for-hashing) defines.
|
||
For a sentence with math or citations, the text is its plain projection ([chapter 5](../spec/05-hashes.md#51-the-five-values)).
|
||
For a sentence of plain text, the whole computation is a one-liner:
|
||
|
||
```sh
|
||
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
|
||
```
|
||
|
||
```text
|
||
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
|
||
```
|
||
|
||
The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic-damb`:
|
||
|
||
```sh
|
||
grep -o '"id":"v3063qv".*' fixtures/packages/hale2026harmonic-damb/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
|
||
```
|
||
|
||
```text
|
||
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
|
||
```
|
||
|
||
## Check a package's manifest
|
||
|
||
This shows how a reader detects that a package's files have changed.
|
||
A package is a folder of JSON (JavaScript Object Notation, a plain-text format for structured data) files describing one paper.
|
||
`MANIFEST.json` lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package ([chapter 12](../spec/12-package.md#123-the-manifest)).
|
||
|
||
```sh
|
||
cd fixtures/packages/okafor2025solvent-acth
|
||
sha256sum nodes.jsonl
|
||
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
|
||
```
|
||
|
||
```text
|
||
09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3 nodes.jsonl
|
||
"path":"nodes.jsonl","sha256":"09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3"
|
||
```
|
||
|
||
## Read a package with nothing but a JSON parser
|
||
|
||
This shows that a package needs no special software to read.
|
||
JSON Lines is a text format with one JSON object per line, here one node (a section, sentence, figure, table or equation) per line of `nodes.jsonl`.
|
||
Save this script as `outline.py`.
|
||
It prints a document's sections, figures, tables and equations.
|
||
|
||
```python
|
||
import json, sys
|
||
|
||
pkg = sys.argv[1]
|
||
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
|
||
by_id = {n["id"]: n for n in nodes}
|
||
children = {}
|
||
for n in nodes:
|
||
children.setdefault(n["parent"], []).append(n)
|
||
|
||
|
||
def text(n):
|
||
"""The plain projection of a node with runs."""
|
||
out = []
|
||
for r in n["content"]:
|
||
if r["kind"] == "math":
|
||
out.append("$" + r["source"] + "$")
|
||
elif r["kind"] == "equation":
|
||
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
|
||
else:
|
||
out.append(r["text"])
|
||
return "".join(out)
|
||
|
||
|
||
for n in nodes:
|
||
if n["kind"] == "section":
|
||
heading = children[n["id"]][0]
|
||
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
|
||
elif n["kind"] in ("figure", "table", "equation"):
|
||
print(n["id"], " ", n["kind"], n.get("label", ""))
|
||
```
|
||
|
||
```sh
|
||
python3 outline.py fixtures/packages/hale2026harmonic-damb
|
||
```
|
||
|
||
```text
|
||
p36k8t6 1 Introduction
|
||
tkxsmts equation (1)
|
||
d400rz3 2 Methods
|
||
v7cx6qs figure Figure 1
|
||
hj4s7pb table Table 1
|
||
```
|
||
|
||
## Follow an address
|
||
|
||
This shows how a citation points inside a sentence.
|
||
An address names a node, and optionally a span of its text ([chapter 6](../spec/06-anchors.md#61-addresses)).
|
||
`hale2026harmonic-damb#hca6dc5[29:44]` names code points 29 to 44 of sentence `hca6dc5`.
|
||
With `text()` from the script above:
|
||
|
||
```python
|
||
print(text(by_id["hca6dc5"])[29:44])
|
||
```
|
||
|
||
```text
|
||
first test case
|
||
```
|
||
|
||
Offsets count Unicode code points, one per character; in Python, string indexes already do.
|
||
JavaScript strings index UTF-16 units, so convert first.
|
||
For text with characters outside the Basic Multilingual Plane the two counts differ, and [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both.
|
||
|
||
## Load packages into SQLite
|
||
|
||
This shows that the same packages can live in a database and be queried with SQL.
|
||
|
||
```sh
|
||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
|
||
```
|
||
|
||
Without pixi, run `python3 tools/load.py` with the same arguments.
|
||
|
||
```text
|
||
hale2026harmonic-damb: 75 nodes
|
||
okafor2025solvent-acth: 15 nodes
|
||
writer2027fixed-ynr7.v3: 15 nodes
|
||
```
|
||
|
||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
|
||
>>>>>>> 4fa326d (First implementation (#1))
|
||
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
|
||
|
||
```python
|
||
import json, sqlite3
|
||
|
||
db = sqlite3.connect("lab.db")
|
||
|
||
# Sentences the converter left below F2. Receipts can raise them; see below.
|
||
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic-damb' "
|
||
"AND kind = 'sentence' AND fidelity IN ('F0', 'F1') ORDER BY seq"):
|
||
print(row)
|
||
|
||
# Every sentence that cites reference 2: the citation graph is sentence-level.
|
||
for row in db.execute("SELECT c.sentence FROM citations c JOIN refs r "
|
||
"ON r.doc_key = c.doc_key AND r.node = c.ref_node "
|
||
"WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2'"):
|
||
print(row)
|
||
|
||
# Who signed what.
|
||
for row in db.execute("SELECT r.signer, r.scope, COUNT(*) FROM receipts r "
|
||
"JOIN receipt_nodes rn ON rn.receipt = r.id GROUP BY r.id ORDER BY r.made_at"):
|
||
print(row)
|
||
|
||
# Table 1's body, row by row.
|
||
rows = db.execute("SELECT r.id FROM nodes r JOIN nodes b ON b.doc_key = r.doc_key AND b.id = r.parent "
|
||
"WHERE r.doc_key = 'hale2026harmonic-damb' AND r.kind = 'row' AND b.kind = 'table-body' "
|
||
"ORDER BY r.seq").fetchall()
|
||
for (row,) in rows:
|
||
cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic-damb' AND parent = ? ORDER BY ord",
|
||
(row,)).fetchall()
|
||
print(" | ".join("".join(r.get("text", "") for r in json.loads(c)) for (c,) in cells))
|
||
```
|
||
|
||
```text
|
||
('ec5jc93', 'F1')
|
||
('a4jrz2t', 'F0')
|
||
('xjns57z', 'F0')
|
||
('hca6dc5',)
|
||
('ana', 'targeted:lode', 3)
|
||
('ana', 'targeted:lode', 23)
|
||
('ana', 'section:p36k8t6', 2)
|
||
('cara', 'page:2', 3)
|
||
('ana', 'targeted:captions', 3)
|
||
('ben', 'qa:y2zw8ctg5s8dp', 10)
|
||
('ben', 'qa:gx2s84z9jf9xv', 3)
|
||
Bond parameters
|
||
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
|
||
CT–HC | 340.0 | 1.090 | 338.7 | 1.0900
|
||
OW–HW | 553.0 | 0.9572 | 548.1 | 0.9570
|
||
```
|
||
|
||
The first query reads the level the converter recorded, not the final one.
|
||
A receipt is a signed record that a reviewer checked certain nodes against the page.
|
||
Receipts by `ana` raise `ec5jc93` and `a4jrz2t` to F3; `xjns57z` stays at F0 because the only receipt that covered it, by `cara`, was revoked.
|
||
To compute the final level, check each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
|
||
The `scope` column of the third query shows the value `targeted:lode`, which is data in the fixture, copied here exactly as the query prints it.
|
||
|
||
## Test your implementation against a vector
|
||
|
||
A test vector is a recorded input together with the output every correct implementation must produce.
|
||
Every vector file has one case per line, with its inputs and expected outputs.
|
||
A harness for the projections looks like this, with your functions in place of the comments:
|
||
|
||
```python
|
||
import json
|
||
|
||
for line in open("vectors/projections.jsonl", encoding="utf-8"):
|
||
case = json.loads(line)
|
||
runs = case["runs"]
|
||
# assert my_plain(runs) == case["plain"]
|
||
# assert my_strict(runs) == case["strict"]
|
||
# assert my_lossy(runs) == case["lossy"]
|
||
# assert my_sha256(case["strict"]) == case["match_hash"]
|
||
print(case["id"], case["strict"])
|
||
```
|
||
|
||
[vectors/README.md](../vectors/README.md) lists every file, what it tests and the chapter that defines it.
|
||
|
||
## Break something on purpose
|
||
|
||
This shows what the checks catch.
|
||
Change one digit in a fixture and run the checks:
|
||
|
||
```sh
|
||
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025solvent-acth/nodes.jsonl
|
||
pixi run quick
|
||
```
|
||
|
||
The checker reports three failures: the SHA-256 of `nodes.jsonl` no longer matches the manifest, unloading the package from the database no longer gives the same bytes, and the package hash differs after unloading.
|
||
Restore the file with `git checkout fixtures/packages/okafor2025solvent-acth/nodes.jsonl`.
|
||
The broken packages in [fixtures/invalid/](../fixtures/invalid/) are this kind of change, each with the rule it breaks.
|
||
`git checkout` restores a tracked file, so run this only in a clone you can reset.
|