10 KiB
Examples
Things to run, each with the output it gives.
Three readers can use this page.
Each section opens with what it shows, in plain terms, for readers new to the problem, and every example uses the fixtures in this repository.
The last sections point implementers (anyone writing a reader, writer, converter or checker) to the test vectors, which pin the exact behavior (the vector files are listed in vectors/README.md).
This page makes no comparison with the neighboring standards.
The formats are version 0.1.0, and each is candidate or provisional in registry/formats.json.
Run every command from the root of the repository, https://git.scient.ing/infra/paperspec.
With pixi, pixi run <task> sets up the environment first.
Every command also works without pixi, with Python 3.14 and two packages, jsonschema and cryptography installed and standard command-line tools (the sed command in the last section is the GNU version of sed).
Run the checks
This shows how to confirm that the repository's specification, fixtures and test vectors agree with each other.
pixi run check
pixi run check lints the Markdown, then runs python tools/check.py, the repository's reference checker.
Each section prints how many checks it ran, and the last line is All checks pass.
pixi run quick, or python3 tools/check.py --quick, skips the slowest part, which rebuilds everything and compares it byte for byte.
Without pixi, the Markdown lint is a separate step (markdownlint-cli2); tools/check.py does not run it.
Recompute a hash by hand
This shows that a hash in a package is not magic: you can reproduce it with a standard tool.
A hash is a fixed-length fingerprint of some text; change one character and the fingerprint changes.
A sentence's text_hash is the SHA-256 of its text after two steps.
The text is converted to Unicode NFC (a standard form that writes each accented or composed character one way), and each run of whitespace is collapsed to one space, as chapter 1 defines.
For a sentence with math or citations, the text is its plain projection (chapter 5).
For a sentence of plain text, the whole computation is a one-liner:
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
The same value is in the package, in the nodes.jsonl of hale2026harmonic-damb:
grep -o '"id":"v3063qv".*' fixtures/packages/hale2026harmonic-damb/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
Check a package's manifest
This shows how a reader detects that a package's files have changed.
A package is a folder of JSON (JavaScript Object Notation, a plain-text format for structured data) files describing one paper.
MANIFEST.json lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package (chapter 12).
cd fixtures/packages/okafor2025solvent-acth
sha256sum nodes.jsonl
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3 nodes.jsonl
"path":"nodes.jsonl","sha256":"09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3"
Read a package with nothing but a JSON parser
This shows that a package needs no special software to read.
JSON Lines is a text format with one JSON object per line, here one node (a section, sentence, figure, table or equation) per line of nodes.jsonl.
Save this script as outline.py.
It prints a document's sections, figures, tables and equations.
import json, sys
pkg = sys.argv[1]
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
by_id = {n["id"]: n for n in nodes}
children = {}
for n in nodes:
children.setdefault(n["parent"], []).append(n)
def text(n):
"""The plain projection of a node with runs."""
out = []
for r in n["content"]:
if r["kind"] == "math":
out.append("$" + r["source"] + "$")
elif r["kind"] == "equation":
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
else:
out.append(r["text"])
return "".join(out)
for n in nodes:
if n["kind"] == "section":
heading = children[n["id"]][0]
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
elif n["kind"] in ("figure", "table", "equation"):
print(n["id"], " ", n["kind"], n.get("label", ""))
python3 outline.py fixtures/packages/hale2026harmonic-damb
p36k8t6 1 Introduction
tkxsmts equation (1)
d400rz3 2 Methods
v7cx6qs figure Figure 1
hj4s7pb table Table 1
Follow an address
This shows how a citation points inside a sentence.
An address names a node, and optionally a span of its text (chapter 6).
hale2026harmonic-damb#hca6dc5[29:44] names code points 29 to 44 of sentence hca6dc5.
With text() from the script above:
print(text(by_id["hca6dc5"])[29:44])
first test case
Offsets count Unicode code points, one per character; in Python, string indexes already do. JavaScript strings index UTF-16 units, so convert first. For text with characters outside the Basic Multilingual Plane the two counts differ, and vectors/offsets.jsonl gives both.
Load packages into SQLite
This shows that the same packages can live in a database and be queried with SQL.
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
Without pixi, run python3 tools/load.py with the same arguments.
hale2026harmonic-damb: 75 nodes
okafor2025solvent-acth: 15 nodes
writer2027fixed-ynr7.v3: 15 nodes
The tables are those of sql/doc.sql and sql/store.sql, which run unchanged on Dolt and PostgreSQL as well (chapter 13); tools/check.py exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
Here are four queries, in Python's sqlite3 so that nothing else is needed.
import json, sqlite3
db = sqlite3.connect("lab.db")
# Sentences the converter left below F2. Receipts can raise them; see below.
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic-damb' "
"AND kind = 'sentence' AND fidelity IN ('F0', 'F1') ORDER BY seq"):
print(row)
# Every sentence that cites reference 2: the citation graph is sentence-level.
for row in db.execute("SELECT c.sentence FROM citations c JOIN refs r "
"ON r.doc_key = c.doc_key AND r.node = c.ref_node "
"WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2'"):
print(row)
# Who signed what.
for row in db.execute("SELECT r.signer, r.scope, COUNT(*) FROM receipts r "
"JOIN receipt_nodes rn ON rn.receipt = r.id GROUP BY r.id ORDER BY r.made_at"):
print(row)
# Table 1's body, row by row.
rows = db.execute("SELECT r.id FROM nodes r JOIN nodes b ON b.doc_key = r.doc_key AND b.id = r.parent "
"WHERE r.doc_key = 'hale2026harmonic-damb' AND r.kind = 'row' AND b.kind = 'table-body' "
"ORDER BY r.seq").fetchall()
for (row,) in rows:
cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic-damb' AND parent = ? ORDER BY ord",
(row,)).fetchall()
print(" | ".join("".join(r.get("text", "") for r in json.loads(c)) for (c,) in cells))
('ec5jc93', 'F1')
('a4jrz2t', 'F0')
('xjns57z', 'F0')
('hca6dc5',)
('ana', 'targeted:lode', 3)
('ana', 'targeted:lode', 23)
('ana', 'section:p36k8t6', 2)
('cara', 'page:2', 3)
('ana', 'targeted:captions', 3)
('ben', 'qa:y2zw8ctg5s8dp', 10)
('ben', 'qa:gx2s84z9jf9xv', 3)
Bond parameters
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
CT–HC | 340.0 | 1.090 | 338.7 | 1.0900
OW–HW | 553.0 | 0.9572 | 548.1 | 0.9570
The first query reads the level the converter recorded, not the final one.
A receipt is a signed record that a reviewer checked certain nodes against the page.
Receipts by ana raise ec5jc93 and a4jrz2t to F3; xjns57z stays at F0 because the only receipt that covered it, by cara, was revoked.
To compute the final level, check each receipt's signature, revocations and spot-checks, as chapter 8 describes.
The scope column of the third query shows the value targeted:lode, which is data in the fixture, copied here exactly as the query prints it.
Test your implementation against a vector
A test vector is a recorded input together with the output every correct implementation must produce. Every vector file has one case per line, with its inputs and expected outputs. A harness for the projections looks like this, with your functions in place of the comments:
import json
for line in open("vectors/projections.jsonl", encoding="utf-8"):
case = json.loads(line)
runs = case["runs"]
# assert my_plain(runs) == case["plain"]
# assert my_strict(runs) == case["strict"]
# assert my_lossy(runs) == case["lossy"]
# assert my_sha256(case["strict"]) == case["match_hash"]
print(case["id"], case["strict"])
vectors/README.md lists every file, what it tests and the chapter that defines it.
Break something on purpose
This shows what the checks catch. Change one digit in a fixture and run the checks:
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025solvent-acth/nodes.jsonl
pixi run quick
The checker reports three failures: the SHA-256 of nodes.jsonl no longer matches the manifest, unloading the package from the database no longer gives the same bytes, and the package hash differs after unloading.
Restore the file with git checkout fixtures/packages/okafor2025solvent-acth/nodes.jsonl.
The broken packages in fixtures/invalid/ are this kind of change, each with the rule it breaks.
git checkout restores a tracked file, so run this only in a clone you can reset.