paperspec/docs/examples.md

# Examples

Things to run, each with the output it gives.
Three readers can use this page.
Each section opens with what it shows, in plain terms, for readers new to the problem, and every example uses the fixtures in this repository.
The last sections point implementers (anyone writing a reader, writer, converter or checker) to the test vectors, which pin the exact behavior (the vector files are listed in [vectors/README.md](../vectors/README.md)).
This page makes no comparison with the neighboring standards.
The formats are version 0.1.0, and each is `candidate` or `provisional` in `registry/formats.json`.

Run every command from the root of the repository, <https://git.scient.ing/infra/paperspec>.
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
Every command also works without pixi, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).

## Run the checks

This shows how to confirm that the repository's specification, fixtures and test vectors agree with each other.

```sh
pixi run check
```

`pixi run check` lints the Markdown, then runs `paperspec-check`, the repository's reference checker.
Each section prints how many checks it ran, and the last line is `All checks pass.`
`pixi run quick`, or `paperspec-check --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `paperspec-check` does not run it.

## Recompute a hash by hand

This shows that a hash in a package is not magic: you can reproduce it with a standard tool.
A hash is a fixed-length fingerprint of some text; change one character and the fingerprint changes.
A sentence's `text_hash` is the SHA-256 of its text after two steps.
The text is converted to Unicode NFC (a standard form that writes each accented or composed character one way), and each run of whitespace is collapsed to one space, as [chapter 1](../spec/01-conventions.md#14-text-normalization-for-hashing) defines.
For a sentence with math or citations, the text is its plain projection ([chapter 5](../spec/05-hashes.md#51-the-five-values)).
For a sentence of plain text, the whole computation is a one-liner:

```sh
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
```

```text
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a  -
```

The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic-damb`:

```sh
grep -o '"id":"v3063qv".*' fixtures/packages/hale2026harmonic-damb/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
```

```text
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
```

## Check a package's manifest

This shows how a reader detects that a package's files have changed.
A package is a folder of JSON (JavaScript Object Notation, a plain-text format for structured data) files describing one paper.
`MANIFEST.json` lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package ([chapter 12](../spec/12-package.md#123-the-manifest)).

```sh
cd fixtures/packages/okafor2025solvent-acth
sha256sum nodes.jsonl
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
```

```text
09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3  nodes.jsonl
"path":"nodes.jsonl","sha256":"09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3"
```

## Read a package with nothing but a JSON parser

This shows that a package needs no special software to read.
JSON Lines is a text format with one JSON object per line, here one node (a section, sentence, figure, table or equation) per line of `nodes.jsonl`.
Save this script as `outline.py`.
It prints a document's sections, figures, tables and equations.

```python
import json, sys

pkg = sys.argv[1]
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
by_id = {n["id"]: n for n in nodes}
children = {}
for n in nodes:
    children.setdefault(n["parent"], []).append(n)


def text(n):
    """The plain projection of a node with runs."""
    out = []
    for r in n["content"]:
        if r["kind"] == "math":
            out.append("$" + r["source"] + "$")
        elif r["kind"] == "equation":
            out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
        else:
            out.append(r["text"])
    return "".join(out)


for n in nodes:
    if n["kind"] == "section":
        heading = children[n["id"]][0]
        print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
    elif n["kind"] in ("figure", "table", "equation"):
        print(n["id"], "  ", n["kind"], n.get("label", ""))
```

```sh
python3 outline.py fixtures/packages/hale2026harmonic-damb
```

```text
p36k8t6 1 Introduction
tkxsmts    equation (1)
d400rz3 2 Methods
v7cx6qs    figure Figure 1
hj4s7pb    table Table 1
```

## Follow an address

This shows how a citation points inside a sentence.
An address names a node, and optionally a span of its text ([chapter 6](../spec/06-anchors.md#61-addresses)).
`hale2026harmonic-damb#hca6dc5[29:44]` names code points 29 to 44 of sentence `hca6dc5`.
With `text()` from the script above:

```python
print(text(by_id["hca6dc5"])[29:44])
```

```text
first test case
```

Offsets count Unicode code points, one per character; in Python, string indexes already do.
JavaScript strings index UTF-16 units, so convert first.
For text with characters outside the Basic Multilingual Plane the two counts differ, and [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both.

## Load packages into SQLite

This shows that the same packages can live in a database and be queried with SQL.

```sh
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
```

Without pixi, run `paperspec-load` with the same arguments.

```text
hale2026harmonic-damb: 75 nodes
okafor2025solvent-acth: 15 nodes
writer2027fixed-ynr7.v3: 15 nodes
```

The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec-check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
Here are four queries, in Python's `sqlite3` so that nothing else is needed.

```python
import json, sqlite3

db = sqlite3.connect("lab.db")

# Sentences the converter left below F2. Receipts can raise them; see below.
for row in db.execute("SELECT id, fidelity FROM nodes WHERE doc_key = 'hale2026harmonic-damb' "
                      "AND kind = 'sentence' AND fidelity IN ('F0', 'F1') ORDER BY seq"):
    print(row)

# Every sentence that cites reference 2: the citation graph is sentence-level.
for row in db.execute("SELECT c.sentence FROM citations c JOIN refs r "
                      "ON r.doc_key = c.doc_key AND r.node = c.ref_node "
                      "WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2'"):
    print(row)

# Who signed what.
for row in db.execute("SELECT r.signer, r.scope, COUNT(*) FROM receipts r "
                      "JOIN receipt_nodes rn ON rn.receipt = r.id GROUP BY r.id ORDER BY r.made_at"):
    print(row)

# Table 1's body, row by row.
rows = db.execute("SELECT r.id FROM nodes r JOIN nodes b ON b.doc_key = r.doc_key AND b.id = r.parent "
                  "WHERE r.doc_key = 'hale2026harmonic-damb' AND r.kind = 'row' AND b.kind = 'table-body' "
                  "ORDER BY r.seq").fetchall()
for (row,) in rows:
    cells = db.execute("SELECT content FROM nodes WHERE doc_key = 'hale2026harmonic-damb' AND parent = ? ORDER BY ord",
                       (row,)).fetchall()
    print(" | ".join("".join(r.get("text", "") for r in json.loads(c)) for (c,) in cells))
```

```text
('ec5jc93', 'F1')
('a4jrz2t', 'F0')
('xjns57z', 'F0')
('hca6dc5',)
('ana', 'targeted:lode', 3)
('ana', 'targeted:lode', 23)
('ana', 'section:p36k8t6', 2)
('cara', 'page:2', 3)
('ana', 'targeted:captions', 3)
('ben', 'qa:y2zw8ctg5s8dp', 10)
('ben', 'qa:gx2s84z9jf9xv', 3)
Bond parameters
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
CT–HC | 340.0 | 1.090 | 338.7 | 1.0900
OW–HW | 553.0 | 0.9572 | 548.1 | 0.9570
```

The first query reads the level the converter recorded, not the final one.
A receipt is a signed record that a reviewer checked certain nodes against the page.
Receipts by `ana` raise `ec5jc93` and `a4jrz2t` to F3; `xjns57z` stays at F0 because the only receipt that covered it, by `cara`, was revoked.
To compute the final level, check each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
The `scope` column of the third query shows the value `targeted:lode`, which is data in the fixture, copied here exactly as the query prints it.

## Test your implementation against a vector

A test vector is a recorded input together with the output every correct implementation must produce.
Every vector file has one case per line, with its inputs and expected outputs.
A harness for the projections looks like this, with your functions in place of the comments:

```python
import json

for line in open("vectors/projections.jsonl", encoding="utf-8"):
    case = json.loads(line)
    runs = case["runs"]
    # assert my_plain(runs) == case["plain"]
    # assert my_strict(runs) == case["strict"]
    # assert my_lossy(runs) == case["lossy"]
    # assert my_sha256(case["strict"]) == case["match_hash"]
    print(case["id"], case["strict"])
```

[vectors/README.md](../vectors/README.md) lists every file, what it tests and the chapter that defines it.

## Break something on purpose

This shows what the checks catch.
Change one digit in a fixture and run the checks:

```sh
sed -i 's/1718 cm/1719 cm/' fixtures/packages/okafor2025solvent-acth/nodes.jsonl
pixi run quick
```

The checker reports three failures: the SHA-256 of `nodes.jsonl` no longer matches the manifest, unloading the package from the database no longer gives the same bytes, and the package hash differs after unloading.
Restore the file with `git checkout fixtures/packages/okafor2025solvent-acth/nodes.jsonl`.
The broken packages in [fixtures/invalid/](../fixtures/invalid/) are this kind of change, each with the rule it breaks.
`git checkout` restores a tracked file, so run this only in a clone you can reset.