Reviewed-on: #1
This commit is contained in:
57 files changed
+11446
No files matched your search
@@ -9,6 +9,7 @@ The formats are version 0.1.0, and each is `candidate` or `provisional` in `regi
|
||||
|
||||
Run every command from the root of the repository, <https://git.scient.ing/infra/paperspec>.
|
||||
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
|
||||
<<<<<<< HEAD
|
||||
Every command also works without pixi once the checkout is installed with `pip install -e ".[tools]"` on Python 3.14, given standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
|
||||
|
||||
## Run the checks
|
||||
@@ -155,6 +156,155 @@ writer2027fixed-ynr7.v3: 15 nodes
|
||||
```
|
||||
|
||||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
|
||||
||||||| parent of 4fa326d (First implementation (#1))
|
||||
=======
|
||||
Every command also works without pixi, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
|
||||
|
||||
## Run the checks
|
||||
|
||||
This shows how to confirm that the repository's specification, fixtures and test vectors agree with each other.
|
||||
|
||||
```sh
|
||||
pixi run check
|
||||
```
|
||||
|
||||
`pixi run check` lints the Markdown, then runs `python tools/check.py`, the repository's reference checker.
|
||||
Each section prints how many checks it ran, and the last line is `All checks pass.`
|
||||
`pixi run quick`, or `python3 tools/check.py --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
|
||||
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `tools/check.py` does not run it.
|
||||
|
||||
## Recompute a hash by hand
|
||||
|
||||
This shows that a hash in a package is not magic: you can reproduce it with a standard tool.
|
||||
A hash is a fixed-length fingerprint of some text; change one character and the fingerprint changes.
|
||||
A sentence's `text_hash` is the SHA-256 of its text after two steps.
|
||||
The text is converted to Unicode NFC (a standard form that writes each accented or composed character one way), and each run of whitespace is collapsed to one space, as [chapter 1](../spec/01-conventions.md#14-text-normalization-for-hashing) defines.
|
||||
For a sentence with math or citations, the text is its plain projection ([chapter 5](../spec/05-hashes.md#51-the-five-values)).
|
||||
For a sentence of plain text, the whole computation is a one-liner:
|
||||
|
||||
```sh
|
||||
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
|
||||
```
|
||||
|
||||
```text
|
||||
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
|
||||
```
|
||||
|
||||
The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic-damb`:
|
||||
|
||||
```sh
|
||||
grep -o '"id":"v3063qv".*' fixtures/packages/hale2026harmonic-damb/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
|
||||
```
|
||||
|
||||
```text
|
||||
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
|
||||
```
|
||||
|
||||
## Check a package's manifest
|
||||
|
||||
This shows how a reader detects that a package's files have changed.
|
||||
A package is a folder of JSON (JavaScript Object Notation, a plain-text format for structured data) files describing one paper.
|
||||
`MANIFEST.json` lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package ([chapter 12](../spec/12-package.md#123-the-manifest)).
|
||||
|
||||
```sh
|
||||
cd fixtures/packages/okafor2025solvent-acth
|
||||
sha256sum nodes.jsonl
|
||||
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
|
||||
```
|
||||
|
||||
```text
|
||||
09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3 nodes.jsonl
|
||||
"path":"nodes.jsonl","sha256":"09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3"
|
||||
```
|
||||
|
||||
## Read a package with nothing but a JSON parser
|
||||
|
||||
This shows that a package needs no special software to read.
|
||||
JSON Lines is a text format with one JSON object per line, here one node (a section, sentence, figure, table or equation) per line of `nodes.jsonl`.
|
||||
Save this script as `outline.py`.
|
||||
It prints a document's sections, figures, tables and equations.
|
||||
|
||||
```python
|
||||
import json, sys
|
||||
|
||||
pkg = sys.argv[1]
|
||||
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
|
||||
by_id = {n["id"]: n for n in nodes}
|
||||
children = {}
|
||||
for n in nodes:
|
||||
children.setdefault(n["parent"], []).append(n)
|
||||
|
||||
|
||||
def text(n):
|
||||
"""The plain projection of a node with runs."""
|
||||
out = []
|
||||
for r in n["content"]:
|
||||
if r["kind"] == "math":
|
||||
out.append("$" + r["source"] + "$")
|
||||
elif r["kind"] == "equation":
|
||||
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
|
||||
else:
|
||||
out.append(r["text"])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
for n in nodes:
|
||||
if n["kind"] == "section":
|
||||
heading = children[n["id"]][0]
|
||||
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
|
||||
elif n["kind"] in ("figure", "table", "equation"):
|
||||
print(n["id"], " ", n["kind"], n.get("label", ""))
|
||||
```
|
||||
|
||||
```sh
|
||||
python3 outline.py fixtures/packages/hale2026harmonic-damb
|
||||
```
|
||||
|
||||
```text
|
||||
p36k8t6 1 Introduction
|
||||
tkxsmts equation (1)
|
||||
d400rz3 2 Methods
|
||||
v7cx6qs figure Figure 1
|
||||
hj4s7pb table Table 1
|
||||
```
|
||||
|
||||
## Follow an address
|
||||
|
||||
This shows how a citation points inside a sentence.
|
||||
An address names a node, and optionally a span of its text ([chapter 6](../spec/06-anchors.md#61-addresses)).
|
||||
`hale2026harmonic-damb#hca6dc5[29:44]` names code points 29 to 44 of sentence `hca6dc5`.
|
||||
With `text()` from the script above:
|
||||
|
||||
```python
|
||||
print(text(by_id["hca6dc5"])[29:44])
|
||||
```
|
||||
|
||||
```text
|
||||
first test case
|
||||
```
|
||||
|
||||
Offsets count Unicode code points, one per character; in Python, string indexes already do.
|
||||
JavaScript strings index UTF-16 units, so convert first.
|
||||
For text with characters outside the Basic Multilingual Plane the two counts differ, and [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both.
|
||||
|
||||
## Load packages into SQLite
|
||||
|
||||
This shows that the same packages can live in a database and be queried with SQL.
|
||||
|
||||
```sh
|
||||
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
|
||||
```
|
||||
|
||||
Without pixi, run `python3 tools/load.py` with the same arguments.
|
||||
|
||||
```text
|
||||
hale2026harmonic-damb: 75 nodes
|
||||
okafor2025solvent-acth: 15 nodes
|
||||
writer2027fixed-ynr7.v3: 15 nodes
|
||||
```
|
||||
|
||||
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
|
||||
>>>>>>> 4fa326d (First implementation (#1))
|
||||
Here are four queries, in Python's `sqlite3` so that nothing else is needed.
|
||||
|
||||
```python
|
||||
|
||||
Reference in new issue
Block a user