infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: documents and session tokens
check / check (default, macos-latest) (push) Canceled after 0s
Details
check / check (default, ubuntu-latest) (push) Canceled after 0s
Details
check / check (py310, macos-latest) (push) Canceled after 0s
Details
check / check (py310, ubuntu-latest) (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
alexm committed 2026-10-12 01:07:50 -04:00
1 parent 62f54ed869
commit 8b2df58266
39 files changed
+1523 -199
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/concepts.md
+262
View File
@@ -0,0 +1,262 @@
# Concepts
Each section introduces one idea, shows it in the example paper `hale2026harmonic` (a synthetic chemistry paper in [fixtures/packages/](../fixtures/packages/)), and links to the chapter that defines it exactly.
## Works, versions, witnesses and documents
Four things are easy to confuse, so the formats keep them apart.
| Object | In the example |
| --- | --- |
| A **work** is the paper across all its versions. | Hale and Osei's paper on harmonic bond parameters, `doi:10.5555/example.hale.2026`. |
| A **version** is one expression of it: a preprint, the version of record, an erratum, the Supporting Information. | The version of record. |
| A **witness** is one file of a version, identified by its SHA-256. | The publisher's PDF, witness `qptsanv`. |
| A **document** is the text established for one version, from one or more witnesses. | `hale2026harmonic`. |
A document's key is its work's key, with a suffix for anything but the version of record: `hale2026harmonic.si` would be its Supporting Information.
See [chapter 2](../spec/02-model.md).
## A document is a tree of nodes
Every part of the text is a node: the document, its sections, headings, paragraphs and sentences, figures and their panels and captions, tables with their rows and cells, equations, footnotes and references.
The example has 75 nodes; its outline is:
```text
bzqvbhd 1 Introduction
qn08tj9 equation (1)
f0tj6ca 2 Methods
ajfvq4s figure Figure 1
gm5z4rx table Table 1
```
`nodes.jsonl` holds one node per line, in reading order, each with its kind, parent, position, content and hashes.
A sentence's content is a list of **runs**: plain text, styled text, inline math, citations, cross-references, footnote marks, links, and display equations the sentence runs through.
This is one table cell, shortened:
```json
{"id":"nkm3xn5","kind":"cell","parent":"zhe4g0k","ord":1,
"attrs":{"column":"…","role":"data","rowspan":1,"colspan":1},
"content":[{"kind":"text","text":"310.0"}],
"fidelity":"F1","text_hash":"…","match_hash":"…","align_key":"…","own_hash":"…","node_hash":"21f13b…"}
```
See [chapter 3](../spec/03-nodes.md).
## IDs are names, not positions
`nkm3xn5` says nothing about where the cell is.
That is deliberate: an ID that encoded a page or an ordinal would change when a better converter ordered blocks differently.
IDs are minted once from the document key, the node's kind, its text and an occurrence count, so a fresh conversion of the same input gives the same IDs, and then they never change.
When two nodes merge, one ID is retired and points at its successor; when an artifact is deleted, its ID is retired with no successor.
Legacy IDs, such as the block IDs older tools used, live on as aliases: in the example, `b-p01-text-003` still leads to sentence `f929eq9`.
See [chapter 4](../spec/04-identity.md).
## Addresses
An address names a node, and optionally pins it to a state and narrows it to a span:
| Address | Names |
| --- | --- |
| `hale2026harmonic#f929eq9` | The sentence about water, whatever it says now. |
| `hale2026harmonic#f929eq9@cd6cdb` | That sentence as it read when its `node_hash` began `cd6cdb`. |
| `hale2026harmonic#f929eq9@cd6cdb[29:44]` | Code points 29 to 44 of it in that state: "first test case". |
| `smithlab/hale2026harmonic#f929eq9` | The same sentence in the `smithlab` store. |
On the web the ID is the fragment, so a link to `…/smithlab/hale2026harmonic#f929eq9` lands on the sentence.
See [chapter 6](../spec/06-anchors.md#61-addresses).
## Five hashes, five jobs
Every node carries five content values.
| Value | Covers | Answers |
| --- | --- | --- |
| `text_hash` | The text exactly, normalized. | Is this the same quotation, character for character? |
| `match_hash` | The text with notation folded to one form. | Do two witnesses really agree? |
| `align_key` | The text folded loosely. | Is this probably the same sentence? |
| `own_hash` | The node's own content, without its children. | Has this section itself changed, as opposed to something inside it? |
| `node_hash` | The node and everything below it. | Has anything here changed at all? |
The middle two exist because one hash cannot do both of their jobs.
Chemistry writes the same thing many ways, and every one of these sentences has the same `match_hash`:
| As stored | Strict projection |
| --- | --- |
| `The solvent is $H_2O$.` | `The solvent is H_{2}O.` |
| `The solvent is H$_2$O.` | `The solvent is H_{2}O.` |
| `The solvent is H` and a subscript `2`, then `O.` | `The solvent is H_{2}O.` |
| `The solvent is H₂O.` | `The solvent is H_{2}O.` |
| `The solvent is $H_2 O$.` in Typst | `The solvent is H_{2}O.` |
But OCR often drops a superscript, and "103" must never be certified as agreeing with 10³.
So "A rate of 103 per second." and "A rate of 10³ per second." share an `align_key`, which proposes that they are the same sentence, while their `match_hash` values differ, which raises a suspect of kind `number` for a person to resolve.
The same care keeps "The energy" followed by citation 3 from matching the text "The energy3".
See [chapter 5](../spec/05-hashes.md) and [vectors/projections.jsonl](../vectors/projections.jsonl).
## Anchors
An anchor is how anything points into a document: a Lode extract, an annotation, a dataset value, a draft's citation.
It is a pinned address with enough context to find its text again after the document changes:
```json
{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic","node":"f929eq9","kind":"sentence",
"own_hash":"d5b2eb…","node_hash":"cd6cdb…","span":[29,44],
"quote":{"exact":"first test case","prefix":"Water, H$_2$O, served as the ","suffix":", with a bending band near 1595 "},
"rev":12}
```
`rev` is the document's revision when the anchor was made, counted in corrections rather than clock time.
When the document changes, an anchor resolves one of five ways:
| Result | Means | In the example |
| --- | --- | --- |
| `current` | Nothing it points at has changed. | A note on the water sentence. |
| `changed-below` | A container is unchanged in itself, but something inside it changed; a drift figure says how much. | A note on Table 1 after a row was inserted. |
| `moved` | Its node was merged into another that still holds the quote. | A note on half of a sentence that was wrongly split, then rejoined. |
| `reattached` | Its node changed, but the quote is still there exactly once, so it is re-pinned. | A note on "treated as zero" in a sentence whose superscript was fixed. |
| `stale` | It can no longer be trusted; a person must re-anchor it. | A note on the cell that read "3l0.0" before it was corrected to "310.0". |
See [chapter 6](../spec/06-anchors.md); the [walkthrough](walkthrough.md#5-what-happened-to-the-notes) shows all five.
## Corrections
A document is never edited in place.
Every fix is a correction: an operation with its evidence, its author and a reason, appended to `corrections.jsonl`.
```json
{"format":"scienting-correction/1","index":1,"op":"replace","target":"nkm3xn5",
"find":"3l0.0","replace":"310.0",
"evidence":[{"kind":"crop","sha256":"9b6fe6…"}],"author":{"key":"ana"},
"reason":"OCR read the digit 1 as the letter l.","at":"2026-10-02T10:00:00Z",
"before":"2fd1e9…","after":"21f13b…","touched":["nkm3xn5"]}
```
The operations are `replace`, `insert`, `delete`, `split`, `merge`, `move`, `reclassify`, `relabel` and `set`.
A gate refuses any correction that would change a digit without a crop of the page as evidence, or that would let a digit change anywhere it did not declare.
See [chapter 11](../spec/11-corrections.md).
## Fidelity
Every leaf has a level that says how far its text was checked.
| Level | Means |
| --- | --- |
| F0 converted | One reading, not corroborated. |
| F1 resolved | Every doubt about it was answered against the page image. |
| F2 corroborated | An independent witness agrees exactly, or the text was born digital. |
| F3 proofread | A named person compared it with the page and signed for its current text. |
A level counts only at the node's current hash, so a correction drops a proofread sentence back until someone signs for it again.
The example reports 28 leaves at F3, 12 at F2, 4 at F1 and 1 at F0.
See [chapter 8](../spec/08-fidelity.md).
## Receipts and spot-checks
Nobody signs three thousand nodes one at a time.
A reviewer signs a **receipt**: one signature over a Merkle root of everything reviewed in a sitting, with the evidence looked at.
In the example, the reviewer `ana` signs one receipt for 23 nodes: Table 1's cells, the equation and the sentences around it.
Signing once per sitting invites rubber-stamping, so a second reviewer spot-checks a sample.
The sample is chosen by the checker's Ed25519 signature over the receipt, which neither reviewer can steer, and anyone can recompute it.
Across samples, the record states a bound on each reviewer's error rate: zero errors in 10 sampled nodes bounds the rate below 26%, and zero errors in 300 bounds it below 1%.
A spot-check that finds an error suspends the receipt; a revocation withdraws it.
Either way the nodes fall back to the level they had without it.
See [chapter 8](../spec/08-fidelity.md#83-receipts).
## Page fingerprints
Publishers stamp each download with the institution and time, so two teammates' copies of one paper rarely share a SHA-256.
Each page therefore also has a fingerprint: a hash of its size, its text with the stamp lines removed, and its images.
Two downloads of the same page stamped "Downloaded via UNIVERSITY OF EXAMPLE on October 11, 2026" and "Downloaded via EXAMPLE PHARMA INC on March 2, 2027" share the fingerprint `6272e6f3…`; change one digit of the text and it becomes `b545d285…`.
Geometry and digitizations point at fingerprints, so a re-download keeps every box and every value read off a plot.
See [chapter 7](../spec/07-geometry.md).
## Keeping text private
A team may annotate a paper it may not share.
A **checks-only** package keeps every ID, label, page and box, but withholds the text, and keys every hash, because a plain hash of a short text such as "310.0" is recovered by hashing guesses.
Inside the team, values are plain SHA-256; exported to another of the team's machines, they are HMACs under the team's key; in a public layer, the key is derived from the paper's page fingerprints, so only someone holding the paper can check them.
[fixtures/packages/hale2026harmonic.checks-only](../fixtures/packages/hale2026harmonic.checks-only) is the example paper exported this way.
See [chapter 5.8](../spec/05-hashes.md#58-keyed-hashes).
## Rights
What a team may do with a paper depends on how it holds the paper.
Every witness carries a rights record: the license as stated, the basis (an open license, a subscription, a collective license, the team's own work), and which uses that allows, including AI uses by purpose and by where the processing runs.
Every processing stage declares what it needs and runs only when the rights allow it.
The example PDF is held on a subscription that allows text mining and internal AI use on the team's own machines.
So conversion by a local model is allowed, conversion by an outside service is refused for lacking `ai/internal/outside`, and model review in the team's cloud is refused for lacking `ai/internal/tenancy?` (the question mark means unknown) until a project owner signs a policy that settles it.
See [chapter 14](../spec/14-rights.md).
## Layers and annotations
Everything people and models add is an annotation, grouped in layers such as "Novice glosses" or "Journal club, March".
Annotations are W3C Web Annotations whose selector is an anchor, so any Web Annotation tool can read them.
This gloss, shortened, explains a phrase for a first-year student:
```json
{"type":"Annotation","motivation":"describing","status":"confirmed",
"body":{"type":"TextualBody","kind":"gloss","audience":"novice",
"value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},
"target":{"type":"SpecificResource","source":"urn:scienting:doc:smithlab/hale2026harmonic",
"selector":[{"type":"AnchorSelector","anchor":{"node":"sxgcc9h","span":[111,130],"…":"…"}},
{"type":"TextQuoteSelector","exact":"bond force constant"}]}}
```
One rule decides what is a claim and what is a layer: a statement true independently of the paper belongs in Lode as a checked claim; a statement about reading the paper belongs in a layer.
Model-written text is always a `generated` annotation, never source.
See [chapter 18](../spec/18-layers.md).
## Exercises
Because every sentence and cell has an address, the answer to "find the evidence" is an address, and an address can be graded.
An exercise asks a student to find the cell giving the Set A bond constant for CT–CT; a student who points at `nkm3xn5` scores 1 of 1 with "That is the cell.", and one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."
Students submit under pseudonyms, and a key whose anchors have gone stale is never used to grade.
See [chapter 19](../spec/19-exercises.md).
## Datasets, digitizations and views
Numbers live in prose, in tables and inside figures; all three should end up as data a team can plot and cite.
A dataset is a Frictionless Data Package with units, uncertainty, and an origin for every value.
The example's Table 1 becomes a CSV whose every value is anchored to the cell that prints it, digits exactly as printed:
```text
key,bond,set,k_r,r_eq
ct-ct-a,CT–CT,A,310.0,1.526
ct-ct-b,CT–CT,B,305.2,1.5260
```
A digitization records how values were read off a plot (axes, calibration, series, uncertainty) and counts only once a person confirms it against an overlay.
A view is a Vega-Lite specification that names its data by dataset ID and hash.
See chapters [15](../spec/15-datasets.md), [16](../spec/16-digitization.md) and [17](../spec/17-views.md).
## Packages, databases and bundles
A document travels as a package: a folder of canonical JSON files.
```text
hale2026harmonic/
MANIFEST.json every other file, with its SHA-256; its own hash pins the package
document.json the work, version, metadata, policy, root hash and fidelity
nodes.jsonl one node per line, in reading order
witnesses.json each witness and its rights
pages.jsonl each page and its fingerprint
locations.jsonl where each node is on each page
references.jsonl parsed references
corrections.jsonl every correction, in order
receipts.jsonl receipts and revocations
ids.jsonl retired IDs and legacy aliases
roster.json the keys that signed
provenance.json tools, versions and where each stage ran
assets/ images, by SHA-256
datasets/ datasets reported from the document
derived/ MathML and other renderings, rebuildable
```
Every line is canonical JSON, so a corrected sentence is a one-line diff, and every file has exactly one valid byte sequence.
The same data loads into tables that run on SQLite, Dolt and PostgreSQL, and unloads back to the same bytes.
A bundle is one SQLite file with every table and a full-text index, which a reader or an MCP server can query offline.
See chapters [12](../spec/12-package.md) and [13](../spec/13-database.md).