paperspec/docs/walkthrough.md

Walkthrough: one paper, start to finish

This guide follows the example paper hale2026harmonic-damb from a flawed first reading to a corrected, reviewed and signed document. The paper is synthetic, written for the fixtures. Its problems are modeled on the failures that the specification's rules address.

Three readers can use the guide, and each gets something different. A newcomer knows what a paper and a citation are, not hashes or JSON, so every term is defined where it first appears. An implementer who will write a reader, writer, converter or checker gets the chapter and test vector for each step. An expert gets the status: paperspec is at version 0.1.0, and each format is candidate or provisional in registry/formats.json (19 candidate, 13 provisional). The scienting- prefix on format names, as in scienting-doc/1, is the formats' namespace. It has nothing to do with the repository's name.

Open fixtures/packages/hale2026harmonic-damb/ beside this page, because every ID below is in it. A package is a folder of files, mostly JSON Lines (.jsonl: one JSON record per line). The hand-written source of the first reading is fixtures/src/hale2026harmonic.json; no converter ran on it, and paperspec build expands it into the package. Each step says what a converter, a person or a checker does.

1. A PDF arrives

The team downloads the publisher's PDF. The file becomes a witness, identified by its SHA-256 (a 64-digit hash that changes if any byte changes), with the short ID qptsanv (spec/02). A person records its rights, meaning what the team may do with the file (spec/14). The library's subscription allows reading, text mining, and AI processing on the team's own machines for internal use, until August 31, 2027. Whether it allows processing in the team's cloud is unknown.

The publisher stamped the download in the margin with the institution and the time, so two downloads rarely share a SHA-256. A page fingerprint avoids the problem of downloads that differ in their stamps. It is a hash of a page's text, size and images after the stamp lines are removed (spec/07), and the rules downloaded-via and sharing-guidelines fired on both pages (see pages.jsonl). Page 101's fingerprint is 6272e6f3… in this copy, stamped October 11, 2026, and in a second download from another institution, stamped March 2, 2027 (vectors/fingerprints.jsonl).

2. The first reading, with its mistakes

A converter reads the PDF into sections, paragraphs, sentences, a figure, a table, an equation, a footnote and references. Each of these is a node, and each node gets a permanent ID: seven characters minted once from the document key, the node's kind, a hash of its text and an occurrence count, then frozen (spec/04). An ID encodes no page, order or kind, so it keeps naming the node after the text is corrected.

Like every reading, this one has mistakes:

What the reading did	What the page says
A cell read 3l0.0.	310.0, with the digit 1.
A sentence read Values below 10-3 kcal mol⁻¹….	A superscript: 10⁻³.
Two sentences ran together: "…treated as zero. Table 1 lists…".	Two sentences.
One sentence was cut in two at a line break: "…and the energies were" and "fitted by least squares."	One sentence.
A paragraph was split at a column break.	One paragraph.
The running head "J. Example Chem. 2026, 14, 101–109" became a paragraph.	A running head, not text.
The equation's number read (l) and the figure's Figure l.	(1) and Figure 1.
The CT–HC row merged two cells into "338.7 1.0900".	Two cells.
The OW–HW row was missing.	It is there.
"Set B" spanned one column.	It spans two.

The converter does not guess. Each doubtful place becomes a suspect, with a crop (an image cut from the page) for a person to answer (spec/05 §5.7). Unresolved suspects travel in suspects.jsonl, which this package does not need. Nothing changes until a correction passes the gate (spec/11).

3. Twelve corrections

Each fix is appended to corrections.jsonl, numbered, with its evidence (a crop, or a person who read the page), its author (ana throughout) and its reason.

#	Operation	On	What it did
1	replace	cell c20cna6	3l0.0 to 310.0. A digit changed, so the gate required a crop, and it has one.
2	replace	sentence ec5jc93	10-3 to the runs 10 and superscript −3.
3	split	sentence ec5jc93	Split before "Table 1"; the first part keeps ec5jc93, the second is new: a4jrz2t.
4	merge	sentence xjns57z	Joined with ayea71r, which is retired with xjns57z as its successor.
5	merge	paragraph gwzbj3s	Joined with ncx2v4f, retired the same way.
6	delete	paragraph v8e4z6z	The running head is gone; its IDs are retired with no successor.
7	relabel	equation tkxsmts	(l) to (1).
8	delete	row rncchma	The garbled CT–HC row and its four cells.
9	insert	row kx2kz2y	The CT–HC row, read again: five new cells.
10	insert	row ke0jxpn	The missing OW–HW row.
11	set	cell xyy7x6c	Set B's colspan from 1 to 2.
12	relabel	figure v7cx6qs	Figure l to Figure 1.

The document is now at revision 12. No uncorrected cell, sentence or row changed its ID or its hashes. Only corrected nodes and their ancestors got new node_hash values; the sentence rr20tev is one of the ancestors, because it holds the relabeled equation. Inserting a row adds a row node. The table's own hash (own_hash) covers only its label and column definitions, so every other cell, and every value already taken from them, is untouched.

4. IDs that never move

The sentence about water is still hca6dc5 after twelve corrections, the corrected cell is still c20cna6, and the sentence that lost its superscript is still ec5jc93. The retired IDs are recorded in ids.jsonl, so anything that pointed at them can follow the trail. Legacy aliases are recorded there too: b-p01-text-003, the ID an older tool gave the water sentence, still leads to hca6dc5.

A fresh conversion of the same input mints the same IDs. A reading by a different converter is aligned with this document sentence by sentence (spec/05 §5.6, pinned by vectors/alignment.jsonl). Sentences the alignment pairs keep their IDs, and every difference becomes a suspect rather than a silent change. Sentences it cannot pair are recorded as dropped or added.

5. What happened to the notes

An anchor is the stored pointer behind a note or citation. It holds a node's ID, the node's hashes when the anchor was made, the revision, and, for part of a sentence, the quoted words with context (spec/06). The team had already made seventeen anchors pointing into the first reading at revision 0, before any correction.

Resolving an anchor gives one of five results:

• current: nothing changed.
• changed-below: the node is unchanged, and something under it changed.
• moved: the node was merged into a successor that still holds the quote.
• reattached: the node changed, and the quote or the cell's headers now find exactly one place.
• stale: nothing is found, so a person must look.

A changed-below result carries a drift, the share of the leaves under the node, counting retired ones, that corrections touched. Above a quarter the drift is high. The table shows how each anchor resolves at revision 12 (vectors/resolve.jsonl records them all).

Anchor on	Result	Why
"H$_2$O" in the water sentence	current	Nothing in that sentence changed.
The cell 1.5260	current	Untouched, though its table changed around it.
The water sentence, through the alias b-p01-text-003	current	The alias leads to the same node.
The cell that read 3l0.0	stale	Its text changed and "3l0.0" is no longer in it; a person must look again.
"treated as zero"	reattached	The sentence changed twice, and the quote occurs in it exactly once, so the anchor is re-pinned.
"fitted constants"	stale	After the split, those words are in the new sentence a4jrz2t, not in ec5jc93.
"least squares", in the half-sentence ayea71r	moved	That ID was merged into xjns57z, which still holds the quote.
The paragraph ncx2v4f	moved	Merged into gwzbj3s, which holds all of its sentences.
The running head	stale	Deleted, with no successor.
The equation	reattached	Only its number changed.
The figure	reattached	Only its label changed, by a relabel.
The CT–HC cell 1.090 in the garbled row	reattached	Its row was deleted, and exactly one cell in Table 1 has the same row and column headers and the same text: the new b5gd0h1.
Section 2	changed-below, drift high	The section itself is unchanged, and 53% of what it holds or held was touched.
Table 1	changed-below, drift high	Rows were replaced and added.
The table body	changed-below, drift high	75% of the cells it holds or held were touched.
The paragraph whose broken sentence was rejoined	changed-below, drift high	Two of the three sentences it has held were touched.
The CT–CT row	changed-below, drift low	One of its five cells was corrected.

A reader shows each note by its result. A checker of exact words accepts only current, moved and reattached.

The W3C Web Annotation Data Model is the general W3C Recommendation for annotating web resources. paperspec's annotation layers build on it, with a custom AnchorSelector followed by a standard TextQuoteSelector as a fallback. It has no permanent IDs for sentences inside the source, no hashes of the target text, and no results such as moved or drift. See comparison.md.

6. Review and signatures

A receipt is a signed record that a person compared a set of nodes with the page. It lifts a node to F3, the top fidelity level (F0 is one reading; F1, every doubtful place answered; F2, an independent witness agrees) (spec/08). One signature covers a Merkle tree, a hash over everything reviewed in a sitting, built as in RFC 6962 (a Request for Comments standard; replaced by RFC 9162, which defines the same tree). The receipts are in receipts.jsonl, and the keys that signed them in roster.json.

• Before the corrections, ana had signed receipt npc15pn6675z5 for three cells at revision 0. Two of them never changed, so the receipt still counts for them. For the corrected cell it no longer counts, because the cell's hash moved on.
• After the corrections, ana signs y2zw8ctg5s8dp at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (ec5jc93, hca6dc5 and rr20tev, the one holding the equation). ben spot-checks it with receipt mg18668nzxvap. His signature, made with his separate spot-check key ben-spot, picks 10 nodes nobody could have chosen, and he finds no errors.
• ana signs mvjmb94nz7rdh for two sentences of section 1, including the new a4jrz2t.
• cara signs q465jpwz8afhe for three sentences on page 2, then revokes it herself (pcbhtpwvysgq6) because she had compared them with the crops of another paper. Those sentences fall back to their earlier levels, which is why xjns57z is still F0.
• ana signs gx2s84z9jf9xv for the figure's three caption sentences. ben's spot-check yt70fcc3qm0xb finds a missing period in caehhp6, which suspends the receipt until she reviews the sitting again and signs a new one.

The document's report, in document.json, counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0. The 27 are the 23 nodes of y2zw8ctg5s8dp, the 2 of mvjmb94nz7rdh, and 2 of the 3 caption sentences. Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right. The third, caehhp6, stays at F2: a spot-check never raises a node it found in error (spec/08 §8.2).

7. Data out of the table

Table 1 becomes a dataset, ds-hale-bond-params, inside the package (spec/15). It is a Frictionless Data Package with units, uncertainty and an origin added, and the format is provisional. Its CSV keeps every digit as printed (1.526 and 1.5260 stay different), and every value has an anchor to the cell that prints it, so a claim that rests on a value can show the cell. The descriptor types the fields as numbers; how a generic Data Package tool treats 1.5260 was not verified. All twelve value anchors resolve as current. The reference checker, paperspec check, confirms each CSV value equals its cell's text.

8. Sharing without the text

The team wants to move its notes to another of its own machines without copying the paper there. hale2026harmonic-damb.checks-only is the same document with every leaf's text withheld and every hash replaced by an HMAC under the team's key (spec/12 §12.5, spec/05 §5.8). An HMAC is a hash mixed with a secret key. Anyone can recover a plain hash of a short text such as "310.0" by hashing guesses; a keyed hash resists that. Corrections, receipts, the roster and the dataset are left out too. IDs, labels, pages and boxes survive, so annotations still land in the right place. Only someone holding the key and the text can check the values. The team key suits an export between the team's own machines. An export to a collaborator outside the team would use a key derived from the pages of the paper instead, so that only someone holding the paper can check it.

9. Recompute, print and query the package

The examples show how to recompute one of these hashes with sha256sum, print the outline with a few lines of Python, and load the whole package into SQLite to query it.